Software must have a purpose[0].

This is both self-evident and controversial. If we're expending serious effort to build it, it better have a purpose, but on the other hand, the purpose (the why) and the behaviour (the what) are often conflated.

Purpose is like a mission statement - a concise description of why something is being done that can be used by anyone to double-check that whatever they're working on is actually relevant and beneficial.

Purpose is not a list of features or requirements. What the software does should always be secondary to why it is needed in the first place. But often, teams miss the forest from the trees and think in terms of functionality - features and requirements.

New vs Improved

The purpose of a software project is either to improve an existing process[1] or to facilitate something that wasn't possible before.

Building software to improve an existing process is often considered boring and uninspiring, but there is a positive side to it - the problems with the current process are usually known and can be analysed and good, reliable data obtained on how to improve it.

When building something new, the people thinking about what needs to be built need to guess. While their assumptions may be close to the target and sometimes they may get lucky, there is simply no way to exactly know what's going to be needed.

Facts vs Beliefs

For the projects in the first category, improving existing system, the designers can rely on hard facts to at least some degree. The important thing here is knowing what exactly is a real fact, and what's a guess. Too often, the designers, the client[2] and the developers add in a number of assumptions, guesses, and "we can also do this easily so why not"-s.

These beliefs are still just guesses and can, and often do, turn out to be wrong. But throughout the software development project, they are treated as reverently as facts.

In some cases, even the real facts are tainted by the analysts filtering them (possibly without even knowing about it) through their own set of preconcieved ideas.

I firmly believe this is one of major reasons for the sad state of affairs where the majority of software projects fail.

A good step in the wrong direction

Agile software development methodologies recognize this, but their workaround is to treat everything as beliefs. Thus they are optimized for changing direction quickly when beliefs are invalidated or new ones introduced. In practice, many organisations adopt Agile in name but not in spirit[3].

In those that do, the pressure is often to focuss at the short term development performance (velocity as a target) and churning out user stories by the week, making it easy to forget about the big picture.

Another problem with Agile in practice is that the feedback loop, supposed to be as small as possible, often isn't. In practice, the feedback is provided by the project managers or QA testing the implemented user stories. The software doesn't actually get used, and beliefs really tested, until much later, turning these projects into waterfalls in Agile's clothing[4].

From facts to requirements

The more facts about the problem being solved, the better, provided that they are analysed correctly, without bias, and that they are precisely and unambigously stated.

For example, a fact might be that using a current software, an operator fills the (imaginary) FT1P form in 10 minutes on average, costing the company $10,000 a month in aggregate.

Similar statement, that it's hard to fill out the FT1P form so the operators are disgruntled, can also be taken as a fact if it's a strong opinion of everyone touching FT1P, and not just a complaint from one person. If so, further analysis can be done to see exactly what about FT1P makes the form hard to fill.

Facts help set constraints on the system. The above facts can produce requirements such as "keyboard shortcuts for moving around the form", "implement autocomplete on fields Foo and Bar" or "the maximum response time for an autocomplete must be under 500ms".

When producing requirements, either from facts or from beliefs, it makes sense to explain why they're needed, as this context is useful information to the system designers and developers. In the above example the explanation could be that these are needed to make it easier for operators to fill out the form in under 5 minutes, which is expected to save the company $5,000 a month.

Avoiding disaster with beliefs

The first step in avoiding in a trap is knowing it exists. Likewise, the first step in avoiding disaster holding a belief is knowing it can turn out to be false, and preparing for it.

The best way to deal with it is to figure out a way to prove or falsify the belief as quickly as possible. In theory, sprint feedback in Agile methodologies should do that. In Lean Startup movement, a Minimum Viable Product (MVP) should do that. In science, we call that an experiment. In engineering, it's called building and testing the prototype.

The main thing here is that the prototype should be as cheap and built as quickly as possible, while still testing it in real-world conditions. For much of the software, it means giving to the real end-users and seeing what they do with it, not asking your boss to try it out and see if she approves.

If this is not possible, one strategy could be to keep both options open (organise the software so it can be easily adapted as the belief is resolved), if it's expected that the complexity cost of keeping them both open is small, or to choose the simpler option[5] and refactor if needed, if the cost of future refactor is expected to be smaller than keeping both options open. Either case will result in techincal debt due to the uncertainty of the situation.

Agile vs Waterfall

It might seem that I favor the old Waterfall approach and believe Agile is overkilling it, but it's not so. I think both are extremes at opposite end of the spectrum. Waterfall assumes everything's a fact. Agile assumes everything's a belief. Sometimes, one or the other is correct. But most of the time, the truth is somewhere in the middle.

Some things are known knowns (facts), some things are known unknowns (beliefs, if correctly recognized), and, no matter how much we try to reduce them, some are unknown unknowns (surprises). So an agile approach (not neccessarily an Agile methodology) and leaving some slack in the schedule and planning is crucial for the success of the project.

---

[0] I mean this in a somewhat narrow context of software that's result of a serious, non-trivial, professional software development activity. Not all software strictly must have a purpose - the development process itself can be a goal, the software medium can be an art form, and so on.

[1] This applies to such niches as games - where the game is either a new one (an original or a new installment in an existing franchise), or a small update to an existing title.

[2] Or someone higher up the food chain if an internal project, or the product manager in case of productized software.

[3] A big multinational household name company's division I worked with practised Scrum with the initial multi-month project plan being put in backlog (with hundreds of tasks) and each sprint we'd move some things from backlog to the current sprint and called that Agile.

[4] In context of producing software to address new and anticipated needs, essentially software startups, this is a matter of life or death for the company. The Lean Startup movement aims to fix this and has some good ideas.

[5] I lean heavily towards the simple option whenever possible, since complexity itself incurrs additional cost. Read more about this in my Progressive Fibonnaci post.