I'm halfway through Patterns of Software, a collection of essays by Richard Gabriel (one of creators of Common Lisp). The book approaches problems in software development from a philosophical standpoint and is heavily influenced by works of Christopher Alexander, an architect that started the entire Design Patterns movement.
As a lead of a software development consultancy, I'm in daily contact with people who find it hard to grasp why a software project can be hard to plan, deadlines and cost hard to estimate, even for experienced developers. In Richard's book I found an excellent explanation:
The true problems of software development derive from the way the organization can discover and come to grips with the complexity of the system being built while maintaining budget and schedule constraints.
He then goes on to explain:
It is not common for organizations to try to put together a novel large artifact, let alone doing it on schedule. When an engineering team designs and builds a bridge, for example, it is creating a variant of a well-known design, and so many things about that design are already known that the accuracy of planning and scheduling depends on how hard the people want to work, not on whether they can figure out how to do it.
This matches my experience well. Any non-trivial software development project is largely a research project as well. If it weren't, it'd already be available as an existing off-the-shelf solution.
The entire book is a great treatise on software development and software quality and I heartily recommend it to anyone interesting in thinking about software, design patterns and code quality. The book is freely available online, in PDF format.
A practitioning bit-shifting magician turned cat herder.