Projects tend to fail because the customer didn't understand what they wanted and because the developers could not adapt to changes as the customer gradually figured it out. Both must be true for the project to fail, and neither are strictly limited to just the specs and the competency of the developers alone; budget, policy, politics and legacy systems are just as much a part of the customer's needs and the developer's restrictions, too.
But customers can be infected with a disease that makes it very tough for developers to work with them and understand their true needs and budget, and that disease is "knowing how programming works". And here's why:
Many users have no clue how software is developed. If they came you and asked "can this check my spelling for me?" you might say "it'll take two months to implement that feature," and they'll take it for granted and go away understanding that Spellcheck == 2 months work. In their minds they might have no clue how software is made. They might as well imagine a software development lab is full of potters wheels and chemistry sets. "If we get just the right balance of dihydrogen monoxide particles in the carbon nanotube matrix we can achieve little red squiggly lines under the misspelled words. HELEN! Charge up the Relativistic Heavy Ion Collider and phone the PHOBOS group to tell them we're doping with dihydrogen monoxide!"
Similarly, when such a user encounters a bug they see it as being an intrinsic property of the "stuff" that software and computers are made of. Like when you learn what happens when you dump cold water into a saucepan of hot fat. After it explodes and burns your skin you learn not to do that anymore, but you don't go complaining that this is a bug in the saucepan that ought to be fixed. In my line of work I've encountered users who have struggled for years to work around a bug that was trivial to fix, but it never occurred to them to report the bug. It's not because they were afraid to, or didn't know how, it was because their internal model of software's nature was quite different from what you or I understand it to be.
The party comes to a crashing halt when the customers get infected with the disease. If you're familiar with the expression "a little knowledge is a dangerous thing" then you'll understand what happens when a customer learns that software development is about writing instructions, and that bugs are caused by "mistakes made by the programmer."
Now everything goes to hell. Every feature becomes trivial because all you have to do is "just write it", and whenever a program crashes or misbehaves it's because those damn lazy programmers weren't being careful, so why the hell should I give them more money when its their damn fault it doesn't work? So you tell them spellcheck will take two months and they explode; "are you trying to bankrupt me?" they say, "just write it! How can it be so hard when Microsoft was able to turn it on in my email program just yesterday? All I had to do was call their tech support and they told me to click a little checkbox. Why can't you just... run it through Outlook or something?"
Scope-growth or feature-creep is another problem with both kinds of customers, but tends to be worse with the latter group. It happens because even the most well-meaning customers cannot accurately translate needs into requirements. Nobody can. Suppose you're a builder and somebody asks you to build something that they can store their lawnmower and gardening tools in.
Builder: "Oh, you mean a shed?"
Customer: "Yeah, is that what you call it? Okay."
And you build them a shed, and then the customer says "it's really hot in here, can you add air conditioning?" So you shrug and you set out to perform the upgrade properly: you add a new breaker to the fuse box in the house and run electrical wire out to the shed, install a pair of outlets and put in a window-mounted air-conditioner.
"Oh, but I wanted central AC like in my house," the customer tells you after inspection. Okay, a bit strange perhaps. But you upgrade the breaker to a higher amperage, find out that the wire you ran is underrated for the current and so rip it out and install a thicker gauge. You cut holes in the wall of the shed to run the pipes through, mount the heat exchanger, then install the compressor on the ground behind the back of the shed.
"That's great," the customer says, but then sees the compressor hidden behind the shed. "Oh, I was thinking it should be like those AC units you see on the roofs of shopping plazas and office buildings."
Now you wonder if he's insane. You explain that this is just a shed and there really isn't a need for air conditioning in the first place, and that the roof was not built to support the weight of a commercial AC unit. And that's when the other shoe drops.
"Yeah, but we need to have a waiting room for people who want to use the tools, and a receptionist. And there has to be parking, I forgot to tell you that. Can you put in some bathrooms, too? And we need to have a drive-thru window and a cash register, and a reservation system for contractors who want to guarantee availability."
You have now entered the world of software development.
Users and customers must be made to understand that software is not magic: it cannot do arbitrary things. Don't even qualify it by saying "it can't do arbitrary things within budget" because after a certain point in the evolution of any software system you will reach a point where certain feature requests imply an infinite budget. You've built too much, and it's going to be a shed no matter how many bathrooms you think you can install in it.
The development process must be transparent enough for the customer to see the moving parts, and the customer must be part of the development process continually, from the very beginning.
In another article I wrote that knowing the scope of the problem is more important than knowing how to solve the problem. So jump into some philosophy with me:
Reality is impossible to perceive directly, we must invent a model of reality and use our senses to test the model for accuracy. And we can go for years, even centuries armed with the wrong model and the wrong view of the universe and have it all ripped out from under us, like a tablecloth in the hands of a magician, by one observation.
The philosopher Ludwig Wittgenstein once posed to a friend, "Why do people say that it was natural to think that the Sun went round the Earth rather than that the Earth turned on its axis?"
His friend replied: "I suppose because it looked as if the Sun went round the Earth."
"But what would it have looked like if the Earth did turn on its axis?"
You, my beloved Intellectual, are surely the disciple of Copernicus. But your customer might be a disciple of Milton Friedman, and it's not that it occurs to them that they might be wrong, it's that the details don't occur to them at all. You can't build a program that can handle an out-of-protons exception, so it never occurs to you that you'd have to. And it doesn't occur, to many customers, that they should have asked for a tool rental business rather than a shed, and they won't realize they're in Friedman's world until they see--early on--that you are in Copernicus's world. Even though they thought they knew how programming worked they'll come to their senses because they can't afford not to. It can also be said that they aren't going from clueless to smart, but that they'll adjust and make you smart.
Or smarter, if that doesn't bruise your ego. Being a good developer is about becoming the customer. They must understand you in order to estimate cost, and you must understand them in order to estimate scope.