Coderific

rss feed

blog - why the software industry is broken

posted by witten on February 17, 2007

The software industry as it currently exists is simply broken. A large percentage of software projects end in utter failure, so something is quite obviously going wrong. This failure often comes in the form of abandoned projects, where the customer doesn't even accept the software. Quite often though the project will simply miss a series of deadlines and/or not do what the customer actually wants, requiring massive reworking.

So what's at fault with these software project failures? Is the culprit the development process, the project management, the requirements gathering, or the complexity of software itself? Well, yes, all of the above. But none of those things are actually the root cause.

Nearly all things that are wrong with the software industry today stem directly from the way most companies are structured. Specifically, I'm talking about the top-down, hierarchical management structure in which decisions made at the top of an organization are handed down to those below, eventually making their way to the software developers in the trenches.

Now, on the face of it, this is a rather bold claim. I'm suggesting that the software industry is so broken today not because of the tools used or the development methodologies employed, but simply because of the way most businesses organize their employees.

If the software industry is broken because of corporate structure, then why aren't other industries similarly out of whack? For instance, manufacturing, construction, airline, and service industries all have a similar corporate structure to most software companies, so why aren't they similarly broken?

The cynic might argue that those industries are similarly broken, but for now let's assume that the software development industry has its own special breed of brokenness. After all, while the occasional bridge might fall into the water shortly after its built, that's certainly the rare exception. Software projects fail all the time.

So why does the corporate structure that seems to work out okay in other industries fail so miserably for software development? Well, there are a couple of reasons.

Firstly, software companies are based around abstract knowledge work. The processes by which coders develop software is not at all clear to the suits leading a company, because software development by its very nature is an abstract endeavor. And even those rare leaders with development experience can't tangibly assess the state of a project just by looking at it. The only people who really have a decent handle on where a project stands and what features can be added to it in a reasonable time frame are the coders in the trenches, and perhaps if the company is lucky, a good development manager or two.

One of my friends has a great analogy to explain this. Imagine you're building a bridge to span a river, and it's nine months into a ten month project and the bridge is nearly complete, and your boss comes along and says, "Oh, by the way, can you add a second deck to that thing?" This of course would be thoroughly absurd, because any schmuck can see that trying to tack a second deck onto an already designed and nearly completely constructed bridge would be a horrendously bad idea.

Now, imagine you're developing a software project, and it's nine months into a ten month project, and the project is nearly complete, and your boss comes along and says, "Oh, by the way, can you bolt on [what amounts to a second bridge deck] to that thing?" There's no physical structure you can point to and say, "Look, you see that support there? It can't handle the weight." Since software is perceived as infinitely malleable due to its abstract nature, and is so often opaque to the managers making the decisions, the suits at the top simply don't see the absurdity of adding a second deck to a software project at the last minute.

One approach you can take in the face of this problem is to simply build a model bridge out of toothpicks or popsicle sticks in parallel with your project. And then at the 8 or 9 month stage, when the software project is nearing completion (or at least its original intended deadline), you'll have built out the model bridge so that it's nearly complete as well. So when someone in a suit tries to slip in "just a tiny feature" at the last minute, you can point at the bridge and try to explain the whole analogy before their eyes glaze over.

But short of having to build a toothpick bridge alongside every software project, what can a developer do when faced with unreasonable demands? Many developers try to push back or to educate their supposed superiors: "The architecture won't support that kind of addition before the deadline," or "That will add too much risk to the project," or "Because of obscure technical details x, y, and z, that doesn't make a whole lot of sense."

And here's where the structure of the organization matters. In a typical top-down hierarchically managed company, where most developers are at the bottom of the totem pole, pushing back without a concrete physical thing to point at as evidence is viewed as not being a team player at best or outright insubordination at worst. Because of the power imbalance between managers and developers, no matter how much the developers might argue, debate, or plead about any particular management decision, the managers have the final say.

And because of the abstract nature of software, the developers are actually the ones with the most information available to make an informed decision. The higher up you go in an organization away from the developers, the less technical information is available with which to make good decisions that affect the development process. So what happens in practice is that the developers with most of the information necessary to make good decisions are told by the people with very little of this information to tack on a second deck to the proverbial bridge right before the deadline. And because of the power imbalance, combined with the abstractness of software, developers have a hell of a time convincing anyone that a second deck might not be such a hot idea.

Now of course, there are some software companies apparently not so strongly afflicted by this problem with corporate structure, where good decisions are often made and software projects often succeed. The reason these sorts of companies are so rare is simply because in a hierarchically managed organization, it takes a tremendous amount of discipline to actually listen to the developers in the trenches when making important decisions. It is a rare organization indeed in which developers have the ability to say "no" when asked to implement things that are technically or temporally unreasonable.

In other industries like manufacturing, the executives at the top do understand what's going on in the trenches to a sufficient degree, and so have sufficient information to make good decisions. In the software industry, unfortunately, the executives just don't have enough information about any particular software development project they're ostensibly overseeing. As a result, bad decisions are made, projects fail, and we find ourselves in an industry which could charitably be described as barely functional.

So why don't other industries with knowledge workers, like accounting or advertising, run into the same sorts of problems? Simply put, software development is just plain difficult. Successfully making software in an organization requires so many things to go right: Requirements gathering, specification, design, scheduling, coder wrangling, and so on. If just one of these things goes wrong, an entire project can be at risk of utter, humiliating failure. The same can be said of other industries, but software as it's currently developed is just so damned complex. Shipping a successful project requires an organization to be run like a well-oiled machine. Anything less, and you'll miss requirements, overlook important design considerations, and eventually miss deadlines.

What's more, communication in most top-down organizations flows downhill, and very little communication flows in the other direction, especially when it's bad news. So by the time any bad news does make its way upward, it's often too late to make a course correction. This is much more important in any software project, again due to complexity. There are so many potential decision points leading up to and during the development of an application, but without honest, unfiltered communication flowing upward, those decision points won't happen. Or if they do, they'll be based on incomplete information.

So if the organization of most organizations is to blame for the state of the software industry today, what's the solution? Stay tuned for next time.

10 comments

Write a comment!
  • Re: why the software industry is broken posted on February 17, 2007 10:51 PM

    Great post.

    reply | quote

  • Re: why the software industry is broken posted by witten on February 18, 2007 10:16 AM

    For those three of you who didn't come here by way of reddit, you might be interested in some of the discussion there on this blog post: http://programming.reddit.com/info/153xl/comments

    reply | quote

  • Re: why the software industry is broken posted by unclebob on February 19, 2007 09:27 AM

    Though I agree that the inversion of knowledge and authority is a problem; I disagree with your conclusion.

    The problem is not that the organization is wrong. The problem is that software developers don't think of themselves as professionals. They think of themselves as grunts.

    Your analogy of the bridge is flawed on the surface of it. Managers *will* ask for the extra deck, and there will certainly be managers who won't understand the physics and mechanics involved. They'll browbeat and cajole and wheedle just like any software manager would. Bridge engineers are just better at saying "NO".

    Indeed, the bridge engineers *must* say no. They simply cannot add the deck. No matter how the managers wail and moan, the deck isn't going on. Software developers, however, feel guilty if their architectures are not flexible enough to add new things. In fact, they have often made guarantees that their architecture *will* be flexible. So managers have the expectation that changes aren't difficult.

    And, in fact, the managers are right to expect this, and the developers are right to try to guarantee it. Because software is not steel. You *can* change software. And the more you are able to deal with last minute changes, the more valuable you are as an architect and developer.

    So, the real issue is why developers think software is steel. Professional developers do not build inflexible structures. They build clean, tidy, flexible systems that willingly accept change. Professional developers also know when, and how, to say NO. They just don't say it very often.

    reply | quote

  • Re: why the software industry is broken posted by witten on February 19, 2007 10:29 AM

    I don't think anyone is advocating building completely rigid, inflexible systems that can't tolerate any amount of change. Obviously there's great "business value" in a software system that's flexible and that easily supports developing new features throughout its life cycle. However, what I am advocating is simply some amount of cost-benefit analysis performed before any management decision is handed down to the developers, whether they're grunts or professionals. For instance, although you might have the most well-designed, flexible application, it simply doesn't make sense from a risk management perspective to add large new features to it right before a deadline. And because of the "inversion of knowledge and authority," as you put it, this cost-benefit analysis usually isn't performed with sufficient information to make good decisions.

    Also, although I am a big fan of building flexible, future-proof software, doing so does come at a cost. Making an application less rigid takes more time in both design and implementation stages, and this is a cost that many businesses aren't ostensibly willing to pay. They want it done, and they want it done now. This is another example of how the power imbalance causes problems. The customer-facing side of the business wants a feature done as soon as possible, and unless the developers are abnormally gifted at convincing their superiors, they'll have a rather difficult time making the case that it's worth paying a little more in time and effort to make a more flexible product.

    reply | quote

  • Re: why the software industry is broken posted by unclebob on February 19, 2007 12:42 PM

    <blockquote>it simply doesn't make sense from a risk management perspective to add large new features to it right before a deadline.</blockquote>

    Agreed. However, failure in this regard is not "Why the software industry is broken." To the extent that the software industry is broken, it is broken because of poor communication between developers and managers. Managers have expectations that are not being managed by the developers, and developers have expectations that are not being managed by the managers. As a result the two factions learn to distrust each other and communicate less and less. (Rather like republicans and democrats nowadays).

    IMHO the solution to this is for developers to start acting like professionals by learning how and when to say NO, learning to focus on the business need, and learning to communicate with managers.

    In other words, I think the software industry is broken because we don't have a high enough bar of professionalism amongst the rank and file. Programmers should act like, and be treated like lawyers and doctors.

    reply | quote

  • Re: why the software industry is broken posted on February 19, 2007 07:11 PM

    I think UncleBob has hit the nail on the head when he wrote "In other words, I think the software industry is broken because we don't have a high enough bar of professionalism amongst the rank and file. Programmers should act like, and be treated like lawyers and doctors."

    Developers, in general, are highly intelligent. But, also in general are highly irresponsible (very low emotional IQ). I am not saying that us developers don't pay taxes, break speed limits and have un-safe sex; what I am saying is that most developers tend to blame everyone around them for their failures and being in a broken industry. The issues of the broken industry are not management's issues; they are the developer's issues. Developers only have themselves to blame.

    Coderific is an example of this "lack of professionalism" in the development arena. Has anyone seen a website with "Employer Ratings for Lawyers, by Lawyers"? Neither have we. There is probably a reason for not seeing sites like Coderific for any other industry.

    It is lack of professionalism that is the root cause, as UncleBob points out.

    I have had very few issues with "management" in my 20 years of development. Why? I think the root cause is professionalism and effective communication skills so I can make my case to management why something needs to change. I still have another 20 years of development in me, and my ability to remain calm and make a clear argument for change is what will make those years the most effective.

    This website exemplifies the root cause of the issues - lack of communication of the developer to management. Nasty messages on websites like Coderific only amplify the lack of professionalism in the development world.

    Sorry for being the person to tell the King he isn't wearing any clothes.

    reply | quote

  • Re: why the software industry is broken posted by witten on February 19, 2007 07:23 PM

    I think you misunderstood the original post. As I've stated elsewhere, I wasn't blaming management for the problems of the software industry. I was specifically blaming the way we as an industry structure our companies, which, among other things, causes communication problems among management and the coders they manage. The root cause of problem lies in the organization, not in any particular people.

    Secondly, although I only mentioned this briefly, another real problem that this corporate structure causes is the unwillingness of developers to push back and in general communicate upward. This is just as much of an indictment of the developers as of managers, but in actuality the fault lies with the structure of the organization itself. In a hierarchically managed organization, developers can't be expected to say "no" to their superiors any more than managers can be expected to accept "no" when asking for unreasonable features.

    And as to your general opinion of the web site, if you don't like it, then why are you still here? I'm sure there are plenty of sites out there that are more "professional" and thus more to your liking.

    reply | quote

  • Re: why the software industry is broken posted by kragen on February 19, 2007 10:51 PM

    The mathematics industry as it currently exists is simply broken. A large percentage of mathematical research projects end in utter failure, so something is quite obviously going wrong. The failure often comes in the form of abandoned projects, where the customer doesn't even accept the paper. Quite often though the project will simply miss a series of deadlines and/or not do what the customer actually wants, requiring massive reworking.

    So what's at fault with these mathematical project failures? Is the culprit the development process, the project management, the requirements gathering, or the complexity of mathematics itself? Well, yes, all of the above. But none of those things are actually the root cause.

    ---

    I think you could say the same things about the poetry "industry", the sculpture "industry", the fashion "industry", the biological research "industry", and the startup-company "industry", and indeed the advertising industry, and it would be just as nonsensical. The thing all these "industries" have in common is that if something has already been done before, even with minor variations, there's no point in doing it again. In the case of software, if there's already software out there that solves your problem, you just manufacture another copy of it (say, with apt-get or InstallShield or gcc) rather than designing a new program.

    By definition, if you're doing something that's never been done before, you don't know if it's possible, and you don't know if it will be good when you get done.

    I'm not saying you can't improve on standard industry practice --- lots of software projects fail for stupid reasons, too. But fundamentally a program is a theorem and a painting, not a bridge, so even if you have an egalitarian management structure with flawless communication and "lean manufacturing"-style planning, any software project worth doing will still have a substantial chance of failure. I know: I've worked in a company that was quite close to that ideal, and we still often had to throw away weeks or months of work, and we never produced anything really earthshattering.

    The advertising industry perhaps is the best we can hope for in the way of "predictability": as I understand it, they iterate frequently with their customers, never undertake big projects (are there any advertisements that took a hundred or ten thousand man-years?), and throw away two thirds of their finished products because the customers don't like them.

    reply | quote

  • Re: why the software industry is broken posted by witten on February 19, 2007 11:04 PM

    kragen wrote:
    In the case of software, if there's already software out there that solves your problem, you just manufacture another copy of it (say, with apt-get or InstallShield or gcc) rather than designing a new program.

    Software definitely has a creative component to it, but the same types of software designed to perform the same tasks are reimplemented time and time again, for a number of reasons (economic, legal, not-invented-here syndrome, etc). For example, there are many development shops out there that just continue cranking out the same old CRUD-oriented web applications.

    But fundamentally a program is a theorem and a painting, not a bridge, so even if you have an egalitarian management structure with flawless communication and "lean manufacturing"-style planning, any software project worth doing will still have a substantial chance of failure.

    This, I agree with. My claim, however, is that the chance of failure in most present-day software projects is way too high, higher than it needs to be, and that this abnormally high failure rate stems from the structure of the organizations in which these projects are developed.

    reply | quote

page 0 1 | next