rss feed
blog - fixing power imbalances in the software industry
posted by witten on February 19, 2007
In my last blog entry (http://coderific.com/blog/post/395), I argued that the software industry is broken because of how most development organizations are structured. The hierarchical management structure means that the people furthest away from the decision making, the developers, are the only ones with much of the relevant knowledge necessary to make good decisions. And because of the power imbalance between developers and their managers, this relevant knowledge doesn't get communicated upward to those actually calling the plays. I'm claiming that this structural problem is the root cause of nearly all problems in the software industry, hackneyed bridge metaphors notwithstanding.Okay, so if there's a problem with our beloved industry, what can we do about it? Well, I'm going to discuss several potential solutions, ranging from minor tweaks to massive organizational refactoring. Those of you hoping for a single silver bullet solution may be a bit disappointed, but hopefully you can settle for the several ideas put forth below.The solution that's perhaps the most difficult to implement but requires the least amount of organizational refactoring is simply to listen to developers. This would involve keeping the same old power structures in place, with managers handing down decisions to developers. But the difference would be in asking developers for their input before decisions are made and taking their feedback to heart after the decisions are handed down.Getting developers into the decision-making process could be as simple as engaging developers to provide a voice of technical reason when features are being promised to customers. Additionally, making it company policy to take developer feedback into account on all decisions affecting development could mitigate the aforementioned structural problems. It also has the added benefit of making developers feel like they're a vital part of the decision-making process rather than an outsider whose only role is to reap the rewards of ill-informed decisions.The immediately obvious problem with this approach, however, is that if the power structures are maintained in place, with developers at the bottom of the totem pole, it would take a tremendous amount of organizational discipline to actually put this approach into practice. It's simply going against the grain to rely on people in a position of power to listen to the opinions of those with little power, even if by doing so they would make software projects more successful. Some businesses may be able to pull this off, but it shouldn't be counted on as a general-purpose solution. So perhaps a more major structural change is necessary.In the previous blog entry, I may have given the impression that developers have all or nearly all of the information necessary to make good business decisions relevant to software development. This is simply not so. One of the other major knowledge silos in any software organization is the team or department responsible for interacting with customers. They have valuable decision-making information about what the customers actually want in the software, and perhaps more importantly, what the customer use cases are for their day-to-day activities. Most developers are simply too far removed from the customers to know these sorts of things except at best second-hand.The problem arises when the customer-facing employees are organizationally above the developers, either directly or by proxy. For instance, often the customer-facing employees in an organization constitute the sales department, which quite frequently is headed up by an executive or even the CEO. Thus, when a big sale is in the works, the salespeople can use their position of authority and proximity to executives to tell developers exactly what to do, whether or not it's technically or temporally feasible. The developers often have no one on their side with nearly as much clout as the executives, and so are at a distinct loss when trying, often quite legitimately, to push back.Any solution to this power imbalance that involves simply reversing this relationship will not actually solve things. In other words, if instead of sales being higher on the org chart than software development, you somehow put software developers above sales, then you run the risk of the valuable decision-making information within sales being similarly stifled the way it usually is within development. Developers making unreasonable demands of salespeople ("Sell this thing the customers doesn't actually want, and I want it done by yesterday!") may have some merit from a strictly comedic perspective, but when it comes to actually running a successful business, it's no better than salespeople making unreasonable demands of developers.So if you can't put sales above development and you can't put development above sales, what can you do? You can put them at relatively equal height on the org chart. Note that since a series of labeled boxes and lines is only a rough approximation of the actual power structure within an organization, moving around some arrows on a Visio chart isn't sufficient to accomplish this goal. You've actually got to get the executives out of the sales department so as to remove the sales department's direct line to the throne, so to speak.With the two departments on a much more even footing, then neither department has a distinct power advantage, and neither can make unreasonable demands of the other. Decisions will have to be made in tandem, thereby taking advantage of the knowledge silos within both departments. This leads to the other part of this solution. Once sales and development are more or less equals, the next step is to improve the communication between the two. Accomplishing this can be as simple as sitting sales next to development instead of across the building or in another timezone altogether, or it can be as formal as having the two departments meet on a somewhat regular basis to discuss upcoming deals and potential directions for development.Improving communication will not only improve joint decision-making, but since knowledge no longer has to flow uphill to get from development to sales, it will actually improve knowledge-sharing between the two departments, thereby making decision-making within a single department that much more effective.Note that in this discussion I've focused on the sales arm of a hypothetical organization. If in your company, the structural power imbalance is felt among development and some other department or even set of departments, then you can follow similar steps with those. Remove or reduce the power imbalance and work on improving communication.There is one rather glaring problem with the solution suggested thus far. It takes even more organizational discipline than with less invasive solutions, not only to affect this level of change but also to maintain it and avoid slipping back into the natural state of structural problems you started with. In fact, the solutions suggested would have to be initiated by those in power, and there is very little obvious incentive for them to give that power up, even if it could conceivably improve decision making and therefore business success as outlined here. It may simply be the case that except in the most enlightened of organizations, there's an insufficient amount of organizational will to put these recommendations into place.So instead of fighting the uphill battle of implementing these ideas in an existing organization, another option is to start a new organization and get the power structure right from the beginning. Assuming that the founders have the financial and management wherewithal to actually pull this off, dictating the structure of a completely new organization is easier than you might think.Think of your first day at your current job or one of your recent programming jobs. Assuming that you were an employee and not a founder, you probably came into an existing development shop with fairly well-established relationships to other departments in the company. You may have given the organizational structure some initial consideration as you tried to understand what departments there were and how they would affect your day-to-day coding job, but chances are you just accepted the current structure as the way things are, and were instead concerned with more pressing issues like where the bathroom is located or where the hell your .vimrc file went. New hires in an existing company try to fit themselves into whatever structures are already in place, so as long as those structures are conducive to successfully pumping out quality software, all you should need to do to keep the ball rolling is perhaps implement some basic training or mentoring.When a new business is founded, the founders have an immense amount of control over all aspects of the company, from the corporate culture to how the organization is structured. When Bill Gore founded W.L. Gore & Associates, he had a new idea for a corporate structure involving a "flat lattice organization" with "no chains of command nor pre-determined channels of communication" (http://www.gore.com/en_xx/aboutus/culture/). Now, this particular structure may be a little hippy-dippy for some of you, and Gore and Associates is by no means a software development shop, but the point is that the founders of a company can come up with any sort of hair-brained org chart they want, and as long as it doesn't drive them out of business, they can quite easily foist it on new employees. Gore & Associates, by the way, made $1.84 billion in sales last year.The cost of implementing structural change in a business decreases as you go back in time towards its founding. When it's just two guys in a garage, it's relatively easy to dictate that any sales team you hire must be on equal footing with your programmers. When, 5 years later, you're a 300-person company, making that policy into practice is much more difficult.Another solution to the structural problem is simply not to grow your organization to a size where it becomes a factor. If your company never gets much larger than the two guys in a garage, like the 37signals web app shop, then you don't have to worry about the power imbalance between your sales and development departments, because you won't even have departments. You won't have to worry about managers having insufficient technical information to make well-informed decisions, because you simply won't have managers.Last, but not least, you can do the whole open source thing. There are a number of problem domains that open source solves well (and a number that it doesn't), but if you're developing Free Software outside the constraints of a typical business, then you don't have to deal with any of the power imbalance problems inherent to a hierarchically managed organization. Of course, developing open source software within the confines of a company is an option too, but then you've still got to contend with all the standard corporate structural problems.So, to solve the structural and power imbalance problems inherent to most businesses, you've basically got three general options: mitigation, reorganization, and reimplementation. They're ordered by increasing cost of implementation but by decreasing difficulty of enforcement. You may find some success with the lower-cost mitigation options, but sometimes you've just got to start over with a clean slate.