Coderific: blog

Coderific is back online with help from the EFF!

witten — 2008-09-12

It's been a harrowing couple of weeks, but here's the big news: After shutting Coderific down last week, I had a good chat with a friendly and knowledgeable attorney from the EFF. And I've decided to bring the site back online.

First things first. The fine folks at the EFF (Electronic Frontier Foundation) are the reason Coderific is back up and running. If you enjoy the service Coderific provides, please go donate to the EFF right now:
https://secure.eff.org/site/SPageServer?pagename=DON_splash

Okay, now for the story. Let me start from the beginning.

Throughout just about the entire life of Coderific, I've been harassed by attorneys representing various companies that are discussed and rated on Coderific. Most of the time these were take-down demands for particular posts. Occasionally it was thinly veiled legal threats.

This correspondence was certainly no fun to handle, but as I understood myself to be on stable legal ground as a mere forum operator, I didn't worry about it too much. It was sort of the cost of running the site.

Then, several months ago, various Coderific users started posting in the forums about an ongoing lawsuit between two companies that happened to be listed on Coderific. I started receiving emails, calls, and snail mail from lawyers on both sides demanding that various posts be removed from the site, and also demanding the identities of the authors who wrote the posts.

Having to drive to the post office to pick up certified mail from a law firm was somewhat annoying, but I wrote a quick response and thought nothing more of it.

Then, a few weeks later, one Saturday afternoon, there was a knock on my door. It was a process server with a subpoena from the same law firm that sent the snail mail earlier. The subpoena demanded all sorts of information on the Coderific users who made posts about particular companies, plus all communications made with Coderific about these companies!

Well, having no legal representation and not realizing that the subpoena was overly broad and could probably be fought in court, I started gathering the requested information to comply with the subpoena. I went through email, log files, voicemail, postal mail, and the Coderific database. Let me assure you, this was not exactly how I had planned to spend my Saturday night.

Fortunately, the Coderific software was designed to protect users' anonymity, so even though I felt I had to send user data in response the subpoena, the data available was somewhat limited. For instance, Coderific does not correlate IP address with user accounts, and those IP addresses that are stored in the web server's log file are rotated very frequently. Also, when you create a Coderific account, entering your email address is completely optional.

Having gathered all of the requested data (or at least what was available), I sent off a burned CD to the address on the subpoena. It was then that I thought that all of this simply wasn't worth it. Here I was, running a free site and paying hosting out of my own pocket, and what do I get for it? Constant legal harassment.

Now don't get me wrong. When I originally started Coderific, I realized that a site where coders could rate their employers would upset certain companies, especially those companies that benefited from keeping word from spreading about the particular brand of dysfunction within their own software development departments.

Here was a site that aimed to make information about bad companies (and good companies) as public as possible, so there were bound to be a few organizations angry over their inability to continue business as usual, hiring developers who didn't know what they were getting themselves into.

In other words, I knew that running a site such as Coderific carried with it certain risks, and that it would not be all beer and skittles.

But the final straw was having to spend my weekend worrying about this subpoena and feeling horrible for essentially snitching on my users, all with the implied threat of jail time if I didn't.

So after I sent off the subpoena data, I took down the site. It simply didn't seem worth it to continue. And to some degree, I felt that by keeping the site up and running, I was risking the ongoing privacy and anonymity of Coderific's users, since Coderific was now a subpoena target. And that was simply not a position I wanted to be put in.

In retrospect, what I should have done as soon as I received the subpoena was to find a pro-bono attorney who was willing to at least answer some questions. But I didn't do that, perhaps because I didn't know what other options I had.

A few days later, I received an email from a senior staff attorney from the EFF who had heard about Coderific shutting down, had actually read the subpoena, and was willing to discuss the situation with me. We arranged a phone call, and the attorney walked me through some of the relevant legal issues with Coderific in general and the subpoena in particular.

He basically assured me that as a forum operator, I was not liable for the posts that Coderific users made (which I already knew). And that the subpoena could quite possibly have been fought (which I didn't know). And perhaps most importantly, he told me that the EFF has "got my back" in the case of any future subpoenas or legal issues (which is awesome). In other words, I can contact the EFF and they will either look at a subpoena for me or refer me to an external attorney.

So, with this new information, and no longer feeling quite so overwhelmed, I'm happy to be the one to tell you that Coderific is back!

And here are some changes I've made to help protect your privacy and anonymity. I've greatly expanded the privacy section of the Coderific terms of service, describing exactly what information is gathered, how it's used, and when it's disclosed. Check out http://coderific.com/about/terms_of_service

Also, I've updated the signup page with a brief mention of some of these issues and a link to the terms of service.

If you have an existing Coderific account and would like any private information (such as your email address) removed from the account, just contact me and let me know.

Thanks for all of your support, and a big thank you to the EFF.

Well, what are you waiting for? Go become an EFF member now:
https://secure.eff.org/site/SPageServer?pagename=DON_splash

Coderific subpoenaed in SuccessFactors v Softscape

witten — 2008-08-30

This is a disclosure that Coderific has been subpoenaed in the ongoing SuccessFactors, Inc. v. Softscape, Inc. lawsuit.

In order to comply with the subpoena, I am commanded to produce all documents in my "possession, custody or control that concern, describe, refer to or relate to the Internet Protocol ("IP") addresses of the individuals who created all posts regarding Softscape and/or SuccessFactors on Coderfic, as well as any login tracker information, user names, passwords and email addresses that relate to same."

Additionally, I am commanded to produce all documents in my "possesion, custody, or control that concern, describe, refer to or relate to COMMUNICATIONS regarding Softscape and/or SuccessFactors."

I should point out that as per section 4 of the Coderific Terms of Service, "By using the Service, you acknowledge that Coderific may, in its sole discretion, preserve or disclose your Content, as well as your information, such as email addresses, IP addresses, timestamps, and other user information, if required to do so by law or in the good faith belief that such preservation or disclosure is reasonably necessary to comply with legal process."

keeping things in perspective

witten — 2008-08-28

Here's a video that looks almost as if the creators went through Coderific looking for choice quotes and then wrote them on cardboard signs for developers to hold by the side of the road: http://develop-one.net/blog/2008/08/27/HugADeveloper.aspx

While the video is quite amusing, it does, whether intentionally or not, put things in perspective. You may have a crappy coding job in a crappy company with a crappy boss and crappy schedules. BUT at least you're not actually on the street without a home, a job, or a family, asking for handouts.

This isn't to say that you should be perfectly content working in a crappy organization. Rather, while you're struggling to improve your lot in your cushy programming job, occasionally you should think about those people whose best day is no better than your worst.

happy employees make businesses succeed

witten — 2008-06-29

There's an interesting article/interview with the founders of Whole Foods and The Container Store: http://www.time.com/time/magazine/article/0,9171,1818183,00.html

The idea is that happy, empowered employees beget happy customers. Happy suppliers help too. All this stakeholder joy eventually redounds to the benefit of shareholders--but the magic fades if shareholders become the focus.

This really is quite a radical idea in today's business world. The conventional wisdom says that in a public company, there are at least three competing interests that an executive has to contend with: The interests of the employees, the interests of the customers, and the interests of the shareholders. And of course the executives are not without their own interests.

But what if this conventional wisdom is all wrong? What if it's not a zero-sum game, and making the shareholders happy doesn't mean that you have to make the employees unhappy? These two CEOs have proposed that if a company focuses on the happiness of its employees, all else will follow: Happy customers, happy shareholders, and presumably happy executives.

So is it only a matter of time until CEOs the world over begin to focus exclusively on employee happiness? John Mackey, CEO of Whole Foods, seems to think so:

"Simultaneously we hit upon the philosophy that I think will be the dominant philosophy in business in the 21st century," Mackey says. "It's this principle that the purpose of business is not primarily to maximize shareholder value."

I don't know if I'm quite so optimistic about that. Hopefully, people will use Coderific to rate those few employers who do adhere to this philosophy. If the word gets out that an employer is one of the few companies actually focusing on employee happiness, imagine the edge that they'll have on hiring good coders.

rating a company on its interview process

witten — 2008-03-05

Recently I had a discussion with a longtime Coderific user who brought up the question of where ratings of a company's interview process belong on this site. Someone who has merely interviewed at a company doesn't have the inside knowledge and experience that an employee would have, so giving their ratings equal weight with ratings by actual employees seems somewhat unfair.

But there definitely is some value to seeing how a company treats its potential employees. Is the handoff from one interviewer to another well-orchestrated? Do the interviewers even know what position they're interviewing the candidate for? Are the interview questions a decent way of judging whether the candidate is a match for the job? Or do the questions more closely resemble "On page 219 of Design Patterns, what is the sixth word of the fourth paragraph?"

So where do interview ratings belong on Coderific? After some consideration, I've come to the conclusion that they belong in the discussion forum for each company. Every employer on Coderific has its own forum, which seems like a perfect place for posting about an interview process, even if you were never a full-blown employee. This has the benefit of still allowing those who interview at a company to provide valuable feedback on it, without encroaching on the ratings of people who have actually worked there.

I should point out that this is already happening. Just today, someone posted a couple of interview commentaries in the respective discussion forums for the companies in question. The only change I would make to promote this approach is to make it more explicit on Coderific that interview ratings are welcome, as long as they're posted in the forums. Right now, interview ratings are barely mentioned.

If you've got any feedback on the topic, make your comments below. I'm interested to find out what people think.

Coderific mentioned in Computerworld

witten — 2008-02-25

Coderific was recently mentioned in a Computerworld article (http://www.computerworld.com/action/article.do?command=printArticleBasic&articleId=9057899) in which they interviewed me and several other people about the topic of rock star coders. In case you don't remember my stance on the issue, please refer to my previous blog post on the subject titled "why you don't want to hire rock star programmers" (http://coderific.com/blog/post/564). Computerworld calls it a diatribe. I'm flattered.

learning and productivity in a micro-ISV

witten — 2007-11-11

Several months ago, I mentioned that I was starting my own software company (http://coderific.com/blog/post/645). Well, I'm pleased to report that I'm just about ready for a 1.0 product launch, even if the software as it stands is currently rather minimalistic. The product is a WYSIWYG personal wiki called Luminotes, and you can check it out at http://luminotes.com/ if you're interested.

Anyway, after several months of slaving away on my own software, I've learned several important lessons that I thought would be worth sharing with you, dear Coderific reader. When I worked in the corporate world, one of the benefits was that I was constantly surrounded by smart software developers, so it was almost guaranteed that I'd pick up new development tips from them. It was in the corporate world that I really internalized test-driven development by working with a team of people who wrote unit tests for a living. And it was working for a corporation that I really became halfway decent at C++, picking up tricks and best practices from other developers.

There were just so many things I learned from other developers on a daily basis. So when I finally left the corporate world to sit in my pajamas all day working on my own micro-ISV, one of my fears was that I'd stop learning. Not being constantly steeped in a team of other developers, I worried that I would fall behind, no longer picking up the tips and tricks that kept me sharp.

Well, I'm happy to report that my fears were completely unfounded. It is possible that by not being around other software developers all day, I missed out on learning some particular topics, but there's a whole slew of things to learn when you're working by yourself, things that simply aren't learnable when you're in a busy office environment.

When you work for a big company, your coding productivity is almost always bounded by things beyond your direct control. Perhaps you get interrupted frequently or have to attend inane meetings all the time. Or maybe you waste time struggling with poorly documented libraries or dealing with office politics, rather actually getting code written.

But when you work for yourself, your coding productivity is mostly bounded by your actual ability to pump out good, working, debugged code. There are still the occasional speed bumps, but if you structure your day properly (and you have immense control over doing so), then you can minimize interruptions and productivity drags. And if you're a one-man software shop, I'm guessing you don't spend a lot of time in meetings.

When your productivity is no longer constrained by your environment, the only constraint is yourself: How well you can concentrate and keep your butt in that chair, how much discipline you can summon to write unit tests, how well you can deal with tricky coding problems. Suddenly, when working for yourself, you are the limiting factor, rather than the organization you work in.

This shift opens up immense opportunities for learning, opportunities that you simply don't have in a corporate environment. Want to learn how to keep yourself coding for 8 hours at a time with only a few breaks to clear your mind? Now's your chance. When you don't have a meeting scheduled every two hours of every day, suddenly you've got to learn how to focus. Want to know how to stay motivated through all the ups and downs of a typical project? When you're on your own, you are the only person cracking the whip, not teammates and certainly not pointy-haired bosses. Ever wonder what time of the day is your most productive for doing specific tasks like debugging, design, algorithmic analysis, and so on? When you run your own micro-ISV, you can structure your day however you like, so it's a perfect opportunity to find out.

Simply put, one of the best ways to improve your productivity, or at least start learning what could use improvement, is to quit your job and start working for yourself.

don't blindly fix bugs before writing new code

witten — 2007-10-07

One of the questions on The Joel Test (http://www.joelonsoftware.com/articles/fog0000000043.html) is "Do you fix bugs before writing new code?" Joel gives a few very reasonable justifications for why this is a good idea:

In general, the longer you wait before fixing a bug, the costlier (in time and money) it is to fix.

Also, Joel points out that:

There's another reason, which relates to the fact that it's easier to predict how long it will take to write new code than to fix an existing bug. ...if you have a schedule with a lot of bugs remaining to be fixed, the schedule is unreliable. But if you've fixed all the known bugs, and all that's left is new code, then your schedule will be stunningly more accurate.

And lastly:

...if your competitor introduces a killer new feature that is stealing your customers, you can implement just that feature and ship on the spot, without having to fix a large number of accumulated bugs.

Fixing bugs before writing new code is certainly a good general rule of thumb for exactly these reasons. But it is not always the best choice in any given situation. Allow me to explain why.

It's often said that highly skilled programmers are an order of magnitude more productive than poor or even average programmers. That may be true. But even within the confines of an individual programmer, productivity can vary greatly over the course of several days.

Whether or not you count yourself among the set of highly skilled programmers, I'm sure you've had days where you've gotten almost nothing done, and not for lack of trying. You stare at your code, looking for the cause of a bug. You get some coffee. You stare some more, adding a few prints. You check Reddit and cycle through your email. You go back to your code. You close your door or put on headphones. You try a few ideas out, but to no avail. Before you know it, it's time to go home, and you've gotten almost nothing done.

Other days, you're amazingly productive. Code falls out fully-formed from your head right into your IDE. You almost intuitively know what's causing a bug, and after a few minutes, it's fixed. You're, well, on fire. Not in the overheating lithium-ion laptop battery sense, but more in the NBA Jam sense of the word (http://www.youtube.com/watch?v=X7dFMbubxr4).

What accounts for this great variability in productivity within a single programmer? Well, as you can imagine, there are a number of factors. Maybe on your unproductive days you're having trouble concentrating due to constant interruptions, or perhaps you just bricked your iPhone and you're too sad to focus on your code. Still, despite these problems, the number one cause of variable productivity within a single programmer is that programmer's motivation.

On your best productivity days, you're interested in the problem you're working on, you've got everything you need to get your job done, and you're invested in the outcome. On your worst days, you're feeling like your efforts will make little impact, you're fed up with management, or you're just tired of the problem at hand. So other than slight variations due to interruptions, bricked iPhones, and so on, an individual contributor's motivation is the single greatest determiner of their personal productivity level.

Now, back to the original topic. When Joel asserts that you should always fix bugs first, he's leaving out a very important component from his "What should I do next?" task selection algorithm. He's making the selection of his next task based solely on the relative priorities of the available tasks. In other words, he's claiming that bugs are higher priority for the reasons stated above, and therefore they should be fixed first.

But just because something is the highest priority doesn't mean it's the next thing you should work on. This sounds counterintuitive, but a programmer's productivity, their individual contribution to making a product customers want, is like a function with many inputs. One input is the priority of the task they're working on. Another input is that programmer's motivation. By just blindly working on bugs first, Joel is only looking at the priority input while completely ignoring the motivation input.

There are two rather interesting properties of a programmer's motivation. One is that you can consciously manipulate your own motivation, with varying degrees of success, and the other property is that your current motivation has a fair amount of momentum. In other words, it tends to change slowly over time, at least in the absence of stimuli that cause it to spike or crash.

So, if you're getting frustrated and upset when unsuccessfully tracking down a particularly difficult bug, and therefore wallowing in the depths of unproductivity, sometimes it can help tremendously to take a break to clear your head and get motivated again. You can often accomplish this simply by going and implementing some small, well-defined feature. By putting aside the bugs for a bit, you can finish an easy task and get your productive juices flowing again, thereby giving a much-needed spike to your morale. Then, once the feature is done, you can go back to tackling that annoying bug with renewed motivation and a clearer head.

In general, whether or not you're working on bugs, you can often get a huge morale boost simply by stopping work on a boring high-priority feature in order to work on a more interesting lower priority feature. You may even end up getting the high-priority task done sooner, and with at a higher level of quality, than if you had just slogged through it without taking a break to code something more fun.

Also, because of the tendency of motivation to stay at its current level for a while (in the absence of shocks like "Oh, by the way, this all needs to be done by tomorrow"), if you take a break to code up something that gives a boost to your motivation, that new level of morale can carry you throughout the day while you bang out less interesting tasks. One might even go so far as to say that the best programmers are experts at personal motivation management. They know how to manipulate their own morale such that they get the most done, even if that means not always working on the absolute highest priority tasks all the time.

So, just to clarify, I'm not suggesting that Joel's "fix bugs first" rule is a bad idea. All I'm proposing is that like with any good rule, it pays to know when to break it.

of umlauts, unicode, and busted rss feeds

witten — 2007-09-10

It came to my attention today that multiple Coderific rss feeds have been busted for quite a while. I really should've noticed earlier, given that I'm subscribed to a few of the feeds myself and none of them have been updating in my rss reader, even though new ratings have been pouring in on the site.

The specific bug turned out to be a CherryPy problem in which one of the handlers barfed unceremoniously on unicode strings (http://www.cherrypy.org/ticket/511). Since Coderific deals with everything in unicode, all that it took to trigger this bug was for someone to enter an employer rating with an umlaut, and then boom, no rss feeds. The fix was to apply one of the CherryPy patches described in that ticket.

The moral of this story is threefold. First, I need more automated tests. Not just unit tests, but end-to-end functional tests for things like umlauts breaking HTML generation. That's really the sort of thing that can be tested ahead of time.

Second, I need to keep a better eye on the site. I shouldn't assume that everything is humming along just because I haven't heard about any problems from Coderific's users. In fact, just this morning I thought to myself, "Why, Coderific is quite an engineering achievement! It seems to keep on running with very little maintenance." I suppose the gods of irony heard that thought and took the matter into their own hands. And I wouldn't mind at all if you, dear Coderific user, emailed me if you notice anything going even slightly awry on this site.

And finally, unicode in the current versions of Python is like a second-class citizen. Everything defaults to 7-bit ASCII by default, and you have to go to great lengths to make a program unicode-aware. Even if you do make the program use unicode strings throughout, as is the case with Coderific, you've still got to worry about all the libraries you're depending on, and how well they play with unicode. I'm really looking forward to Python 3, in which all strings will be unicode by default.

So if you suddenly received a deluge of Coderific rss updates in your reader today, now you know why: Insufficient tests, attention, and unicode!

the resume keyword game

witten — 2007-09-07

Slogging through all the pics and video links on Reddit (or as my wife likes to call the site, "Sawit"), I came across an actual textual article with words and such. It was titled "Why your resume annoys employers" (http://www.cnn.com/2007/LIVING/worklife/08/29/cb.resume.irks/index.html). As can be expected of anything from Careerbuilder, the article is fairly worthless, and full of well-duh resume suggestions like "...your r�sum� could do without: 1. Spelling mistakes and grammatical errors." No shit! I'm glad I checked Careerbuilder before sending out my resume, because I thought spelling mistakes would make me stand out and also demonstrate my creativity!

Anyway, the reason I'm writing about this is because the image included with the article is absolutely priceless (http://i.l.cnn.net/cnn/2007/LIVING/worklife/08/29/cb.resume.irks/art_resume_irks.jpg). Amid the text of the article quoting HR experts about their resume pet peeves is an image of a resume-scanning machine, which appears to be picking keywords out of the resume and ignoring all other text.

This is mildly amusing, because while the HR experts are trying to tell us that we have to do this and that to make our resumes look "professional", what they're not telling us is that 99% of successful resume writing is coming up with keywords that the HR computer is looking for. Then, once the computer has found acceptable keywords, you need the right buzzwords to get past the human HR resume filter.

But the image embedded in the article is slyly revealing this secret of the keywords, a secret that the actual article text never mentions. Oh, sweet, sweet juxtaposition!

So yeah, spell-checking your resume isn't a bad idea. And it's probably prudent to skip the perfume/cologne spritz. But these foibles pale in comparison to the importance of having the correct keywords in your resume. Read the job description, match your resume to its keywords, and you'll have a better chance of getting your foot in the door than anyone who merely follows expert resume advice and neglects to play the keyword game.

obnoxious jerks in comic format

witten — 2007-08-31

A while back I posted about how obnoxious jerks are a pox upon our civilization (http://coderific.com/blog/post/597). In it I included an imagined verbal exchange between a micromanaging obnoxious jerk caveman boss and his caveman subordinate who was trying to invent fire. And then today I found a comic by Joe Sayers in which he (unwittingly) illustrated that very exchange! Check out http://www.jsayers.com/thingpart/thingpart119.html

if your manager ran a role-playing game

witten — 2007-08-29

In my previous post (http://coderific.com/blog/post/659), I wrote about the striking similarities among software development managers and Game Masters:

Perhaps management training courses should consist of all the managers running their own D&D games. For a chuckle, try to imagine some of your previous managers adjudicating a dispute between two players about grapple rules. I'll bet that your best managers could handle such a situation without breaking a sweat.

Then Ravenhawk made an interesting point in the comments:

If only having those qualities which immediately would cause a lousy Game Master to lose their players would equally negate the existence of poor management...

So what if a typical manager did become a GM and went about looking for players for their campaign? Well, here's what the posting on craigslist might look like:

Looking for a rewarding Healing Engineer position in a fast-paced, team-oriented environment? Are you a self-starter who constantly seeks out challenges? Do you have a passion for healing? Look no further. Come help us build the D&D campaign that will be leveraged to change the world forever.

Founded in 2007, iDungeon, Inc. is a leader in fantasy-based offline social networking. As a successful, growing campaign, iDungeon is poised to become one of the largest granter of experience points in the world, with over 700% growth in experience awards this year alone.

As a member of the iDungeon team handling Hit Point recovery, your responsibilities will include:
- answering healing-related support requests from other team members
- diagnosing viruses, diseases, and poisons
- implementing enterprise undead-turning solutions

Required:
- 10 years experience in HEAL
- 7 years experience in MACE and/or SHIELD
- Extensive Knowledge Religion skills
- Bachelor's degree in Pelor or related field

Desired:
- Strong turning DC
- Experience in Lich-infested environments

iDungeon offers the following benefits:
- Comprehensive Spot/Listen/Heal coverage
- Matching GP contributions
- Very competitive XP awards
- Tavern membership included
- Many exciting opportunities for character growth
- And much, much more!

Principals only. This position may involve some travel. iDungeon is an equal opportunity employer. But please, no gnomes. Anything but gnomes.

what Game Masters and managers have in common

witten — 2007-08-26

Yesterday I spent about 16 hours at Penny Arcade Expo (http://pennyarcadeexpo.com), a video game and tabletop gaming convention. I think I consumed more caffeine there than I did in the entire month of July. Anyway, while hopped up on coffee and energy drinks, I attended one of the panel talks on the future of tabletop RPGs. One of the topics discussed was what makes a good Game Master (http://en.wikipedia.org/wiki/Game_master).

It struck me while listening to the discussion that a good Game Master has a whole lot in common with a good software development manager. Here are some of the similarities.

Both a good manager and a good GM are expert illusionists. The best GMs create the illusion of a fully interactive, well-detailed, living, breathing game world where the players can do whatever they want. They often do this by coming up with a detailed plan for the world and the adventure, while still being able to improvise and veer off course at times to handle whatever crazy ideas the players can dream up. An expert GM knows how to prevent reality from seeping in to undermine the illusionary world they've created.

The best managers, likewise, create the illusion of the ultimate software development environment. Acting as a sort of shit umbrella, they shield their developers from the ongoing mid-to-upper-level management political machinations. Shielded developers write their code under the illusion that they only need to digest requirements, check in code, fix bugs, and hold the very occasional meeting. The expert manager doesn't let the reality of all the other ongoing bullshit seep through.

Good GMs and managers also have in common the ability to cede control without giving it up completely. The best GMs know when to step back and let the players take the reigns of storytelling, rather than railroading them down a particular path. That way the players feel like they're part of the story rather than simply being told what to do, and everyone has more fun playing. And yet a good GM can still keep the players from going too far off the beaten path, gently leading them back into the story as necessary.

Similarly, the best managers know that the Command and Control style of software development management (http://www.joelonsoftware.com/items/2006/08/08.html) simply leads to pissed off, unproductive developers. Instead, they give all their developers a voice in what gets developed and how it's implemented, but without letting anyone run completely wild. That way, everyone feels like their opinion matters and has a stake in the final product.

Finally, the best GMs and the best managers are excellent arbiters. When it comes time to decide which of two developers' mutually exclusive ideas actually gets implemented, the veteran manager will facilitate the discussion such that a decision can be reached, without alienating anyone or making arbitrary rulings. And the veteran GM will know how to resolve conflicts such that the players feel the solution is fair and rational, even if they don't completely agree with it.

Anyway, I could go on with the parallels between good GMs and managers for a while. The analogy isn't perfect, but I do think it's a fairly compelling one. Perhaps management training courses should consist of all the managers running their own D&D games. For a chuckle, try to imagine some of your previous managers adjudicating a dispute between two players about grapple rules. I'll bet that your best managers could handle such a situation without breaking a sweat.

Oh, one last similarity: Both good GMs and good managers are worth their weight in gold.

the real reason I started Coderific

witten — 2007-08-14

I'm going to let you in on a little secret. I had an ulterior motive when I created Coderific.

When I first tell people about this whole employer rating web site thing, they usually assume that I created it simply for the benefit of my fellow coders the world over, or that I was attempting to reward the good employers and punish the bad ones, or possibly that I'm just a dirty hippy with no respect for the fine, upstanding corporations that employ the very software developers writing nasty things about them on this web site.

Well, that's all more or less correct, although I think I missed the hippy movement by a good forty years. Even though I did create Coderific for all of these reasons, there was still another driving force behind the creation of the site, the real reason I slaved away after I got home from work every day, coding into the wee hours of the morning until the site was finally complete and ready to wow the world with its breathtaking utilitarian design and cutting-edge, populist ideals.

I wanted a job.

That's it. That's the big secret. I created Coderific to get hired. I didn't just want any job, however. I started this site because I wanted to find a good job, at a good company, working on interesting problems with interesting people. I started this web site for myself. If anyone else happened to benefit from it, well, that would just be a really cool side-effect.

Here's how the plan was hatched. Some time in 2006, I thought to myself, "I would really like to work for a company that knows how to treat software developers well. If only there was some way to harness the collective wisdom of all the coders out there..." And thus was born the idea of putting together an employer ratings web site. Then, once the site was in place, I would only have to sit back and wait for the employer ratings to start rolling in, and before I knew it the best employers would be automatically tabulated, sorted, and ripe for the resume-sending.

It seemed like a brilliant plan. In hindsight, it was really quite a daft idea. I created a whole web site because I didn't know how to go about finding a good place to work. It's like Jeff Bezos starting a little web site with product reviews called Amazon.com because he didn't know how to go about finding a good vacuum cleaner.

So a few months ago, my job slinging code at a soulless corporation ended. Rather abrubtly. I didn't feel too bad about it, because although the timing wasn't exactly of my choosing, the feeling was certainly mutual. Oh, and half the workforce was let go a couple of days later. And there was some sort of hostile takeover. And then there was that whole business about not getting paid.

Anyway, that's all the past. Parting ways with my employer did give rise to an opportunity. I could now finally put Coderific to the test and find myself that dream job! All I had to do was send in my resume to the best-rated employers, ace a couple of interviews, and before I'd know it I'd be writing code on a beefy dual-head box in a forward-thinking, well-run organization.

That was the theory anyway. I applied a few places, I did a few interviews. After several weeks, it began to dawn on me that perhaps I had gone about this problem of finding my dream job all wrong. Maybe, since I had such specific ideas about what a good software development organization should be like, my ideal role wasn't within an existing company. Perhaps, I was better suited to blazing my own path.

This thought wasn't entirely without precedent. When I was a teenager, I owned my own software company. Now, before you start getting all impressed, let me explain exactly what I mean by that. I produced a tiny DOS shareware program that, for the low, low registration cost of $5, would uudecode binary files off of Usenet for you. Unless there was an error. In which case it wouldn't. (Barbara, if you're reading this, try the "/d" option and see if that helps!) I think a total of four hapless souls actually registered the program, which worked out to something like a whopping 25 cents an hour.

Despite the rather modest revenue, going into "business" for myself was a huge rush. People not only downloaded and used my program, they actually voluntarily parted with legal tender in exchange for it! At the time I was fed and housed by my parents, so actually living off the "income" wasn't a concern. I got to write what I wanted, how I wanted, and the only real constraint was that the software had to be moderately useful in order to be a viable product.

So all of this got me thinking. Even though my reason for starting Coderific was to land a job, maybe I'm going about things incorrectly. Maybe, since I presume to know how a software company should be run, I should "put up or shut up", and try to start a company myself. I'm being granted this golden opportunity in the form of sudden unemployment. All I have to do is bathe irregularly, whittle down my life savings, and find within myself an affinity for all things ramen. Oh, and spend several months trying to write some kick-ass software that other people will use of their own volition.

So that's pretty much what I've been doing the past couple of months. My goal is to make a viable software product that enough people will find interesting and useful such that I'll be able to pull down wages slightly exceeding 25 cents an hour. I'll let you know how that works out.

In the meantime, I hope Coderific continues to serve the needs of those who are still seeking that dream job within the sphere of companies that actually exist. For the rest of you, well, I hope you're at least enjoying the posts.

your source code is worthless

witten — 2007-07-14

Yes, it's that time again: Time for another tech industry bedtime story.

A few years ago (so, back in the dawn of time), I used to sling code for a small software company. We had developed a consultingware product for one of our customers that had absolutely nothing to do with our main business model nor our core competencies. Let's call it a widget database application.

One day, one of our executives was looking for new sources of revenue, as executives are often wont to do. He had a brilliant idea: What if we took this widget database application that we had developed with requirements specific to the whims of a single customer, cleaned it up a bit, and then resold it to dozens of other customers as an enterprise widget database application? The obvious answer, and the one that spurred all further decision making on the matter, was that this scheme when implemented would make us all exceedingly rich, such that we could bathe in actual cash.

The inherent flaw in this brilliant idea.. Wait, let me back up a bit.. One of the many inherent flaws in this brilliant idea was the assumption that a group of developers can somehow start with code written to soothe the temper of one particular cranky customer, wave some sort of magical "cleanup" wand, and thereby generalize the code so that it soothes the simultaneous tempers of many dozens of cranky customers.

Allow me to quote Dennis Forbes (found via http://cysquatch.net/blog/?p=43):

The question every organization needs to ask itself, then, is what value they could sell their "reusable code" for - what, realistically, would competitors and new entrants in the field offer for it? The answer, in almost every case, is $0, and they wouldn't want it even at that price. There is extraordinarily little code theft in this industry (even though we're in the era of burnable DVDs and USB keys) because most code - above and beyond the industry-wide frameworks and libraries - has no value at all outside of a specific project with a specific group of developers. Trying to use it for other projects is often worse than starting with nothing at all.

Let me ask you this. Think of all the proprietary code you've left locked up within a company as you walked out the door for the last time. Of all that code, spread over all those years, how much of it do you think was potentially valuable for use in other software? In other words, what percentage of that code could you honestly reuse if you made a copy of it before you left the building? 20%? 10%? 5%? I don't know about you, but at most places I've worked, I'd be lucky to break 1%.

Anyway, back to the tale. Undaunted by this minor flaw in his plan, the intrepid executive started shopping around the widget database application to other potential victims/customers, even before any cleanup development work began. After all, once we had a signed contract, generalizing the application to actually be useful to other customers would be a cinch, right?

While all of this was going on, some of our developers gave notice. Specifically, it turned out, these developers were working on the very widget database application that our executive was so hot to resell. The executive wondered aloud whether these quitting developers, now obviously harboring ill intent, could potentially make off with the precious widget database source code like so many bandits in the night. The implication being, of course, that the widget database source code was something of great value that a departing developer would be interested in stealing for fun and profit.

At this point in the saga of the widget database application, I was tasked with monitoring the departing developers to make sure they didn't actually steal any highly valuable source code in their last couple of weeks employed by our illustrious organization. I'm sorry to say that, as far as I was able to discover given the limited means at my disposal, no highly proprietary widget application source code was stolen, nor was it sold on the black market for untold millions. And by the time I eventually left the company, the hapless executive still hadn't managed to sell a single copy of the widget software, despite his guaranteed recipe for success.

So what went wrong here? Our executive was laboring under the false assumption that our one-off, internally produced software was something of value, something that other customers would find worthwhile. Every one of his moves was based on the a priori premise that our recycled software would sell like hotcakes, and therefore its source code should be jealously guarded.

This sort of widespread belief grips the software industry like a disease. Most software companies revere their source code as if its escape would somehow give their competitors the keys to immediate riches. It is this very fallacy that prevents many software shops from even considering open sourcing their software. If their source is in and of itself a thing of great value, subject to reuse and thereby profit, then open sourcing it would be like just giving everything away!

Of course, once you realize that the majority of your organization's source code is essentially worthless outside the very artificially constructed womb that is your particular development team with your specific tribal knowledge, you begin to realize that open source is not at all a threat to the sanctity of your source code, nor does it in any way threaten your bottom line.

A few weeks ago, a friend asked me if he could reuse the Coderific source code for another type of unrelated rating site. Coderific is licensed under the GPL, so he didn't even have to ask, but I of course gave him my blessing anyway. The last I heard, he was having trouble getting it setup, and even if he does get the source installed and configured, there still remains the not inconsiderable challenge of adapting software written specifically for employer ratings to a completely different type of rating site.

So even Coderific source code is essentially worthless. The only real value is in the site that's built on top of it, and user contributions in the form of employer ratings. Without that, it's just a pile of project-specific, internally produced code, very similar to code produced by companies all throughout the industry. The main difference is that most proprietary software shops think that the only reason no one is stealing their source is because they keep it hoarded away in a cave somewhere. The reality is that no one wants to reuse the vast majority of source code out there, even if it's completely open and ripe for the taking.

we're better off without obnoxious jerks

witten — 2007-06-15

I just read Ian Olsen's post entitled "Civilization Was Built By Obnoxious Jerks" (http://ianolsen.wordpress.com/2007/06/14/civilization-was-built-by-obnoxious-jerks/). The basic gist is that the people who actually get important things done are those with an inflated sense of self worth, and so having those people around is a good thing. Here's his justification:

Have you noticed the people heavy on bravado and light on skill often wend their way into positions of authority? Is this simply the typical workings of a clueless bureaucracy or something else? Both. Here's the thing: if people did only what they were qualified to do, things for which they had already proven some aptitude, we'd still be living in caves. The caveman that built the first fire was certainly not qualified. What a pompous jerk this guy was. The other cave-people were doing just fine huddling together for warmth and eating raw meat, thanks. And it was so obvious that he had no idea what he was doing. Rubbing sticks and banging rocks for hours? Nice going there, Lothar. Do us all a favor and spear a woolly mammoth or something, would ya? But you have to admit that the whole fire thing worked out pretty well.

Given my earlier post discouraging the hire of arrogant "rock star" programmers (http://coderific.com/blog/post/564), you can probably guess what my take on this topic is. So I could give you a well-reasoned, fact-checked argument for why Ian is wrong and obnoxious jerks are indeed the bane of civilization. But that would be boring, especially at one in the morning. So instead, I'll treat you to an alternate rendition of the events leading up to fire's invention, starring Lothar the obnoxious jerk, and Morg the average cave-dwelling Joe.

Lothar: How's that fix for the whole "cold" issue coming along? I promised people that it would be done today.
Morg: Well I'm researching a number of techniques, one of which involves...
Lothar: You're not using that rock there are you?
Morg: Well yeah, I thought that maybe if I...
Lothar: You can't keep people warm with a rock. I just read this cave painting that said all the big caves are switching to mud for their warming needs. Why don't you try that?
Morg: But, you see, I think with a rock I could...
Lothar: Start working on a real solution. The chieftan is expecting results.
Morg: Okay, but I think I could use some help if you don't mind.
Lothar: Sorry, no time. The chieftan and I are going out for woolly mammoth.
Morg: I'll just get started on the mud then.
Lothar: You do that.

Now I'm not arguing against trying new things or attempting tasks for which you're not traditionally qualified. And I'm in no way promoting passive coders who can't stand up to management. But I will claim that often, in many organizations, the very people who are too big for their britches are the ones who are actually detrimental to productivity. People who rise into positions of authority are only occasionally rewarded these positions based on merit. It's usually the appearance of confidence and the firm handshake that gets you the promotion, not any sort of actual ability. But I promised I wouldn't try to make any sort of coherent argument. So, back to Lothar.

Lothar: Okay, are you about done?
Morg: Well, the mud idea doesn't seem to be working out all that well.
Lothar: Are you kidding me? It's past dark! We need a warming solution now.
Morg: I could go back to the rock approach. If you gave me a few more rocks to work with, maybe some flint...
Lothar: That will never work. Hmm.. Have you tried trees yet?
Morg: Trees?
Lothar: Yeah, try working something out with that tree over there.
Morg: Uh, I suppose I could try...
Lothar: Well, look at the time. I'm going to catch some shuteye.
Morg: I'll just be right out here... With this tree...

project management is a problem

witten — 2007-06-10

In a recent post, Reg Braithwaite shared a post-mortem on his software designed to help project managers, and in the course of doing so, opined about what's wrong with project management (http://weblog.raganwald.com/2007/06/still-failing-still-learning.html):

Project management is a social problem. It is 99.5% about getting everyone who knows something about the state of the project to share what they know with everyone else. Getting all the relevant information is 99.5% of the problem, analyzing the information is 0.5% of the problem.

Putting aside for the moment the unintentional implication that "project management is a problem", I'm immediately reminded of a particularly appropriate example of this sort of project management problem at my last coding job.

My team was working on a big, important release. Our project manager had this Agile project management reporting tool that ostensibly tracked bugs and development tasks. At the start of the release cycle we had written out all our tasks on 3x5 index cards. (Yes, this was Agile with a capital A. Well, except for the whole lack of communication part.)

As the dev cycle progressed, we handed this manager cards for completed tasks as we finished them. All she had to do was input this data into the project management reporting tool, and it would magically crunch some numbers and show a big green, yellow, or red light depending on whether the numbers indicated that the project was on track, veering off course, or crashing and burning, respectively.

Wait, that sounds familiar. What exactly was it that Reg was developing?

So I set out to write a piece of software that, pure and simple, would look at a software development project and show you a traffic light: a green light would mean that the project looks like it's on track, a yellow light would mean that the project needed help, and a red light would mean that there is no hope.

Now I'm sure Reginald's project was much more sophisticated than the crappy little web-based Agile reporting tool that my manager was using, but the difference doesn't matter, because analyzing all of the information that a project produces really is only a tiny part of the problem. The rest of it is actually coming up with and communicating the information you're going to analyze.

So, back to the story. My manager, after crunching the numbers and coming up with a shining green light, scurried over to her own manager and proudly displayed the good news. Green light: the project is on track! The software says so! Nothing to worry about! Everyone was happy. That is, everyone in management was happy.

Fast forward several weeks. The release was late. Very late. And the end was still not in sight. My manager's manager was furious. He had been shown green lights every day up until the very date of deadline. Why was my manager reporting that the project was on track if it so obviously wasn't?

Ah, and therein lay the problem. Up until the end, it was anything but obvious that the project was veering off course. It may have been obvious to a developer or two that there was this or that problem, this or that unmitigated risk. But since no one was really communicating via anything other than hastily scrawled notes on 3x5 index cards, none of this distributed knowledge was really captured. Any good, meaty project management information such as potential risks and areas of uncertainty were simply not gathered together into one person's head. No amount of analysis would've helped because, well, garbage in, garbage out.

Long story short, my manager was unceremoniously fired soon after the release. Not because the deadline was missed, but because the deadline was missed while she was telling everyone that the project was on track. She definitely had a hand in the poor communication, but the developers were at fault too. Everyone was too blindly following the "process" without really thinking about anything that might lie outside of it. Like, say, communicating their own perception of whether things were actually on course.

I am continually amazed at how, after a project fails, anyone bothering to do even a cursory post-mortem can easily pick out many of the warning signs in retrospect. Such as.. The new platform hadn't been tested at all a week before the release. Or, one of the riskiest parts of any project, integration of software from different teams, wasn't included at all in original time estimates. Or maybe a major bug that QA found early on wasn't getting any attention from developers.

In fact, it's often the case that these warning signs are actually acknowledged by developers or testers during the release cycle as potential trouble areas, but for whatever reason (social, political, or cultural), these people fail to report the risks upward to the managers responsible for gathering and analyzing all of this data. In other words, at any given point during development, all the information you need about whether a release is on track is out there distributed among the minds of the developers and testers. It's just not being communicated.

Reg's post suggests that prediction markets may be the solution:

Sitting here typing this, I think the company who can do the best job of predicting the outcome of software development projects is Inkling Markets. That's because their entire business is about finding a way for people to communicate what they really think of something, not just what they think other people want them to say about something.

That would actually be pretty cool: Setting up a prediction market within an organization to represent everyone's collective confidence in the release being on course. However, I think there's an even easier solution. Simply ask people what they think! No need for fancy buying and selling of virtual shares. If the project manager just went around periodically asking people working on a release about their opinion on the state of the release, a lot of the bottled-up warning signs and valuable risk information could actually be communicated appropriately, well before the post-mortem.

With this approach there's still the potential of social and cultural factors getting in the way of communication. People rarely like being the bearer of bad news, and some corporate cultures go out of their way to discourage anyone from piping up. But despite these challenges, simply asking people their opinions on an upcoming release would yield a hell of a lot more valuable risk information than hoping that index cards or Bugzilla tickets can accurately capture and communicate this sort of data. As most projects are run now, people might grumble that this or that risk isn't being addressed, but unless they're directly asked for their opinion, they'll continue filling out 3x5 cards and writing code without mentioning a word about their concerns.

The other side of this is that any manager polling the opinions of the populace has to actually be receptive to what he or she discovers. If three people say that the Foobar component isn't getting sufficient testing and could very well blow up right before the release, well then the manager has to be willing to turn the green light to red and make sure something's being done about the problem.

Because project management is a social problem, everyone needs to be part of the solution. Not just managers, not just devs. Solving communication problems falls on everyone's shoulders.

the power of code samples in interviewing

witten — 2007-05-30

My admittedly rambling post the other day about not hiring "rock star" programmers (http://coderific.com/blog/post/564) generated a fair amount of feedback, both positive and negative. So I thought I'd go a step further and outline exactly how I recommend looking for good, non-rock-star programmers. Fortunately, Giles Bowkett has already done this for me (http://gilesbowkett.blogspot.com/2007/05/programmer-interviews-two-warning-signs.html):

There are a lot of interviewers who will ask you trivial syntax questions, like, what does ^ mean in Ruby? ... Unfortunately, a lot of companies know that these questions are trivial questions, and they know they're supposed to abandon the practice, but they don't know what they're supposed to do instead, so they just ratchet it up a notch. Instead of asking you questions which are trivial because any idiot could answer them by going on Google, they ask you questions which are trivial because any idiot could answer them by pulling Knuth off the shelf. Questions like, "what is a tree sort? What is it good for?" You can give them credit for having higher standards, because they make the hypothetical idiot go to a more sophisticated source, but they're still asking questions which any idiot could answer with access to the right reference materials.

Giles then goes on to say:

The best way to hire programmers is to read their blogs and look at their code.

I completely agree with this way of judging programmers. But why is this a good approach for accurately assessing the schmuck who you've brought in for an interview? Why isn't it sufficient to simply ask about the running time of heap sort and have the candidate solve a few coding problems on the whiteboard? Well, let's look at the criteria you should be using to judge any potential hire. To quote Joel (http://www.joelonsoftware.com/articles/fog0000000073.html), the criteria for getting hired at Fog Creek, and therefore implicitly the criteria the whole programming world should be using for hiring someone are:

Smart, and
Gets Things Done.

I'd like to propose a third criterion:

Good fit for the team and organization.

It doesn't matter how smart you are and how good you are at cranking out code. If you're a complete and utter asshole, if you don't play well with others, if you're just not a good cultural fit for the organization, then you'll drag down the team rather than improve its productivity.

This is why it can be helpful to read the blog of any potential software development candidate. By doing so, you can at least somewhat get a sense for someone's personality, how they think, what their cultural proclivities are. Hell, whether they use vi or emacs. You won't get any of this information by asking a candidate about Ruby operators or how to implement Floyd's cycle detection algorithm. If you don't somehow test for a candidate's cultural fit, you could very well end up with a submarine asshole on your team that only surfaces well after their start date.

Okay, so that's one reason for reading someone's blog and/or undertaking any other means you can think of to scope out their potential "fit". But what about looking at a coder's code?

One of the most successful ways I've found to accomplish this task is to ask for a code sample before the interview, and then actually go over the code with the candidate during the interview! All you need to do is ask for some relatively small, self-contained piece of code that solves a non-trivial problem.

This is a much more realistic test for exhibiting coding prowess than some sort of artificial string manipulation problem. Additionally, and this is a hugely important point when performing any interview, it tends to put the candidate at ease to talk about their very own pre-written code, so they're not sweating bullets and thereby giving you an inaccurate reading.

When you do go over someone's code with them during an interview, here are the sorts of questions to ask:

How would you explain this code to someone unfamiliar with it?
Why did you make this particular design decision here?
What are some trade-offs involved with this design decision?
How would you go about profiling this code?
Can you think of a way to optimize this particular section?
How would you change things if the requirements were altered in this particular way?
Seriously, man, tabs for indentation?

Basically, ask all the same sorts of questions you would normally ask after a candidate solves a stupid string manipulation problem on the whiteboard, but instead ask those questions about the candidate's own pre-submitted code, written on a proper computer with the use of a proper compiler.

Now one variation on this approach I've seen is asking all candidates to submit a program to solve the exact same set of requirements, e.g. implement Tic-Tac-Toe for two players (no AI). This is dumb for many reasons, the most prominent of which is that any programmer worth his salt is not going to waste the time writing some completely new code just on the off chance that you'll bring him or her in for an interview. Only the desperate programmers will spend their time doing that. So by putting such a psychologically sizable hurdle smack in the middle of your hiring process, you're almost ensuring that you'll only get crappy candidates.

On the other hand, if you ask for a code sample of something the candidate has already written, then the good coders will be happy to send you some of their best existing work and will love to talk about it in detail during the interview. It takes very little effort to dust off some already written code and send it out, at least compared to implementing a code-sample-worthy Tic-Tac-Toe from scratch.

One objection to this whole scheme is that some candidates don't do the whole open source thing and simply don't have any code available that's not under NDA or what have you. For these candidates without any existing code to send you, you can always give them the option of a standard problem for them to code up before the interview, although for the love of all that is holy, please don't choose Tic-Tac-Toe.

debugging Erlang-style concurrency

witten — 2007-05-27

Although Erlang is not a widely used language today, you'll find that many developers heap praise upon its concurrency model. Instead of traditional and error-prone shared-everything locking semantics, Erlang employs a sort of lock-free asynchronous message passing between threads. If you haven't yet read about this approach or need a refresher on the subject, I recommend the excellent article "Erlang Style Concurrency" at http://www.defmacro.org/ramblings/concurrency.html

As cool as this style of asynchronous concurrency happens to be, it is not without its own set of problems. In a previous post (http://coderific.com/blog/post/489), I argued that:

A hugely important and oft-overlooked factor in selecting a code library is the relative ease of tracking down and exterminating bugs when using it. If a library you're using makes debugging unnecessarily difficult, then you're going to spend more time hunting down bugs and less time actually writing code.

As it turns out, one of the problems introduced by Erlang-style concurrency is a decrease in debuggability. Let's say that you're used to standard Java-style locks-all-over-the-place synchronous concurrency. When something goes wrong, such as a thrown exception or a deadlock, you have easy access to a complete call stack. With one glance you can see which thread initiated the call and each function that was subsequently synchronously invoked, all the way down to where the exception actually occurred. There's no question about the order of execution because the execution is ordered.

But in a program written in the style of Erlang's asynchronous message passing, when something goes wrong, there's no call stack to fall back on for debugging purposes. Each successive passing of a message from one thread to another is performed without any blocking whatsoever, so as soon as a thread sends a message to its destination, that caller immediately goes on to do other things without waiting to find out whether the message eventually results in an exception of some sort. If an exception does occur within the destination, there's no record of how you got there.

The result is that naively written asynchronous concurrency code can be more difficult to debug than its synchronous lock-based counterpart. Sure, lock-based concurrency has its own host of debugging problems such as deadlocks and race conditions during access to shared data, but despite these problems you still have access to a call stack for ease of debugging.

Given this disparity in debuggability, some programmers shake their heads and resign themselves to writing lock-based concurrency in Java for the rest of their lives. However, neither the baby nor the bathwater requires throwing out just because Erlang-style concurrency makes it difficult to get at call graph data. The simple solution is to take the little bit of extra effort required to make that call graph data available to the programmer, even if it's not available "for free" as it is with synchronous concurrency.

You can start by just adding logging everywhere. The logging can optionally be compiled out in production code for performance reasons, but the general idea is that by looking at the program's log, you can get a sort of poor man's call stack as each successive thread passes messages. When thread A sends a message to thread B, log it. When thread B then, as a result, sends a message to thread C, log that too. And so on. That way with some basic log analysis you can discern the series of events just as if you had an actual stack dump. For instance:

thread A sent message M to thread B
thread B sent message N to thread C
thread C sent message O to thread D

Now this logging approach isn't entirely fool-proof. If you have multiple messages firing all at once, which is a fairly realistic use case for any program written with asynchronous concurrency, then the messages will be interleaved in the logs and you won't have any idea which messages fired as a result of which other messages:

thread A sent message M to thread B
thread A sent message N to thread B
thread B sent message O to thread C
thread C sent message P to thread D
thread B sent message Q to thread C
thread C sent message R to thread D

In this example, unless the contents of the messages themselves somehow indicate a correlation between successively fired messages, you'll have no idea which message sent from thread A to B caused which message to be sent from B to C.

So logging has its limitations as far as debuggability goes. Another slightly more complex solution involves actually building up a faux call stack associated with each path of messages. When thread A passes a message to thread B, tack on a bit of additional information with that message that basically includes the sender, message, and receiver. Then when thread B dispatches the message to one of its handlers and decides as a result to send its own message to thread C, tack on additional info for the new sender, message, and receiver. Repeat this, tacking on call information to each new message that is passed. Include as much or as little information as you think will be useful during debugging.

In this manner you can build up a record of the call graph as with the logging above, but the list of previous calls will be associated with each message individually rather than just dumped to the log for all messages. Then when an exception occurs, simply print out the simulated call stack for the message you've just received, and you'll have a full list of all the messages passed that got you to the handler you're currently in.

One concern with this scheme that immediately jumps to mind is performance. If you're doing all this extra work just to simulate a call graph that you would otherwise get for free with traditional synchronous concurrency, then is it really worth it? Well, think of it this way. When you use standard lock-heavy concurrency, your call graph isn't really free. Every time you make a function call, you've got to push a fair amount of data onto the stack. So in order to be able to dump the stack within a deeply nested synchronously invoked function, you're relying on call data that was copied to the stack previously. With the simulated call stack approach, you can store that exact same call data, and if you like, you can store it in the same way on a stack data structure of some sort. That should hopefully allay any concerns about performance differences between the two approaches.

Another valid concern is that the various simulated call stacks will grow too large and therefore eat up undue amounts of memory. Since calls with this approach aren't synchronous and therefore never return such that call data is popped off the stack, a series of asynchronous messages will just cause the simulated call stack to increase in size without bound. Whether this actually turns out to be a problem in practice depends on your message passing usage. If your program really does have a need for very long chains of message passing, such as with bounded but deeply nested recursion, then you could always truncate the simulated call graph at some maximum size. This will save some memory at the cost of make debugging slightly more difficult.

However if a series of passed messages in your program tends to terminate after a certain point, then you shouldn't run into unbounded call stack problems. This could look like thread A messaging thread B, which as a result messages thread C, which then decides not to do any message passing in its own handler. The simulated call stack could then at that point be manually or automatically freed, and so won't grow any further.

Hopefully this post has given you some ideas for making your own use of Erlang-style concurrency easier to debug, whether that involves Erlang itself, Python's generator-based microthreads, or something else entirely.

why you don't want to hire rock star programmers

witten — 2007-05-25

Previously (http://coderific.com/blog/post/556) I wrote about a particular kind of programmer with a certain unhealthy affinity for excessive abstraction. Apparently though, in some circles this trait is viewed as a positive.

Let's play a little game. I'll list off a few recent job posting titles from Craigslist, and you tell me what they have in common:

DHTML rock star wanted for hot consumer facing product
AJAX Rockstar: Search & Discovery
Technical Architect / Rockstar
Windows Mobile Developer - looking for rock star

Yes, that's right! All these companies are looking for rock stars! Let's assume for a moment that they're talking about rock star developers and not someone with long hair galloping around on stage with a guitar. (Although the latter would probably have a more beneficial impact on the company's bottom line. At least then they could perform at trade shows.)

Anyway, let me just get this out of the way now: You do not want to hire rock star programmers! If someone tells you that you should hire rock star developers, or that their company is so awesome because they only hire rock star developers, or that it's worth every penny to hire rock star developers because blah blah 10x productivity blah blah, then just save them the trouble and push them into oncoming traffic.

Let's explore what exactly a rock star developer is. First and foremost, a rock star developer is arrogant. Most believe that they're god's gift to gcc and/or Visual Studio. Rock star programmers tend to equate knowledge of a particular programming language with being a good software engineer, as if somehow knowing the contents of K&R by heart and strutting around the office with one's nose in the air are sufficient prerequisites for producing solid software products.

Rock stars also tend to hold a marked disdain for everything other than their chosen area of supposed expertise. They can talk all day about RTTI or factory methods, but ask them to write unit tests, document their code, or participate in requirements gathering, and they simply can't be bothered. If it isn't something that takes every ounce of clever in their body to do it, it won't get done.

Now here's why you don't want to hire a rock star. Observe the oft-quoted Kernighan's Law:

Debugging is twice as hard as writing the program, so if you write the program as cleverly as you can, by definition, you won't be clever enough to debug it.

Rock star programmers absolutely revel in writing tricky code and using esoteric language features. Uncommented bit twiddling? Par for the course. Inane inheritance hierarchies? You betcha. Template metaprogramming up the yin-yang? Oh, you better believe it.

This worship of cleverness means that not only will the rest of the team have trouble with rock-star-produced code, but the rock star himself often will be unable to read and debug his own programs! But that's okay, because the only thing rock stars love more than writing unmaintainable code is throwing out all their old unmaintainable code and rewriting it, replacing it with newer, shinier unmaintainable code. Why try to understand or fix bugs in old code when the constant churn will just cause it to be replaced within a week's time anyway?

"But, but!" I can hear you crying, "Paul Graham in http://paulgraham.com/avg.html says that with smarter people using a better language, you can get everything done faster and therefore get bought by Yahoo and become independently wealthy and never have to work for corporate schmucks again! What do you have to say to that?!"

Well, first let me say: Stop being such a PG fanboy. It's just not becoming on a grown man. Secondly, there is a difference between an arrogant rock star programmer as described above and a coder simply chosing the right tool for the job. You can use a powerful language without being an arrogant prick, without writing code half again as clever as you are actually able to read, and without ignoring all relevant aspects of software engineering. An argument against rock stars is not an argument for dumb programmers. An argument against rock stars is an argument for best practices when it comes to actually producing software.

"But what about Joel of On Software fame?" I can hear you scream at the monitor. "Doesn't he only hire the absolute best and the brightest and therefore get so much productive work done that they put out a full point release of Wasabi-based FogBugz once every other hour?"

I will contend that the best and the brightest are not rock star programmers. The best software developers play nicely with others, recognize the importance of all aspects of the software development process, and generally make high quality products. Sure, they may even be more productive than the average programmer by some large factor. I'll go so far as to say that this anti-rock-star programmer, the cream of the crop, will actually be more productive than a supposed rock star.

So the next time you're surreptitiously browsing job postings on Craigslist during your lunch hour with your trigger finger on alt-tab, be especially cautious of any company that claims to be looking for a rock star programmer. It is possible they're just trying to convey that they don't hire crappy Blub programmers, that they want people who really know their stuff. But more often than not, the company is saying they're looking for someone to strut around the office and make the CTO feel like they've got a "rock star" development team. Please, save the strutting for the guys with the guitars.

[Update: Follow-up posted here: http://coderific.com/blog/post/582]

Python, Java, and abuses of abstraction

witten — 2007-05-22

A few months ago I wrote about a former coworker with a particular propensity for churning out mountains of code (http://coderific.com/blog/post/351):

Let's call him Bob. Don't worry, if you're one of my former coworkers, and you're reading this, then it isn't you. Bob liked to write code. He liked to write lots of code. He would invent problems where there weren't any, not because he genuinely liked solving problems, but because he liked to write copious amounts of code. He relished big, sweeping architectural changes, and simply based on the frequency with which he caused them, you'd think that he knew no other way to write software.... Bob was simply too fascinated with the minutiae of his language of choice, and so as long as he was lost in reams of code he had written containing overly complicated component interactions to sort out, he was happy.

Today I found this post from Paul Buchheit that explains this type of programming mentality better than I could (http://paulbuchheit.blogspot.com/2007/05/amazingly-bad-apis.html):

I also have a second theory about what happened to Java: it attracted people who like to write piles of code filled with endless abstractions. When I worked at Google, one of the engineers on our Java mailing list suggested that we ban the keyword 'new'! I was puzzled by this suggestion, but after a little research I learned that one of the hot fashions in Java is to use factories instead of 'new' (not that factories are bad, but "sometimes good" does not imply "always best"). Better yet, you don't reference the factories directly (not enough abstraction), instead you have some framework that injects the factories into your classes according to what is written in an XML configuration file. In this way, you can make your code perfectly unreadable, but more importantly, it is very abstract.

There is a very tempting trap that many programmers fall into. In my post quoted above I called it the Bob Trap, but perhaps a better name would be the Abstraction Trap. Abstraction in general is an immensely powerful tool, and as its magic has been bestowed upon the land, programmers have realized that they can write complex programs in C++ or Java or Python rather than assembly language and therefore be much more productive. But the tendency with anything good, as Paul explains, is to try to apply it to absolutely everything. And this is where programmers like Bob start to go astray. When abstraction becomes an end unto itself, it begins to interfere with the original goal of producing maintainable software.

Anyway, I don't have a whole lot to add to this. The idea just really resonated with me. One of the reasons I dig Python is that it tends to use abstraction as a force for good.

Further reading on Python versus Java, especially as related to abstraction and code bloat: http://dirtsimple.org/2004/12/python-is-not-java.html

making technology instead of products

witten — 2007-05-20

I once worked at a software startup where it was announced that the engineering team was to have a meeting across the street at the local bar. This was a little out of the ordinary, especially since we were only given a vague idea about the meeting's topic: Future directions after the current release is shipped. So at around 4 in the afternoon, a motley group of developers, testers, and managers filed into a nearly empty Cuban-themed bar with congo drums for bar stools. We sat uncomfortably around several wooden tables, sipping sugary mixed drinks and waiting for whatever announcement was coming.

The announcement, as delivered by one of our own venerable managers, was as follows. The company has awesome technology that's worth millions of dollars. That was assumed as a given. However, if some employee comes up with a killer app to showcase this technology, then the killer app would have a multiplier effect, making the millions of dollars of company value into tens of millions of dollars. The implication being that since we all have some stock options, coming up with a killer app will make us even richer when the eventual and inevitable huge payout happens. Names of various inspirational killer apps were bandied about, such as MySpace and dodgeball.com. It was left as an exercise to the listener as to what the hell these existing web sites had to do with our company's technology.

We squinted at each other in the afternoon sun, wondering why this announcement merited taking us all away from coding for two hours. Then, a challenge was put forth. The first person to make a killer app will be paid one thousand dollars! This proclamation was made with neither any hint of irony nor a Dr.-Evil-style pinky-to-mouth gesture. By busting our butts and producing a killer app to make the company's technology into something actually useful, and thereby lining the pockets of the owners and investors with tens of millions of dollars in value, one of us would be paid a whole thousand dollars for our trouble! And our infinitesimally small stock option grants would be worth slightly more. It sounded like quite a deal.

As the inspirational ramblings by various managers wound down and the meeting degenerated into general chatting and mojito-sipping, some of us wondered aloud what the hell was going on. And it soon occurred to us: Desperation. The management realized that all this time the company had been producing cool technology rather than products that actually meet customer needs. And while developing software simply because the technology it's built on happens to be cool is a noble goal, it also has the unique property of not producing any measurable revenue. In other words, being technologically cool counts for something, but if your goal involves producing software that people will voluntarily exchange for money, then it's more important to make something that customers actually want.

Management realized this, albeit a bit late in the game, and so they were desperate for something, anything that would convert their multi-year investment into an actual product that customers would pay good money for. But rather than revamp our product suite or cut out "cool" features that no customers actually wanted, they settled on the quick fix idea of a killer app. If only our developers invent something like MySpace running on top of our awesome software, our otherwise useless technology will become an actual product, and we'll be rich! And we can reward one of them with a thousand dollars as a motivational tool!

The company made the classic mistake of focusing on technology rather than products. Unless you're making products like high-end audio equipment for people with too much discretionary income, customers don't care about whiz-bang technology. They care about whether the product solves a problem and makes their life easier. Whiz-bang technology might make a product that's already useful even more appealing, but technology on its own is rarely sufficient.

Here's a simple one step process detailing how to make a product that's focused around the customer's needs rather than your need to make cool technology. Step one: Actually focus on the customer's needs! When coming up with a new feature, first step back and figure out which customer problem it solves. What's that? It doesn't solve, address, or have anything to do with an actual customer problem? Don't implement it!

Okay, this advice of listening to your customers might sound trite, and the quite legitimate objection can be raised that sometimes customers don't know what they want. This is certainly a concern, but that doesn't excuse foisting upon your customers software that doesn't meet any of their needs. Even if a particular customer doesn't know what he or she wants, it is your job as the developer and all-around problem solver to guess whether a feature you're developing has a non-zero chance of being remotely useful to that customer. If in your esteemed opinion the feature is technologically cool but not terribly useful, well, then you have your answer.

Why do so many companies fall into this trap of making technology rather than products? Simply put, because it's easy. Actually focusing on the customer and gathering requirements and distilling them down into actual features is just plain hard, especially for a group of logically minded coders. Developers by their nature deal in technology, and it takes another set of skills that many developers don't have to think in terms of what would satisfy their users. So many software companies take the easy route of focusing exclusively on the technology because it's what they know. Doing otherwise would not only require stepping outside of their comfort area but would require much more up-front effort.

One of the interesting claims in the Mythical Man-Month is that turning a piece of software into an actual product takes three times the development effort of just producing the software to begin with. It is this sort of transition from a simple technology-oriented program to a shippable, customer-oriented "solution" that many companies neglect. A company might get their software to market faster by focusing on the technology rather than the customer, but the software they do ship won't make any customers happy.

Fortunately, these problems have a way of correcting themselves. At the software startup I worked for with the "cool" technology, no one ever came up with the killer app and so no one claimed the thousand bucks of prize money. The company ended up missing payroll, nearly went out of business, and was eventually sold off at fire sale prices to a holding company. No one got rich, and all the stock options that were to be worth millions of dollars turned out to be more useful when printed out and used as toilet paper.

ignore developers at your own peril

witten — 2007-04-29

First, a little background. One of the development teams I used to work on was a fairly tight-knit group of coders. We got along well, and we even hung out occasionally outside of work. Our specialty was Python and C++ on Linux, but we did Windows development as well. This was problematic, because the rest of the organization in which we worked was populated largely by Microsoft apologists. However, we did generally produce good code. So other than the occasional half-hearted attempt to convince us to rewrite all our products in C#, we managed to get our work done in cross-platform Python and C++. And it was usually done on schedule despite our having little or no input into the scheduling process.

Our team was also fairly vocal about development process changes. We came up with new ways to improve productivity, and we weren't shy about expressing them to management. We introduced a ticket tracking system that was eventually adopted company-wide. In general, we weren't content to just sit and code. We also wanted to be part of the decision-making process insofar as it affected development. This was also problematic, because management preferred that developers were seen and not heard.

So with that in mind, imagine being told the following by your manager:

If it wasn't for the fact that you guys wrote high-quality code, got it done on time, and worked well together as a team, you'd all be fired.

In other words, if it weren't for the minor fact that you guys are completely competent at your jobs, you'd all be canned. The quote above is indeed a near word-for-word utterance from our development manager. Apparently, it wasn't enough to be good at what we did. It was also expected that we keep our mouths shut and do as we were told, rather than coming up with improvements to our development process or pushing back in the face of backseat coding (http://coderific.com/blog/post/474).

Developers in general are highly educated people. Many have college degrees. Most have years of development under their belts. There exists a huge amount of perspective and knowledge among any team of developers. Then, why is it that developers are so often ignored when it comes to organizational decision-making?

Well, there are many reasons, but a big one is that corporations in this country are typically structured with a command and control style of management (http://www.joelonsoftware.com/items/2006/08/08.html). To quote Joel:

Primarily, the idea is that people do what you tell them to do, and if they don't, you yell at them until they do, and if they still don't, you throw them in the brig for a while, and if that doesn't teach them, you put them in charge of peeling onions on a submarine, sharing two cubit feet of personal space with a lad from a farm who really never quite learned about brushing his teeth.

This sort of approach might work in the military, but it doesn't work so well with knowledge workers such as software developers. Firstly, a developer who feels that they can provide valuable input into decision-making is usually right. No one is in a better position to understand the technical development trade-offs that may affect a particular organization than the person actually writing the code. Sure, there are many bits of information that a developer in the trenches cannot see, such as most things related to sales. But there is even more information that most managers cannot fathom, and that's what makes a developer's perspective so incredibly valuable.

How about an example. Let's say that management hands down a request to your team to develop a poorly specified new product, and you have under three weeks in which to do it, on top of all your existing development responsibilities. Now, any developer giving even a cursory glance to the particular request can tell that three weeks won't be sufficient, and any coder who bothers to task out the work will realize that it's a multi-month endeavor.

This exact scenario happened at one of my prior development jobs, and so my team pointed out how long the task would really take, about 13 weeks. We even backed this number up with a document detailing our estimates and the rationale for them. As a result, the project was yanked from my team, handed to another team comprised primarily of yes-men, and my team was then bad-mouthed as non-team-players. I'll let you guess how long the team of yes-men ended up taking to implement the new application. That's right. A little over 13 weeks.

Why doesn't management have sufficient perspective and knowledge to figure this out on their own? Simply put, they're not developers, and they don't have the necessary experience and knowledge to make accurate estimates. This is especially so given that development management is usually under direct pressure from above to come up with estimates that are as short as possible, or even shorter than possible. Developers, on the other hand, while often under similar pressure, have the direct technical knowledge and familiarity with the code base to at least somewhat accurately break the task into its constituent parts and come up with good estimates. Developers are simply better at any decision-making involving technical considerations because they have more information directly relevant to those decisions.

So ignoring the opinions of developers has obvious implications when it comes to good organizational decision-making. But a perhaps unintended consequence is that it causes otherwise good, hard-working developers to become alienated. Developers are some of the people most uniquely qualified to make decisions on technical matters, and yet their opinions are often given the least credence of anyone in the organization. This disconnect causes developers to feel like they're just being treated as replaceable resources rather than valuable members of the organization. If a company is at all interested in retaining its employees, there's no better way to do so than to actually listen to their opinions.

motivation, Reddit, and Mechanical Turks

witten — 2007-04-10

In an earlier post (http://coderific.com/blog/post/437) I wrote about how adding a tipping system to Coderific would potentially corrupt people's intrinsic motivations for posting employer ratings. Yesterday, Jeff Atwood of Coding Horror wrote about Amazon's Mechanical Turk project as a failure (http://www.codinghorror.com/blog/archives/000828.html), suggesting that paying for people to perform various menial tasks interferes with their internal motivations for doing those tasks. He then goes on to compare the pay-for-content Mechanical Turk with Amazon's freely provided customer reviews:

The strength of the user reviews is one of the main reasons I frequent Amazon. That's user-submitted content that people invested countless thousands of man-hours on. And Amazon didn't pay anyone a dime to do it.

This might very well be the case. People write Amazon reviews or Coderific employer ratings for a number of reasons, but usually it's because there's some intrinsic motivation, such as an urge to get some experience with a product or employer off their chest. And it's entirely possible that the Mechanical Turk's apparent failure is due to its extrinsic motivation, paying people for content. When you attach a small price tag to a task rather than making the task its own reward, suddenly the task is that much less appealing.

To illustrate this idea, here's an old joke from http://www.gnu.org/philosophy/motivation.html

An elderly man, harassed by the taunts of neighborhood children, finally devises a scheme. He offered to pay each child a dollar if they would all return Tuesday and yell their insults again. They did so eagerly and received the money, but he told them he could only pay 25 cents on Wednesday. When they returned, insulted him again and collected their quarters, he informed them that Thursday's rate would be just a penny. "Forget it," they said - and never taunted him again.

The Coding Horror article claims that Amazon, when making the Mechanical Turk, should have foregone the extrinsic rewards and instead chosen a reward system based on intrinsic motivation. The post then goes on to give supposed examples of intrinsic motivation such as Folding@Home's leaderboards or Reddit's front page.

The problem here is that getting onto "top contributor" lists or seeing your links moderated upward on sites like Reddit or Digg involve the exact opposite of intrinsic motivation. When you post a link to Reddit, you're usually not doing so for the sheer fun of it, but rather for the extrinsic reward of seeing your post get moderated upward by other people, thereby overshadowing any internal motivations.

There is actually very little difference in the reward system employed by the Mechanical Turk and Reddit. Both replace your internal driving motivation to do some task for its own sake with an external motivating factor. Both involve tying a reward directly to performing the task. The only reason that projects like Reddit and Folding@Home are so much more successful than those like the Mechanical Turk at employing extrinsic rewards to drive user-provided content is that the successful sites rely on the weighty reward of peer recognition.

Which would you rather have: Ten cents from Amazon for providing some small amount of content, or the knowledge that some link you provided to Reddit was well-received by a large group of semi-anonymous peers? In either case, you can assume equal intrinsic motivation for the respective tasks (not much), but in Reddit's case, there's the possibility that the content you submit will be greeted with positive recognition.

Every time you post something to Reddit and get it moderated up, that's like a little referendum on your worth as a human being. The external reward of course doesn't make you want to do the task any more for its own sake. It's just a very effective means at getting people to share links that they otherwise might not care about sharing.

the importance of debuggability

witten — 2007-04-01

There are a number of ways in which a library can actively get in the way of efficient debugging. A common one is the overuse of clever C++ template meta-programming. Genericity is an incredibly useful language feature, and you can do all sorts of cool tricks with templates to make things happen at compile-time rather than runtime. However, you can go overboard with meta-programming.

Consider Boost Spirit (http://spirit.sf.net), a parser framework that can be used for compiler development. Spirit makes extensive use of template meta-programming so that you can declare your parser grammar in a BNF-like syntax directly in your C++ code. But like with so many template-heavy projects, make one mistake with Spirit, and during compilation you'll be treated to an error message like this one:

/usr/include/boost/bind/mem_fn_template.hpp: In member function 'R boost::_mfi::mf0<R, T>::call(U&, const T*) const [with U = const Grammar, R = bool, T = Grammar]':
/usr/include/boost/bind/mem_fn_template.hpp:50: instantiated from 'R boost::_mfi::mf0<R, T>::operator()(U&) const [with U = const Grammar, R = bool, T = Grammar]'
/usr/include/boost/bind.hpp:224: instantiated from 'R boost::_bi::list1<A1>::operator()(boost::_bi::type<R>, const F&, A&, long int) const [with R = bool, F = boost::_mfi::mf0<bool, Grammar>, A = boost::_bi::list0, A1 = boost::_bi::value<Grammar>]'
/usr/include/boost/bind/bind_template.hpp:26: instantiated from 'typename boost::_bi::result_traits<R, F>::type boost::_bi::bind_t<R, F, L>::operator()() const [with R = bool, F = boost::_mfi::mf0<bool, Grammar>, L = boost::_bi::list1<boost::_bi::value<Grammar> >]'
/usr/include/boost/spirit/core/composite/epsilon.hpp:47: instantiated from 'typename boost::spirit::parser_result<boost::spirit::condition_parser<CondT, positive_>, ScannerT>::type boost::spirit::condition_parser<CondT, positive_>::parse(const ScannerT&) const [with ScannerT = boost::spirit::scanner<const char*, boost::spirit::scanner_policies<boost::spirit::iteration_policy, boost::spirit::ast_match_policy<const char*, boost::spirit::node_val_data_factory<boost::spirit::nil_t> >, boost::spirit::action_policy> >, CondT = boost::_bi::bind_t<bool, boost::_mfi::mf0<bool, Grammar>, boost::_bi::list1<boost::_bi::value<Grammar> > >, bool positive_ = true]'
/usr/include/boost/spirit/dynamic/impl/conditions.ipp:86: instantiated from 'ptrdiff_t boost::spirit::impl::condition_evaluator<ConditionT>::evaluate(const ScannerT&) const [with ScannerT = boost::spirit::scanner<const char*, boost::spirit::scanner_policies<boost::spirit::iteration_policy, boost::spirit::ast_match_policy<const char*, boost::spirit::node_val_data_factory<boost::spirit::nil_t> >, boost::spirit::action_policy> >, ConditionT = boost::_bi::bind_t<bool, boost::_mfi::mf0<bool, Grammar>, boost::_bi::list1<boost::_bi::value<Grammar> > >]'
/usr/include/boost/spirit/dynamic/if.hpp:78: instantiated from 'typename boost::spirit::parser_result<boost::spirit::impl::if_else_parser<ParsableTrueT, ParsableFalseT, CondT>, ScannerT>::type boost::spirit::impl::if_else_parser<ParsableTrueT, ParsableFalseT, CondT>::parse(const ScannerT&) const [with ScannerT = boost::spirit::scanner<const char*, boost::spirit::scanner_policies<boost::spirit::iteration_policy, boost::spirit::ast_match_policy<const char*, boost::spirit::node_val_data_factory<boost::spirit::nil_t> >, boost::spirit::action_policy> >, ParsableTrueT = boost::spirit::epsilon_parser, ParsableFalseT = boost::spirit::nothing_parser, CondT = boost::_bi::bind_t<bool, boost::_mfi::mf0<bool, Grammar>, boost::_bi::list1<boost::_bi::value<Grammar> > >]'
/usr/include/boost/spirit/core/composite/sequence.hpp:54: instantiated from 'typename boost::spirit::parser_result<boost::spirit::sequence<A, B>, ScannerT>::type boost::spirit::sequence<A, B>::parse(const ScannerT&) const [with ScannerT = boost::spirit::scanner<const char*, boost::spirit::scanner_policies<boost::spirit::iteration_policy, boost::spirit::ast_match_policy<const char*, boost::spirit::node_val_data_factory<boost::spirit::nil_t> >, boost::spirit::action_policy> >, A = boost::spirit::rule<boost::spirit::scanner<const char*, boost::spirit::scanner_policies<boost::spirit::iteration_policy, boost::spirit::ast_match_policy<const char*, boost::spirit::node_val_data_factory<boost::spirit::nil_t> >, boost::spirit::action_policy> >, boost::spirit::parser_context<boost::spirit::nil_t>, boost::spirit::parser_tag<23> >, B = boost::spirit::impl::if_else_parser<boost::spirit::epsilon_parser, boost::spirit::nothing_parser, boost::_bi::bind_t<bool, boost::_mfi::mf0<bool, Grammar>, boost::_bi::list1<boost::_bi::value<Grammar> > > >]'
/usr/include/boost/spirit/core/non_terminal/impl/rule.ipp:233: instantiated from 'typename boost::spirit::match_result<ScannerT, ContextResultT>::type boost::spirit::impl::concrete_parser<ParserT, ScannerT, AttrT>::do_parse_virtual(const ScannerT&) const [with ParserT = boost::spirit::sequence<boost::spirit::rule<boost::spirit::scanner<const char*, boost::spirit::scanner_policies<boost::spirit::iteration_policy, boost::spirit::ast_match_policy<const char*, boost::spirit::node_val_data_factory<boost::spirit::nil_t> >, boost::spirit::action_policy> >, boost::spirit::parser_context<boost::spirit::nil_t>, boost::spirit::parser_tag<23> >, boost::spirit::impl::if_else_parser<boost::spirit::epsilon_parser, boost::spirit::nothing_parser, boost::_bi::bind_t<bool, boost::_mfi::mf0<bool, Grammar>, boost::_bi::list1<boost::_bi::value<Grammar> > > > >, ScannerT = boost::spirit::scanner<const char*, boost::spirit::scanner_policies<boost::spirit::iteration_policy, boost::spirit::ast_match_policy<const char*, boost::spirit::node_val_data_factory<boost::spirit::nil_t> >, boost::spirit::action_policy> >, AttrT = boost::spirit::nil_t]'
Test.cc:55: instantiated from here
/usr/include/boost/bind/mem_fn_template.hpp:31: error: invalid conversion from 'const Grammar*' to 'Grammar*'
scons: *** [Test.o] Error 1
scons: building terminated because of errors.

I'll wait while your eyes finish bleeding.

Okay, welcome back.

That error message is the actual compilation error I get by leaving out a single "&" operator in a Spirit grammar definition. And this isn't even close to the longest or most incomprehensible error message I've ever gotten when developing Spirit-based parsers.

Where do you even begin with an error like this one? Well, you can start at the end, and try to figure out what's going on with the invalid conversion. Given enough effort, you'll probably eventually determine what went wrong, but not until you've wasted gobs of time that you could've spent actually doing productive work.

When you're writing any kind of code, an almost universally good practice is to fail early. In other words, if you encounter some exceptional case or invalid data, you should throw an exception or return an error code as early as you possibly can, thereby ensuring that the error is the most relevant to the root cause of the problem.

This policy is even more important when you're implementing a general-purpose library. If the library fails as soon as the caller passes in invalid data, then it should be pretty clear to the programmer what went wrong and why. But if instead a library only throws an undocumented exception from somewhere deep within the bowels of the library's own code, then you'll end up with incomprehensible template garbage like the error above. And any programmer making use of the library will have to go spelunking through the library's source (if it's even available) to figure out how their use of the API corresponds to the particular error they're seeing.

Another way of putting this is that the worst libraries have leaky abstractions. In order to debug problems with such a library's black box API, you have to open up the box and start rifling through its contents to figure out what went wrong. This takes time and makes programmers pull out their hair.

Using a difficult-to-debug library has implications beyond just wading through error messages. The best way to release software based on any sort of schedule is to actually break the development down into small, granular tasks, and then estimate how long each task will take to complete. (See http://www.joelonsoftware.com/articles/fog0000000245.html for more info.) This approach isn't anything revolutionary, and it's a pretty well-understood practice in the software industry despite the fact that coming up with even remotely accurate estimates is as much art as science.

But a major problem with making schedules based on granular estimates arises when you try to estimate debugging. First of all, when you're trying to figure out how much time to schedule for a release, you know how many development tasks you have, and so you can somewhat accurately estimate how long actual development will take. But how can you possibly estimate debugging if you don't know how many bugs you'll have? And even if you have a rough idea of how many bugs you're going to have based on how much code you're writing, it's impossible to know how long fixing a particular bug will take.

There are basically three phases to fixing a bug:

1. You try to reproduce the reported problem.
2. You try to find the root cause of the bug.
3. You fix the root cause of the bug.

When you attempt to reliably reproduce a problem, you typically just try different actions or configurations until the bug magically reappears, and then you try to confirm that the actions you took caused the bug and it wasn't all just a coincidence. Similarly, when you attempt to track down the root cause of a bug, you run through the reproduction steps with prints in the code or while stepping through the source in a debugger. After some indeterminate amount of time, you'll hopefully come up with a root cause. Both of these steps are more like detective work than anything else. You basically do them until they're done. And because you don't know what's causing the bug until after this detective work is complete, you have no idea about how long it will take to fix the bug until after you've found the cause. In other words, the first two steps of debugging both have the interesting property that together they take most of debugging time and they're also nearly impossible to estimate in advance.

When you use libraries that are difficult to debug, you're not just wasting time. You're actually killing any chance of your development schedule resembling reality. Hard-to-debug code adds more uncertainty to the debugging phase of the release cycle, and even with decent libraries, this debugging phase is already the most difficult part of the schedule to accurately estimate. The more uncertainty you introduce into the debugging phase, the less likely you'll be able to predict when any given release will be out the door.

So when evaluating a programming library, don't just examine its API, its efficiency, and the quality of its code. Also evaluate how easy it is to debug code written to use the library. Because if you trade debuggability for efficiency, you may end up with speedy code that never gets released.

5 ways to deal with backseat coding

witten — 2007-03-14

I was riding the bus to work today, and as we drove into downtown Seattle, a sea of stationary cars filled the road before us. The bus slowed to a stop, and over the next several minutes I'm pretty sure we made a good ten feet of forward progress.

A man sitting near the front of the bus turned to the driver, and the following conversation ensued.

passenger: My stop's right here.
driver: We're not at the stop yet.
passenger: But my stop's here!
driver: We'll get there.
passenger: You're in the wrong lane.
driver: Just wait one minute.
passenger: What about my stop?
driver: Listen, man, my job is to do all the driving, navigate through traffic, and get you to your stop. Your job is just to sit there and wait til I open the doors. That's all you gotta do. It's real easy. Just sit there and wait til the doors open.

The man grumbled, and we continued to inch forward. When the bus finally got to the man's stop, he muttered a sarcastic "thanks" and stalked out. But the driver had the last word.

driver: Have a nice day. I'll let you take it from here.

This sort of backseat driving shouldn't be unfamiliar to anyone who writes code for a living. It's not at all transparent to non-developers what you're doing all day clackity-clacking away on your keyboard. People who have a stake in getting their hands on the results of your labor, whether that's managers, salespeople, or customers, often feel like they need to tell you how to do your job.

When backseat driving happens, it's hard to miss the signs. Usually it takes the form of someone not at all involved in actual development reaching down into the dev process and handing you a pre-engineered solution for some pet feature. By "pre-engineered" I mean a solution that is already designed in detail before it ever gets to your inbox, and quite often before an actual engineer ever sees it.

These solutions can be vague, ill-conceived, poorly considered, or all three. But backseat engineering never comes without a rationale. A common one is that timely completion of the feature is of the utmost importance to satisfy some demanding customer or a predetermined release date, and so there's no time to get questions answered or, heaven forbid, assess the existing "design" to see if it's any good. This rationale for backseat coding, by the way, is called playing the schedule card.

Another common rationale for backseat engineering is that "It's what the customer wants." In other words, some scrawled ideas for the pre-engineered solution have been briefly flashed in front of the customer at some point, and without actually sitting down to hash out requirements, the customer has agreed to the solution. There's no need to think about the design or raise any questions, because after all, it's already exactly what the customer wants. This rationale for backseat coding is called playing the customer card.

So what's a poor software engineer to do when faced with a pre-designed feature to implement? Well, there are a couple of options. Let's run through them in increasing order of ending your day unemployed.

option 1: Keep your big mouth shut, and code the damned feature.

You know that the customer was never asked what they are really trying to accomplish with this feature. You also know that for that reason alone, your implementation will probably not turn out to meet the customer's needs. But no one likes a boat rocker, so you just develop the feature anyway. This has the benefit of keeping everyone happy at first, but when you deliver the exact turd they asked for, the customer won't be grinning anymore. Oh, and each time you just grit your teeth and neglect to point out glaring design problems, your die a little on the inside.

option 2: Code the damned feature, but document and publish the questions and concerns you have with the "design".

This is more of a CYA technique. If at some point it becomes apparent that the solution as pre-engineered is utter crap, well, at least then you can wave around the emails you wrote months before containing all your objections. There is a certain amount of soul rot to this option as well, because although you can smugly say you knew it was a bad idea all along, well, you did nothing to prevent it from getting implemented.

option 3: Firmly request some additional requirements gathering before you start coding the feature.

This are a few ways this can go. Depending on the tone you take, and the questions you ask, and whether the party attempting to engage in backseat coding is in a good mood, you can sometimes actually get answers to your questions and somewhat refine the granularity with which the pre-designed solution is specified. If the pre-engineering wasn't too terrible to start with, this can even get you enough information about the design to code something moderately reasonable. You'll look like a hero for being thorough and asking good questions, the customer will actually get something roughly resembling what they want, and everyone will go home happy.

However, if you're unlucky enough to have a real stinker of a pre-designed solution, then no amount of asking for clarifications will magically turn it into a good design. You'll end up looking bad for turning out a poor solution, even if that was just because the design sucked, and the customer will be very angry and jump up and down.

Often though, especially when people are playing the schedule card, they don't want to hear questions. No matter how good your intentions may be, backseat coders in a hurry think you're not a team player if you ask questions. You have a design already, what are you waiting for? Just start coding already. In this situation, once questions have fallen on deaf ears, coders often resort to options 1 or 2.

option 4: Provide a detailed estimate for how long you'll take to implement the feature as designed.

This option can be a good one if one of the reasons why the pre-engineered design is so bad is because no one has stopped to consider how long it will actually take to implement. The idea here is to communicate to the backseat coder, ideally in unbearable detail, the true cost of what they're asking for. Don't inflate numbers, just call it like you see it. Pie-in-the-sky feature? Well, break that up into pie-in-the-sky task 1, pie-in-the-sky task 2, and so on.

This option can be particularly effective if you couple it with a set of clarification questions, pointing out that the answers to the questions may reduce or increase the time estimate. Even better, provide some alternate design ideas and estimate their development time as well. That way, you can first point to the original pie-in-the-sky design that will take an estimated 6 weeks, and then draw everyone's attention to your bold, alternative design that will take only 3 weeks to develop.

If you choose this option, be aware that you are playing with fire. If it works, the backseat coder will reconsider their request and maybe revise things, ideally taking your input into account. And you'll get to code something that actually makes some sort of coherent sense. But if it backfires, you will face the wrath of an angry backseat driver, who will not only jump up and down, but will mutter a sarcastic "thanks" as he or she stalks out of your bus.

option 5: Flat out refuse to implement the feature as designed, but offer a good, alternative design.

This is not generally recommended, because unless you have a tremendous amount of accumulated good will and/or other job security, it will probably get you canned. But it often really is the most honest option. In case you're having trouble coming up with what to say, consider the following:

"No offense intended, but your design is crap, largely because it was created by someone completely untrained in software development. I wouldn't expect myself to be able to do your job any more than I expect you to be able to do mine. But don't worry, I've come up with an alternate design that is vastly superior."

Well, at least if you get fired over this you'll know what website to use to find a better employer.

in defense of anonymity

witten — 2007-03-08

Fairly regularly I receive emails from employers upset with their ratings on Coderific. And inevitably these employers express a rather marked distaste for the ratings being posted anonymously. Some employers even go so far as to suggest that I banish anonymous ratings altogether by requiring email validation, confirming that rating authors work for the companies they're rating.

There are a number of issues at stake here, so allow me address each one individually. Firstly, there's the question of whether rating authors should be restricted to current employees of the employer being rated. By requiring email validation matching the employer being rated, all former employees would be barred from posting ratings.

This is a horrible idea, and here's why. When you're actively developing software for an employer, you have a very limited perspective about that employer. The heady days of successful product releases are too new, and the wounds you have from mismanaged projects are too fresh. It's only some time after a job ends that you can even somewhat objectively begin to analyze what went well and what didn't.

Think about being in a relationship. And imagine for a moment there's a Relationshiperific site. You could rate a person you're dating or had dated, and so could everyone else. Now imagine that you had just gotten in a big fight or had a wonderful, romantic evening, or perhaps both. You could go on this hypothetical Relationshiperific and write a passionate review of your boyfriend, girlfriend, or spouse. If you had a big fight, then it might be a vitriolic, rage-filled post. If you had just gotten home from a romantic date, then it might instead take the form of a flowery love poem, glossing over any personality faults.

Fast-forward a few years. You're no longer dating that person. You go back and read your rage-induced review or your embarrassingly emotion-filled poem. You'd laugh at your former self for not seeing the big picture, for not seeing both the good and the bad. With your increased perspective, you would be able to write a much more thorough, well-reasoned rating.

Employer ratings are not all that different from relationship ratings. You, as a software developer, have a relationship with your employer. That relationship might be friendly, trusting, distant, or even abusive. And only after the relationship ends do you have sufficient perspective to make rational judgments on the matter.

Now this isn't to say that ratings by current employees are not valuable. These ratings tend to be more passionate, emotional, and in-the-moment. Much like that love letter, a rating by a current employee captures their feelings at a snapshot in time. Their opinion of that employer might very well change over the years, especially as they move on to other employers that they can hold up for comparison.

It's also important to point out that although leaving a job doesn't immediately heal all employer-inflicted wounds, and some ex-employees might maintain a certain level of bitterness for some time, that bitterness becomes more tempered with time. Even bitter ex-employees will only write bile-filled employer ratings for so long after their employment ends.

So abolishing anonymity and only allowing ratings by current employees is pretty clearly a bad idea. But what about ratings written by people who didn't even work at the employer being rated, or didn't write software there? Wouldn't some sort of validation scheme prevent these fake ratings?

Okay, time for another thought experiment. Let's say we established some sort of magical mechanism for ensuring that every person who writes an employer rating on Coderific is a current or former employee of that employer, and actually wrote code there. (Let's put aside for the moment the minor fact that this sort of validation would be slow, cumbersome, and well out of the range of Coderific's budget. Some employers also won't tell random, unidentified callers anything about your job description.)

So suppose you're considering writing an employer rating on the alternate reality version of Coderific, and you find out that we require identity validation, a resume, and calls placed to your former place of employment. How many milliseconds would it take for you to close the browser window in disgust? Would you seriously consider jumping through those hoops just to review your former employer? Perhaps more importantly, would you feel comfortable giving all of this private information to some random web site? Would you trust your employer not to retaliate when they realize that they confirmed your dates of employment around the same time that an "anonymous" rating of them appeared on Coderific?

Think of the chilling effect that banning anonymity would have on ratings being posted. You think Coderific has only a few ratings now? Imagine Coderific without any ratings containing anything even slightly critical of any employers. Now, of the handful of glowingly positive ratings that remain, cut out all those written by busy developers who don't like jumping through invasive validation hoops. The three ratings that remained on the entire site would be highly validated, assiduously fact-checked, and thoroughly useless.

I have a lot of fun watching Coderific web server logs as people access the site. (Don't worry, these logs are rotated away into oblivion fairly quickly.) By looking at the site URLs that a particular IP address accesses, I can get a rough idea of how a particular user is browsing the site.

One common navigation pattern is a user clicking on "browse employers", and then clicking through all the individual letters of the alphabet. Man, people really like their alphabet. I often see logs like:

GET /employers/browse_letter/a
GET /employers/browse_letter/b
GET /employers/browse_letter/c

... And so on, all the way to z. Some people even like browsing backwards, from z to a. Maybe they're practicing for the next time they're pulled over for a suspected DUI.

Anyway, occasionally someone clicks a link that requires an account, and so redirects to the login page. That shows up in the log like so:

GET /employers/new
GET /users/login_page

Quite often that's enough to make a prospective rating author leave the site. No more hits show up in the logs from that IP. I can just imagine the thought process, "What, I need an account to write a rating? Lame."

Sometimes the user is brave enough to click the user creation link from the login page, and I see this:

GET /users/signup

But then the prospect of having to come up with a username, a password, and optionally enter an email address is simply too much. Goodbye, Coderific! No more log entries from that IP. It's the rare user that makes it past the login page, through the signup page, and finally gets up the courage to type in an employer rating.

What I'm getting at here is that every additional impediment that you throw up in front of prospective rating authors dramatically reduces the number of people who will actually perform all the hoop jumping and write a rating. Require a username? Cut out, say, 50% of potential posters. Require a password? There goes another 25%. What, I need to submit my resume? There goes everyone but one guy in Idaho. And the only reason he puts up with it is because he thinks Coderific is a job site.

With any sort of site containing primarily user-provided content, you've got to strike a careful balance between putting restrictions on your users and ensuring quality content. If you clamp down on your users too much, you won't have much content. If you don't have any controls in place (such as a flagging system), your content will consist of posts like "HAHA NOOB U R SO LAME." I'm simply claiming that the right balance for Coderific is struck when anonymity is allowed.

Much of this post has dealt with the cost of removing anonymity from employer ratings. But an equally interesting topic is the cost of keeping anonymity. When anonymous ratings are allowed, we do run the risk of people engaging in the internet equivalent of hiding behind a mask and shouting insults. However, we're not powerless to sit idly by when this does happen. Through the magic of flagging, the Coderific community has the ability to delete those ratings that violate the Terms of Service, such as ratings full of insults.

Anonymity also allows people to post the aforementioned fake ratings, written by non-developers or even people who have never worked for the subject of the rating. Fortunately, anyone has the ability to post responses to ratings or even write their own ratings. The hope is that any false ratings will be answered by posts pointing out the falsehoods and that they will be drowned out in the sea of legitimate ratings. Sure, any reader will have to do a bit more work to decide which ratings to believe for themselves. But this is as it should be. You don't read product ratings on Amazon or Froogle without evaluating their opinions and objectiveness, and you shouldn't do so on Coderific either.

moderation and motivation

witten — 2007-03-04

One of my friends suggested that to encourage Coderific users to provide content in the form of employer ratings and forum posts, I should look into implementing some sort of site currency like the "Geek Gold" present on the http://boardgamegeek.com site. The idea there is that when people provide good content, users can "tip" them with Geek Gold. Anyone can tip, as long as they're logged in and have some gold to spend.

Initially this sounded like a cool system for Coderific. I was thinking that users could tip other users for well-written, interesting ratings or forum posts. Each post could indicate how much in tips it has generated, to give you a general idea of how much people liked it.

However, as I thought through such a currency system, it became apparent that there were some practical problems. If new users started out with some currency (or we were using a LETS-based currency model where currency could go negative), someone could create a bunch of accounts and funnel the cash toward the user of a particular rating, driving the score for that rating artificially high. Even starting new users out with zero cash would mean that someone with a bunch of fake accounts could earn a little cash for each fake user (perhaps by posting some pithy forum responses), and then would still be able to funnel cash towards one user and drive up their rating's score.

The way that most sites work around this problem (Slashdot, Reddit, etc.) is to allow both up-moderation and down-moderation. In other words, not only can you tip someone in order to say "wow, that's a great post", but you can do the opposite, and detract from the score. This cuts down on potential abuse because even with a bunch of fake accounts up-modding a particular rating, people will hopefully see the artificial score inflation for what it is and down-moderate accordingly. Of course, down-moderation is kind of difficult to do with a tipping system. You can't very well reach into someone's tip jar and take money out.

Anyway, as I was searching around for articles on site moderation and thinking about potential solutions or workarounds to these problems, I came across this article by Derek M. Powazek: http://designforcommunity.com/essay8.html

This was the point that really stood out for me, and changed my mind on the whole prospect of site currency:

I assert that any time you manufacture a reason for people to participate in a community area, you corrupt the usual altruistic motivation for participation, in turn corrupting the content itself. Instead of trading in social capital, members begin to trade in a fake currency with no real bearing on the content of the community. When that happens, you invite your members to treat the community like a game.

In other words, introduce a community moderation system like site currency or up- and down-moderation, and suddenly the incentive shifts from writing quality content for standard social and creative reasons (garnering respect of one's peers, the fun of writing, etc.) and becomes a game to earn the most points. This impacts the quality of the posts in a negative way.

So after reading that article, I'm leaning away from implementing any sort of site currency or community moderation system (other than the existing flagging system). I'm actually reminded of an article I once read at http://www.gnu.org/philosophy/motivation.html titled "Studies Find Reward Often No Motivator". The basic premise is that introducing an external incentive to do some creative act (such as writing an employer rating) can cause any intrinsic interest one has in the task to decrease. External motivators also tend to put a damper on creativity, thereby killing the quality of any creative endeavor.

Now, this does raise a rather interesting question about motivation as it relates to software development. If external motivation tends to quash creativity and intrinsic interest, and writing software is inherently a creative task that requires intrinsic interest to perform well, then how can an employer motivate coders with cash?

The short answer is they can't. Either you're interested in the coding problem you're solving or you aren't. Cash is sure a nice perk, and a company underpaying its coders won't have the good ones around for long. But cash in and of itself can't make writing a boring web app more interesting. It can only make your life slightly more tolerable while you waste it writing the boring web app.

By the way, I'd like to mention an IRC channel for discussion of MicroISVs. (That's a micro independent software vendor, to you.) The IRC server is irc.freenode.org and the channel name is ##microisv. Don't forget the double pound. See you there!

software consulting and the fixed-price contract

witten — 2007-02-27

I once worked for a company with a pretty basic service-oriented business model. Customers downloaded a small client program that allowed them to connect to our servers via the internet, and we charged them for the privilege on a recurring basis.

Our potential customer base used all sorts of operating systems, so we had ported our client program to just about all of them. After we'd gotten a new customer signed on, we'd send the client program to the customer's engineers, and quite often, they'd figure out how to get it up and running without any help from us. They'd start happily using our services, and we'd happily take some money off their hands every month.

However, not all of our customers proved quite so self-sufficient. One day we got a call from a new customer that one of our sales guys had just signed up for service. Here's how the conversation went with one of our developers. (I got most of this second-hand.)

customer: We just signed up for your service, but apparently your client program doesn't support our operating system.
developer: Oh that's not a problem. We have a very simple network protocol, so you can write your own client-side program for whatever operating system you happen to have.
customer: Oh yeah?
developer: Yup, it's easy. You just open a socket and send the data as...
customer: Uh, a socket?
developer: Yeah, you know, a TCP socket?
customer: Oh.. Well, I don't think we have those.
developer: !!!

It finally came out that the customer was using a mainframe with an OS that simply didn't support sockets! Our sales guy had signed up this customer without giving even a cursory thought as to whether the customer could actually use our service. Now the burden was on our developers to either figure out some technical magic for a completely new platform very quickly, or risk making the customer very angry that they had given us money for a service we couldn't deliver on.

Although this sort of disconnect between sales and software development is all too common in the software industry, it's not the only way to do business. There are basically three ways to sell consulting or some sort of service to a customer:

1. Do all the specification and planning up front, with a fixed price tag for the development.
2. Skip the specification and planning up front, with an open-ended hourly price tag for the development.
3. Skip the specification and planning up front, with a fixed price tag for the development.

The first option is the best from a developer's perspective, as long as they're actually in on the design and planning. Customers often like this option too, since they know exactly what they're getting and exactly how much they're paying for it. The idea is you sit down with the customer and hash out their requirements in great detail, whether that includes screen mockups of the application you're developing or simply details on all the platforms they need to support. After everything is written down and agreed upon, and only then, do you start coding.

The problem with this approach is that although developers like it, and customers like it, many salespeople hate it. The job of the typical salesperson, especially one that operates on commissions, is to reduce the amount of time between initially contacting a potential customer and cashing a check from that customer. Any time spent between those two events is time that's not being spent also convincing other customers to empty their wallets. The highest volume salespeople in a commision-based environment operate by selling quickly. As soon as one customer's in the bag, they're on to the next.

So salespeople often hate the first option above for two reasons. First, it increases the amount of time between the initial contact and the check cashing. If requirements gathering and specification take an additional three weeks, that's an extra three weeks before they see their commission bonus. That three weeks could be the difference between getting that customer on their January numbers instead of their February numbers.

But perhaps more importantly, often the result of requirements gathering, specification, and planning is that you find out you simply can't meet the customer's needs, or that meeting the customer's needs will cost the customer a lot more than they had originally thought. It's in the specification and planning phase, if there is one, that little minor details come out like the customer's OS not supporting sockets. So by allowing developers to spec things out for several weeks before the deal is signed, the salesperson increases the chance of that deal evaporating and the customer walking away with their bulging wallet intact.

The second option for consulting or selling a service is to skip the up-front specification phase, and charge an ongoing hourly rate for any development work. Salespeople like this option, because without spending time on specification and planning, they see their bonus check sooner and can move on to the next customer. Developers often don't mind this option too much, because even though they're working with zero specs, at least they can take as long as they need to figure out what the customer actually wants and develop it for them, billing them all the while. But as you can expect, customers usually can't stand not knowing how much they're going to end up paying for something, especially if they're a small company. Customers have this thing about wanting to know what price they're agreeing to before agreeing to it.

The third and final option is to also to skip the specification phase, but then include a fixed price tag for the development work. And therein lies madness. Salespeople get the same benefit of getting the customer's cash sooner, and the customers are more likely to spring for it because it's got an actual price tag on it, but then the developers get royally screwed. Here's how: As soon as the sales guy hangs up the phone and does a little dance around the office to celebrate his victory, the clock starts ticking. The developers, starting with zero specs, have to figure out what the customer wants and build it in a very finite amount of time. Even worse, that finite amount of time wasn't determined after sitting down and figuring out how long the task will actually take. The development time is based on a number that the sales guy pulled out of his ass, hoping that it'd be equal to or slightly lower than the highest price the customer would be willing to pay.

So then the developers have to squeeze in what little amount of design, planning, and implementation they can into that fixed-size window of time, and once the time's up, any additional development they do isn't being paid for by the customer. Suddenly, there's a whole lot of pressure to just polish the turd as well as they can, and ship the damn thing so they can move on to the next customer. As you can imagine, this approach isn't terribly conducive to producing quality software or services that customer's love.

Unfortunately, this zero planning with a fixed price approach is one of the most common consulting models in the software industry. But if it ends up working so poorly for the developers and the customer, why exactly is it so common? Simple: Many software shops are run by salespeople. I've worked many places where the CEO is essentially the head of the sales department. And often the other salespeople are the CEO's buddies. So what you end up with is the sales team dictating what sort of consulting approach is going to be taken, and so naturally they choose the model that's most likely to make them sales: Zero time "wasted" on planning, and a fixed price tag to convince the customers to open their wallets.

So the next time you're asked to port your program to a completely new platform after one of your sales guys has convinced a new customer to sign up, and you're told you only have eight hours in which to do it, you'll have a pretty good idea as to why.

why people add developers to late projects

witten — 2007-02-24

If you've participated in the release of a software product, and you have any amount of ability to observe events in the world and draw logical conclusions based upon them, then you should've figured out by now that adding additional developers to a late software project only makes it later. Even without the ability to form coherent thoughts or reason about cause and effect, you could've had someone read The Mythical Man-Month aloud to you as you gazed with wonder at your own navel. There is no excuse for ignoring the ramp-up time, increased communication requirements, and other costs inherent in scaling up a development effort.

I once worked for a consulting company that sent me and an armada of other developers to a customer's site to work on a large software project. We plodded away for several months, ticking off the implemented features one by one. But like so many poorly planned projects, it never quite reached completion. As the project dragged on, the customer got antsy. The management at my company, in their infinite software management wisdom, decided to speed things up by adding more developers to the project.

But not just any developers. They actually pulled several people off of technical support and sent them to the customer's office as developers. I'm talking about guys who answered phones for a living and hadn't written a line of code in their lives. Now don't get me wrong, these were good guys and eager to learn. But throwing them at a late project only caused the developers already working on it to waste a lot of time bringing them up to speed, not only training them on the project itself but also teaching them the fundamentals of computer programming!

One day at lunch, several of the tech support guys masquerading as developers and I had gone to one of those vaguely fancy Italian restaurants with butcher paper over the table. We had also gotten our hands on some of the crayons they had available so kids can draw pictures on the table while their parents ignore them. I spent the lunch hour alternately wolfing down mouthfuls of spaghetti bolognese and writing out regular expressions in Burnt Sienna on the butcher paper table cloth, trying to teach these guys some basic programming concepts. They were smart and they asked good questions, but there was no way they were going to be anything but a huge drag on the development effort.

I don't need to tell you that the project ended with complete and utter failure, with my company losing their consulting contract. Unfortunately, with the exception of technical training relying on Crayolas, this sort of experience is alarmingly common in the software industry. By now, all developers and managers should have seen this sort of trainwreck firsthand, and should know better than to try the same sort of shenanigans themselves. Even throwing experienced developers at a late project is only marginally better than adding complete noobs.

So if every developer, manager, or executive who has spent more than a few consecutive minutes in the software industry should know that throwing manpower at a late project isn't a solution typically associated with hitting deadlines, then why do so many people persist in adding developers to a struggling project in the hopes of getting it done faster? Why do so may people act irrationally when it comes to late software products?

The first answer is perception. Imagine you're a middle manager in a consulting company, and your customer is giving you grief about the late project they're paying you to develop. Perhaps the customer is also putting pressure on the executives above you in your own company, who are in turn putting pressure on you. You need to do something to make the project get done faster, or at least do something very visible that looks like you're serious about the problem. Suddenly, the idea dawns on you that a sudden influx of new developers scurrying about would look awfully productive. Both your executives and the antsy customer would see that you're serious about the problem and that you're doing something about it. You might even be aware of the small fact that adding additional developers is only going to delay the eventual ship date, but that won't be evident for at least a few months. Right now, with more developers to type away at keyboards, perception will improve.

Another reason people add programmers to a late project is out of simple ignorance. Managers truly inexperienced with software projects, such as those originally from other industries, often discount the communication and scaling problems inherent to growing software teams. But inexperience can also come from those who actually have read the relevant literature on the subject, but simply haven't internalized it.

One of my friends used to work for a large, well-known software company where his team would regularly be assigned books to read on the topic of software development. It was almost like a book club. Everyone from the development managers on down to the newest coders would be expected to read the same book about best practices in large-scale software projects, and then they would hold a meeting during work hours specifically to discuss the concepts in the book. When the meeting was over, everyone would walk out of the room and promptly forget all the lessons learned, continuing to make all the same mistakes they had been making all along.

During your everyday software development, it takes conscious effort to apply the lessons you've learned, whether they're from books or hard-earned experience in the trenches. For many people, it simply requires less initial thought and effort to lapse into reactionary fire-fighting mode instead of preventing fires to begin with. Even if the fire-fighting involves dousing things with gasoline.

Finally, a major reason why people add developers to a late project is to stave off a feeling of powerlessness. Imagine you're that manager again, presiding over a late project. All questions of perception aside, if a project is running late, you might feel that you need to pull some sort of proverbial lever, any lever, to exert an effect on the project and nudge it in the desired direction. You don't want to stand idly by without taking some sort of action, because doing so would leave you feeling powerless. So you pull one of the only levers within reach, the "add developers" lever. Of course, adding programmers will actively harm the already late project, but hey, at least you're doing something.

The right levers to pull on a software project to make it successful come near the beginning of development. For starters, you can try the "requirements gathering" lever. It's also not a bad idea to give "design and planning" a tug. But when a project is late, and you're feeling powerless, what levers can you pull that will actually improve matters? Two big ones are "cutting features" and "realigning the schedule with reality" (a.k.a. "slipping the release date"). If you don't have enough time to get all of the features implemented that you've promised, reducing the scope and cutting some features can get the project done sooner.

And as unpleasant as it may be, it's often easier to sit down with the customer and renegotiate a new release date that you can actually stick to, rather than going through the charade of adding manpower and hoping that, this time, the laws of physics won't apply.

fixing power imbalances in the software industry

witten — 2007-02-19

In my last blog entry (http://coderific.com/blog/post/395), I argued that the software industry is broken because of how most development organizations are structured. The hierarchical management structure means that the people furthest away from the decision making, the developers, are the only ones with much of the relevant knowledge necessary to make good decisions. And because of the power imbalance between developers and their managers, this relevant knowledge doesn't get communicated upward to those actually calling the plays. I'm claiming that this structural problem is the root cause of nearly all problems in the software industry, hackneyed bridge metaphors notwithstanding.

Okay, so if there's a problem with our beloved industry, what can we do about it? Well, I'm going to discuss several potential solutions, ranging from minor tweaks to massive organizational refactoring. Those of you hoping for a single silver bullet solution may be a bit disappointed, but hopefully you can settle for the several ideas put forth below.

The solution that's perhaps the most difficult to implement but requires the least amount of organizational refactoring is simply to listen to developers. This would involve keeping the same old power structures in place, with managers handing down decisions to developers. But the difference would be in asking developers for their input before decisions are made and taking their feedback to heart after the decisions are handed down.

Getting developers into the decision-making process could be as simple as engaging developers to provide a voice of technical reason when features are being promised to customers. Additionally, making it company policy to take developer feedback into account on all decisions affecting development could mitigate the aforementioned structural problems. It also has the added benefit of making developers feel like they're a vital part of the decision-making process rather than an outsider whose only role is to reap the rewards of ill-informed decisions.

The immediately obvious problem with this approach, however, is that if the power structures are maintained in place, with developers at the bottom of the totem pole, it would take a tremendous amount of organizational discipline to actually put this approach into practice. It's simply going against the grain to rely on people in a position of power to listen to the opinions of those with little power, even if by doing so they would make software projects more successful. Some businesses may be able to pull this off, but it shouldn't be counted on as a general-purpose solution. So perhaps a more major structural change is necessary.

In the previous blog entry, I may have given the impression that developers have all or nearly all of the information necessary to make good business decisions relevant to software development. This is simply not so. One of the other major knowledge silos in any software organization is the team or department responsible for interacting with customers. They have valuable decision-making information about what the customers actually want in the software, and perhaps more importantly, what the customer use cases are for their day-to-day activities. Most developers are simply too far removed from the customers to know these sorts of things except at best second-hand.

The problem arises when the customer-facing employees are organizationally above the developers, either directly or by proxy. For instance, often the customer-facing employees in an organization constitute the sales department, which quite frequently is headed up by an executive or even the CEO. Thus, when a big sale is in the works, the salespeople can use their position of authority and proximity to executives to tell developers exactly what to do, whether or not it's technically or temporally feasible. The developers often have no one on their side with nearly as much clout as the executives, and so are at a distinct loss when trying, often quite legitimately, to push back.

Any solution to this power imbalance that involves simply reversing this relationship will not actually solve things. In other words, if instead of sales being higher on the org chart than software development, you somehow put software developers above sales, then you run the risk of the valuable decision-making information within sales being similarly stifled the way it usually is within development. Developers making unreasonable demands of salespeople ("Sell this thing the customers doesn't actually want, and I want it done by yesterday!") may have some merit from a strictly comedic perspective, but when it comes to actually running a successful business, it's no better than salespeople making unreasonable demands of developers.

So if you can't put sales above development and you can't put development above sales, what can you do? You can put them at relatively equal height on the org chart. Note that since a series of labeled boxes and lines is only a rough approximation of the actual power structure within an organization, moving around some arrows on a Visio chart isn't sufficient to accomplish this goal. You've actually got to get the executives out of the sales department so as to remove the sales department's direct line to the throne, so to speak.

With the two departments on a much more even footing, then neither department has a distinct power advantage, and neither can make unreasonable demands of the other. Decisions will have to be made in tandem, thereby taking advantage of the knowledge silos within both departments. This leads to the other part of this solution. Once sales and development are more or less equals, the next step is to improve the communication between the two. Accomplishing this can be as simple as sitting sales next to development instead of across the building or in another timezone altogether, or it can be as formal as having the two departments meet on a somewhat regular basis to discuss upcoming deals and potential directions for development.

Improving communication will not only improve joint decision-making, but since knowledge no longer has to flow uphill to get from development to sales, it will actually improve knowledge-sharing between the two departments, thereby making decision-making within a single department that much more effective.

Note that in this discussion I've focused on the sales arm of a hypothetical organization. If in your company, the structural power imbalance is felt among development and some other department or even set of departments, then you can follow similar steps with those. Remove or reduce the power imbalance and work on improving communication.

There is one rather glaring problem with the solution suggested thus far. It takes even more organizational discipline than with less invasive solutions, not only to affect this level of change but also to maintain it and avoid slipping back into the natural state of structural problems you started with. In fact, the solutions suggested would have to be initiated by those in power, and there is very little obvious incentive for them to give that power up, even if it could conceivably improve decision making and therefore business success as outlined here. It may simply be the case that except in the most enlightened of organizations, there's an insufficient amount of organizational will to put these recommendations into place.

So instead of fighting the uphill battle of implementing these ideas in an existing organization, another option is to start a new organization and get the power structure right from the beginning. Assuming that the founders have the financial and management wherewithal to actually pull this off, dictating the structure of a completely new organization is easier than you might think.

Think of your first day at your current job or one of your recent programming jobs. Assuming that you were an employee and not a founder, you probably came into an existing development shop with fairly well-established relationships to other departments in the company. You may have given the organizational structure some initial consideration as you tried to understand what departments there were and how they would affect your day-to-day coding job, but chances are you just accepted the current structure as the way things are, and were instead concerned with more pressing issues like where the bathroom is located or where the hell your .vimrc file went. New hires in an existing company try to fit themselves into whatever structures are already in place, so as long as those structures are conducive to successfully pumping out quality software, all you should need to do to keep the ball rolling is perhaps implement some basic training or mentoring.

When a new business is founded, the founders have an immense amount of control over all aspects of the company, from the corporate culture to how the organization is structured. When Bill Gore founded W.L. Gore & Associates, he had a new idea for a corporate structure involving a "flat lattice organization" with "no chains of command nor pre-determined channels of communication" (http://www.gore.com/en_xx/aboutus/culture/). Now, this particular structure may be a little hippy-dippy for some of you, and Gore and Associates is by no means a software development shop, but the point is that the founders of a company can come up with any sort of hair-brained org chart they want, and as long as it doesn't drive them out of business, they can quite easily foist it on new employees. Gore & Associates, by the way, made $1.84 billion in sales last year.

The cost of implementing structural change in a business decreases as you go back in time towards its founding. When it's just two guys in a garage, it's relatively easy to dictate that any sales team you hire must be on equal footing with your programmers. When, 5 years later, you're a 300-person company, making that policy into practice is much more difficult.

Another solution to the structural problem is simply not to grow your organization to a size where it becomes a factor. If your company never gets much larger than the two guys in a garage, like the 37signals web app shop, then you don't have to worry about the power imbalance between your sales and development departments, because you won't even have departments. You won't have to worry about managers having insufficient technical information to make well-informed decisions, because you simply won't have managers.

Last, but not least, you can do the whole open source thing. There are a number of problem domains that open source solves well (and a number that it doesn't), but if you're developing Free Software outside the constraints of a typical business, then you don't have to deal with any of the power imbalance problems inherent to a hierarchically managed organization. Of course, developing open source software within the confines of a company is an option too, but then you've still got to contend with all the standard corporate structural problems.

So, to solve the structural and power imbalance problems inherent to most businesses, you've basically got three general options: mitigation, reorganization, and reimplementation. They're ordered by increasing cost of implementation but by decreasing difficulty of enforcement. You may find some success with the lower-cost mitigation options, but sometimes you've just got to start over with a clean slate.

why the software industry is broken

witten — 2007-02-17

The software industry as it currently exists is simply broken. A large percentage of software projects end in utter failure, so something is quite obviously going wrong. This failure often comes in the form of abandoned projects, where the customer doesn't even accept the software. Quite often though the project will simply miss a series of deadlines and/or not do what the customer actually wants, requiring massive reworking.

So what's at fault with these software project failures? Is the culprit the development process, the project management, the requirements gathering, or the complexity of software itself? Well, yes, all of the above. But none of those things are actually the root cause.

Nearly all things that are wrong with the software industry today stem directly from the way most companies are structured. Specifically, I'm talking about the top-down, hierarchical management structure in which decisions made at the top of an organization are handed down to those below, eventually making their way to the software developers in the trenches.

Now, on the face of it, this is a rather bold claim. I'm suggesting that the software industry is so broken today not because of the tools used or the development methodologies employed, but simply because of the way most businesses organize their employees.

If the software industry is broken because of corporate structure, then why aren't other industries similarly out of whack? For instance, manufacturing, construction, airline, and service industries all have a similar corporate structure to most software companies, so why aren't they similarly broken?

The cynic might argue that those industries are similarly broken, but for now let's assume that the software development industry has its own special breed of brokenness. After all, while the occasional bridge might fall into the water shortly after its built, that's certainly the rare exception. Software projects fail all the time.

So why does the corporate structure that seems to work out okay in other industries fail so miserably for software development? Well, there are a couple of reasons.

Firstly, software companies are based around abstract knowledge work. The processes by which coders develop software is not at all clear to the suits leading a company, because software development by its very nature is an abstract endeavor. And even those rare leaders with development experience can't tangibly assess the state of a project just by looking at it. The only people who really have a decent handle on where a project stands and what features can be added to it in a reasonable time frame are the coders in the trenches, and perhaps if the company is lucky, a good development manager or two.

One of my friends has a great analogy to explain this. Imagine you're building a bridge to span a river, and it's nine months into a ten month project and the bridge is nearly complete, and your boss comes along and says, "Oh, by the way, can you add a second deck to that thing?" This of course would be thoroughly absurd, because any schmuck can see that trying to tack a second deck onto an already designed and nearly completely constructed bridge would be a horrendously bad idea.

Now, imagine you're developing a software project, and it's nine months into a ten month project, and the project is nearly complete, and your boss comes along and says, "Oh, by the way, can you bolt on [what amounts to a second bridge deck] to that thing?" There's no physical structure you can point to and say, "Look, you see that support there? It can't handle the weight." Since software is perceived as infinitely malleable due to its abstract nature, and is so often opaque to the managers making the decisions, the suits at the top simply don't see the absurdity of adding a second deck to a software project at the last minute.

One approach you can take in the face of this problem is to simply build a model bridge out of toothpicks or popsicle sticks in parallel with your project. And then at the 8 or 9 month stage, when the software project is nearing completion (or at least its original intended deadline), you'll have built out the model bridge so that it's nearly complete as well. So when someone in a suit tries to slip in "just a tiny feature" at the last minute, you can point at the bridge and try to explain the whole analogy before their eyes glaze over.

But short of having to build a toothpick bridge alongside every software project, what can a developer do when faced with unreasonable demands? Many developers try to push back or to educate their supposed superiors: "The architecture won't support that kind of addition before the deadline," or "That will add too much risk to the project," or "Because of obscure technical details x, y, and z, that doesn't make a whole lot of sense."

And here's where the structure of the organization matters. In a typical top-down hierarchically managed company, where most developers are at the bottom of the totem pole, pushing back without a concrete physical thing to point at as evidence is viewed as not being a team player at best or outright insubordination at worst. Because of the power imbalance between managers and developers, no matter how much the developers might argue, debate, or plead about any particular management decision, the managers have the final say.

And because of the abstract nature of software, the developers are actually the ones with the most information available to make an informed decision. The higher up you go in an organization away from the developers, the less technical information is available with which to make good decisions that affect the development process. So what happens in practice is that the developers with most of the information necessary to make good decisions are told by the people with very little of this information to tack on a second deck to the proverbial bridge right before the deadline. And because of the power imbalance, combined with the abstractness of software, developers have a hell of a time convincing anyone that a second deck might not be such a hot idea.

Now of course, there are some software companies apparently not so strongly afflicted by this problem with corporate structure, where good decisions are often made and software projects often succeed. The reason these sorts of companies are so rare is simply because in a hierarchically managed organization, it takes a tremendous amount of discipline to actually listen to the developers in the trenches when making important decisions. It is a rare organization indeed in which developers have the ability to say "no" when asked to implement things that are technically or temporally unreasonable.

In other industries like manufacturing, the executives at the top do understand what's going on in the trenches to a sufficient degree, and so have sufficient information to make good decisions. In the software industry, unfortunately, the executives just don't have enough information about any particular software development project they're ostensibly overseeing. As a result, bad decisions are made, projects fail, and we find ourselves in an industry which could charitably be described as barely functional.

So why don't other industries with knowledge workers, like accounting or advertising, run into the same sorts of problems? Simply put, software development is just plain difficult. Successfully making software in an organization requires so many things to go right: Requirements gathering, specification, design, scheduling, coder wrangling, and so on. If just one of these things goes wrong, an entire project can be at risk of utter, humiliating failure. The same can be said of other industries, but software as it's currently developed is just so damned complex. Shipping a successful project requires an organization to be run like a well-oiled machine. Anything less, and you'll miss requirements, overlook important design considerations, and eventually miss deadlines.

What's more, communication in most top-down organizations flows downhill, and very little communication flows in the other direction, especially when it's bad news. So by the time any bad news does make its way upward, it's often too late to make a course correction. This is much more important in any software project, again due to complexity. There are so many potential decision points leading up to and during the development of an application, but without honest, unfiltered communication flowing upward, those decision points won't happen. Or if they do, they'll be based on incomplete information.

So if the organization of most organizations is to blame for the state of the software industry today, what's the solution? Stay tuned for next time.

the Next Big Language doesn't matter

witten — 2007-02-11

In a recent post (http://steve-yegge.blogspot.com/2007/02/next-big-language.html), Steve Yegge describes the features the "Next Big Language" must have in order to take over the world, including things like C-like syntax, dynamic typing with optional static types, and a decent IDE.

When I refer to NBL, the Next Big Language, I specifically mean the next popular language. I.e., a language that you yourself will wind up learning and actually using.

Any language lucky enough to support all of these must-have features would presumably be accepted much more quickly than less well-endowed languages, and therefore would eventually be wildly popular among developers.

But I really need to stop spending time telling people individually why their languages are doomed to fail. Instead, I'll summarize it in today's blog entry.

His premise is that any language you invent, unless it has these certain features and is backed by industry heavyweights, is doomed to utter obscurity and therefore failure.

This is a familiar refrain. In an excellent series of blog posts on what features a more popular Haskell-like language might have (http://toomuchcode.blogspot.com/2007/02/part-1-programming-and-metaphorical.html), Ryan Brush claims that:

A language must be usable by mediocre developers to achieve widespread use.

And that:

Widespread adoption is important so good programmers can use better languages in their day jobs.

However, I simply don't accept the premise. It may have been that ten years ago, your choice of languages was restricted by what binaries or VMs your customer's desktops could run. And in that case, the relative popularity of a language really did matter. If you chose a language whose runtime none of your customers could install or use, or didn't come pre-installed on your customer's computers, then you ran the risk of not being very successful.

But today, many applications are web apps served off of remote servers. Except for client-side UI code, large swaths of application logic live on a climate-controlled, colocated server that the customer never touches except via HTTP. That means that the server can equally well run Lisp, Python, Befunge, you name it, without having any adverse impact on your customers. As long as it serves up the web application the customer wants, it doesn't matter whether your language is the Next Big Language or the Obscure Dialect Du Jour.

Now, before you start getting all in a tizzy, let me state some rather important caveats.

I am fully aware that local, native applications aren't going away anytime soon, or ever. And for those, Yegge's Next Big Language will probably be very handy. And yes, I know that he's probably talking about the next version of JavaScript, which will even play a large role in web applications.

I'm also aware that some web applications are shipped to customers, and so do have runtime and platform restrictions. Witness Fog Creek's much-lambasted Wasabi.

But even so, there will still remain a large contingent of server-side application components that you run on your own servers, and so you'll be able to write those apps in any language you want. In other words, although there will always be many areas of applications in which the language does matter, there will similarly always be many server-side applications in which it simply doesn't.

So, if you can use any language you want on the server, does language popularity matter at all for server-side applications? You better believe it does. As Ryan Brush points out, language popularity has a direct impact on whether your risk-averse employer will let you use it at work. Nobody has ever gotten fired for choosing Java.

But again, I'm not making an argument that you can happily get by using Bob's Podunk Lisp Variant when you work at IBM or Sun, but simply that there are still some specific areas of application development in which you can use pretty much any language you want. Now, this may be relegated to the server in the form of web apps, and you may have to be working at a smaller and/or less stuck-in-the-mud company to convince your boss to let you use your language of choice, but there still are many areas in which the Next Big Language just doesn't matter one whit.

Okay, so then how does popularity affect writing server-side apps for progressive organizations that will let you use something other than Java, C#, or ECMAScript 2007 Enterprise Edition? Well, there's still the question of standard and third-party language libraries, where the network effect has a large impact.

For instance, if your language is so obscure that no one's writing libraries for it, then it will be extremely difficult for any developers, new or experienced, to get anything done. In other words, if you have to write your own HTTP client every time you want to pull something down from a URL, because you're only the eighth person who has ever used the language and the first seven used it to write nothing more variations on the Fibonacci sequence, well then you're going to be giving up on it pretty quickly in favor of Java.

But if the language happens to be moderately obscure while still supporting enough of a self-sustaining community to contribute, maintain, and document the language's standard libraries, then the language can be useful enough to developers for day-to-day use. Who cares if Microsoft is using your language as long as it's got built-in libraries for doing everything you need to do and runs on all the machines you need it to run on.

So unlike Steve Yegge's characterization, that language popularity is an all-or-nothing proposition, there is a fine gradient between total obscurity and complete corporate adoption. And along that gradient there lie varying degrees of usefulness for varying types of software development. If it so happens that your language of choice is just popular enough to be useful for the kind of applications you're developing, then you can continue to code in peace without having to worry about what might be next or big.

Oh, and for all you Linux/BSD nuts out there, as an exercise, run through the above argument substituting "OS" for "language". You'll notice that the Next Big OS doesn't matter either.

what the worst employers have in common

witten — 2007-02-08

If you've worked in the software industry for any length of time, you can probably rattle off a long list of problems found at the worst places you've worked. But to save you the trouble of dredging up those painful memories, I've collected a list of what the worst employers have in common, generated from the bottom of the barrel of Coderific employers: http://coderific.com/employers/worst

First, let's look at the highest-rated aspects of the worst-rated employers. In other words, these are areas in which the worst employers don't completely suck. All scores are out of 4 stars, and employers lacking multiple ratings are excluded:

1. casual dress code: 3.3
2. nearby food: 2.9
3. internal team communication: 2.8
4. location: 2.8
5. business model: 2.8
6. quality of developers: 2.7
7. product quality: 2.7
8. source control: 2.6
9. health coverage: 2.5
10. paid time off: 2.4

Much like the best employers (http://coderific.com/blog/post/368), the worst employers also are rated highly for casual dress policies. Granted, the casual dress score for bad employers (3.3) is a bit lower than for the best employers (3.6), but I would've expected a lot more suits and ties at the bad development shops.

It is notable that even at the worst software shops, the quality of developers metric still gets a moderately decent rating. Is it that developers are unlikely to blame themselves for any problems that a company might have? Or are all of Coderific's users from Lake Wobegon, and therefore all above average? Obviously it's the latter.

But what about the lowest-rated aspects of the worst employers? Why exactly are the bad organizations so bad to work for?

1. hitting deadlines: 1.9
2. team-to-team communication: 1.9
3. automated testing: 1.8
4. quality of upper management: 1.8
5. other perks: 1.8
6. prevention of crunch time: 1.7
7. warm fuzzy feeling: 1.7
8. employee retention: 1.6
9. overall: 1.6
10. development of Free Software: 1.5

Besides the fact that these scores are almost a full star lower than those for the best employers, the lists of the lowest-rated aspects are eerily similar. Both the best and worst employers have trouble with team-to-team communication, automated testing, hitting deadlines, and prevention of crunch time. A natural conclusion from this data is that perhaps all development organizations have these problems in one form or another. Developing software is a relatively new discipline, and neither the best nor the worst organizations have quite figured out how to do it without missing deadlines and slave-driving their developers.

So if the best and the worst software shops have so much in common, then what differences are there to explain why the worst shops are so bad from a coder's perspective? Well, if the items present on the list above but missing from the list for best employers is any measure, then quality of upper management, employee retention, and that warm fuzzy feeling are all rather significant.

They say that you don't quit your job. You quit your boss. This little aphorism might be more true than you would otherwise think. Quality of upper management is rated so poorly at the worst employers, in fact, that it could be the primary reason driving developers away, leading to the employee retention problems these organizations have.

When coders feel like their management is incompetent, out-of-touch, or otherwise incapable of basic management tasks, they will be much less likely to go along willingly when asked to work that inevitable crunch time. Simply put, developers become resentful when they're negatively impacted by poor management decisions.

But at a good employer, with decent or even passable development and upper management, coders feel much more like everyone's rowing in the same direction and there's less of an "us versus them" mentality between developers and management. So even though there are still missed deadlines and crunch time at the good employers, the code monkeys won't feel so resentful about it, and so will be much less likely to hop jobs to try to improve matters.

Ah, and then there's the matter of the warm fuzzy feeling. It's the feeling that the code you're cranking out is worthwhile, that the products you're making are going to make users' lives better. There are a million tiny ways to erode this feeling, including everything from poorly justified management decisions to neglecting to give developers any actual software development requirements.

Now, let's get to the OSes. Of the worst-rated employers, here are the platforms used:

1. use of Free Software: 2.7
2. use of Windows: 2.6
3. use of GNU/Linux: 2.4
4. use of Mac OS: 1.9
5. use of Solaris: 1.6
6. use of BSD: 1.0

Wow, the ranking of platforms used for best employers and worst employers are identical! I guess we can discount the platforms used in an organization as a major factor in how good the employer is for developers. Sure, some developers prefer one platform over another, but whether an employer is overall good or bad does not appear to correlate strongly with the OSes used.

Of the worst-rated employers, here are the languages used:

1. use of C++: 3.1
2. use of legacy languages: 2.8
3. use of Java: 2.7
4. use of C: 2.3
5. use of Perl: 2.0
6. use of ASP: 1.6
7. use of PHP: 1.6
8. use of C#: 1.6
9. use of Python: 1.4
10. use of Ruby: 1.1
11. use of Lisp: 1.1
12. use of Objective-C: 1.0

Now this is interesting. The poorest development shops use a heck of a lot more C++ and legacy languages than the best employers do. It appears that the good employers use what can be glibly categorized as trendier languages, while the worst employers use languages like Fortan, COBOL, and C++. The heavy use of legacy languages isn't too terribly surprising. Bad development shops don't have the time, motivation, or leadership to refactor or upgrade, and so are often stuck supporting old cantankerous systems written in languages not mentioned in polite conversation since the Ford administration.

But what about the large disparity in C++ use among the best and the worst employers? To put it plainly, higher-level languages like Java, C#, Ruby, and Python are more fun for most developers for most general-purpose programming. When your day-to-day debugging involves fixing the actual logic in your program rather than chasing down memory allocation problems, you tend to be a happier coder.

So what can development organizations learn from this data? Well, if you have bad management in your company, throw them out. It could be costing you more than you think to keep incompetent managers running things, especially in terms employee turnover and general coder productivity.

What if you are the bad management? Well, then either ask your developers for a couple of good books to read on the topic or think about another occupation. After all, you can sit in your office all day being ineffective just as well in any other line of work.

what the best employers have in common

witten — 2007-02-06

The coders have spoken, and through the rigorous scientific process of anonymous forum posts, they've selected the best employers to work for: http://coderific.com/employers/best

So what makes these employers so good? What traits do they have in common? What flavors of gummi bears do they have in their snack rooms? I could try to make some generalizations about these organizations, perhaps to shed some light on what makes them so kick-ass. Unfortunately, it would be simply irresponsible to attempt to draw conclusions about the employers on this site based on the small amount of data currently available.

Fortunately, I've never been big on responsibility. So here goes.

As of this writing, the average employer rating on Coderific is about 2.5 out of 4 stars. Not too shabby, considering. (In all data used here, employers lacking multiple ratings are excluded.) Now, let's break this down by the various aspects within each rating. Of all the ratings for best-rated employers, here are the highest-rated aspects:

1. casual dress code: 3.6
2. quality of developers: 3.5
3. cool technology: 3.4
4. overall: 3.4
5. development hardware: 3.3
6. cultivation of creativity: 3.3
7. taking responsibility: 3.3
8. internal team communication: 3.3
9. product quality: 3.3
10. source control: 3.3

Many of these aren't too surprising. Developers don't like wearing ties, because restricting blood flow to the brain isn't conducive to, well, anything. And so obviously the best places to work have a casual dress policy, at least for coders.

It's also not a terrible shock that developers think highly of other developers at these good workplaces, or that such employers have cool technology. Developers are at heart propeller heads, and in general they'd much rather work on whiz-bang technology than on pumping out tired business solutions that have already been implemented a thousand times before.

It is rather interesting to note that good employers have many of the touchy-feely corporate culture things pegged, including cultivation of creativity and people taking responsibility for their actions. Nothing kills a developer's perception of their workplace more than an oppressive, unfair, or otherwise crappy corporate culture.

So what about the lowest-rated aspects at the best-rated employers? Here's where the top employers fall down:

1. team-to-team communication: 2.9
2. snacks: 2.8
3. reasonable workload: 2.8
4. issue tracking: 2.8
5. mitigation of risk: 2.8
6. prevention of crunch time: 2.7
7. development of Free Software: 2.7
8. hitting deadlines: 2.6
9. clear requirements: 2.6
10. automated testing: 2.6

So even the best employers aren't immune from the standard software development problems so prevalent in this industry: Release cyles ending in crunch time and missed deadlines. There are many reasons why a release cycle can go south like this, but I'll give you a hint as to what might be going on in this case. What one item, if not present in sufficient dosage towards the start of a release cycle, dooms all the remaining development to missed milestones and last-minute surprises? Look at that list again. Yup, I'm talking about good, clear requirements. Without those, even the best employers risk tossing their schedules out the window.

And for all the attention various popular development methodologies pay to unit testing and other automated tests, the best employers still haven't gotten that one quite down. This is an obvious area to work on, probably for all development organizations.

Gummi bear flavors could use some work, too.

In terms of platforms, the best-rated employers fare as follows:

1. use of Free Software: 3.1
2. use of Windows: 3.0
3. use of GNU/Linux: 2.9
4. use of Mac OS: 1.8
5. use of Solaris: 1.6
6. use of BSD: 1.2

Free Software isn't really a platform per se, but it is interesting to see that while not many of the best development shops actually develop free software, a whole lot of them use it. And though Linux claims a rather small percentage of the desktop market, it apparently has a strong showing among developers at the top employers.

And as for programming language use at the best-rated employers:

1. use of C#: 2.6
2. use of C: 2.6
3. use of Java: 2.6
4. use of ASP: 2.5
5. use of Python: 2.4
6. use of Perl: 2.3
7. use of C++: 2.3
8. use of PHP: 2.2
9. use of Ruby: 1.9
10. use of Objective-C: 1.4
11. use of legacy languages: 1.4
12. use of Lisp: 1.2

One moderate surprise, at least to me, is that the hydra language that is Java/C# has apparently eclipsed C++ (no pun intended, honest). C is still going strong, and even Python is making a showing. Functional languages like Lisp are getting no love though.

That's all for now, but stay tuned for next time: What the worst employers have in common!

only good programmers don't like coding

witten — 2007-02-04

Jonathan "Wolf" Rentzsch claims the following:

Programmers don't like coding, they like problem solving. If programmers liked to code, we'd all be writing in machine language to this day. You can write that stuff all day and get precious little of the real problem solved.

More at: http://rentzsch.com/notes/programmersDontLikeToCode

I'd like to amend this statement in a very important way: *Good* programmers don't like coding in and of itself; they like problem solving. *Bad* programmers find all sorts of ways to code for the sake of coding, and just use problem solving as a convenient excuse to write voluminous amounts of code.

I once worked with a particular programmer from this last camp. Let's call him Bob. Don't worry, if you're one of my former coworkers, and you're reading this, then it isn't you. Bob liked to write code. He liked to write lots of code. He would invent problems where there weren't any, not because he genuinely liked solving problems, but because he liked to write copious amounts of code. He relished big, sweeping architectural changes, and simply based on the frequency with which he caused them, you'd think that he knew no other way to write software.

Bob would often come up with supposed problems to solve and pose them to his manager in order to justify his little coding expeditions. He'd justify the solution by using some metric that hadn't occurred to anyone but Bob himself up to that point, such as using performance considerations to justify making changes to something where performance wasn't really an issue. Right before Bob's team was about to start on a bout of coding that Bob really didn't feel like participating in, he'd find a way to embark on one of his own solo development efforts of dubious utility.

Sure, Bob solved some problems in the course of his coding for coding's sake, but if he did so it was almost by coincidence. Bob was simply too fascinated with the minutiae of his language of choice, and so as long as he was lost in reams of code he had written containing overly complicated component interactions to sort out, he was happy.

But we're not all like Bob, or at least we strive not to be. There are times when even the best programmer falls into the Bob Trap, coding without keeping the problem being solved in sight. And certainly it is the minority of programmers that are good enough to avoid falling into that trap on a regular basis.

So no, Jonathan, it's not that all programmers like problem solving over coding, it's that only *good* programmers do.

a weblog for coders, by a coder

witten — 2007-02-04

Welcome to the inaugural issue of the Coderific blog. First, let me tell you a little about myself. I develop software for a living, and by that I mean that people in suits pay me to press little square lettered buttons for about eight hours each day to produce a series of zeros and ones that they then sell to other people in suits. I also develop software for fun, which means that I press little square lettered buttons for another several hours each day to produce a different series of zeros and ones that for some odd reason is extremely edifying.

One of those series of zeros and ones constitutes the fine web site you are currently enjoying. (Hint: No one in a suit paid me to make it.) Coderific originally arose to fill a gap, the gap between knowing some basics about a company and knowing whether you'd like to work there. And anyone who has worked in the tech industry for any length of time knows that asking a company for its opinion of itself is no way to get a good, balanced impression of whether they're a good employer. The only way to find out the real facts about an employer is to ask the people who have actually worked there.

But you already know the basic premise of this site. So how about this. I'll list off some of the topics that I intend to eventually cover in this blog, and you'll sit there and nod and smile and pretend you're interested in them.

Firstly, you can bet your favorite text editor and/or IDE that this blog will cover the tech industry from the perspective of the people who actually write code in it. This won't be an IT manager rag, and it certainly won't be a blog gushing about the latest regurgitated press releases from consumer electronics and/or software giants. All I really care about is the people who slave away to make the cogs turn, and I'm guessing that unless you're one of the people in suits, you do too.

Secondly, this blog will be technical. I will cover geeky topics. I will slant unabashedly in the open source / free software direction. This is first and foremost a coder's blog, so don't expect anything less.

Last but not least, this blog will be about Coderific itself. I will announce new and prospective site features here first. This blog has an rss feed, and as can be expected, you're free to post comments on any particular blog entry.

So welcome to Coderific, and please let me know if you'd like me to cover any specific topics of interest.