Coderific

blog - the importance of debuggability

posted by witten on April 01, 2007

There are a number of ways in which a library can actively get in the way of efficient debugging. A common one is the overuse of clever C++ template meta-programming. Genericity is an incredibly useful language feature, and you can do all sorts of cool tricks with templates to make things happen at compile-time rather than runtime. However, you can go overboard with meta-programming.

Consider Boost Spirit (http://spirit.sf.net), a parser framework that can be used for compiler development. Spirit makes extensive use of template meta-programming so that you can declare your parser grammar in a BNF-like syntax directly in your C++ code. But like with so many template-heavy projects, make one mistake with Spirit, and during compilation you'll be treated to an error message like this one:

Okay, welcome back.

That error message is the actual compilation error I get by leaving out a single "&" operator in a Spirit grammar definition. And this isn't even close to the longest or most incomprehensible error message I've ever gotten when developing Spirit-based parsers.

Where do you even begin with an error like this one? Well, you can start at the end, and try to figure out what's going on with the invalid conversion. Given enough effort, you'll probably eventually determine what went wrong, but not until you've wasted gobs of time that you could've spent actually doing productive work.

When you're writing any kind of code, an almost universally good practice is to fail early. In other words, if you encounter some exceptional case or invalid data, you should throw an exception or return an error code as early as you possibly can, thereby ensuring that the error is the most relevant to the root cause of the problem.

This policy is even more important when you're implementing a general-purpose library. If the library fails as soon as the caller passes in invalid data, then it should be pretty clear to the programmer what went wrong and why. But if instead a library only throws an undocumented exception from somewhere deep within the bowels of the library's own code, then you'll end up with incomprehensible template garbage like the error above. And any programmer making use of the library will have to go spelunking through the library's source (if it's even available) to figure out how their use of the API corresponds to the particular error they're seeing.

Another way of putting this is that the worst libraries have leaky abstractions. In order to debug problems with such a library's black box API, you have to open up the box and start rifling through its contents to figure out what went wrong. This takes time and makes programmers pull out their hair.

Using a difficult-to-debug library has implications beyond just wading through error messages. The best way to release software based on any sort of schedule is to actually break the development down into small, granular tasks, and then estimate how long each task will take to complete. (See http://www.joelonsoftware.com/articles/fog0000000245.html for more info.) This approach isn't anything revolutionary, and it's a pretty well-understood practice in the software industry despite the fact that coming up with even remotely accurate estimates is as much art as science.

But a major problem with making schedules based on granular estimates arises when you try to estimate debugging. First of all, when you're trying to figure out how much time to schedule for a release, you know how many development tasks you have, and so you can somewhat accurately estimate how long actual development will take. But how can you possibly estimate debugging if you don't know how many bugs you'll have? And even if you have a rough idea of how many bugs you're going to have based on how much code you're writing, it's impossible to know how long fixing a particular bug will take.

There are basically three phases to fixing a bug:

1. You try to reproduce the reported problem.
2. You try to find the root cause of the bug.
3. You fix the root cause of the bug.

When you attempt to reliably reproduce a problem, you typically just try different actions or configurations until the bug magically reappears, and then you try to confirm that the actions you took caused the bug and it wasn't all just a coincidence. Similarly, when you attempt to track down the root cause of a bug, you run through the reproduction steps with prints in the code or while stepping through the source in a debugger. After some indeterminate amount of time, you'll hopefully come up with a root cause. Both of these steps are more like detective work than anything else. You basically do them until they're done. And because you don't know what's causing the bug until after this detective work is complete, you have no idea about how long it will take to fix the bug until after you've found the cause. In other words, the first two steps of debugging both have the interesting property that together they take most of debugging time and they're also nearly impossible to estimate in advance.

When you use libraries that are difficult to debug, you're not just wasting time. You're actually killing any chance of your development schedule resembling reality. Hard-to-debug code adds more uncertainty to the debugging phase of the release cycle, and even with decent libraries, this debugging phase is already the most difficult part of the schedule to accurately estimate. The more uncertainty you introduce into the debugging phase, the less likely you'll be able to predict when any given release will be out the door.

So when evaluating a programming library, don't just examine its API, its efficiency, and the quality of its code. Also evaluate how easy it is to debug code written to use the library. Because if you trade debuggability for efficiency, you may end up with speedy code that never gets released.

2 comments