On The Gantt Chart, Clients, and the Stuff in Between

There are some really insightful points over at CodeSqueeze. Though, the starting point is agreed on by managers in the know – Gantt Charts lie. Lies are not useful.

To reduce the number of lies and unknowns you have to gather data. Find out what you don’t know, and then know it. This is hard when everyone is trying to save face, and show that they know everything there is to know. As a client facing person you have to first find out about the ‘domain’, what the project needs to do. This is vastly different from what the client wants the project to do. Descrepancies between these two ideas are what causes the most problems in a software development cycle. Software solves problems, it doesn’t do tasks. Find out the problems you need to solve. This is why Gantt charts are not useful, they only know tasks, not problems.

The most common objection is pointed out by Max:

We don’t want to waste time or money on a client or project we don’t know if we have yet. Let’s just figure out how much it is going to cost so we can give them a bid and we will figure out the rest later.

Do I even have to go into depth why this is stupid and counter intuitive? How can we even be remotely close in our estimations if we have no context of what we are building. We obviously are slow learners because numbers still fly out of our mouths when given an elevator pitch’s worth of information.

The real problem here is a problem of perception, perception of what a project really is. A project is not a clients wish list. A project is not a vague idea. A project is a solvable problem that will save or make the client money (and time is money, remember). A vague idea is not solvable, nor is a wish list. If the client is convinced they have a problem that costs them money, or a problem they can solve and therefore make money – they should be equally convinced to do real discovery. But, if projects are seen as things they can do, or not do, depending on the day of the week, which side of the bed they woke up on, or how much it costs (problems are recurring, cost is a one-time thing) then of course you’re not going to waste time and money doing discovery. Just give me a number and I’ll tell you go, or no go. Usually by then the client is checked out and unable to actually tell you something about the project.

On to the other side of the coin. As a developer, it is hard to pin down the difference between large and small time blocks:

Sure we can accurately estimate the difference between a 1 hour task and a 2 hour task, but can we really estimate the difference between a 23 hour and a 24 hour task? No, not in the least.

As Spolsky points out, you have to break down problems into small tasks. Of course that is impossible if the first part is not done right. If you don’t know what you’re solving, you don’t know how to find a solution. It is hard to break down dogmatic tasks into usable pieces. It is hard to break down a fake-problem, or a problem constructed right on the spot (which doesn’t match reality) into smaller, solvable problems.

Our job cannot be imagined as a factory. Work and progress is not made linearly. It is made in chunks. It doesn’t always go in, what seems from the outside to be, a logical pattern. And hopefully, your job as a programmer requires creativity. Programmers are responsible for the coding, and that includes the scheduling as well. Management shouldn’t be allowed to dictate which parts are logically before and after something. Progress is usually visible only near the end, much like an iceberg. You start with the entire project, the entire solution visually solved. If its a web-app, then you have HTML and CSS. If it is a desktop app, you have all your windows, menus, and pull-downs arranged. That is the agreed upon solution. (This doesn’t mean, of course, that these things can’t change, solutions to problems change as the problems are better understood. Which is the whole point of this article. To commit to a deadline and price without understanding anything is like playing Russian roulette. You might kill youself, your developers, or your client. They each get a turn with the gun). To continue with the iceberg, only near the end does it look like anything has been done at all. That is distinctly different than a factory where incremental changes are made piece by piece until something is completed.

Now, how to compliment managements desires with the realities of software development? That really means, how to schedule a project so 1) the client gets it by their deadline, 2) the developers don’t kill themselves, and 3) everyone likes each other when it is all said and done. Until I’m convinced otherwise, I think scheduling can only be done to satisfy all those criteria in one way: the Spolsky Monte Carlo method. I’ve whipped up one of these and applied it to our time sheets. And I’ve noticed that it fails to deliver in certain circumstances.

If the sample size is too small, the results aren’t that informative. If you have less than one hundred tasks, you’re going to have to duplicate some time entries to get to a hundred. You want to be taking a subset of the time entries, as Joel argues. But, if you do not have a large sample size, taking a subset will give you wildly erratic data. If the tasks are wildly divergent the results aren’t that informative either. This also results in wildly erratic data. If you’ve got a lot of one hour tasks that are finished in fifteen minutes, and fifteen hour tasks finished in thirty hours you will create a lot of outliers in estimating for other tasks. These problems are all explainable though.

If you don’t have enough tasks, and you have fifteen hour tasks, you’re not making smaller solvable problems. If some projects are wildly under and others wildly over, you can have any number of problems, usually all of them; communication failure to development, failure of discovery with client, failure of knowing the consequences of decisions, feature creep, or flat out feature changes mid-project.

Again, if you can change the perception (this is very hard to do) of how people look at projects, you can erase the failures of discovery, feature creeps and (gross) feature changes. Consequences and communication just take experience and a willingness to improve.