Time, Value and the dark side of Agile

(This post is a follow-up to Agile is Antifragile.)

What is the best way to organise a group of people who are working together to make some software? This must be the central question of software engineering, but we tend to think about it in a very narrow way. It's rather odd that we are happy to talk about processes, from Waterfall to Agile, but we don't like to talk about the kind of practical problems that always arise when people come together to make something for money.

These practical problems are standard fare in business schools and in some mainstream engineering courses, where they are discussed under the label Organisation Theory. This in turn has its roots in ideas of "political economy" that go back over 200 years. Once you understand these ideas, you can see clearly the cause of some tensions that arise in software development. So, let's see if I can introduce you to these ideas — I don't expect in this post to get to the twentieth-century fashions started by Taylor and Mayo, but we should be able to cover enough of the nineteenth century ideas to understand the different feelings people have about Agile and to recognise its dark side.

Where to start? Lisp-hacker Paul Graham, in his essay How to Make Wealth, uses the following example to explain how wealth or "value" can be created by people out of nothing:

"Suppose you own a beat-up old car. Instead of sitting on your butt next summer, you could spend the time restoring your car to pristine condition. In doing so you create wealth. The world is — and you specifically are — one pristine old car the richer. And not just in some metaphorical way. If you sell your car, you'll get more for it. In restoring your old car you have made yourself richer. You haven't made anyone else poorer."

Graham uses this parable to explain where the wealth comes from when people work for themselves in a start-up company: it comes from people spending their time working on something that's useful to themselves and to others. This is scarcely a new thought, and although it would be more traditional to say that "value" comes from "socially necessary labour time", Graham's explanation of wealth is essentially the same as the traditional one.

However, there is a subtlety here which Graham glosses over, a subtlety that the old-time political economists were well aware of: it's the issue of how things change when, rather than working for yourself, you work for someone else. Following the culinary theme of this blog, let's invent our own parable, with people working for someone else in the kitchen of a restaurant.

In our make-believe restaurant there is a proprietor who pays the rent and provides the furniture and most of the kitchen equipment. Every day the proprietor buys the ingredients for the meals and pays the wages of the staff. In return, the staff work a certain number of hours each day, during which they transform those raw ingredients into the meals which customers buy. Value has certainly been created, and in return for this newly created value, the customers leave money after they finish their meals, money which is collected by the proprietor.

The key point here is to notice where the new value comes from: it comes from the staff who spend their time preparing and cooking the food. Not from the proprietor, who merely acts as a facilitator or catalyst by providing the money that makes their work possible. In Graham's car-refurbishment example, imagine that over the summer you keep the stripped-down car in your uncle's garage as you do the work. Does the fact that he is providing the garage mean that he is directly creating value? No. He is facilitating the creation of value, but unless he works on the car with you, that's all he is doing. Without him, you would not be able to create value, but without you, he would just have an empty garage and there would be no newly created value, no new wealth.

Now we come to the central tension between the proprietor and the restaurant staff: how does the newly created value get shared out? Assuming the restaurant makes a profit, then after necessary running expenses are paid, how much of the remaining money goes to the staff as wages and how much to the proprietor? They both have a moral right to that money, since they both have to play their part in order for the new value to be created. There is an unavoidable tension here: they are both in the right.

From the point of view of the proprietor, wanting to keep more of the money for themselves, there is no reason to pay the staff more than the current going-rate. Look around: what are other restaurants paying? You really don't need to pay any more than that. If you do, you are certain to find people knocking on your door asking to work for you, regardless of what fraction of the takings you keep for yourself and what fraction you distribute to your staff as wages. And of course, if you pay less than the going rate, your staff will be tempted away by better offers elsewhere. You won't actually have a restaurant if all your staff leave.

So, the necessary level of wages is determined by the world outside the restaurant, and provided the proprietor pays those wages, they can take whatever money is left for themselves. We would naturally expect the proprietor to try and take more rather than less, and there are essentially two main methods to achieve this: the proprietor can either try to increase the productivity of each member of staff — but still pay the same wages — or they can try to "de-skill" the work, then hire less-skilled employees who can be paid less to produce the same value.

The classic way to achieve the first of these aims is to increase productivity by the "division of labour", described by Adam Smith in the eighteenth century and celebrated today on the back of the British 20 pound note. The idea is that if particular people specialise in particular tasks, then they will perform these tasks quicker than if they continually switch from one thing to another. This is the essence of the twentieth-century factory, with jobs reduced to fitting a particular part or tightening a particular bolt all day. And in the kitchen we do see division of labour to some extent, with specialists in sauces and deserts, for example.

More interestingly, and less often noticed, the division of labour can also achieve the second of the proprietor's aims: it can change the nature of the work, de-skilling it and allowing the proprietor to pay reduced wages, at least to some of the staff. This was well understood by computing hero Charles Babbage, who in the 1830s took time off from building the difference engine to write a treatise on political economy, based on the observations he made while touring the factories and workshops of England. In On the Economy of Machinery and Manufactures (1832), Babbage explains:

"... the master manufacturer, by dividing the work to be executed into different processes, each requiring different degrees of skill or force, can purchase exactly that precise quantity of both which is necessary for each process; whereas if the work were executed by one workman, that person must possess sufficient skill to perform the most difficult, and sufficient strength to execute the most laborious, of the operations into which the art is divided." (pp175-176)

Babbage took Smith's example of a pin factory and went one better in his analysis: he detailed the precise division of work and the wages paid to each of the workers. Some processes in pin manufacture required both skill and strength; some were so simple that a child could do them — and since this was Victorian England, a child did do them. So the highest paid worker on the staff was a man, at 5s. 4½d. per day; the lowest paid a boy, at 4½d. per day. Which meant, as Harry Braverman points out in Labor and Monopoly Capital (1974):

"If the minimum pay for a craftsman capable of performing all operations is no more than the highest pay in the above listing, and if such craftsmen are employed exclusively, then the labour costs of manufacture would be more than doubled, even if the very same division of labor were employed and even if the craftsmen produced the pins at the very same speed as the detail workers." (p80) [italics in original]

This explains why it makes sense for the proprietor of the restaurant to employ a less skilled person just to peel potatoes and wash the dishes. The proprietor could hire a skilled cook to do this work, but it will be cheaper to hire someone less skilled, preferably the least capable person who can still do those jobs. A skilled cook would be more expensive because their wage is determined by what they could earn elsewhere, outside this restaurant, not by what they do inside. And exactly the same is true of any job done for a wage — a point which (finally) brings us back to software development.

The proprietor of a software development organisation would obviously like to benefit in the same way as the proprietor of the restaurant. They would like to use division of labour, so that they can reduce their wage-bill by hiring a staff of cheaper, less-skilled people who can each only just do their own particular jobs. Wherever people make things in return for wages there will be this pressure. Software is no different.

But there's a catch. In software, is this really possible? Division of labour makes sense in a mass-production environment, but is software development such an environment? The proprietors of the software world clearly wish it were so, but that wish can only come true to the extent that software development is predictable and routine — and for the most part it isn't. Just like other forms of engineering, software development is around 70% repair and maintenance. Repair is never routine, and in software even the initial production is often unpredictable, because it is actually mostly design work, not manufacture in the normal sense. (This is perhaps the main way in which coding differs from cooking: coding is always about process creation, cooking is usually about process execution.)

However, this practical difficulty doesn't prevent software proprietors from yearning for predictability. It might even explain their unreasonable enthusiasm for the Waterfall process over the years. Any process that promises predictability is also implicitly promising a less skilled staff and a lower wage-bill. In contrast, an Agile process is implicitly demanding a more skilled staff and hence a higher wage-bill — because when you don't believe in prediction you need a flexible, broadly competent staff who can step-up and do things that nobody expected. (Of course, to the extent that a particular project really is unpredictable, the Agile enthusiasts will be correct when they claim that in the end their way is actually better, because the less skilled staff would actually take longer and cost more, if they finished at all.)

What started as a tension between the proprietor and the staff over how to share out the newly created value has turned into a tension over skills and competence. Developers might naturally be expected to take a pride in their craft skills, building them up over the years like any artisan. Perhaps the best expression of this desire can be found in open-source software, where there is never any attempt to make the work of creating software routine, predictable or de-skilled. But the difference here is that open-source software is written by hackers for themselves, not for a wage.

Agile processes promise the developers an opportunity to practice and grow their craft skills. Agile processes promise the proprietor nothing but an extra expense, accepted only because predicting the future is too hard. But some Agile processes implicitly make another promise to the proprietor, a promise with a dark side. And I think this dark side is perhaps why some developers are right to be suspicious of the rhetoric of Agile development.

Remember the two ways the proprietor can make more money? I've been concentrating on de-skilling, but there's also the other way: the proprietor can keep more money for themselves if they can increase the productivity of each member of staff, but still pay the same wages. With a given division of labour and a given level of skill, the proprietor can keep more money for themselves if they can get their staff to work for longer each day or to work with greater intensity. And when we deconstruct the terminology used by the most popular Agile process, Scrum, we find an unfortunate allusion to work at a greater intensity.

For example, the week-by-week cycle of development is called an "iteration" in practically every other software process, including Extreme Programming. In Scrum this time period is called a Sprint. What associations does that conjure up? Well when you sprint you make an extreme effort to go as fast as possible, an effort that you can only keep up for a short time. That's what "sprint" means in everyday life.

The list of unimplemented features is just called the "user stories" in Extreme Programming. In Scrum this list is called the Backlog. What does "backlog" mean in everyday life? It's the stuff that you should have already finished. The metaphor is clear: Rush! Rush! Finish it off quickly! You're late already.

Notice that I'm not saying that this is what these names should mean, or that this is what Scrum practitioners want them to mean, it's just that words come with everyday meanings, and people will tend to understand them this way. Imagine if you went to a restaurant and your waiter told you that the proprietor had introduced a new kind of organisation in the kitchen, called Scrum. The kitchen staff now have a "backlog" of dishes to prepare and they are "sprinting" to get them done. As a customer, I think you would take the metaphor at face value. Now imagine that you are a customer or a manager of a software development team using Scrum. Is there any reason for you not to take the metaphor at face value?

So this is the dark side of Agile: the danger that people outside the development team will hear these metaphors and misunderstand them — as a promise of work at a greater intensity and nothing more. Thinking that, apart from that, Agile is business as usual. Still yearning to make more money out of predictability, lower skills and lower wages. Not understanding that the point of Agile is to be Antifragile.

It's easy to imagine, if you were a developer in such an environment, how you would feel, how you would resist. I think when Agile fails, it must often fail in this way.