(This post is a follow-up to
Agile
is Antifragile.)
What is the best way to organise a group of people who are working
together to make some software? This must be the central question of
software engineering, but we tend to think about it in a very narrow
way. It's rather odd that we are happy to talk about processes,
from Waterfall to Agile, but we don't like to talk about the kind of
practical problems that always arise when people come together to make
something for money.
These practical problems are standard fare in business schools and in
some mainstream engineering courses, where they are discussed under
the label Organisation Theory. This in turn has its roots in ideas of
"political economy" that go back over 200 years. Once you understand
these ideas, you can see clearly the cause of some tensions
that arise in software development. So, let's see if I can introduce
you to these ideas — I don't expect in this post to get to the
twentieth-century fashions started by Taylor and Mayo, but we should
be able to cover enough of the nineteenth century ideas to understand
the different feelings people have about Agile and to recognise its
dark side.
Where to start? Lisp-hacker Paul Graham, in his essay
How to Make Wealth,
uses the following example to explain how wealth or "value" can be
created by people out of nothing:
"Suppose you own a beat-up old car. Instead of
sitting on your butt next summer, you could spend the time restoring
your car to pristine condition. In doing so you create wealth. The
world is — and you specifically are — one pristine old car the
richer. And not just in some metaphorical way. If you sell your car,
you'll get more for it. In restoring your old car you have made
yourself richer. You haven't made anyone else poorer."
Graham uses this parable to explain where the wealth comes from when
people work for themselves in a start-up company: it comes from people
spending their time working on something that's useful to
themselves and to others. This is scarcely a new thought, and although it
would be more traditional to say that "value" comes from
"socially necessary labour time", Graham's explanation of wealth
is essentially the same as the traditional one.
However, there is a subtlety here which Graham glosses over, a
subtlety that the old-time political economists were well aware of:
it's the issue of how things change when, rather than working for
yourself, you work for someone else. Following the culinary theme of
this blog, let's invent our own parable, with people working for
someone else in the kitchen of a restaurant.
In our make-believe restaurant there is a proprietor who pays the
rent and provides the furniture and most of the kitchen
equipment. Every day the proprietor buys the ingredients for the meals and
pays the wages of the staff. In return, the staff work a certain number
of hours each day, during which they transform those raw ingredients
into the meals which customers buy. Value has certainly been created,
and in return for this newly created value, the customers leave money
after they finish their meals, money which is collected by the
proprietor.
The key point here is to notice where the new value comes from: it
comes from the staff who spend their time preparing and cooking the
food. Not from the proprietor, who merely acts as a facilitator or
catalyst by providing the money that makes their work possible. In
Graham's car-refurbishment example, imagine that over the summer you
keep the stripped-down car in your uncle's garage as you do the
work. Does the fact that he is providing the garage mean that he is
directly creating value? No. He is
facilitating the creation of
value, but unless he works on the car with you, that's all he is
doing. Without him,
you would not be able to create value, but
without you,
he would just have an empty garage and there would
be no newly created value, no new wealth.
Now we come to the central tension between the proprietor and the
restaurant staff: how does the newly created value get shared out?
Assuming the restaurant makes a profit, then after necessary running
expenses are paid, how much of the remaining money goes to the staff
as wages and how much to the proprietor? They both have a
moral right to that money, since they both have to play their part in
order for the new value to be created. There is an
unavoidable
tension here: they are both in the right.
From the point of view of the proprietor, wanting to keep more of the
money for themselves,
there is no reason to pay the staff more than the current going-rate.
Look around: what are other restaurants paying? You really don't need
to pay any more than that. If you do, you are certain to find people
knocking on your door asking to work for you, regardless of what fraction
of the takings you keep for yourself and what fraction you distribute
to your staff as wages. And of course, if you pay less than the going
rate, your staff will be tempted away by better offers elsewhere. You
won't actually have a restaurant if all your staff leave.
So, the necessary level of wages is determined by the world outside the
restaurant, and provided the proprietor pays those wages, they can
take whatever money is left for themselves. We would naturally expect the
proprietor to try and take more rather than less, and there are
essentially two main methods to achieve this: the proprietor can
either try to increase the productivity of each member of staff
— but still pay the same wages — or they can try to
"de-skill" the work, then hire less-skilled employees who can be paid
less to produce the same value.
The classic way to achieve the first of these aims is to
increase productivity by the "division of labour", described by Adam
Smith in the eighteenth century and celebrated today on the back of
the British 20 pound note. The idea is that if particular people
specialise in particular tasks, then they will perform these tasks
quicker than if they continually switch from one thing to
another. This is the essence of the twentieth-century factory, with
jobs reduced to fitting a particular part or tightening a particular
bolt all day. And in the kitchen we do see division of labour to some
extent, with specialists in sauces and deserts, for example.
More interestingly, and less often noticed, the division of labour can
also achieve the second of the proprietor's aims: it can change the
nature of the work, de-skilling it and allowing the proprietor to pay
reduced wages, at least to some of the staff. This was well understood
by computing hero Charles
Babbage, who in the 1830s took time off from building the difference
engine to write a treatise on political economy, based on the
observations he made while touring the factories and workshops of
England. In
On
the Economy of Machinery and Manufactures (1832), Babbage explains:
"... the master manufacturer, by dividing the work to be executed
into different processes, each requiring different degrees of skill or
force, can purchase exactly that precise quantity of both which is
necessary for each process; whereas if the work were executed by one
workman, that person must possess sufficient skill to perform the most
difficult, and sufficient strength to execute the most laborious, of
the operations into which the art is divided." (pp175-176)
Babbage took Smith's example of a pin factory and went one better in
his analysis: he detailed the precise division of work and the wages
paid to each of the workers. Some processes in pin manufacture
required both skill and strength; some were so simple that a child could do
them — and since this was Victorian England, a
child
did do them. So the highest paid worker on the staff was
a man, at 5s. 4½d. per day; the lowest paid a boy, at
4½d. per day. Which meant, as Harry Braverman points out in
Labor and
Monopoly Capital (1974):
"If the minimum pay for a craftsman capable of performing all
operations is no more than the highest pay in the above listing, and
if such craftsmen are employed exclusively, then the labour costs of
manufacture would be more than doubled, even if the very same
division of labor were employed and even if the craftsmen produced
the pins at the very same speed as the detail workers." (p80)
[italics in original]
This explains why it makes sense for the proprietor of the restaurant
to employ a less skilled person just to peel potatoes and wash the
dishes. The proprietor
could hire a skilled cook to do this
work, but it will be cheaper to hire someone less skilled, preferably
the least capable person who can still do those jobs. A skilled cook
would be more expensive because their wage is determined by what they
could earn elsewhere, outside this restaurant, not by what they do
inside. And exactly the same is true of any job done for a wage
— a point which (finally) brings us back to software
development.
The proprietor of a software development organisation would obviously
like to benefit in the same way as the proprietor of the
restaurant. They would like to use division of labour, so that they
can reduce their wage-bill by hiring a staff of cheaper, less-skilled
people who can each only just do their own particular jobs. Wherever
people make things in return for wages there will be this
pressure. Software is no different.
But there's a catch. In software, is this really possible? Division of
labour makes sense in a mass-production environment, but is software
development such an environment? The proprietors of the software world
clearly
wish it were so, but that wish can only come true to
the extent that software development is predictable and routine
— and for the most part it isn't. Just like other forms of
engineering, software development is around 70% repair and
maintenance. Repair is never routine, and in software even the initial
production is often unpredictable, because it is actually mostly design
work, not manufacture in the normal sense. (This is perhaps the main
way in which coding differs from cooking: coding is always about
process creation, cooking is usually about process execution.)
However, this practical difficulty doesn't prevent software proprietors
from yearning for predictability. It might even explain their
unreasonable enthusiasm for the Waterfall process over the years. Any
process that promises predictability is also implicitly promising a
less skilled staff and a lower wage-bill. In contrast, an Agile
process is implicitly demanding a more skilled staff and hence a
higher wage-bill — because when you don't believe in prediction
you need a flexible, broadly competent staff who can step-up and do
things that nobody expected. (Of course, to the extent that a
particular project really is unpredictable, the Agile enthusiasts will
be correct when they claim that in the end their way is actually
better, because the less skilled staff would actually take longer and
cost more, if they finished at all.)
What started as a tension between the proprietor and the staff over
how to share out the newly created value has turned into a tension
over skills and competence. Developers might naturally be expected to
take a pride in their craft skills, building them up over the years
like any artisan. Perhaps the best expression
of this desire can be found in open-source software, where there is
never any attempt to make the work of creating software routine,
predictable or de-skilled. But the difference here is that open-source
software is written by hackers for
themselves, not for a wage.
Agile processes promise the developers an opportunity to practice
and grow their craft skills. Agile processes promise the proprietor
nothing but an extra expense, accepted only because predicting the
future is too hard. But some Agile processes implicitly make another promise
to the proprietor, a promise with a dark side. And I think this dark
side is perhaps why some developers are right to be suspicious of the
rhetoric of Agile development.
Remember the two ways the proprietor can make more money? I've
been concentrating on de-skilling, but there's also the other way:
the proprietor can keep more money for themselves if they can
increase the
productivity of each member of staff, but still pay the same
wages. With a given division of labour and a given level of skill, the
proprietor can keep more money for themselves if they can get their
staff to
work for longer each day or to work with greater intensity. And when we
deconstruct the terminology used by the most popular Agile process,
Scrum, we find an unfortunate allusion to work at a greater
intensity.
For example, the week-by-week cycle of development is called an
"iteration" in practically every other software process, including
Extreme Programming. In
Scrum this time period is called a Sprint. What associations does that
conjure up? Well when you sprint you make an extreme effort to go as
fast as possible, an effort that you can only keep up for a short
time. That's what "sprint" means in everyday life.
The list of unimplemented features is just called the "user
stories" in Extreme Programming. In Scrum this list is called the
Backlog. What does "backlog" mean in everyday life? It's the stuff
that you should have already finished. The metaphor is clear: Rush!
Rush! Finish it off quickly! You're late already.
Notice that I'm not saying that this is what these names
should
mean, or that this is what Scrum practitioners
want them to mean,
it's just that words come with everyday meanings, and people will
tend to understand them this way. Imagine if you went to a restaurant
and your waiter told you that the proprietor had introduced a new kind
of organisation in the kitchen, called Scrum. The kitchen staff now have a
"backlog" of dishes to prepare and they are "sprinting" to get them
done. As a customer, I think you would take the metaphor at face
value. Now imagine that you are a customer or a manager of a software
development team using Scrum. Is there any reason for you not to take
the metaphor at face value?
So this is the dark side of Agile: the danger that people outside the
development team will hear these metaphors and misunderstand
them — as a
promise of work
at a greater intensity and nothing more. Thinking that, apart from
that, Agile is business as usual. Still yearning to make more money
out of predictability, lower skills and lower wages. Not understanding
that the point of
Agile is
to
be Antifragile.
It's easy to imagine, if you were a developer in such an environment,
how you would feel, how you would resist.
I think when Agile fails, it must often fail in this way.
