Agile is Antifragile

Earlier this year I read the excellent book Antifragile by Nassim Nicholas Taleb. He's a bit full-of-himself and has a somewhat quirky writing style, but the ideas in the book are so good that this is easy to forgive. (And in fact, when I had finished the book, I turned around and read it through again, which I never do, so that must say something about the content.)

I've been thinking about the connection between Taleb's ideas and software development for some time — something like five years in the case of my recent essay Bugs with Long Tails — but his previous books don't make it quite so clear how to put into practice the maxim that one should "invest in preparedness, not prediction". In the face of uncertainty, what exactly does it mean to be well prepared?

Taleb has invented the new term "antifragile" as a label for those things which are not merely robust to uncertainty, but actually relish it and benefit from the uncertainty. For example, a tourist who follows a schedule is fragile, because if the bus doesn't arrive it's a disaster. By contrast a "flâneur" who makes no decisions ahead of time doesn't care. If the bus doesn't arrive, it's an opportunity to see things and do things that would never have been on any schedule. Better than robust, they are antifragile: when the unexpected happens, it's an advantage.

Antifragile cherishes "optionality": the ability to do something if it benefits you without the obligation to do it when it doesn't benefit you. You don't need to predict what will happen in the future, provided that you have placed your bets so that you incur a small fixed loss but in return you have the chance to make a large gain. (And because you have an option, not an obligation, if it goes wrong you get to walk away. You never suffer a large loss.) When losses are certain but small and gains uncertain but large, a trial-and-error approach will in the long term bring more than survival: it will bring success.

The problem with this approach is that it takes a kind of moral courage. People hunger for apparent predictability, for a reassuring chain of small gains even at the risk of a huge loss that wipes them out. Day-to-day it's harder to endure the drip-drip of the small losses incurred by the antifragile approach, even when you consciously recognise the long-term benefits. (See Thinking, Fast and Slow by Daniel Kahneman for an explanation of this curious effect.)

In business, the people responsible for each part of a firm will tend to "play it safe": they will try to ensure that they usually do make a small gain, even if they run the risk of a large loss. But from the point of view of the whole business this is a poor choice: it would be much better if the people in charge of each part accepted small losses, provided that in return they had a fair chance of a large gain and no chance of a large loss. They usually don't make this choice, because they don't want to be blamed for the small failure if their bet doesn't pay off. Since they avoid blame, they get to keep their jobs for now, but maybe not for long, because the business as a whole becomes fragile.

A business which is fragile has no choice but to try to predict the future. But in many cases, predicting the future or even just estimating the probability of future events is much harder than most people think. And as I have argued in Bugs with Long Tails, there is evidence that software development is one of those cases where prediction is too hard. We can't do it. And if we can't do it, we shouldn't try. We should "invest in preparedness, not prediction". We should be antifragile. What does that mean in software? It looks a lot like what people usually call "agile".

I'll give just a few examples here, because I'm sure you can fill in more for yourself:

• Don't try to guess what the customer/product-owner wants. Instead of trying to predict what you can't predict, insist that they are always available to say what they want. (Or admit that they don't yet know, in which case you can help them find out.)


• Big projects are known to have worse cost over-runs than small projects and this is because they rely so much on prediction. So agile development splits big projects into short iterations/sprints a few weeks in length, each of which delivers some finished product, worthwhile on its own. A few of these iterations may fail, but that's fine. Learn your lesson and move on: small, isolated losses; large overall gains. To paraphrase Kent Beck: "The purpose of planning is not to predict the future, but to know that you are working on the most important thing right now."


• In Extreme Programming there are the principles "You Aren't Going To Need It" and "Do The Simplest Thing That Could Possibly Work". The least productive development work is building code that is never needed. But we don't have to predict what code we need right now, because we already know. For code that we don't need right now, we have to predict what people in the future will need our code to do, how efficient it must be in time and space, and lots of other details that we currently know nothing about. Very often we are wrong in these predictions. Better to make a bet that we will never need that code and accept the small loss of changing the code later if it proves inadequate. The big payoff if we win the bet is less code and less overall work.

Another relevant example of antifragilility can be found in the architectural work of Christopher Alexander. (Though it's curious that Taleb doesn't mention Alexander, especially since he does mention How Buildings Learn by Stewart Brand.) It's well known that in the 1990s, the same people in the object-oriented programming community who came up with the idea of agile development also worked on "patterns", inspired by Alexander's books The Timeless Way of Building and A Pattern Language: Towns, Buildings, Construction.

Less well-known in the software development world is that Alexander became dissatisfied with patterns because they did not reliably generate the kind of "living structure" of buildings that he describes in The Timeless Way of Building: buildings that have the "Quality Without A Name". This was a problem, but a problem with a happy ending, because he describes the solution in his massive follow-on work The Nature of Order. (Over 2000 pages in four volumes: Book 1, Book 2, Book 3, Book 4.)

Alexander has even more to say in this magnum-opus than Taleb in Antifragile, but the key point for us here is the idea of a "generative sequence". The patterns are useful, but to do their work properly they must be used in a disciplined way, in a specific generative sequence where nothing has to be predicted ahead of time, but decisions only made based on what is visible right now, in the situation at hand. For example, Alexander gives a 24-step generative sequence for building a Japanese tea house (vol. 2, p303-304) starting with a secluded garden. Within this sequence, step 12 is to lay out a particular path, step 13 to choose a stone water basin and place it on this path, and so on. The trick is that having made a decision at one step it is never necessary to go back and revisit an earlier decision, nor is it necessary to consider a later decision ahead of time. There is no planning.

It's surprising that this approach works at all. Why should it be possible to find generative sequences which allow things to be designed step by step and never go back to revisit an earlier decision? And yet it does seem to be possible to find these sequences, and they do seem to be able construct things which have the "Quality Without A Name". Expert knowledge therefore encompasses not just the patterns, but the generative sequences that make them work and the principles used to discover generative sequences. (And most of The Nature of Order is about these principles.) As Alexander notes:

"The significance of generated structure lies in the concept of mistakes. Fabricated plans always have mistakes — not just a few mistakes, but tens of thousands, even millions of mistakes. It is the mistake-ridden character of the plans which marks them as fabricated — and that comes from the way they are actually generated, or made, in time. Generated plans have few mistakes." (vol. 2, p186)

Generated structures don't avoid mistakes — their construction involves many mistakes. But these are cheap mistakes, easy to notice and quick to correct. Always corrected before going on to the next step. Planned structures probably involve the same number of mistakes, but a plan doesn't give you any feedback, so every mistake persists and is built into real-life.

We borrowed the idea of patterns from Alexander. We could frame agile development as another borrowing, an application of Alexander's idea of generated structure in the world of software development. We still seem quite a long way from having reliable generative sequences in software, apart from the tacit craft-knowledge embodied in individuals and small groups, but maybe in the future we can do better. Maybe we could see the agile "movement" as a search for such generative sequences. However, I think an even better framing would be to see both agile development and generative sequences as instances of antifragility. This perspective emphasises what gives them both their power: avoiding plans and prediction when we can't know enough to predict correctly.

(I've noticed that I'm not the only one to have seen a link between Taleb's ideas and agile development: see Agile=Antifragile. If you'd like to see Taleb talking about his ideas, you might like to watch this video filmed at the New York Public Library. You can also find much better account of how Alexander's ideas relate to software in Part One of Patterns of Software by Richard Gabriel.)