Decision Culture

Chapter 11 asked what an individual decision is. Most decisions that are made today with the help of artificial intelligence, however, are not made alone but in teams, departments, administrations and companies. This chapter therefore asks how groups decide, and what happens when a machine enters an existing decision culture.

A widespread expectation is that good technology makes good decisions. Kentaro Toyama, a computer scientist, spent years leading projects in which technology was deployed in schools, hospitals and administrations in poor regions. He summed up his experience in a principle that he calls the Law of Amplification: the most important effect of technology is to amplify human forces. Like a lever, it amplifies people's capabilities in the direction of their intentions. Where people and institutions are well-intentioned and capable, it improves a great deal; where they are indifferent, divided or corrupt, it achieves little or does harm.

If Toyama is right, then the bottleneck in the use of artificial intelligence does not lie primarily in the technology. It lies in the way people decide together. A system that enters a culture of clarity makes it clearer. A system that enters a culture of silence makes the silence more efficient.

History supports this view. When the electric motor replaced the steam engine in factories, productivity barely rose for decades. The economic historian Paul David showed in 1990 why. The factories were built around a single large drive shaft, and as long as only the drive was replaced, little changed. Only when the factories were reorganised around many small motors did the gain come. The economist Robert Solow mocked in 1987 that one could see the computer age everywhere, only not in the productivity statistics. With artificial intelligence the same pattern is appearing. A study by the Massachusetts Institute of Technology found in 2025 that around 95 per cent of companies using generative artificial intelligence achieved no measurable gain; only about one in twenty brought such a system into regular operation. The researchers attribute this above all to the fact that the systems were not embedded in daily work and did not learn from it. The figure is disputed because it measures only the effect on profit, and the research group itself develops technology for such systems. But the direction is clear: where an organisation is not prepared to change its processes, even the best technology remains an experiment.

Why do groups of clever people sometimes decide astonishingly badly? Irving Janis, a social psychologist, examined this question in 1972 using decisions in American foreign policy, such as the failed invasion of the Bay of Pigs in 1961. He called the pattern groupthink: a way of thinking in which the striving for unanimity in a close-knit group becomes so strong that it displaces the sober examination of the alternatives.

Janis described eight signs. Among them are the illusion of being invulnerable, the unquestioned belief in one's own morality, the self-censorship with which individuals hold back their doubts, and the illusion of unanimity, in which silence counts as agreement. In addition there are members who make themselves guardians and shield the group from inconvenient information.

As remedies, Janis proposed explicitly assigning someone the task of dissent, dividing the group temporarily, seeking criticism from outside and, before implementation, scheduling a second meeting in which remaining doubts may be voiced.

Research has since tested Janis's model and partly refuted it. The conditions he named as causes could not be confirmed as clearly as he had assumed. The phenomena themselves, self-censorship and illusory agreement, occur, however, in far more groups than he thought. For this book this more cautious reading suffices: agreement is not the same as insight.

More recent research reaches similar conclusions by another route. Garold Stasser and William Titus showed in experiments in 1985 that groups talk above all about what everyone knows anyway. What only individuals know hardly gets mentioned, even when it would make the better decision possible. Cass Sunstein and Reid Hastie summarised in 2015 that groups often do not compensate for the errors of their members but amplify them. Like-minded people who deliberate together end up more extreme than they began; in an experiment in the American state of Colorado, liberal groups became more liberal after the discussion and conservative groups more conservative.

The examples from practice are numerous. On the evening before the launch of the space shuttle Challenger in January 1986, engineers from the manufacturer of the booster rockets warned that the sealing rings could fail in the cold. Their superiors conferred, one of them asked a colleague to take off his engineer's hat and put on his manager's hat, and the company approved the launch. The shuttle broke apart shortly after launch; seven people died. With the Boeing 737 MAX aircraft type, two aircraft crashed in 2018 and 2019; 346 people died. An automatic control system that relied on a single sensor had pushed the aircraft downwards, and the pilots had barely been informed about it. A report of the American Congress spoke in 2020 of a culture in which concerns did not get through. And in Switzerland the investigation report on Credit Suisse's loss with the hedge fund Archegos showed that warnings from risk specialists were not passed on and that no one really felt responsible. In all three cases there was no lack of knowledge. The knowledge was there, but it did not arrive where the decision was made.

What protects a group from these patterns was shown by Amy Edmondson, an organisational researcher, in 1999 in a study of 51 work teams in an industrial company. She called it psychological safety: the shared belief within a team that one can take interpersonal risks without being punished or exposed. Whoever asks a question, admits a mistake or contradicts a superior takes such a risk.

Edmondson found that teams with high psychological safety learned more from mistakes, and that this learning explained their performance. A team's confidence in its own capability, by contrast, no longer played any role of its own once psychological safety was taken into account. Decisive for its emergence was, among other things, leadership that accompanies and empowers rather than merely giving orders.

How powerful this finding is had already been shown by an earlier study by Edmondson in hospitals. To her surprise, the best nursing teams reported more medication errors than the weaker ones. They did not make more errors; they were more willing to speak about them. From 2012 onwards, Google investigated in a project called Aristotle what distinguishes its most successful teams, and found in first place not the talent of individuals but psychological safety. Aviation learned this lesson bitterly. In 1977 two jumbo jets collided on the island of Tenerife; 583 people died. The flight engineer of the aircraft that was taking off had doubts whether the runway was clear and asked, but the experienced captain brushed the objection aside. Afterwards the airlines introduced a training that teaches crews to voice concerns and to listen to them, whatever a person's rank. In hospitals a checklist of the World Health Organization has a similar effect: before every operation the team introduce themselves by name and role and go through the most important points together. In a large study, mortality after operations subsequently fell by almost half.

The finding fits an observation that runs through this whole book. In Chapter 9 a single person at a platform noticed an irregularity in the logs and stopped agents that were more capable than almost all security researchers. Whether someone reports such an observation does not depend on their intelligence. It depends on whether they can expect to be listened to.

If artificial intelligence amplifies a decision culture, what exactly does it amplify? Four effects can be derived from the previous chapters.

First, it can become a new guardian. Chapter 11 showed that people tend to follow automated recommendations, even as experts and even in teams. In a group that already presses towards unanimity, the sentence “The system recommends it” easily becomes the last word. Whoever still objects then contradicts not only their colleagues but apparently the data as well.

Second, it accelerates. Chapters 10 and 11 showed how the time for checking shrinks when systems deliver extensive material in a short time. The second meeting that Janis recommended for voicing doubts is precisely the time that is cut first under pressure.

Third, it condenses. A summary that a system produces from many contributions reproduces the majority, not the minority. What a single person noted on the margins easily drops out in the condensing, although it may have been the most important objection.

Fourth, however, it can also help. A system can make visible dissent that no one would voice out loud, for instance when contributions come in without names. It can explicitly formulate the opposing view, as Janis demanded of the advocate of dissent. And it can record the assumptions on which a decision rested, so that one can later check whether they were correct. Whether the machine amplifies silence or dissent is decided not by the machine but by the culture it enters.

Organisations learn, but they also forget. Pablo Martin de Holan and Nelson Phillips, organisational researchers, examined in 2004 how and why organisations forget. They distinguish whether the forgetting happens unintentionally or intentionally and whether it concerns old or newly acquired knowledge. This yields four forms: proven knowledge decays, for instance when experienced people leave. What has newly been learned is never recorded in the first place. Outdated habits are deliberately unlearned. And bad new habits are deliberately never adopted in the first place. The first two forms are a loss, the last two a capability.

How costly forgetting can become is shown by two examples from the United States. For the maintenance of nuclear warheads, a material code-named Fogbank was needed in the 2000s that had last been produced in the 1980s. The records were still there, but the people who had known what mattered had gone. It cost around 69 million dollars and at least a year's delay before the material could be produced again; only late did it become clear that an unintended impurity in the old process had been decisive. And when the space shuttle Columbia broke apart on re-entry in 2003, the investigating commission found that patterns had repeated themselves at the space agency that had already led to the Challenger disaster seventeen years earlier. The lessons had been written down, but not retained.

A third example concerns automation itself. In 2009 an Air France aircraft on its way from Rio de Janeiro to Paris crashed into the Atlantic; 228 people died. When the autopilot disengaged because of iced-up sensors, the pilots did not find their way back to flying by hand quickly enough; the investigation report criticised, among other things, a lack of training for precisely this situation. What the machine always takes over, human beings unlearn, and with them the organisation.

There are organisations that resist forgetting, and their means are similar. In the 1970s the US Army introduced, after every operation, a debriefing with four simple questions: What was supposed to happen? What happened? Why? What do we learn from it? Since 1976 aviation has collected reports of near misses in a confidential reporting system operated by NASA, by now well over a million, and whoever reports is, as a matter of principle, not punished. Swiss hospitals know the same thing under the name CIRS: professionals report critical incidents before any harm occurs. And in software development it has become customary to record important decisions, together with their context, rejected options and expected consequences, in short records.

Here something matters that is easily overlooked: the reasons must be recorded before the outcome is known. The psychologist Baruch Fischhoff showed in 1975 that people believe in hindsight that they had foreseen the outcome. Whoever writes things down only afterwards rewrites history. And whoever records only what was decided loses what was deliberately not done, and with it often the most important reasons. Recording good and bad decisions alike, including those not taken, together with their reasons, is therefore not bureaucracy. It is the precondition for an organisation being able to learn.

For decision culture this means: an organisation can decide well and nevertheless learn nothing from it if it forgets why it decided as it did and what came of it. Then at the next crisis it starts again from the beginning. Artificial intelligence can have a twofold effect here. It can record what people would forget: reasons, assumptions, outcomes. And it can make the recording so extensive that no one can find what counts any more.

Many organisations today try not to leave their decision culture to chance. Certain principles appear again and again.

Clear decision rights: before an important decision it is settled who decides, who prepares it, who must be heard beforehand, who is informed afterwards and what happens if no agreement is reached. Whoever is to bear responsibility needs the necessary information and authority to do so.

A decision record: what is recorded is not only the outcome but also the question, the known facts, the open assumptions, the options examined, the reasons for the chosen direction and the point in time at which it is reviewed. In this way an organisation can later learn whether it decided well, and not only whether it was lucky. A good decision is not automatically one with a good outcome, but one that was taken carefully and responsibly with the information available at the time.

Invited dissent: dissent is not merely tolerated but explicitly sought, through counter-questions, through roles that represent the objection, and through separate or confidential conversations where dependencies make speaking openly difficult.

And finally the willingness to stop: a well-founded termination can be a success if it prevents further misguided investment. Groupthink often shows itself precisely in the fact that no one any longer says what everyone suspects.

Two objections can be raised against this chapter, and they contradict each other.

The first says that Toyama underestimates technology. Technology, it argues, does not only amplify what is already there; it changes human beings and their culture themselves. Melvin Kranzberg, a historian of technology, coined a sentence for this in 1986 as the first of his laws: “Technology is neither good nor bad; nor is it neutral.” Whoever works with a system that asks certain questions and not others learns to think differently. Chapters 3 and 4 showed that what we look at changes us. The Air France example from section 12.5 shows it: automation did not only amplify what the pilots could do, it changed what they were still able to do.

The second objection says the opposite: culture cannot be built. Rules for decision rights, records and roles for dissent, it holds, would remain ineffective if the stance is lacking; they would then become rituals that are ticked off. This is the critique of rules without inner disposition that Chapter 13 develops, applied to organisations. There is evidence for this too. When the Canadian province of Ontario made the checklist of the World Health Organization mandatory for all hospitals in 2010, mortality after operations did not fall measurably. The same list that had achieved so much in the first large study apparently became a formality where it merely had to be followed.

Both objections hit on something. Technology amplifies and changes at the same time. And structures do not create culture, but they can give it room or take that room away. Aviation shows how the two go together. Its training for crews works not because it prescribes rules but because it practises a stance that is modelled by example and discussed without punishment after every incident (sections 12.3 and 12.5).