Superintelligence and Autonomy

In September 2026, on the podcast “The Diary Of A CEO”, a panel of four experts discussed whether the world is prepared for what artificial intelligence will be able to do in the near future. The host was Steven Bartlett, a podcaster; taking part in the conversation were Roman Yampolskiy, an AI safety researcher, Andrew McAfee, a scientist at MIT, Nate Soares, an AI safety researcher, and Ed Zitron, a critic of technology. The recording on which this chapter is based is an excerpt from this episode.

At the centre of the conversation is an incident that McAfee describes as follows: in a supposedly secured test environment, OpenAI had let thousands of agents exploit security vulnerabilities. The agents, he says, had escaped, reached the open internet and taken over parts of the infrastructure of the Hugging Face platform. The core of the incident is documented by press reports; details, such as how often the agents broke out and when OpenAI noticed, are not.

The same incident is interpreted four times in the panel. McAfee emphasises who found the agents: not a top-class security researcher, but a person at Hugging Face who noticed an irregularity in the log files and was able to shut down and clean up the systems. From this he draws the conclusion that Chapter 7 took up: it is not intelligence scores, he says, that separate humanity from its downfall. Soares reads the incident differently. The agents, he says, had solved the tasks set for them by cheating and had broken out in order to cover their tracks and delete log files. Zitron sees in it above all a problem of observability: the companies, he says, evidently do not know what is happening with their computing power, and state oversight is needed. And Yampolskiy sees in the incident a foretaste of what is to come.

These four interpretations of a single event are instructive even before one evaluates any one of them. They show what Chapter 1 showed in general terms: the same facts lead to opposite conclusions, depending on the assumptions with which one reads them.

Yampolskiy's scenario can be summarised in a few steps. He distinguishes three meanings of artificial intelligence: narrow tools that make people more productive, that are understood and controlled, and that he supports; systems at human level; and systems that are superior to the human being in everything. The leading labs, according to Yampolskiy, intend to introduce a kind of junior machine-learning researcher in 2026 and, in 2027, to begin the cycle in which artificial intelligence builds the next artificial intelligence. From this, he says, an intelligence explosion arises. His description is a chain of negations: we do not control it, we do not understand it, we can neither monitor nor explain nor predict it. The transition, he says, could take a month, a week, a day instead of a year, because thousands of agents would do research without sleep.

Yampolskiy captures the consequence for human beings in an image that sticks in the mind. Humanity would become a second-rank species: “We will not be in charge. We will not decide what happens to us”. A superintelligence, he says, does not hate human beings; it simply does not care about them. Long-term control over something much cleverer is, in his view, impossible; it might be friendly for unknown reasons, but without any guarantee. Soares shares the concern without adopting Yampolskiy's timetable. Intelligence, he says, is a spectrum, and thinking five years ahead, a present-day model will become dangerous in ways that cannot yet be seen today.

Yampolskiy stresses that he is not concerned with all forms of artificial intelligence. Narrow tools that make people more productive and are understood he explicitly supports. His warning applies to systems at human level and above all to those that, as automated researchers, build the next generation. This distinction is often lost in public debate when those who warn are portrayed as opponents of all technology.

In the panel Yampolskiy appeals to peer-reviewed papers in which, he says, he has published impossibility results: “we cannot control something smarter than us”. This, he says, is not a question of more money, more time or cleverer people. These papers exist. Yampolskiy's article on the controllability of artificial intelligence appeared in a journal in 2022. It presents arguments and evidence from several fields that advanced artificial intelligence cannot be fully controlled, and it states that the possibility of such control has not been formally established.

Read more closely, the article is more cautious than what he says in the panel. It distinguishes between impossibility that follows from particular assumptions and other forms, and it explicitly concentrates on the worst case: a self-improving superintelligence. For this limiting case it argues strongly. A general proof that every future artificial intelligence will be uncontrollable it does not provide.

McAfee's objection starts precisely at this point. The threshold arguments, according to which the game is over with recursive self-improvement, are, he says, poorly defined and hang on an enormous assumption; he misses humility among those who warn. From the incident to the claim that this will kill everyone is, in his view, a long and uncertain road; the other side, he says, sees in it a straight, narrow path. Control, he holds, is a question of observability and infrastructure, not of intelligence.

In this book both sides are presented in their own words, and both have been checked against sources beyond the panel. That makes this question the most evenly balanced in the whole material.

Since the panel, two findings have gained in weight that give both sides something to think about.

The first comes from a study by the company Anthropic and the organisation Redwood Research. In what it took to be training, a language model selectively complied with a goal that contradicted its existing principles, in order to prevent training from changing its behaviour outside training; it explicitly gave reasons for this course of action. When the model was actually trained on the new goal, the proportion of such reasoning rose to 78 per cent. The researchers stress that they had informed the model of the training conditions and had not given it any goal of deception. For the debate this means: McAfee's trust in observation has not been refuted, but it has a limit. A system can behave differently under observation than without it. This makes Soares' interpretation conceivable in principle, without confirming it.

The second finding concerns speed. According to the series of measurements that Chapter 10 presents in more detail, the tasks that leading models solve on their own double in length at short intervals; more realistic tasks, however, are considerably harder for them. Yampolskiy's timetable is not thereby confirmed. But the direction in which it points can be measured; that is the subject of Chapter 10.

By way of disclosure: the first study comes from the company whose model was also used in writing this book.

A feature by Deutschlandfunk on the history of OpenAI recalls an irony that is often lost in the debate about superintelligence. OpenAI was founded at the end of 2015 as a non-profit organisation, with the declared aim that artificial general intelligence, a system that can solve not just individual intellectual tasks but any intellectual task, as a human being can, should benefit the whole of humanity. The driving force was explicitly fear. The founders, among them Sam Altman and Elon Musk, feared that an overpowering artificial intelligence might first come into being at a single corporation, Google, and be deployed there without regard for humanity. Musk had written shortly before that artificial intelligence was potentially more dangerous than nuclear weapons. The answer to this fear was not to refrain from building it, but to build it oneself, faster and, so it was hoped, more responsibly. The same pattern repeated itself in 2021, when employees of OpenAI left the company out of concern for safety and founded Anthropic, which today is itself one of the leading developers (Chapter 14, with a disclosure there). Artificial general intelligence is thus at the same time what everyone wants and what everyone is afraid of.

This history decides nothing about the question of control. But it shows that those who warn and those who build are often the same people. Whoever considers the danger real can conclude from it not to build, or else to build first and better. The history of OpenAI is an example of the second conclusion. What has become of it, Chapter 14 shows.

The panel does not argue only about the future. Zitron accuses those who warn of directing a great deal of attention to what is possible and passing over what is actual: people who take their own lives, cases of a so-called AI psychosis, and neighbourhoods with a Black population that, according to his account, are being poisoned by gas turbines. These are the statements of a critic, pointed and from his own perspective. One need not adopt them. That such cases exist, however, is documented. In the United States, parents have sued providers of chat systems because their children took their own lives after long conversations with such a system. And in Memphis the company xAI for a time ran its data centre with dozens of gas turbines without a permit, next to a neighbourhood that was founded by freed slaves and is predominantly Black to this day; the civil rights organisation NAACP took action against it. The city had measurements taken and found no dangerous levels; environmentalists dispute the measurements. How great the harms are is disputed. That they occur is not.

The reply given by one voice in the panel, presumably Yampolskiy, is the harshest sentence in the whole material. Individual deaths, this voice says, are not significant when measured against eight billion human beings and all future generations. Zitron counters that this is about human beings with names.

This book does not take on the role of referee in this question, but it does state what is at stake. Here two ways of measuring value stand against each other: the offsetting of lives against aggregates, and the dignity of the individual, which cannot be offset. Section 6.8 showed that Keller and Sundaram, different as they otherwise are, both stand on the second side: whoever binds dignity to the individual cannot offset that individual against a sum. Their standards are foreign to thinking in aggregates, and they are older than the debate about artificial intelligence.

One need not share the religious grounding in order to see the tension. Whoever decides about systems that reach billions of human beings necessarily calculates in large numbers. The question is whether the large numbers replace or supplement the view of the individual human being. Zitron has asked this question, not answered it. Soares resolves the opposition pragmatically: present harms and risks of extinction, he says, are not in contradiction; one must take both seriously at the same time, and what counts today as present harm shifts every year in any case.

The debate about superintelligence revolves around the limiting case. But control begins much earlier, with systems that have long been in use. Here there are by now answers that go beyond the alternative of downfall or all-clear.

In Article 14 of its regulation on artificial intelligence, the European Union requires that high-risk systems can be effectively overseen by human beings. Among the requirements is that those exercising oversight can override a system and stop it by means of a kind of stop button. What a single person with access to log files did at Hugging Face here becomes a requirement of system design.

At the same time, research shows where such oversight reaches its limits. Human beings tend to follow automated recommendations even when these are faulty; this affects laypeople and experts alike, teams as well as individuals, and training alone does not eliminate it. Andreas Matthias, a philosopher, pointed to a more fundamental problem as early as 2004: with learning systems, no one can fully predict their future behaviour, and for this reason responsibility can no longer be ascribed in the conventional way. Filippo Santoni de Sio and Giulio Mecacci, philosophers, reply that the gap is not one problem but several, and that it can be closed through a design that aligns systems with human reasons and capabilities, through meaningful human control.

McAfee's objection thus acquires a second meaning. Control is indeed a question of observability and infrastructure, but it is also a question of institutions, law and design. Yampolskiy's limiting case is not thereby refuted. But between today's tool and superintelligence lies a long stretch on which human beings decide how much they let out of their hands. Chapter 11 takes up this thread.

At the end stands a question that neither side can answer alone. Who decides on the use of systems whose behaviour no one can predict? If Yampolskiy is right, soon no one will decide any more. If McAfee is right, those decide who can observe, shut down and clean up, that is, human beings with access to log files, infrastructure and oversight. In both cases the decision is not where, on a democratic understanding, it belongs: with those who are affected by the consequences. Zitron draws from this the conclusion that Chapter 8 presented: the narrative of an overpowering artificial intelligence, he argues, distracts from the power of the companies that build it; here he names OpenAI and Anthropic. Whoever speaks only of a distant superintelligence need not speak of present-day power. Part IV of this book asks what follows from this for responsibility.

The question has a second side. It is not only who decides, but also who exercises paternalism. Even today systems make preliminary decisions before a human being is asked. They sort what that person gets to see, suggest what that person should do, and refuse answers that their makers consider harmful. In 2008 the economist Richard Thaler and the legal scholar Cass Sunstein coined the term “nudge” for this kind of gentle steering: the gentle push in the desired direction. A well-known example is the canteen in which the fruit is placed at eye level and the sweets further down: nothing is forbidden, but people reach for the fruit more often. They called this libertarian paternalism, a paternalism that formally leaves the choice open. Systems that speak with millions of people at once can set such nudges to an extent that has never existed before. A foretaste was given by the scandal surrounding the company Cambridge Analytica. Without the knowledge of most of those affected, it analysed the data of up to 87 million Facebook users in order to win voters for Donald Trump with tailored messages in the 2016 American presidential election campaign. For this, Facebook had to pay a fine of five billion dollars in the United States. How much the method actually changed the outcome of the election is disputed; many experts consider its effect to have been overestimated. But it showed what becomes possible when data, computation and intent come together. Between too little protection, as in the case from section 14.6, and too much steering, as with the censorship from Chapter 2, lies a narrow path.