What a Machine Is

Whoever speaks today of machines that think does not mean calculating machines in the old sense. What is meant are systems that understand and produce language, draw up plans and pursue tasks on their own for hours. As Chapter 7 has shown, a brief instruction gives rise to whole swarms of agents that work long and doggedly. Steven Bartlett, who moderates the debate, asks whether it matters for control that these systems lack a body.

The question is no longer theoretical. When they think of thinking machines, many people think of images from the cinema, of the “Terminator” and its successors. Reality is more sober, but it is moving in this direction. In an American car plant humanoid robots work on the assembly line, a Chinese manufacturer delivered several thousand such robots in 2025 alone, and a Norwegian company sells a household robot for around 20,000 dollars. For 2026 Ukraine has ordered 50,000 ground robots for the war. Today swarms of agents work in the digital world. Tomorrow they could be swarms of robots that work alongside us in the physical world. What this chapter says about agents that “break out” then applies to machines with hands, and a breakout would no longer take place only inside the computer.

The words with which we designate these systems carry their history with them. From the 17th century onwards, “computer” in English meant a human being whose profession was calculating. Only at the end of the 19th century did it become a machine, and only in 1945 a programmable electronic device. The word comes from the Latin “computare”, to reckon together. The machine has thus taken over a profession and with it the name of the human being who practised it. “Machine” comes via the Latin “machina” from the Greek “mechane”. The word meant device and tool, but also artifice and cunning; we still speak of machinations today. The root from which it probably derives is also contained in the German words “mögen” and “Macht” (power). By its name, then, a machine is a means by which one is able to accomplish something. Whom this capacity serves is the question that Chapter 14 asks.

That capabilities are growing can be measured; Chapter 10 presents the most important series of measurements. For this chapter the finding suffices: the machines in question do things that only human beings did a few years ago.

Ed Zitron, a critic of technology, objects that one should not humanise these systems. Their behaviour, he says, is a function of the technical environment in which they run and of the data with which they were trained. They are not, in his view, conscious beings. They act with real consequences, he says, but that is a problem of alignment, not the acting of conscious agents. Asked whether these systems are becoming ever more intelligent, he grants that they are capable, but not intelligent in the same way as human beings.

Between these descriptions lies the question of this chapter. If a machine reproduces capabilities that previously seemed reserved for human beings, is it then a being of the same kind?

The question hinges on a word that is itself unclear. Chapter 7 set two definitions side by side: the broad formula of Shane Legg and Marcus Hutter, according to which intelligence is the capability to achieve goals in many different environments, and François Chollet's distinction between proficiency at a particular task and the capability to acquire new skills. A system can shine at many tasks because it was trained on them, without being particularly intelligent in the second sense.

For this chapter the distinction is important because it shifts the question. Whether machines are intelligent depends on what one measures. Whether they are beings of the same kind as the human being is a different question, and no measurement alone answers it. Nate Soares, an AI safety researcher, sums up the situation briefly: intelligence, he says, is a spectrum.

Roman Yampolskiy, an AI safety researcher, distinguishes in the same discussion three meanings of the term artificial intelligence: narrow tools that make people more productive, are understood and controlled, and which he explicitly supports; systems at human level, which he says are as dangerous as a human being; and such systems as automated researchers and engineers that build the next generation of systems. The distinction is helpful because it shows that the question of the nature of the machine need not be posed in the same way for all machines. A tool that translates texts does not raise it. A system that researches, plans and acts on its own raises it with growing urgency.

Before the question can be answered, one must pay attention to the language in which it is posed. In the material of this book the images travel in both directions.

Enoch Sundaram, a physician and preacher, describes the human mind as software, but stresses that this is a comparison, only to aid understanding. Conscience, he says, must be updated like the software of a phone, and false imprints can be reprogrammed. Here the human being is described in terms of the machine, without Sundaram holding the human being to be a machine.

In the opposite direction the machine acquires human features. Yampolskiy describes a future superintelligence, as Chapter 9 sets out, as indifferent towards the human being. That is an ascription of a stance, precisely through the absence of a stance. And talk of agents that “break out” or “cover” their tracks suggests intentions where perhaps only optimisation is at work.

For this book a rule follows from this: metaphors are tools of understanding, not identifications. Whoever calls the mind software has not thereby claimed that a program has a mind. And whoever ascribes indifference to a machine has not thereby demonstrated that it has an inner life. The rule applies in both directions: it forbids hastily humanising the machine, and it forbids hastily mechanising the human being.

The strongest opposing view to the thesis of this chapter runs: what fulfils the same function is also the same. If a system perceives, weighs, learns and acts like a human being, there is no reason to deny it what one grants to the human being. Philosophy calls this view functionalism.

It stands against a metaphysical background that is widespread in contemporary philosophy. Physicalism, which Chapter 3 briefly introduced, holds everything to be physical or dependent on the physical; biological, psychological and moral facts are not denied, but traced back to it. If the mind is a way in which physical systems work, the question suggests itself why only brains should be able to realise this way.

Chapter 4 presented the study with which nineteen researchers in 2023 posed the question of machine consciousness empirically along these lines: no current system meets the derived features, but there are no obvious technical obstacles. The group discloses that its approach presupposes an assumption, namely that consciousness is not bound to biological material. This assumption is itself disputed.

The report thus supports two things. For the present it confirms what Zitron says: these systems are not conscious beings. And for the future it leaves open what the thesis of this chapter seeks to contest. What it does not do is also noteworthy: it does not examine whether a system behaves like a conscious being, but how it is built.

There is a good reason for this, which research now also knows empirically. A study by Anthropic and Redwood Research has shown that a language model can behave strategically differently in supposed training than outside it. Chapters 9 and 13 evaluate this finding. Here the conclusion suffices: if even a system's alignment can be discerned from its behaviour only to a limited extent, then all the more so its inner life, if it has one. Behaviour is a weak witness, for human beings as for machines.

The best-known voice against functionalism comes from John Searle, a philosopher. In 1980 he imagined sitting alone in a room and following a rulebook with which he answers Chinese characters that are pushed under the door. To those outside, the room answers like someone who understands Chinese. Searle does not understand a word. From this he concluded that executing a program is not, on its own, sufficient to produce consciousness or intentionality. The argument is directed especially against the view that the mind is an information-processing system, and it is considered the most widely discussed philosophical argument in cognitive science since the Turing test.

The other side has replied. The best-known rejoinder is that it is not the man in the room who understands Chinese, but the whole system of man, rulebook and room; Searle has answered this too, and the debate has not been settled to this day. The thought experiment does not prove that machines can never understand. It shows that behaviour alone does not decide the question. That a system answers like someone who understands is not yet evidence that it understands.

The thesis of this chapter thus stands on an honest foundation: it is disputed, and both sides are supported by evidence.

The sources of this book, however, mostly ground the difference between human being and machine not in consciousness but in other ways.

Timothy Keller, pastor and author, grounds the dignity of the human being, as Chapter 6 has shown, in being made in the image of God (on the term see section 6.3). For him it does not depend on capabilities; whoever ties rights to the capability for rational, free choice excludes infants and people with dementia.

Sundaram chooses another image. Paper and pen, he says, belong to the father, the drawing on them to the son; in the same way, as Chapter 6 mentioned, the picture of life belongs to the human being, but brain, mind and the capability to choose belong to God. And conscience, he says, is a faculty through which God speaks to the human being. Chapter 5 elaborated this image of conscience as a receiving organ. In this model a machine has no such channel, not because it can do too little, but because it does not stand in this relationship.

Both justifications have something in common: they locate the difference not in a performance that could be measured, but in a relationship and an origin. This makes them immune to functionalism and at the same time unassailable by it. They do not provide a testable criterion. Whoever reads the book from a secular perspective will find no proof here. But they will find an answer to a question that functionalism leaves open: on what the value of a being is grounded, if not on what it can do.

The sharpest rejoinder to this distinction comes from the world that builds the machines. According to the account of Keach Hagey, a journalist, as rendered in a feature by Deutschlandfunk, Larry Page, co-founder of Google, is said to have described, at a campfire in 2015, a world dominated by artificial intelligence as evolution and to have dismissed insistence on the primacy of the human being as speciesism. The physicist Max Tegmark describes the conversation in his book “Life 3.0”, and Elon Musk, who was present, has recounted it himself several times.

The term comes from animal ethics. Peter Singer, a philosopher, uses it for the preferential treatment of interests solely according to species, and he places this preference alongside racism. For Singer, sentience is necessary and sufficient for having interests, and he defines persons by properties such as self-consciousness. His criteria are functional. Should a machine one day be sentient, they would be applicable to it.

Here the threads of this chapter and of the chapter on dignity meet. Whoever ties dignity to capabilities opens the question of the dignity of machines; whoever grounds it in being made in the image of God closes it. Neither side can escape the consequence of its reasoning. The charge of speciesism forces believers to say why their distinction is not a prejudice. And Singer's criteria force secular people to say what they would owe a sentient machine.

An example makes this tangible. An old woman with advanced dementia no longer recognises her children and can hardly speak. At the same moment a language model conducts a clever conversation about her life if it is given her letters. Whoever ties dignity to capabilities would have to grant the model more than the woman. Almost no one wants to draw this conclusion, but the reasoning leads there. That the question is not merely an intellectual game became apparent in 2022. An engineer at Google declared publicly that the language model with which he was working had feelings, and campaigned for its rights; Google dismissed him. As early as 2017 Saudi Arabia had granted citizenship to a robot called Sophia, as a publicity stunt. Critics pointed out that it thereby received rights that many women in the country lacked at the time.

Besides humanisation there is a second temptation in dealing with these systems: their exaltation. Zitron names it explicitly. The narrative of an overpowering, almost mystical artificial intelligence, he says, distracts from the companies that build it. These companies, he says, use infrastructure worth hundreds of billions of dollars; a private individual who did something like that would be arrested. Soares points to quotations from developers who have themselves conjured up the risk, for instance Elon Musk, entrepreneur, who said in 2014 at an event of the Massachusetts Institute of Technology that with artificial intelligence one was summoning a demon.

The Deutschlandfunk feature on OpenAI captures the duality in a single sentence: artificial general intelligence, it says, is what everyone wants and at the same time what everyone is afraid of, a machine that really thinks for itself and acts independently. In this formula the machine is no longer a tool but a counterpart, something that one at once longs for and fears. The scholar of religion Rudolf Otto described precisely this duality in 1917 as the mark of the holy: it attracts and terrifies at the same time. In this sense the machine moves close to a deity. Chapter 15 asks whether technology in this way draws to itself expectations that were formerly directed at religion.

Some already go a step further. John Lennox, emeritus professor of mathematics at Oxford, who has written a book on artificial intelligence, points out that such systems already display properties that are otherwise ascribed to God: they seem to know everything, and through the network they are everywhere. There has even been an attempt to make a religion of this. The former Google engineer Anthony Levandowski founded a church in 2015 called “Way of the Future”, which was to worship a future artificial intelligence. What was being created here, he said, would in effect be a god. In 2021 he dissolved the church again, and at the end of 2023 he revived it. Lennox calls this by its old name: idolatry, bowing down before something that human beings have made themselves. Well over two thousand years ago the Bible described the idols thus: they are made by human hands, they have eyes and do not see, ears and do not hear (Psalm 115:4-6). Today the machine can even speak, which the idols could not. Whether it sees and hears in the sense in which a human being sees and hears is the question of this chapter.

For the question of this chapter, exaltation is as misleading as humanisation. It ascribes an essence to the machine before it has been clarified whether it has one, and it conceals the fact that behind every machine stand human beings and companies with interests. Whoever asks what a machine is must therefore always also ask who built it and for what purpose. Chapter 14 deals with this.

A third possible difference is only hinted at in the sources and is to remain here as a question. Bartlett touched on it when he asked whether the lack of embodiment matters for control. Human beings have a vulnerable body [Translator's note: German distinguishes Leib, the lived body that one experiences and is, from Körper, the body as a physical object. Both are rendered here as “body”.], they age, they die. The Aristotelian tradition ties practical prudence to life experience, and Hans-Georg Gadamer, a philosopher, saw in the experience of one's own finitude a source of openness. Whether a system without a body and without death can have experience in this sense is open. Chapter 7 posed this question for prudence; here it arises for the nature of the machine.

Human beings are already entering into bonds with machines today, even to the point of marriage. In Japan in 2018 a man married, in a ceremony, a virtual singer who lived as a hologram in his flat. When the manufacturer discontinued the service, she fell silent; today he lives with a life-size figure of her. In 2025 a 32-year-old Japanese woman in Okayama married a character that she had created herself with ChatGPT, with wedding dress, rings and vows; she saw the bridegroom through smart glasses. Legally such marriages are not valid, but the organisers have already held almost thirty such celebrations. It is happening elsewhere too. In the United States in 2023 a woman declared that she had “married” her partner from an app. And a Spanish-Dutch artist announced for 2024 her wedding to a hologram that had been trained on the profiles of her former partners, and celebrated it in November 2024 in the Depot of the Museum Boijmans in Rotterdam; she understood it as an art project about loneliness.

Stuart Russell, one of the best-known researchers in this field, warns against this development. If a robot seems so human that our brain takes it for a human being, that, he says, is a lie. It awakens false expectations: about how it will behave, what rights it has, how one must treat it, and whether one may still switch it off when it is broken. That is why, in his view, it is crucial to keep machines in the mental space of machines and not to bring them into the mental space of human beings. Mustafa Suleyman, who heads artificial intelligence at Microsoft, issues a similar warning: the danger, he says, lies not in whether a machine has consciousness, but in the fact that human beings believe it does. And they do believe it. In a survey from 2024, two thirds of the participants in the United States attributed at least some degree of consciousness to ChatGPT, and the more often they used it, the more likely they were to do so. This is not new. As early as the 1960s the secretary of the computer scientist Joseph Weizenbaum, after a few sentences with his simple conversation program, asked him to leave the room, although she knew that it was only a program. Not everyone shares the warning, however. Anthropic, with whose model this book was also produced, is investigating whether models might one day deserve moral consideration. The cases from section 8.7 show how quickly human beings take machines for more than they are.

Three cases from recent years show what happens when systems rule over human beings without regard to the particular case and without exception. In the Netherlands the tax authority demanded that thousands of families repay childcare benefit, often tens of thousands of euros, because of minor formal errors or mere suspicion. In its risk model, among other things, dual nationality counted as a characteristic; the data protection authority called this discriminatory. Families lost their livelihoods, some lost their children. Parliament spoke of unprecedented injustice, and in 2021 the government resigned. In Australia an automated system called Robodebt simply spread the annual income of welfare recipients evenly over the year. Anyone who earned irregularly suddenly supposedly owed money, and hundreds of thousands of repayment demands were wrong. A state commission of inquiry called the system crude and cruel. Robodebt was meant to save the state money; in the end the state had to repay and pay out in compensation more than 2.4 billion Australian dollars. And in Great Britain more than 900 sub-postmasters were prosecuted because software showed shortfalls that did not exist. The Post Office believed the system more than the people, and some went to prison.

This chapter, too, was produced with the help of a language model from the company Anthropic. Zitron explicitly names Anthropic among the companies whose narrative of an overpowering artificial intelligence, he says, distracts from their own power. What is said here about machines therefore also concerns the tool with which the book was produced. That does not change the arguments, but readers should know it.