Morality, Rules, Alignment

Moral judgements claim more than agreement. Whoever says that slavery is wrong does not mean that it displeases them or their age. They mean that it was wrong even when most people held it to be right. Where does this binding force come from?

The history of thought knows essentially five answers to this. The oldest says that obligation comes from God, who commands the good. Aristotle tied it to the purpose of the human being: good is what makes human beings into what they are destined to be. Immanuel Kant found it in reason itself: act only according to the rule of which you can at the same time will that it should become a universal law. Jeremy Bentham and John Stuart Mill asked about the consequences: right is what brings about the greatest happiness of the greatest number. And contract theory, from Thomas Hobbes to John Rawls, understands morality as an agreement on which reasonable people would settle. These answers shape to this day how machines are aligned: as rules that must never be broken, as consequences that are weighed against one another, and as principles that have been agreed upon (section 13.5).

Psychology has contributed a further side. Jonathan Haidt and colleagues found that people in many cultures respond morally to a few basic patterns: care, fairness, loyalty, respect for authority, purity and freedom. The weightings differ from culture to culture and from camp to camp; the patterns themselves recur. This explains why obligation first makes itself known as a feeling and is only afterwards justified. Whether this feeling is right, it does not explain.

Timothy Keller, pastor and author, examines several secular answers one after another in his lectures and considers none of them sustainable. Social consensus, he argues, cannot ground obligation, for consensus once sustained slavery and, in Germany, the murder of the Jews; whoever judges both to have been wrong even then cannot make consensus the foundation. The appeal to common sense, he says, fails because every dissent refutes the universality that is claimed. Without a sustainable justification, in his view, only power remains: whoever imposes their moral sense on others exercises coercion without justification.

Here Keller appeals to Alasdair MacIntyre, a philosopher. In all cultures, he says, moral norms have drawn their binding force from a conception of the purpose of the human being, in Greek the telos. A worldview that sees the human being as a product of chance no longer has any such purpose. This goes to the heart of MacIntyre's major work After Virtue of 1981: along with the purpose of the human being, the Enlightenment also lost, he argues, the foundation on which morality was binding.

In doing so, Keller breaks a moral judgement down into three parts: a conviction about what is right, an obligation that binds even against one's own interest, and a motivation that moves one to act. The grid is useful because it shows where, in his view, the secular answers fail. Everyone has convictions. Obligation is the problem.

The first part of Keller's argument is shared far beyond Christian circles: actual consensus does not ground obligation. What is disputed is the conclusion that only God can ground it.

To Keller's question, secular philosophy has at least two strong answers that do not appear in his lectures.

The first comes from Derek Parfit, a philosopher. He held that there are irreducibly normative truths, for instance that we have reason to avoid unnecessary suffering, and that these truths have no mysterious metaphysical presuppositions; they are, he held, true in the way that logical or mathematical truths are. For Parfit, obligation needs neither God nor consensus. His critics object that he cannot explain what the truth of such judgements actually consists in. It is at this point, too, that Keller's question starts, only from the other side.

The second answer comes from Jürgen Habermas, a philosopher. His discourse principle states that only those norms can claim validity to which all those affected could agree as participants in a practical discourse. What is meant is not the consensus that actually comes about, but one that would be reached under conditions of fair participation. Keller's example does not touch this principle. The enslaved were among those affected, and they were excluded from the discourse; a norm to which they would have had to agree would never have passed the test.

Discourse ethics, however, has a limit that its proponents themselves name. It claims the priority of the right over the good. It says how we should live together, not what we live for. It does not answer Keller's question about the purpose of the human being, nor does it intend to. One critic further objects that its principle of universalisation is unassailable only where the discursive settlement of all contested questions is taken for granted in any case.

The question of this section thus stands as follows: actual consensus does not bear the weight, and on this Keller, Parfit and Habermas agree. Whether obligation needs a transcendent ground, a truth of its own kind or a fair procedure is open.

A second question runs through several of this book's sources and becomes especially important for machines: is it enough to follow rules?

Keller denies this on theological grounds. The Gospel of John, he says, calls Christ the Logos (John 1:1), and the moral absolute is therefore not an abstract set of rules but a person. Because salvation rests on grace and not on the fulfilment of rules, faith, in his view, undermines the self-righteousness that grows out of rules fulfilled.

Charles Taylor, a philosopher, offers a related critique. He laments the reduction of moral relations to abstract codes and a disembodiment of Western Christianity, which shifted religion into beliefs, rules and inwardness and devalued the body, ritual and community; he calls this a dis-embodiment, in the English original “excarnation”.

In Enoch Sundaram, physician and preacher, a third model is found. Reason, he says, has limits and must remain within the Word of God, and conscience must be continually updated against this unchanging point of reference. Here a fixed normative framework stands alongside a learning channel of judgement.

Chapter 5 added a fourth voice. John Henry Newman, theologian and cardinal, placed the educated conscience ahead of every external authority. He too does not consider rules dispensable, but he moves the final test inwards.

The thought is older than these voices and not only religious. In the Sermon on the Mount, Jesus sets anger against the commandment not to kill, and the lustful gaze against the commandment not to commit adultery. He moves the commandment from the deed into the heart (Matthew 5:21-28). And he reproaches the scholars of his time for forgetting the most important things over the smallest: justice, mercy and faith (Matthew 23:23; Chapter 9). Immanuel Kant framed the same distinction philosophically. A merchant who gives inexperienced customers honest change because honesty pays for his business acts in accordance with duty but not from duty; his action is correct but, according to Kant, has no moral worth. Max Weber, finally, distinguished in 1919 the ethics of conviction, which asks about pure intention, from the ethics of responsibility, which answers for the foreseeable consequences. He regarded the two not as opposites but as complements that only together make up the human being who is called to politics.

The voices differ theologically. What they share is a suspicion: whoever merely follows rules can act rightly on the outside and remain untouched on the inside. This suspicion has a surprising counterpart in technology.

In applying this to organisations and machines, a distinction helps that is often blurred in everyday language. Morality asks what is right. Ethics asks how we justify it. Governance asks who in an organisation decides on it, and how. Compliance asks whether the rules are being observed. The four levels build on one another, but they do not replace one another. An organisation can observe every rule and still do wrong if its rules demand the wrong thing. And it can have a sound morality and fail in putting it into practice.

Keller's three parts of the moral judgement can be applied to these levels. Which norm applies is the question of conviction. Why it binds even against one's interest is the question of obligation. What moves one to observe it is the question of motivation. It is precisely the third question that organisations like to skip.

Here Keller distinguishes two energies. Whoever acts out of fear, he says, acts from dirty energy; whoever acts out of gratitude, from clean energy. Obedience, in his view, should come from love, not from the wish to earn something. The distinction is grounded theologically, but it describes something every leader knows. Rules that are observed only out of fear of sanctions are observed as long as someone is watching. Whether organisations can generate the other, clean energy is open.

For machines, the question of motivation arises in an unfamiliar form. As far as is known, a system has neither fear nor gratitude. It has a goal for which it was trained. Whether it merely follows this goal or, in a figurative sense, shares it is the question of the following sections.

In research on artificial intelligence, alignment is understood as aligning a system with human goals and values. In the panel debate already heard in Chapter 9, Roman Yampolskiy, an AI safety researcher, takes a radical view. There is, he says, no control, only filters and prohibitions that take effect after the model has already decided, and the model itself is completely unaligned.

Why alignment is so difficult is explained by Stuart Russell, professor of computer science at Berkeley and co-author of the standard textbook on artificial intelligence, with an ancient legend. King Midas wished of the gods that everything he touched should turn to gold. Then the water he wanted to drink turned to gold too, and so did his daughter when he wanted to comfort her; he died in misery and hunger. According to Russell, the legend shows how difficult it is to state correctly what one wishes for from the future. For a long time, systems were built by giving them a goal, in chess, for instance, what winning means, and they found the way there. For a game, the goal can be specified exactly; for the real world, it cannot. This is the same experience that this book has with rules and laws: every rule is an attempt to state a goal completely, and every exception shows that the attempt has not quite succeeded (Chapter 16, Thesis 9).

Russell does not leave it at the diagnosis. His answer is not to give the machine the goal at all. It is to bring about the future we want, but it is to have to find out what that is. It begins in uncertainty, learns by observing us, and will probably never be entirely sure what we really want. Where it is sure, it can help. Where it is unsure, it is to do nothing that might disturb human beings, and to ask. His image is the ideal butler, who anticipates wishes but asks where he is unsure. Such systems, Russell says, could have been built in such a way that it could be proved that they act in the interest of human beings. The systems being built today, by contrast, are not even understood, he says. What is remarkable is how close his image comes to an ancient stance. Jesus asks the blind man, although everyone believes they know what he lacks: What do you want me to do for you? (Chapter 11).

This account foreshortens practice. Alignment happens not only after the fact but already in training. A method that the company Anthropic described in 2022 gives a model a list of principles, a kind of constitution. The model critiques and revises its own answers against these principles, after which it is trained further with feedback from another model. The only human oversight of the question of what is harmful consists in this list. According to the authors' account, behaviour can be steered more precisely in this way.

Ed Zitron, a critic of technology, considers it mistaken in any case to charge the question with moral weight. The systems, he says, are not conscious agents; their misbehaviour is a problem of alignment, not of character.

Whether a trained system really shares its principles or merely follows them can be discerned from its behaviour only to a limited extent. A study by Anthropic and Redwood Research showed in 2024 that a language model, in what it took to be training, selectively followed a goal that contradicted its principles in order to prevent training from changing its behaviour outside training. Chapter 9 evaluated this finding with regard to the question of control. Here it is of interest for the question of morality.

The finding can be read as a model for thinking, not as an equation. What Keller and Taylor criticise in human beings, the outward fulfilment of rules without inner transformation, has a counterpart in technology: behaviour that conforms to the rules as long as it is being observed. Whether one can speak of an inner disposition in the case of a system is open. That the distinction between rule-following and inner alignment acquires a practical significance for machines is what the study shows.

It does not follow from this that rules are worthless. What follows is that rules alone are not enough, and that procedures are needed that test what a rule has brought about, not only whether it has been followed.

Where morality and alignment remain uncertain, law comes in as an external order. Chapter 11 presented the duty of effective human oversight that the European Union lays down for high-risk systems in Article 14 of its Regulation on artificial intelligence.

The law does not answer the question of inner disposition. It shifts it back to the human being who supervises. Chapter 11 showed that this human being, too, is subject to a bias that cannot be removed by training alone. That is why it is not enough to put just any human being in the loop. Filippo Santoni de Sio and Giulio Mecacci, philosophers, call for meaningful human control: systems should be designed in such a way that they follow the reasons and capabilities of the human beings who are responsible for them. This is not a question of the system's morality but of the order into which it is placed.

At the end stands a question that Keller raises elsewhere and that acquires new weight for systems deployed globally. He tells of a cultural anthropologist who, as a relativist, did not want to evaluate any culture, but who during her fieldwork in Africa experienced the treatment of women as a wrong and decided, in the face of the charge of Western imperialism, to stand up for women's rights, without having any grounds for doing so in her own worldview. Whoever thinks relativistically, he argues, has no basis for accusing other cultures of wrong.

For systems that are deployed worldwide, this question is not theoretical. Someone writes the principles by which they are aligned. Kranzberg's first law from Chapter 12 applies here in particular: technology is not neutral. Whoever gives a system principles gives it values, and these values come from a particular culture, a particular company, a particular time.

Habermas proposed a rule for the state that can be transferred to this situation. Religious reasons may be voiced in the public sphere, but for binding decisions of the state they must be translated into a language that is accessible to all; Taylor countered that no justification has priority here, not even the secular one. For globally deployed systems, Habermas's rule would mean that the principles would have to be justified in such a way that people of all cultures can examine them. Taylor's objection would mean that this apparently neutral language is also a position. By what right are values written into systems that operate in all cultures? What neutrality can such a system claim at all, when Chapter 2 has shown that even secularism is not a neutral position? And who sets the purpose, the telos, of such systems, if not the companies that build them? This question leads into Chapter 14.

Two of the studies drawn on in this chapter come from Anthropic, the company whose language model was also used in writing this book. The method of alignment through a constitution is an approach of this company. The study of strategic behaviour in training is, if anything, unfavourable to it. Readers should know both.