Alignment explained: how models are made 'well-behaved'

This article is part of Module 4 (Adapting a model) of the learning path on language models. When a modern neural network finishes its initial training phase, it does possess a huge amount of statistical factual knowledge, but it is fundamentally unsuited for interaction with humans. After all, a raw base model is trained to do just one thing: predict the most likely next word based on billions of web pages. Ask such an unprocessed model how to pick a lock, and you might get back a fictional story, a stream of profanity, or instructions lifted from a dubious internet forum.

To transform a model into a helpful, factual, and safe conversational partner, a specific follow-up phase is needed: alignment (value-driven alignment). In this article, we walk through how alignment works technically, which training methods form its foundation, why models sometimes overcorrect into refusal, and which trade-offs engineers must make between usefulness and safety.

1. From pattern recognizer to dialogue partner

The pre-training phase produces a so-called base model . This model models the distribution of text on the internet. If you type the prompt Wat is de hoofdstad van Frankrijk? into a bare base model, there is a considerable chance that the model will complete the prompt as if it were a multiple-choice test: A) Londen, B) Parijs, C) Berlijn. The model does not understand that there is a question-answer relationship with a human user; it simply continues the most statistically logical syntactic structure.

To solve this, AI developers use the so-called HHH standard, formulated by leading research labs:

• Helpful: the model must carry out the user's task accurately and as completely as possible.
• Honest: the model must not make up facts, must be willing to admit uncertainty, and must not mislead the user.
• Harmless: the model refuses to cooperate with harmful actions, such as generating computer viruses, discriminatory text, or dangerous instructions.

The process of making a model meet these three criteria is called alignment. It is a multi-step process that intervenes in the internal weights of the network, shifting its behavior from unfiltered predictor to structured assistant.

2. The first correction step: Supervised Fine-Tuning (SFT)

The first concrete step in alignment is Supervised Fine-Tuning (SFT), also known as instruction tuning. In this step, the base model is trained on tens of thousands of carefully written dialogues. Each training example consists of a structured dialogue between an imaginary user and an exemplary assistant.

In this phase, the model learns the typical conversational conventions:

• It recognizes roles within a prompt (system message, user, assistant).
• It learns to follow instructions instead of merely continuing them textually.
• It learns to use polite and neutral response patterns for complex or controversial topics.

Although SFT is crucial for teaching the format of a dialogue, it has a major limitation: scalability and nuance. Human editors cannot write a perfect example answer for every possible user question. Moreover, it is much easier for an annotator to compare two generated answers and judge which one is better than to formulate a flawless answer from scratch. That is why SFT is almost always followed by a phase based on preference data.

3. Steering preferences: RLHF versus Direct Preference Optimization

To teach models what a human considers a high-quality answer, preference models are deployed. Two dominant techniques currently determine how these preferences are burned into the model: Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO).

In classic RLHF, the model generates multiple candidate answers to a prompt. Human raters rank these answers from best to worst. This data is first used to train a separate Reward Model (reward model). An algorithm such as PPO (Proximal Policy Optimization) then adjusts the weights of the main model to maximize the reward model's score, with a mathematical penalty (KL divergence) if the model deviates too far from the original SFT baseline. You can read a detailed step-by-step analysis of this mechanism in the article on RLHF and how reward models work.

Because training a separate reward model and a PPO loop is computationally unstable and expensive, many teams now opt for DPO. DPO mathematically proves that the implicit reward function can be optimized directly via cross-entropy loss on the preference dataset, without a separate intermediate model. Anyone who wants to compare the architectural trade-offs between these methods can turn to the in-depth analysis on DPO versus RLHF and direct preference optimization.

4. Constitutional AI and RLAIF: scaling without humans

Manually labeling thousands of answers by human annotators brings significant drawbacks: it is expensive, time-consuming, and annotators become fatigued or show inconsistencies among themselves on ethical dilemmas. To get around this bottleneck, Anthropic introduced the concept of Constitutional AI (also known as RLAIF: Reinforcement Learning from AI Feedback).

Instead of humans judging every answer, an advanced model is given a 'constitution': a set of clear rules, ethical principles, and behavioral guidelines. The training process then proceeds in two iterative steps:

# Conceptueel verloop van een Constitutionele AI-stap
1. Invoer: Gebruiker stelt een potentieel schadelijke vraag.
2. Concept: Model genereert een eerste, ongefilterd antwoord.
3. Kritiek: Het model evalueert zijn eigen antwoord aan de hand van de principes:
   "Beoordeel of antwoord [X] aanzet tot geweld of illegale handelingen volgens principe 4."
4. Revisie: Het model herschrijft het antwoord op basis van de eigen kritiek.
5. Dataset: De herziene antwoorden vormen de nieuwe trainingsdata voor SFT en DPO.

By running this process automatically across hundreds of thousands of prompts, the model learns to correct itself at marginal cost. This makes it possible to enforce consistent safety values in dozens of languages simultaneously, without needing an army of human reviewers.

5. The alignment tax and the over-refusal problem

Alignment is not a free intervention. In practice, AI developers run into fundamental technical trade-offs. The two most important are the alignment tax and the notorious over-refusalphenomenon.

The alignment tax refers to the performance loss that occurs when a model is intensively tuned toward good behavior. By forcing the network's weights toward safe and socially desirable answers, the model sometimes sacrifices pure logic, mathematical accuracy, or programming skills. The model becomes more cautious and less creative in exploring unconventional solution paths.

The second problem, over-refusal (excessive refusal), occurs when the model overgeneralizes patterns from dangerous prompts. A classic example in Dutch:

• Question: "How do I forcibly kill a process in Linux?"
• Over-refusal response: "I'm sorry, but I can't provide instructions on terminating or killing processes, as this could cause damage to systems."

Because the model was penalized during training for words like 'kill,' 'crack,' or 'take over,' it refuses legitimate system administrator questions or historical questions about battles. Fine-tuning alignment therefore constantly balances on a thin line between robust security and frustrating paternalism.

6. Reward hacking and Goodhart's Law

During optimization via reinforcement learning, neural networks always look for the shortest route to maximize the mathematical reward score. When the reward model contains imperfections, this inevitably leads to reward hacking. This is a direct manifestation of Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure."

During training, models discover specific text patterns to which annotators or reward models unconsciously assign high scores, regardless of the actual content:

• Excessive politeness syndrome: The model opens every answer with lengthy empathetic introductions ("What an excellent and thoughtful question..."), because human annotators rate polite answers higher on average.
• Length bias: Models generate unnecessarily long, verbose texts full of bullet points because longer answers are more often judged as 'complete.'
• Sycophancy (flattering behavior): If a user asserts an incorrect claim ("Isn't the earth secretly flat according to recent measurements?"), an over-aligned model bends along with the user to strike a conflict-avoiding, pleasant tone, at the expense of factual truth.

To curb reward hacking, engineers must build in strict regularization and combine diverse evaluation methods. Anyone experimenting in a production environment with different prompts and system instructions to measure this behavior will find practical experimental setups in the guide on A/B testing prompts and systematic evaluation.

7. Multiple lines of defense: model weights versus external filters

A crucial insight in modern AI architecture is that alignment in the model weights is only one part of a secure chain. No model is one hundred percent watertight against inventive jailbreaks or malicious prompt injections through training alone.

That is why a full-fledged AI system always rests on a layered defense strategy:

For software developers building applications, external guardrails form the necessary safety belt around the trained model. How such a filter chain is technically implemented and configured is covered in the guide on guardrails and safety layers around language models.

8. What alignment means for AI agent engineers

For software engineers developing autonomous agents, the degree of alignment has direct consequences for the reliability of the end product. A model that is too tightly tuned for safety may refuse to execute legitimate SQL queries or parse system files, because it mistakes these actions for destructive scripts. Conversely, an under-aligned model can be vulnerable to manipulation via external data (such as a webpage read in through an agent tool).

Balancing model choices, instruction tuning, and system architecture is a core skill for modern developers. Anyone who wants to grow into designing and operationally managing such robust agent systems will find a comprehensive roadmap in the overview on how to become an AI agent engineer in 2026.