Reasoning models: what changes when thinking becomes a separate step

By Ivo Donker — compiled with AI assistance · Last updated: August 7, 2026

Welcome to module 5 of the learning track, titled "Under the Hood". In this chapter, we explore a fundamental shift in artificial intelligence architecture: the emergence of reasoning models. Classical language models generate responses token by token in a single forward pass, where required compute is directly tied to the length of the generated output. Reasoning models introduce an explicit, internal thinking phase before producing the final response. Through this application of so-called test-time compute, the processing pipeline changes fundamentally: thinking becomes a distinct, measurable, and often visible intermediate step. In this article, you will explore exactly what changes under the hood, how these models are trained, when leveraging additional compute is cost-effective, and what implications this holds for software architectures.

Prerequisites

To contextualize how reasoning models work, it is helpful to be familiar with several core concepts covered in earlier modules of this learning track. The following articles provide the necessary foundation:

• Chain-of-thought as a prompting technique — Read this article if you want to understand how users manually elicit intermediate steps via prompts, whereas the current article covers how models automate this process internally.
• Inference in classical language models — Check out this explanation to understand traditional single-pass generation, which serves as a baseline comparison for reasoning models.
• Speculative decoding explained — Refer to this guide if you want to explore how the increased latency of longer token sequences can be mitigated using complementary techniques.

What changes under the hood?

From single-pass inference to test-time compute

In classic autoregressive language models, query processing occurs via a continuous, single-pass inference. When a prompt is entered, the network calculates the probability distribution across the entire vocabulary for each subsequent token. As soon as the model selects a token, this token is immediately appended to the output sequence and serves as input for computing the next token. This process repeats until the model generates a stop token. Consequently, the computational budget a classic model allocates to a task scales linearly with the length of the generated text. In this traditional setup, a short question requiring a complex mathematical answer receives exactly the same computational power per generated token as a question asking for a simple fact.

Reasoning models break away from this rigid coupling between output length and computational power. They leverage the principle of test-time compute (computational power during inference). Before the model begins generating the final answer visible to the user, it executes a series of internal computations. These computations take the form of intermediate 'thinking tokens' or 'reasoning tokens'. The model uses these internal tokens to analyze the problem, explore potential paths, validate intermediate results, and correct any errors before finalizing the ultimate answer.

Processing a request thus shifts from a purely pattern-recognition process to a hybrid form of pattern recognition and dynamic search and compute operations. Whereas a classic model often stumbles on complex queries because the initial tokens must be immediately correct to reach the right end result, a reasoning model can discard flawed hypotheses within its thinking phase and pursue a new path.

Training Methodology: Reinforcement Learning on Reasoning Traces

The ability to think in a structured and efficient manner does not emerge spontaneously. It requires a specific training approach that differs significantly from standard pre-training and Supervised Fine-Tuning (SFT) used for generic language models.

The foundation of reasoning models relies on large-scale Reinforcement Learning (RL) applied to reasoning traces. During the training phase, the model is presented with complex problems across domains such as mathematics, formal logic, and programming. The model is not only rewarded for producing the correct final answer (Outcome-based Reward Models or ORM), but also—via Process-Supervised Reward Models (PRM)—for the quality, correctness, and efficiency of each individual intermediate step.

Under the influence of these reward signals, the network learns autonomous reasoning strategies, including:

• Decomposition: Breaking down a comprehensive problem into small, independently solvable sub-questions.
• Self-Correction: Identifying an inconsistency or computational error within its own intermediate steps, after which the model explicitly backtracks to an earlier point in the reasoning phase.
• Verification: Validating an intermediate result along the way by executing a cross-check or boundary condition verification.
• Backtracking and MCTS-like exploration: Exploring multiple potential solution paths (similar to Monte Carlo Tree Search) to select the most promising path.

A crucial insight with this training method is that the generated reasoning traces evolve throughout the training process. Initially, the model often produces long, rambling texts to reach a conclusion. As the RL process progresses, the model develops a more compact and effective internal 'language' for logical steps, thereby optimizing the ratio between the volume of thinking tokens and the final accuracy of the answer.

Inference budget, thinking tokens, and test-time scaling

The term test-time scaling laws refers to the empirical observation that model accuracy on complex tasks scales not only with training dataset size and parameter count, but also with the amount of compute allocated during the inference phase. By allowing the model to generate more thinking tokens, its success rate on challenging logical and mathematical benchmarks increases significantly.

In practice, this compute is managed via an inference budget. This budget determines the maximum number of thinking tokens a model may generate for an individual query. Thinking tokens consume space within the model's total context window. When a model generates 4,000 thinking tokens before beginning its output, it leaves 4,000 fewer tokens available for the input context or follow-up interactions within the same session.

Developers or users typically control this via settings that govern how deeply the model delves into finding a solution. For low-complexity tasks, a large inference budget is not only unnecessary, but also introduces latency and drives up costs. For highly complex tasks, however, an adequate budget prevents the model from prematurely halting its reasoning and falling into incorrect assumptions.

Cost spectrum and latency: the price of deep thinking

Introducing an explicit thinking phase carries direct implications for operating costs and the user experience of AI applications. Compared to classical models, two main factors come into play:

1. Token amplification: For a prompt that takes 50 input tokens and 100 output tokens in a classical model, a reasoning model might consume 50 input tokens, 2,000 thinking tokens, and 100 response tokens. Since API providers charge per processed token (with thinking tokens typically billed at the standard output rate), the operational cost per request can increase by a factor of ten to fifty.
2. Increased latency (Time-To-First-Token and total turnaround time): Because the model must first generate hundreds or thousands of thinking tokens before the first token of the actual response is returned to the user, Time-To-First-Token (TTFT) rises significantly. While a classical model starts streaming within a few hundred milliseconds, a reasoning model may remain 'silent' for several seconds or even tens of seconds while completing its internal reasoning trace.

These characteristics make reasoning models unsuitable for real-time applications such as voice assistants or interactive autocomplete features in user interfaces, unless specific measures are taken to bridge the latency.

Settings: reasoning effort, temperature, and top-p

Controlling reasoning models requires a different approach to hyperparameters than traditional LLMs.

In classical models, the settings temperature and top_p are the primary dials for controlling the creativity or determinism of the response. A higher temperature introduces more variation into token selection. With reasoning models, the situation is different. Since the internal thinking phase relies on rigorous logical steps, an excessively high temperature during reasoning can lead to hallucinations within intermediate steps, causing the entire reasoning path to derail.

Many API providers therefore lock the temperature for the internal thinking phase at a low value, or advise developers to leave the temperature at its default setting (often 1.0 or 0.0, depending on the specific model). Instead, APIs introduce a new parameter: reasoning effort (often configurable as low, medium, or high). This parameter sets a limit on the number of thinking tokens the model is allowed to use, enabling developers to directly balance cost/latency against reasoning depth.

Evaluate and choose the right architecture

For a well-considered design of your application landscape, it is important to weigh reasoning models against generic models and stay up to date with the state of the art. Consult the following resources within the llmnet network for further depth:

• Overview of reasoning models — Use this comparison to determine which specific reasoning models are currently available and what characteristics they possess.
• Reasoning quality benchmark on GSM8K and MATH — Explore these benchmarks to see how accuracy on complex mathematical problems is measured in a standardized way.
• Model routing in API environments — Consult this guide to learn how to dynamically route requests to reasoning models or standard models using logic in your code.
• Market developments in reasoning models — Read this overview to stay informed about the latest commercial and open-source releases in reasoning architectures.

When to use and when to avoid?

It is a common misconception to assume that reasoning models are superior for every type of task. Because test-time compute incurs additional cost and latency, a strict division of tasks is essential.

Suitable use cases (When to use):

• Multi-step arithmetic and mathematical problems: Tasks where intermediate variables must be tracked and calculations depend on earlier results.
• Complex logic puzzles and planning: Tasks with numerous constraints, such as creating a complex schedule or analyzing dependencies in software architectures.
• Code generation and refactoring: Scenarios where the syntax must not only be correct, but edge cases, memory management, and algorithmic efficiency must also be taken into account.
• Tax and legal rule application: Interpreting complex legislation for a specific case where multiple exceptions and threshold amounts play a role simultaneously.

Unsuitable use cases (Do not use):

• Simple factual prompts: Questions such as "What is the capital of France?" do not require a thinking phase. The model can retrieve this directly from its parametric memory.
• Text processing and creative writing: Summarizing a short text, rephrasing an email, or writing a poem does not benefit from mathematical logic and is only made slower and more expensive by thinking tokens.
• Classification and entity recognition: Categorizing customer queries or detecting brand names in a text can be handled more efficiently by lighter, classic language models.

Concrete Dutch practical example: VAT and discount calculation with exceptions

To illustrate the difference between a classic single-pass approach and a reasoning model, we examine a case from Dutch business practice. It involves an order calculation with stacked rules, tiered discounts, and different VAT rates.

The case

A wholesaler applies the following rules to an order:

• Item A (electronics) costs € 150.00 excluding VAT (standard rate 21%).
• Item B (professional books) costs € 40.00 excluding VAT (reduced rate 9%).
• For a total order value above € 500.00 excluding VAT, the customer receives a 10% volume discount on the entire goods amount before VAT. This discount is distributed proportionally based on goods value across the two item types to determine the correct VAT base.
• Shipping costs are € 25.00 excluding VAT (taxed at the standard rate of 21%). Shipping costs do not count toward reaching the € 500.00 threshold for the volume discount.

The prompt for the model: The customer orders 3 units of Item A and 3 units of Item B. What is the exact total invoice amount including VAT, and how much VAT is calculated per tax bracket (21% and 9%)?

Classic approach versus reasoning approach

A classic language model immediately begins formulating the text. It calculates "as it writes" and tends to apply discount percentages directly to the total amount including shipping costs, or to incorrectly weight the reduced and standard VAT rates across the discounted amounts. This happens because the model cannot validate intermediate results before the text is generated on screen.

A reasoning model, by contrast, starts with a hidden or visible thinking phase. Below is a faithful representation of what the model calculates in its thinking tokens before generating the final answer:

[Thinking tokens / Reasoning trace]
Step 1: Calculate the gross goods value excluding VAT.
- Item A: 3 * € 150.00 = € 450.00.
- Item B: 3 * € 40.00 = € 120.00.
- Total goods value excl. VAT = € 450.00 + € 120.00 = € 570.00.

Step 2: Check the discount condition.
- Is goods value excl. VAT > € 500.00? Yes (€ 570.00 > € 500.00).
- Volume discount is 10% on € 570.00 = € 57.00.
- Net goods value excl. VAT = € 570,00 - € 57,00 = € 513,00.

Step 3: Allocate the discount proportionally between Item A and Item B to determine VAT bases.
- Item A's share of gross total = 450 / 570 = 0.78947368 (78,95%).
- Item B's share of gross total = 120 / 570 = 0.21052632 (21,05%).
- Net tax base Item A (21% VAT) = € 450,00 - (0.78947368 * € 57,00) = € 450,00 - € 45,00 = € 405,00.
- Net tax base Item B (9% VAT) = € 120,00 - (0.21052632 * € 57,00) = € 120,00 - € 12,00 = € 108,00.
- Check: € 405,00 + € 108,00 = € 513,00. Correct.

Step 4: Add shipping costs.
- Shipping costs = € 25,00 excl. VAT (taxed at 21%).
- Total base for 21% VAT = Net Item A + Shipping costs = € 405,00 + € 25,00 = € 430,00.
- Total base for 9% VAT = Net Item B = € 108,00.

Step 5: Calculate VAT amounts.
- 21% VAT on € 430,00 = € 430,00 * 0,21 = € 90,30.
- 9% VAT on € 108,00 = € 108,00 * 0,09 = € 9,72.
- Total VAT = € 90,30 + € 9,72 = € 100,02.

Step 6: Calculate and verify the final amount.
- Total excl. VAT = € 513,00 (goods) + € 25,00 (shipping) = € 538,00.
- Total incl. VAT = € 538,00 + € 100,02 = € 638,02.
- Verification via individual incl. VAT amounts:
* Item A incl. VAT: € 405,00 * 1,21 = € 490,05.
* Item B incl. VAT: € 108,00 * 1,09 = € 117,72.
* Shipping incl. VAT: € 25,00 * 1,21 = € 30,25.
* Sum = € 490,05 + € 117,72 + € 30,25 = € 638,02. Everything aligns perfectly.
[/Thinking Tokens / Reasoning Trace]

Following this internal verification, the model generates the final response to the user. Because all intermediate steps and boundary conditions have already been verified in the thinking process, the final answer is free from calculation errors and incorrect assumptions.

When we simplify the same question to: "What is 21% VAT on € 150,00?", initiating such a reasoning process is unnecessary. A classic model delivers the result immediately (€ 31,50) without extra compute time or token consumption. Unnecessarily deploying thinking tokens for simple calculations causes latency without increasing accuracy.

Comparison Table: Classic Models versus Reasoning Models

The table below provides an overview of the technical and operational differences between classic language models and reasoning models.

Architectural impact on software integrations

The introduction of reasoning models does not only change how prompts are written, but also impacts the design of software applications that utilize AI APIs.

1. Adapting timeouts and asynchronous patterns

Traditional HTTP REST integrations often assume a quick response. When invoking a reasoning model with a high reasoning_effort the processing time can exceed a minute. Client libraries and API gateways that apply default timeouts of 10 or 30 seconds will encounter network errors as a result.

Developers should adapt their architecture by:

• Setting more generous HTTP timeouts on API connections.
• Switching to asynchronous processing patterns (such as message queues or WebSockets) where the user interface tracks the status of the reasoning process without blocking the connection.
• Displaying specific status indicators in the UI ("Model is analyzing the problem...") to prevent the user from perceiving the application as unresponsive.

2. Cost control through dynamic routing

Because thinking tokens substantially increase the total token volume, routing all incoming requests to a reasoning model by default is financially impractical. An effective architecture utilizes a routing layer (model router).

This routing layer evaluates incoming queries using a lightweight classification model or heuristic rules. Simple queries (such as rewriting a sentence or looking up a fact) are forwarded to a standard, cost-effective language model. Only when the query exhibits characteristics of complex mathematics, multi-step logic, or extensive code analysis is the request forwarded to the reasoning model.

3. Visibility and handling of thinking tokens

Reasoning model APIs often provide the option to include generated thinking tokens in the response (some providers conceal these to protect intellectual property or only provide an anonymized summary). When thinking tokens are available, developers must decide how to handle them:

• Storage: Storing thinking tokens in the database can be valuable for auditability (for instance, to verify how tax advice was derived).
• Session context: Thinking tokens must be carefully filtered before being sent along as historical context in follow-up queries, to prevent the context window from filling up unnecessarily fast.

Conclusion

Reasoning models mark an important evolution in the architecture of language models. By transforming thinking into an explicit, measurable intermediate step powered by test-time compute, the limitations of traditional single-pass models on complex, multi-step problems are largely overcome. However, this gain in accuracy comes with higher token consumption and increased latency. Successful implementation therefore requires a thoughtful trade-off of tasks, adapted software architectures, and effective cost control.

Up next

Continue your learning journey on AI architectures and reasoning models with the following in-depth articles:

• View the overview of reasoning models — Compare the current performance and specifications of various reasoning models to make the right choice for your applications.
• Implement dynamic model routing — Learn how to optimize cost-efficiency by routing simple queries to classic models and complex tasks to reasoning models.