# Scaling laws: compute and data size

[Skip to content](#lm-inhoud)Network/[NL](/en/scaling-laws-hoe-rekenkracht-en-datagrootte-prestaties-voorspellen)EN[Hubhub.llmnet.nlCompare models on task, language, cost and licence.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs in production: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlRolling out AI in an organisation, pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, explained for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, on your own tasks.](https://benchmark.llmnet.nl/en/)[Careersvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for builders.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/en/)[](https://x.com/intent/post?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fscaling-laws-hoe-rekenkracht-en-datagrootte-prestaties-voorspellen&text=Scaling%20laws%3A%20compute%20and%20data%20size)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fscaling-laws-hoe-rekenkracht-en-datagrootte-prestaties-voorspellen)[](https://www.reddit.com/submit?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fscaling-laws-hoe-rekenkracht-en-datagrootte-prestaties-voorspellen&title=Scaling%20laws%3A%20compute%20and%20data%20size)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fscaling-laws-hoe-rekenkracht-en-datagrootte-prestaties-voorspellen&text=Scaling%20laws%3A%20compute%20and%20data%20size)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fscaling-laws-hoe-rekenkracht-en-datagrootte-prestaties-voorspellen)[](https://www.reddit.com/submit?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fscaling-laws-hoe-rekenkracht-en-datagrootte-prestaties-voorspellen&title=Scaling%20laws%3A%20compute%20and%20data%20size)[](#)

 
# Scaling laws: how compute and data size predict performance

 By Ivo Donker — compiled with AI assistance (Claude & Gemini)

 
 What you should know beforehand: This article belongs to Module 1 (Foundations). For a solid grasp of training processes, first consult the overview of [how an AI learns during the training phase](https://leren.llmnet.nl/en/hoe-leert-een-ai). It also helps to be familiar with network architectures through the primer on [parameters and weights in a model](https://leren.llmnet.nl/en/parameters-en-gewichten).

 

 When developing large language models, guesswork is not an option. Training a modern neural network costs millions of euros in hardware time and electricity. To justify those investments up front, researchers rely on empirical mathematical formulas: the so-called scaling laws . These laws describe how loss (the test loss or cross-entropy) declines predictably when three crucial dials are turned up: the number of parameters in the network, the number of training-data tokens processed, and the total compute budget expressed in floating-point operations (FLOPs).

 The discovery that model performance is not erratic chance but follows tight power laws across many orders of magnitude is the theoretical engine behind the current AI wave. This article dissects how these mathematical relationships arise, which historical shifts occurred between early assumptions and present-day training strategies, where the physical limits lie, and why the focus is moving toward inference time.

 
## The foundations: the mathematical formulation of power laws

 Scaling laws in autoregressive language models are modeled mathematically as power laws. Instead of a linear relationship between compute and performance, every additional step in quality improvement requires exponentially more resources. When we plot test loss $L$ against the number of non-embedding parameters $N$, the dataset size in tokens $D$, or the total training budget $C$, the graph shows a strikingly straight line on a double logarithmic scale (log-log plot).

 In practice, performance loss is modeled with a formula that accounts for an unavoidable minimum of noise in natural language, the so-called irreducible loss ($L_\infty$ or Bayes error):

 L(N) = (N_c / N)^(alpha_N) + L_infinity
L(D) = (D_c / D)^(alpha_D) + L_infinity
L(C) = (C_c / C)^(alpha_C) + L_infinity

 Here $N_c$, $D_c$ and $C_c$ represent scaling factors that depend on the architecture, optimizer and data distribution. The exponents $\alpha_N$, $\alpha_D$ and $\alpha_C$ determine how quickly loss falls as the respective factor increases. Because natural language contains inherent ambiguity (given earlier words, the next word can never be predicted with one hundred percent certainty), loss can never fall to zero. The value $L_\infty$ captures this inherent entropy of human language.

 Crucially, these power laws only hold reliably when the dataset is clean and representation is uniform. If you want to know exactly how raw text is converted into the discrete units these formulas operate on, read the guide on [how tokenization splits text for AI](https://leren.llmnet.nl/en/tokenisatie-uitgelegd). A faulty assumption about token distributions can substantially distort the empirical exponents in a scaling experiment.

 
## The Kaplan era: focus on model size (2020)

 In 2020, a research team led by Jared Kaplan at OpenAI published a landmark paper that would dominate the field for years. Their central conclusion was that model size ($N$) had a far stronger influence on final performance than the amount of training data ($D$). According to Kaplan's empirical fit, model builders facing a growing compute budget should invest primarily in larger networks, while dataset size only needed to grow marginally alongside.

 Concretely, Kaplan's law held that when the compute budget increases by a factor $X$, the number of parameters $N$ should grow by roughly $X^{0.73}$, while the number of tokens $D$ only needed to grow by $X^{0.27}$. This produced a clear consensus in the AI industry: build gigantic models, even if you lack enough data to train them at length.

 The best-known result of this philosophy was GPT-3, a network with 175 billion parameters trained on only 300 billion tokens. In hindsight, that ratio proved badly out of balance. The model was enormous in capacity but had seen relatively little data during its training phase, leaving a large share of the representational power of its weights unused.

 
## The Chinchilla correction from DeepMind (Hoffmann et al., 2022)

 In 2022, the DeepMind team (led by Jordan Hoffmann) showed that Kaplan's earlier conclusions rested on a methodological flaw. Kaplan and colleagues had not re-optimized the learning rate schedules for each individual compute budget, but used a fixed schedule with varying training duration. As a result, models that trained on data for a long time appeared to perform worse than was mathematically possible.

 DeepMind trained more than 400 models with varying combinations from 70 million to 16 billion parameters on 5 to 500 billion tokens. These rigorous experiments revealed a fundamentally different optimum: model size and data size should move roughly in lockstep, in a 1:1 ratio. When the compute budget increases by a factor $X$, both the number of parameters and the number of tokens should grow by about $X^{0.5}$.

 
 
 
 
 Property | 
 Kaplan approach (2020) | 
 Chinchilla optimum (2022) | 
 Current practice (over-training) | 
 

 
 
 
 Parameter growth rate ($N$) | 
 $N \propto C^{0.73}$ (fast growth) | 
 $N \propto C^{0.50}$ (balanced) | 
 Stagnation / fixed size (e.g. 8B, 70B) | 
 

 
 Token growth rate ($D$) | 
 $D \propto C^{0.27}$ (slow growth) | 
 $D \propto C^{0.50}$ (balanced) | 
 $D \propto C^{1.00+}$ (extreme data growth) | 
 

 
 Tokens per parameter ratio | 
 Roughly 1.7 to 2 tokens per parameter | 
 Roughly 20 tokens per parameter | 
 100 to 200+ tokens per parameter | 
 

 
 Example model | 
 GPT-3 (175B parameters / 300B tokens) | 
 Chinchilla (70B parameters / 1.4T tokens) | 
 Llama 3 (8B parameters / 15T tokens) | 
 

 
 
 

 The Chinchilla law showed that a 70-billion-parameter model trained on 1.4 trillion tokens consistently outperformed the more than twice as large 175-billion-parameter model that had run on only 300 billion tokens. The so-called Chinchilla ratio dictates a rule of thumb: compute-optimal pre-training requires roughly 20 tokens per model parameter.

 
## The shift toward inference optimality

 While the Chinchilla formula mathematically computes how to achieve the lowest test loss with a fixed training budget, it does not account for operational costs after training. In practice a model is trained once, but then queried billions of times through APIs and local systems.

 The industry has therefore deliberately departed from pure training-compute optimality and moved to inference-optimal scaling. Running a model with 70 billion parameters demands considerably more memory bandwidth and compute per generated token than an 8-billion-parameter model. To push inference costs down, organizations choose to train smaller models extremely long (over-training) on 15 trillion tokens or more. This costs considerably more pre-training FLOPs than Chinchilla prescribes, but yields a compact network that competes in performance with much larger architectures and is drastically cheaper in production.

 The sheer scale at which these clusters run has a direct impact on physical infrastructure, however. If you want to understand the ecological and infrastructural consequences of these training and server farms, read the analysis of [why AI compute and power consumption matter](https://nieuws.llmnet.nl/en/ai-en-energie). Continuous power and cooling questions are now just as hard a boundary as the mathematical limits of the model.

 
## Data walls: the physical limit of pre-training data

 Now that the math dictates we need tens of trillions of high-quality tokens to make full use of compact architectures, labs are running into a hard physical boundary: the so-called data wall. The total volume of publicly available, high-quality human text on the internet (books, scientific articles, Wikipedia, curated code bases) is finite and is close to exhausted.

 When a training run iterates several times over the same data (multi-epoch training), overfitting sets in quickly, causing the scaling law to stagnate and the test loss to stop following the theoretical power law. To get around this barrier, researchers are turning to synthetic data generation and advanced filtering pipelines. Training on unfiltered web scrapes introduces noise, however, which reduces the exponent $\alpha_D$. Quality therefore weighs exponentially more heavily than quantity as models grow larger.

 
## Emergence versus continuous metrics

 A much-discussed phenomenon when scaling models up is so-called emergence: the sudden appearance of complex abilities (such as mathematical reasoning, translation or code generation) that seemed entirely absent in smaller models. Initially this was thought to indicate qualitative phase transitions inside neural networks.

 Further research has since shown that emergence is largely an illusion caused by non-linear evaluation metrics. When a task is scored with an all-or-nothing metric (such as accuracy, where an answer scores 1 only if it is exactly right and 0 otherwise), the curve appears to shoot up suddenly. If the same performance is measured with continuous metrics (such as token probability or edit distance), the improvement across successive model sizes turns out to be perfectly smooth and consistent with the underlying scaling laws.

 
 
 
 
 Metric type | 
 Example | 
 Perceived behavior | 
 Actual underlying trend | 
 

 
 
 
 Discontinuous (non-linear) | 
 Exact match accuracy, 0/1 code compiling | 
 Sudden jump (emergent ability) | 
 Distorted by threshold effect | 
 

 
 Continuous (linear) | 
 Cross-entropy loss, log-likelihood, Brier score | 
 Gradual, predictable decline | 
 Follows the log-linear power law exactly | 
 

 
 
 

 
## Test-time compute: scaling during the inference phase

 Now that the returns of classic pre-training scaling laws are hitting data and energy limits, the center of gravity for scalability has shifted to the inference phase: test-time compute scaling. Instead of burning all compute up front in the weights of the base model, the model is allotted extra compute time at the moment it answers a question.

 Through techniques such as chain-of-thought, tree-search algorithms (such as Monte Carlo Tree Search) and self-correction loops, a more compact model can explore and verify different reasoning paths during inference. This introduces an entirely new scaling law: performance scales logarithmically with the amount of compute deployed per token during generation. As a result, a smaller base model can outperform larger models on complex mathematical or logical tasks purely by "thinking" longer before formulating a final answer.

 
## Methodology: how researchers measure scaling laws

 Determining the exact exponents ($\alpha$) for a specific architecture requires a strict experimental setup. Researchers typically follow a standardized procedure to keep noise or suboptimal hyperparameters from contaminating the measurements:

 First, one trains a family of small models, ranging from 10 million to roughly 1 billion parameters, on different subsets of data. In each experiment the learning rate is optimized through a grid search and tied to a cosine decay schedule that ends exactly at the planned number of tokens. The resulting test losses are then plotted on a log-log diagram.

 Non-linear least-squares optimization (fitting) is used to estimate the parameters $\alpha_N$, $\alpha_D$, $N_c$ and $D_c$. If the fit remains stable across three or four orders of magnitude, researchers extrapolate the line to the intended production model (for example 70B or 400B parameters). This makes it possible to predict with great precision what loss the final model will reach and what budget in GPU hours has to be reserved for it.

 
## Limitations and blind spots of classic scaling laws

 While scaling laws are an indispensable compass, they carry considerable limitations that engineers should not lose sight of:

 First, a low cross-entropy loss does not automatically predict a safe or usable model. A neural network can achieve an exceptionally low test loss on general web text and still hallucinate, be vulnerable to prompt injection, or fail on basic instructions. Scaling laws measure purely the statistical predictive power of the next token, not factual accuracy or reasoning ability after fine-tuning.

 Second, standard scaling laws assume a uniform data distribution. As soon as the data mix changes (by adding more code or specialist mathematical sources, for instance), the exponents shift. Finally, classic formulas take no account of hardware efficiency: networks that scale optimally in theory in terms of FLOPs can in practice run into memory bandwidth bottlenecks (memory-bound operations) on specific GPU clusters, making actual training time per token deviate sharply from the mathematical projection.

 
 Continue with: Now that you understand how models scale on compute and data, you can dig deeper into how these parameters are organized internally. Read how the computational load is distributed across multiple circuits in the in-depth article on [Mixture-of-Experts (MoE) architectures](https://leren.llmnet.nl/en/mixture-of-experts-uitgelegd), or find out how to shrink models after the scaling phase in the guide on [quantization with INT4 and FP8](https://leren.llmnet.nl/en/kwantisatie-uitgelegd-hoe-modellen-krimpen-met-int4-en-fp8).