Activation Functions in LLMs: Why SwiGLU and GELU Replaced ReLU

In the early days of deep neural networks, the Rectified Linear Unit (ReLU) was the undisputed standard. It replaced traditional sigmoid and hyperbolic tangent functions because it resolved the notorious vanishing gradient problem for positive activations and was computationally extremely cheap. However, if you open the configuration files of modern open-weight and commercial language models—from LLaMA and Mistral to Gemma and GPT variants—you will hardly find standard ReLU anywhere. Instead, functions like GELU (Gaussian Error Linear Unit) and GLU derivatives such as SwiGLU dominate the feed-forward networks (FFNs) across every transformer layer.

This shift is neither an arbitrary trend nor a cosmetic optimization. Selecting a specific activation function fundamentally determines how gradients backpropagate during training, how neurons decide which representations to pass through, and how many floating-point operations (FLOPs) are required to process a token. In this article, we dissect the mathematical limitations of ReLU, explore why smooth curvatures like GELU yield richer representations, and analyze why multiplicative gating mechanisms such as SwiGLU have become the current industry standard.

The Role of Non-Linearity in Transformer Architectures

Without non-linear activation functions, a neural network—regardless of its depth or width—mathematically collapses into a single linear transformation. After all, executing ten matrix multiplications in sequence, W_n · ... · W_2 · W_1 · x, is algebraically identical to a single multiplication with a composite matrix W_total. To model complex linguistic relationships, syntax, logic, and abstract contexts, a network must act as a continuous, non-linear function approximator.

Within a classic transformer layer, the activation function resides in the Feed-Forward Network (also known as the MLP layer), which directly follows the multi-head attention mechanism. While attention enables tokens to exchange information across the entire context window, the FFN is responsible for token-wise processing and knowledge storage. Typically, a first matrix W_1 projects the hidden dimension d_model to a wider intermediate space (traditionally 4 × d_model), after which the activation function σ is applied element-wise, before a second matrix W_2 compresses the dimension back down to d_model:

FFN(x) = \sigma(x W_1 + b_1) W_2 + b_2

The activation function σ acts as a fine-grained filter here. It determines which features remain active in the expanded vector space and with what intensity they contribute to the final token representation. How this filter behaves at the boundaries and for negative values dictates the learning dynamics of the entire model.

The rise and shortcomings of ReLU in large models

ReLU is defined as the maximum of zero and the input value: f(x) = max(0, x). Its derivative is trivial: 1 when x > 0, and 0 when x < 0. This simplicity brought massive breakthroughs to convolutional networks and early sequence models because forward and backward passes required minimal computational power. Nevertheless, modern language models with billions of parameters run into three fundamental issues when using pure ReLU.

The first problem is the 'Dying ReLU' phenomenon. When a neural network trains with high learning rates or large batches, a neuron's weights can be updated in such a way that the input value x becomes negative for virtually all representative training examples. Because the gradient of ReLU for negative inputs is exactly zero (f'(x) = 0), no error signal can flow back to this neuron via backpropagation. The neuron effectively 'dies' and remains permanently inactive throughout the remainder of training, resulting in wasted model capacity.

The second bottleneck is the lack of differentiability at the origin. The abrupt transition from 0 to x introduces a hard kink at x = 0. In deep networks with dozens of layers, this creates sharp discontinuities in the optimization landscape. As a result, optimization algorithms like AdamW have a harder time navigating toward stable, flat minima, increasing the risk of instabilities during large-scale training runs.

Finally, ReLU enforces strict 'hard sparsity': all negative activations are pushed strictly to zero. Although sparsity can theoretically offer memory benefits, language modeling shows that subtle, negative activation values often contain valuable contextual information about what a concept is precisely not representing. By abruptly wiping these values out, the network loses gradual expressiveness.

GELU: probabilistic activation and soft gating

To soften the hard threshold of ReLU, Dan Hendrycks and Kevin Gimpel introduced the Gaussian Error Linear Unit (GELU) in 2016. GELU bridges the deterministic nature of ReLU with probabilistic regularization like dropout. Instead of applying a binary gate based on the sign of x, GELU weights the input value by the probability that the input is greater than a random variable from a standard normal distribution.

Mathematically, GELU is defined using the cumulative distribution function (CDF) of the standard normal distribution Φ(x):

GELU(x) = x \cdot \Phi(x) = x \cdot P(X \le x), \quad \text{waarbij } X \sim \mathcal{N}(0, 1)

Written out with the Gaussian error function erf(x), this yields:

GELU(x) = 0.5 \cdot x \cdot \left(1 + \text{erf}\left(\frac{x}{\sqrt{2}}\right)\right)

Because directly computing the error function erf(x) is computationally heavy on GPUs, frameworks in practice often use a fast polynomial tanh approximation:

GELU(x) \approx 0.5 \cdot x \cdot \left(1 + \tanh\left(\sqrt{\frac{2}{\pi}} \cdot \left(x + 0.044715 \cdot x^3\right)\right)\right)

The crucial advantage of GELU is that the function is non-monotonic and continuously differentiable everywhere. For strongly negative values, the function converges to zero, but for slightly negative values (between 0 and -2), the function dips below the x-axis to a minimum of approximately -0.17. As a result, small negative gradients are preserved during the backpropagation phase. Models such as BERT, GPT-2, and GPT-3 defaulted to GELU because it demonstrably led to faster convergence and lower validation loss on large-scale datasets.

The principle of Gated Linear Units (GLU)

Although GELU solved the gradient issue of ReLU, the structure of the FFN layer remained linearly tied to the activation: one multiplies by a weight matrix, applies the function, and multiplies again. In 2016, Dauphin et al. introduced the Gated Linear Unit (GLU) as an alternative approach for language modeling.

A GLU splits the transformation into two parallel linear projections whose elements are multiplied element-wise (Hadamard product, ⊗). One branch acts as the signal path, while the other branch acts as a gate that regulates how much of the signal is allowed to pass:

GLU(x, W, V, b, c) = (x W + b) \otimes \sigma(x V + c)

In the classic GLU, σ is the standard sigmoid. The revolutionary insight behind this is that the gradient of the gating function scales directly with the activation of the signal path, and vice versa. Instead of a static threshold that operates the same way for every input element, the network dynamically learns per input token how strictly to open or close the gate. This significantly increases the expressiveness of the layer without increasing the overall depth of the model.

SwiGLU dissected: why modern models choose Swish-Gated units

In 2020, Noam Shazeer (then a researcher at Google) published the influential paper "GLU Variants Improve Transformer". Shazeer systematically investigated what happens when the traditional sigmoid in a GLU is replaced with modern non-linear functions such as ReLU, GELU, and Swish (also referred to as SiLU, where Swish(x) = x · sigmoid(β x)). Comprehensive benchmarks revealed that the variant named SwiGLU consistently outperformed all alternatives.

The mathematical definition of a SwiGLU FFN layer is as follows (bias terms are often omitted in modern architectures):

\text{SwiGLU}(x) = \left(\text{Swish}_1(x W_{gate}) \otimes (x W_{up})\right) W_{down}

Here, three distinct matrix projections take place:

• Gate projection (W_gate): Projects the vector into the intermediate space and is subsequently transformed by the Swish function.
• Up projection (W_up): Projects the same input vector in parallel into the intermediate space without an immediate non-linearity.
• Down projection (W_down): Projects the element-wise product of the activated gate and the up-projection back to the original model dimension d_model.

In Python with PyTorch, a clean implementation of a SwiGLU-based MLP looks compact:

import torch
import torch.nn as nn
import torch.nn.functional as F

class SwiGLUFeedForward(nn.Module):
  def __init__(self, d_model: int, d_ff: int):
    super().__init__()
    # Drie lineaire projecties zonder bias
    self.w_gate = nn.Linear(d_model, d_ff, bias=False)
    self.w_up = nn.Linear(d_model, d_ff, bias=False)
    self.w_down = nn.Linear(d_ff, d_model, bias=False)

  def forward(self, x: torch.Tensor) -> torch.Tensor:
    # Swish(x * W_gate) * (x * W_up)
    gate = F.silu(self.w_gate(x))
    up = self.w_up(x)
    return self.w_down(gate * up)

The mathematical synergy between the soft truncation of SiLU and the multiplicative interaction of the linear up-projection enables the network to learn very fine-grained decision boundaries. When the model formulates facts or reasoning steps during inference, it can almost completely suppress specific representation channels without permanently pinching off gradient flow during the training phase.

The computational trade-off: FLOPs, memory, and parameter compensation

An intuitive drawback of SwiGLU is the introduction of a third weight matrix. Whereas a traditional GELU or ReLU layer uses two matrices (W_1 and W_2), SwiGLU requires W_gate, W_up, and W_down. If one were to keep the intermediate dimension d_ff equal to the classic factor of 4 × d_model, the parameter count and required compute in the FFN layer would increase by 50% (from 8 d_model^2 to 12 d_model^2).

To enable a fair comparison and keep the parameter count and FLOPs identical to a standard transformer, architects scale down the intermediate dimension d_ff in SwiGLU. Instead of an expansion factor of 4, a factor of approximately 8/3 ≈ 2.67 × d_model is typically chosen (often rounded to the nearest multiple of 64 or 256 for optimal GPU matrix multiplication via Tensor Cores).

Even when the total parameter count is kept exactly identical through this rescaling, SwiGLU achieves significantly lower perplexity and better downstream benchmark scores than a standard 4x-GELU FFN. The computational overhead per FLOP thus translates into higher statistical efficiency per processed token.

Impact on training stability and hardware efficiency

Beyond model capacity, training stability and hardware execution play a decisive role in the choice of an activation function. In models trained on trillions of tokens, numerical instabilities (such as loss spikes or floating-point overflows) can cause millions of euros in wasted compute. Once we run models in production environments, it is essential to understand what happens under the hood during inference, especially since memory bandwidth and kernel fusion determine the final throughput speed.

SwiGLU requires intermediate activation vectors to be retained in fast GPU SRAM (Static Random-Access Memory) to perform the element-wise multiplication. Modern compilation tools and optimized kernels (such as Triton or CUDA-fused kernels) combine the SiLU computation, the up-projection, and the multiplication into a single memory pass. As a result, the theoretical memory latency of the extra matrix operation is virtually eliminated.

Furthermore, the smooth derivative of SiLU/SwiGLU ensures that gradient values during mixed-precision training (with FP16 or BF16) are less prone to underflowing to zero or exploding to infinity, reducing the need for aggressive gradient clipping and enabling more stable learning rate schedules.

Practical comparison across modern architectures

The table below illustrates how leading transformer architectures have transitioned to advanced activation functions and corresponding FFN dimensions in recent years:

The transition to SwiGLU without bias terms has now been universally adopted by virtually all leading open model families. Omitting bias vectors simplifies hardware optimization and reduces the risk of representational drift in very long context windows.

The role of activations in model reliability and evaluation

Although activation functions are primarily architectural components, they indirectly influence how models reason and maintain factual consistency. A model with a smoother representation space can separate concepts with greater nuance, reducing the risk of subtle representational errors. When evaluating outcomes in practice, the methodology for systematically fact-checking AI responses helps verify whether internal representations actually lead to consistent output.

The shift from ReLU to GELU and ultimately SwiGLU shows that architectural progress in deep learning systems is not merely about blindly adding more compute or larger datasets. By refining the mathematical flow of signals and gradients through the network via non-linear gating mechanisms, modern LLMs achieve significantly better performance within the same hardware and compute budget.