Why do Transformer models, like BERT, use the GELU function instead of the ReLU function? According to an article published by Analytics Vidhya, the answer lies in how each processes information. While ReLU completely discards negative inputs, GELU applies a smooth mechanism, dependent on the value of each input, to decide what portion of the information should pass through.
GELU, for Gaussian Error Linear Unit, is a non-linear activation function introduced in 2016 by Dan Hendrycks and Kevin Gimpel. An activation function determines how a neuron transforms its input before passing it to the next layer. Without it, stacking linear layers would only produce an overall linear transformation, limiting what a neural network can learn.
GELU does not apply a strict threshold. It scales each input based on its value. Strong positive values pass through almost unchanged, values near zero are partially attenuated, and strong negative values are pulled towards zero. Moderately negative values can retain a small negative output. The article notes that the original GELU paper evaluated the function in computer vision, natural language processing, and speech recognition, and reported improvements over several traditional functions in its experiments.
GELU vs. ReLU
ReLU, for Rectified Linear Unit, returns 0 for a negative input and the value itself for a positive input. So, for an input of -1, it returns 0, and for an input of 2, it returns 2. GELU behaves differently: an input of -1 yields approximately -0.159, and an input of 2 yields about 1.954. The function is therefore not strictly increasing over its entire domain, its negative region showing a slight dip before approaching zero for very negative values.
GELU is written as x × Φ(x), where Φ(x) is the cumulative distribution function of the standard normal distribution, i.e., a distribution with zero mean and unit standard deviation. This function is expressed using the error function erf. For an input of 1, the value of Φ is approximately 0.8413: the positive signal is thus largely preserved, while the negative signal of -1 is heavily reduced. The text specifies that GELU is deterministic. It does not randomly drop neuron outputs during execution, even though its operation is sometimes described as a probabilistic filtering mechanism.
ReLU remains popular because it is simple and computationally inexpensive. GELU offers a smoother transformation, but its mathematical form requires more computation. The article further emphasizes that neither is automatically better: the model architecture, dataset, hardware, and implementation all influence the final outcome.
Why GELU is Present in Transformers
A Transformer, the architecture underlying BERT and many large language models, combines attention mechanisms, normalization layers, and so-called feedforward neural networks, or FFNs. GELU is commonly used in the FFN, not in the computation of attention probabilities. In the BERT architecture, it is the default activation function for hidden layers, as indicated by Hugging Face’s BERT configuration. It also appears in vision Transformers, but some recent models use gated variants based on SiLU, such as SwiGLU.
The exact formula for GELU involves the Gaussian cumulative distribution function, which is more expensive to compute than simpler functions. The article presents a common approximation based on the hyperbolic tangent function, which produces values close to exact GELU. PyTorch allows the use of both formulations via its approximate parameter.
A Python Experiment
The author compares two identical networks, one with ReLU and the other with GELU, on scikit-learn’s make_moons dataset, consisting of 2,000 samples. Both models share the same architecture, the same Adam optimizer, the same learning rate of 0.001, and the same random seed, over 100 epochs. The article cautions that a single run is not enough to establish the best function: it recommends repeating the experiment with multiple random seeds and reserving the test set for final evaluation. Its goal is not to demonstrate that GELU is universally superior.
For a pre-trained model, the text generally recommends keeping the original activation function. Replacing GELU with ReLU in a pre-trained Transformer alters its computations and may degrade its performance, unless the modified model is properly retrained or fine-tuned.
