ReLU and Its Variants

Keep positive evidence, zero negative evidence

The rectified linear unit (ReLU) returns max(0, x): negative inputs become zero and positive inputs pass through unchanged. It is cheap, nonlinear, and has derivative 1 on its positive side, helping deep networks train more effectively than stacks of saturated sigmoids.

x:       -2   -1    0    1    2
ReLU(x):  0    0    0    1    2

A unit can become a dead ReLU if updates leave it negative for every input, producing zero output and zero gradient forever. Leaky ReLU keeps a small negative slope, while smoother variants such as GELU are common in Transformers. Each variant changes optimization behavior more than the basic purpose: inject useful nonlinearity.

Scenario: A very high learning rate pushes many hidden-unit biases far negative. Those ReLUs stop activating, capacity collapses, and training stalls. Reducing the learning rate or using a nonzero negative slope can help.

Warning: ReLU outputs are not probabilities. Their range is unbounded above; output-layer activation must match the task and loss.