Sigmoid and Tanh

Smooth bounded activations

The sigmoid maps any input to (0,1), making it natural for a binary output interpreted as probability. The hyperbolic tangent (tanh) maps to (-1,1) and centers outputs around zero. Both are smooth and historically important.

input:    -4    -1     0     1     4
sigmoid: .02   .27   .50   .73   .98
tanh:   -.99  -.76   .00   .76   .99

At large positive or negative inputs, these curves saturate: output changes very little, so derivatives become near zero. During backpropagation, repeatedly multiplying by tiny derivatives makes gradients fade through many layers - the vanishing gradient problem.

Analogy: A saturated volume knob is already at its stop. Push harder and output barely changes, so the system receives almost no signal about how to adjust earlier controls.

Tip: Use sigmoid for a binary output probability. Hidden layers more commonly use ReLU-family activations because their useful gradient does not shrink across the whole positive range.