Activation Functions

Without an activation function, a neural network is just stacked linear algebra that can only ever draw a straight line through data. This paper walks through the five functions that introduce the non-linearity deep learning actually depends on — what each one computes, why it was adopted, and where it breaks down in practice.

• Sigmoid squashes any input into (0, 1), which is great for binary probabilities but suffers from vanishing gradients on large or small inputs.
• Tanh is sigmoid's zero-centered sibling, mapping to (-1, 1) and training faster — though it shares the same vanishing-gradient weakness.
• ReLU is the default for hidden layers today: fast, simple, and non-saturating for positive inputs, at the cost of neurons that can permanently 'die' on negative inputs.
• Leaky ReLU patches the dying-neuron problem by letting a small negative slope through instead of flattening to zero.
• Softmax turns a layer's raw output scores into a normalized probability distribution, which is why it's reserved for the final classification layer.