Table of Contents
What an Activation Function Does
An activation function is the non linear step that you place between linear layers so a neural network can model non linear relationships. If you stack only linear layers, the whole network collapses into a single linear transformation, no matter how many layers you add. Adding an activation after a linear layer breaks that limitation.
In a typical feedforward block you will see a pattern like $h = \phi(Wx + b)$, where $Wx + b$ is produced by a linear layer and $\phi(\cdot)$ is the activation function applied elementwise.
If you do not include non linear activations between linear layers, your network is effectively a linear model and cannot learn non linear decision boundaries.
The Most Common Activations in Practice
ReLU, written as $\mathrm{ReLU}(x) = \max(0, x)$, is the default choice for many beginner models because it is simple and usually trains well. ReLU is typically used in hidden layers, not in the final output layer.
LeakyReLU is a small variation that keeps a small slope for negative inputs, commonly $\mathrm{LeakyReLU}(x)=\max(x, \alpha x)$ with a small $\alpha$ like $0.01$. It can reduce cases where units stop updating because they only see negative inputs.
GELU is popular in transformer style models. You will often see it as a drop in replacement for ReLU in modern architectures. For beginners, the key takeaway is that GELU behaves like a smoother ReLU and is used mainly because it tends to work well in certain deep models.
Sigmoid, written as $\sigma(x)=\frac{1}{1+e^{-x}}$, maps values to $(0,1)$. It is mostly used to represent probabilities for binary outputs, rather than as a hidden layer activation in modern deep networks.
Tanh, written as $\tanh(x)$, maps values to $(-1,1)$. It appears in some recurrent networks and older architectures, and is less common than ReLU family activations for generic hidden layers today.
Softmax converts a vector of scores into a probability distribution across classes. For a vector $z$, $$\mathrm{softmax}(z_i)=\frac{e^{z_i}}{\sum_j e^{z_j}}.$$ Softmax is used at the output for multi class classification when you want normalized class probabilities.
Do not add a softmax layer at the end if you plan to use torch.nn.CrossEntropyLoss. That loss expects raw, unnormalized logits and applies the correct normalization internally.
Choosing the Right Activation for the Output Layer
The activation you use at the final layer depends on what the model is supposed to predict. For regression, the final layer is often left with no activation so it can output any real number. For non negative targets you might choose an activation that enforces non negativity, but that is a modeling decision tied to the data.
For binary classification, you commonly output one logit per example and use a sigmoid only when you need probabilities. During training, it is typical to keep the output as a raw logit and use torch.nn.BCEWithLogitsLoss, which is numerically stable.
For multi class classification, you typically output $C$ logits for $C$ classes and train with torch.nn.CrossEntropyLoss. You can convert logits to probabilities later with softmax when you want to interpret results.
For training stability, prefer losses that combine the activation with the loss in one function, such as BCEWithLogitsLoss and CrossEntropyLoss, instead of manually applying sigmoid or softmax before the loss.
Activation Functions in PyTorch Code
In PyTorch you can use activations as layers in torch.nn or as functional calls in torch.nn.functional. Layer style is convenient inside nn.Sequential and when you want a clean module definition, functional style can be convenient inside a custom forward.
A common pattern for a small network is to alternate nn.Linear and an activation like nn.ReLU(), then end with a final nn.Linear that produces logits or regression outputs. The activation itself typically has no trainable parameters, with the exception of variants like PReLU which learn the negative slope.
Practical Rules of Thumb for Beginners
ReLU is a safe default for hidden layers in basic feedforward and CNN models. If training seems to stall and many activations might be stuck at zero, trying LeakyReLU can be a simple experiment.
Use sigmoid only to interpret binary probabilities, not necessarily inside the model before a logits based loss. Use softmax only for interpreting multi class probabilities, not before cross entropy loss.
If you are unsure, decide the final layer based on the target type, then start with ReLU in the hidden layers and iterate from there.
Hidden layer default: ReLU or a close variant. Output layer: match the task, and feed logits into BCEWithLogitsLoss or CrossEntropyLoss rather than applying sigmoid or softmax before the loss.