Table of Contents
What a Loss Function Does
A loss function turns the model’s predictions into a single number that measures how wrong those predictions are compared to the targets. Training then tries to minimize this number by adjusting parameters using gradients. In practice, the loss defines what it means to be correct, and it strongly influences how stable and effective training will be.
A typical supervised learning setup has model outputs $\hat{y}$ and targets $y$. The loss computes $L(\hat{y}, y)$, often first per example and then aggregated across a batch, usually by a mean.
A loss function is not just a reporting metric. It is the objective you differentiate. If you choose a loss that does not match your task or your output layer, training can fail even if your accuracy metric looks sensible.
Regression Losses
For regression, the target is a real value, and the model output is usually a real number with no activation at the end.
Mean squared error, often called MSE, penalizes large errors more strongly. For one example it is $ (\hat{y} - y)^2 $, and for a batch it is commonly
$$
\text{MSE} = \frac{1}{N}\sum_{i=1}^N (\hat{y}_i - y_i)^2.
$$
In PyTorch you typically use torch.nn.MSELoss().
Mean absolute error, often called MAE or L1 loss, is
$$
\text{MAE} = \frac{1}{N}\sum_{i=1}^N |\hat{y}_i - y_i|.
$$
It is more robust to outliers than MSE but has a non smooth point at zero error. In PyTorch this is torch.nn.L1Loss().
Huber loss is a compromise that is quadratic near zero and linear for large errors. It can be a good default when you suspect outliers but still want stable gradients. In PyTorch, torch.nn.SmoothL1Loss() is commonly used for this idea.
For basic regression in PyTorch, do not apply a sigmoid or softmax to the final output. Use a linear output and a regression loss like MSELoss, L1Loss, or SmoothL1Loss.
Binary Classification Losses
Binary classification targets are usually $0$ or $1$. You will see two common ways to structure the model output and loss.
The most numerically stable approach is to output a single logit, which is an unconstrained real number, and use torch.nn.BCEWithLogitsLoss(). This loss internally applies the sigmoid in a stable way. Conceptually it corresponds to binary cross entropy with $\sigma(z)$ where $z$ is the logit and $\sigma$ is the sigmoid.
An alternative is to apply a sigmoid yourself and then use torch.nn.BCELoss(), but this is easier to get wrong and can be less stable.
Use BCEWithLogitsLoss with raw logits. Do not apply torch.sigmoid before it, or you effectively apply sigmoid twice and training will behave poorly.
For shapes, a common convention is predictions of shape [batch_size] or [batch_size, 1], and targets with the same shape and floating dtype.
Multiclass Classification Losses
For multiclass problems with $K$ classes, the standard choice is cross entropy. In PyTorch this is torch.nn.CrossEntropyLoss(). It expects raw logits of shape [batch_size, K] and integer class indices of shape [batch_size].
Conceptually, cross entropy combines a softmax and a negative log likelihood. If logits are $z$, softmax probabilities are
$$
p_k = \frac{e^{z_k}}{\sum_{j=1}^K e^{z_j}}.
$$
The loss for a single example with true class $c$ is
$$
L = -\log p_c.
$$
PyTorch computes this in a numerically stable way directly from logits.
With CrossEntropyLoss, do not apply softmax in your model. Pass logits directly. Also, targets must be class indices, not one hot vectors, unless you are explicitly using a different setup.
If you have one hot labels, convert them to class indices with targets.argmax(dim=1) before using CrossEntropyLoss, assuming the one hot vectors are valid.
Multilabel Classification Losses
Multilabel classification means each example can have multiple independent labels, like tags. Here you usually output one logit per label, shape [batch_size, K], and use torch.nn.BCEWithLogitsLoss() with targets of floats in {0, 1} with the same shape.
This is different from multiclass classification. Multiclass uses one label out of many, and uses CrossEntropyLoss. Multilabel uses multiple independent binary decisions, and uses BCEWithLogitsLoss.
Reduction, Batch Aggregation, and Loss Scale
Most PyTorch loss modules have a reduction argument, usually "mean", "sum", or "none". With "mean", the batch loss is averaged, which makes the scale less sensitive to batch size. With "sum", the loss grows with batch size, which can require changing the learning rate. With "none", you get per example losses and you can apply custom weighting.
If you change reduction from "mean" to "sum", you often must adjust the learning rate or gradient accumulation settings, because gradient magnitudes change.
Handling Class Imbalance with Loss Weighting
When some classes are rare, a model can minimize loss by mostly predicting the majority class. Many loss functions allow weighting.
For CrossEntropyLoss, you can pass weight= as a tensor of shape [K] to upweight rare classes.
For BCEWithLogitsLoss, you can use pos_weight= to upweight positive examples per label, or weight= for per element weighting.
The key idea is that weighting changes how much each error contributes to the final objective. This is often a cleaner first step than changing your metric or threshold.
Label Smoothing and When Not to Use It
Label smoothing slightly softens hard targets in multiclass classification, which can improve generalization and calibration. In PyTorch, CrossEntropyLoss supports label_smoothing= in recent versions.
It is not always beneficial, and you generally skip it when you need very confident outputs for downstream decisions or when the dataset labels are already uncertain in a structured way that needs different treatment.
Picking the Right Loss for the Output You Produce
A simple matching guide is based on what the model outputs represent.
If the model outputs an unconstrained real value and the target is real, use regression losses.
If the model outputs one logit and the target is 0 or 1, use BCEWithLogitsLoss.
If the model outputs $K$ logits and exactly one class is correct, use CrossEntropyLoss.
If the model outputs $K$ logits and any subset of classes can be correct, use BCEWithLogitsLoss with shape [batch, K].
Always decide the loss and the target format together. Many training bugs come from a mismatch among activation, logits, and target dtype or shape.
A Minimal PyTorch Pattern
In code, the typical pattern is to create the loss module once and call it inside the training step.
For multiclass classification, it looks like calling loss_fn(logits, targets) where logits are [N, K] floats and targets are [N] long integers.
For binary or multilabel classification, it looks like calling loss_fn(logits, targets_float) where logits are raw and targets are float tensors with the same shape.
For regression, it is loss_fn(preds, targets) where both are float tensors of matching shape.
The details of the training loop, optimizer steps, and backpropagation belong to the training loop chapters, but choosing and wiring the correct loss is the core step that makes those loops work.