Table of Contents
What a Linear Layer Computes
A linear layer in a neural network applies an affine transform to an input vector. “Affine” means it is a linear transformation plus a bias shift. For one input example with $d$ features, written as a row vector $x \in \mathbb{R}^{d}$, a linear layer with $m$ outputs computes
$$
y = xW^T + b
$$
where $W \in \mathbb{R}^{m \times d}$ is the weight matrix and $b \in \mathbb{R}^{m}$ is the bias vector. Each output unit is a weighted sum of the inputs plus its own bias.
For a batch of inputs $X \in \mathbb{R}^{N \times d}$, the computation becomes
$$
Y = XW^T + b
$$
with $Y \in \mathbb{R}^{N \times m}$. The bias $b$ is broadcast across the batch dimension, so each row in $Y$ gets the same $b$ added.
A “linear layer” in deep learning is almost always an affine transform: it uses both weights and a bias. If you set the bias to false, then it becomes a purely linear map $y = xW^T$.
How This Maps to `torch.nn.Linear`
In PyTorch, nn.Linear(in_features, out_features, bias=True) implements exactly this affine transform. If you create
import torch
import torch.nn as nn
layer = nn.Linear(in_features=4, out_features=3, bias=True)then the parameter shapes are
layer.weight.shape # (3, 4)
layer.bias.shape # (3,)
PyTorch stores weight as (out_features, in_features), which matches the math form $W \in \mathbb{R}^{m \times d}$. During the forward pass, given an input x shaped (N, 4), the output will be shaped (N, 3).
Shape rule to memorize: for nn.Linear(d, m), inputs are (..., d) and outputs are (..., m). The last dimension is the feature dimension that gets transformed.
Batches and “Any Number of Leading Dimensions”
A useful PyTorch feature is that nn.Linear works with inputs that have more than two dimensions, as long as the last dimension equals in_features. For example, if x is shaped (batch, time, features) = (N, T, d), then
x = torch.randn(32, 10, 4)
y = layer(x)
y.shape # (32, 10, 3)The layer treats everything except the last dimension as batch-like dimensions and applies the same affine transform independently at every position.
Manual Computation With `matmul`
Sometimes it helps to see the layer as explicit tensor operations. For a batched input X shaped (N, d):
X = torch.randn(5, 4)
W = layer.weight # (3, 4)
b = layer.bias # (3,)
Y_manual = X @ W.T + b # (5, 3)
Y_layer = layer(X)
torch.allclose(Y_manual, Y_layer)This equivalence is a good debugging tool when you suspect a shape mismatch or an unexpected broadcast.
If you write the computation manually, remember to use W.T because PyTorch stores weight as (out_features, in_features), while X @ W.T expects W.T shaped (in_features, out_features).
Interpreting Weights and Biases
Each output unit corresponds to one row of the weight matrix and one bias value. If $W_i$ is the $i$th row of $W$ and $b_i$ the $i$th bias entry, then the $i$th output for a single example is
$$
y_i = \sum_{j=1}^{d} x_j W_{i,j} + b_i
$$
This is why linear layers are often described as “weighted sums plus bias.” The weights control how strongly each input feature contributes to each output unit, and the bias shifts the output baseline.
When You Might Disable the Bias
You might set bias=False when the next operation already includes a learnable shift, or when you want the model to have fewer parameters. A common example is using normalization layers that include their own offset parameter. The key practical point is that turning off bias changes the hypothesis class, because it forces the mapping to pass through the origin.
Common Pitfalls Specific to Linear Layers
A frequent source of bugs is mixing up feature dimensions. Beginners often accidentally pass tensors shaped (d, N) instead of (N, d). In PyTorch, the default convention is that the batch dimension comes first, so a typical tabular batch is (batch, features).
Another common issue is flattening too much. If you have structured inputs, you often need to reshape them so that the last dimension matches in_features. For example, if you want to feed an image tensor (N, C, H, W) into a linear layer, you must first convert it into (N, C·H·W) by flattening per example. The details of reshaping and working with images belong elsewhere, but the linear layer requirement remains the same: the last dimension must equal in_features.
If you see an error like “mat1 and mat2 shapes cannot be multiplied,” check the last dimension of your input and confirm it matches in_features.
Summary of the Core Rules
A linear layer computes an affine transform $y = xW^T + b$, applied to the last dimension of the input. In PyTorch, layer.weight has shape (out_features, in_features), outputs replace the last dimension with out_features, and the bias broadcasts across all leading dimensions.