3.1. Linear Layers and Affine Transforms
Table of Contents
What a Linear Layer Computes
A linear layer in a neural network applies an affine transform to an input vector. “Affine” means it is a linear transformation plus a bias shift. For one input example with $d$ features, written as a row vector $x \in \mathbb{R}^{d}$, a linear layer with $m$ outputs computes
$$
y = xW^T + b
$$
where $W \in \mathbb{R}^{m \times d}$ is the weight matrix and $b \in \mathbb{R}^{m}$ is the bias vector. Each output unit is a weighted sum of the inputs plus its own bias.
For a batch of inputs $X \in \mathbb{R}^{N \times d}$, the computation becomes
$$
Y = XW^T + b
$$
with $Y \in \mathbb{R}^{N \times m}$. The bias $b$ is broadcast across the batch dimension, so each row in $Y$ gets the same $b$ added.
A “linear layer” in deep learning is almost always an affine transform: it uses both weights and a bias. If you set the bias to false, then it becomes a purely linear map $y = xW^T$.
How This Maps to `torch.nn.Linear`
In PyTorch, nn.Linear(in_features, out_features, bias=True) implements exactly this affine transform. If you create
import torch
import torch.nn as nn
layer = nn.Linear(in_features=4, out_features=3, bias=True)then the parameter shapes are
layer.weight.shape # (3, 4)
layer.bias.shape # (3,)
PyTorch stores weight as (out_features, in_features), which matches the math form $W \in \mathbb{R}^{m \times d}$. During the forward pass, given an input x shaped (N, 4), the output will be shaped (N, 3).
Shape rule to memorize: for nn.Linear(d, m), inputs are (..., d) and outputs are (..., m). The last dimension is the feature dimension that gets transformed.
Batches and “Any Number of Leading Dimensions”
A useful PyTorch feature is that nn.Linear works with inputs that have more than two dimensions, as long as the last dimension equals in_features. For example, if x is shaped (batch, time, features) = (N, T, d), then
x = torch.randn(32, 10, 4)
y = layer(x)
y.shape # (32, 10, 3)The layer treats everything except the last dimension as batch-like dimensions and applies the same affine transform independently at every position.
Manual Computation With `matmul`
Sometimes it helps to see the layer as explicit tensor operations. For a batched input X shaped (N, d):
X = torch.randn(5, 4)
W = layer.weight # (3, 4)
b = layer.bias # (3,)
Y_manual = X @ W.T + b # (5, 3)
Y_layer = layer(X)
torch.allclose(Y_manual, Y_layer)This equivalence is a good debugging tool when you suspect a shape mismatch or an unexpected broadcast.
If you write the computation manually, remember to use W.T because PyTorch stores weight as (out_features, in_features), while X @ W.T expects W.T shaped (in_features, out_features).
Interpreting Weights and Biases
Each output unit corresponds to one row of the weight matrix and one bias value. If $W_i$ is the $i$th row of $W$ and $b_i$ the $i$th bias entry, then the $i$th output for a single example is
$$
y_i = \sum_{j=1}^{d} x_j W_{i,j} + b_i
$$
This is why linear layers are often described as “weighted sums plus bias.” The weights control how strongly each input feature contributes to each output unit, and the bias shifts the output baseline.
When You Might Disable the Bias
You might set bias=False when the next operation already includes a learnable shift, or when you want the model to have fewer parameters. A common example is using normalization layers that include their own offset parameter. The key practical point is that turning off bias changes the hypothesis class, because it forces the mapping to pass through the origin.
Common Pitfalls Specific to Linear Layers
A frequent source of bugs is mixing up feature dimensions. Beginners often accidentally pass tensors shaped (d, N) instead of (N, d). In PyTorch, the default convention is that the batch dimension comes first, so a typical tabular batch is (batch, features).
Another common issue is flattening too much. If you have structured inputs, you often need to reshape them so that the last dimension matches in_features. For example, if you want to feed an image tensor (N, C, H, W) into a linear layer, you must first convert it into (N, C·H·W) by flattening per example. The details of reshaping and working with images belong elsewhere, but the linear layer requirement remains the same: the last dimension must equal in_features.
If you see an error like “mat1 and mat2 shapes cannot be multiplied,” check the last dimension of your input and confirm it matches in_features.
Summary of the Core Rules
A linear layer computes an affine transform $y = xW^T + b$, applied to the last dimension of the input. In PyTorch, layer.weight has shape (out_features, in_features), outputs replace the last dimension with out_features, and the bias broadcasts across all leading dimensions.
Views: 0
Where to Move
Move chapter:
- ☰ 1. Orientation and Learning Setup
- ☰ 1.1. Course Goals and How to Use This Outline
- ☰ 1.2. What You Need to Know Before Starting
- ☰ 1.3. Installing Python and PyTorch
- ☰ 1.4. Choosing CPU vs CUDA vs Apple Silicon
- ☰ 1.5. Using Jupyter, VS Code, and Scripts
- ☰ 1.6. Reproducibility, Seeds, and Determinism
-
- ☰ 2. Python and PyTorch Fundamentals
- ☰ 2.1. Tensors and Basic Operations
- ☰ 2.2. Tensor Shapes, Dtypes, and Devices
- ☰ 2.3. Indexing, Slicing, and Broadcasting
- ☰ 2.4. Autograd and Computational Graphs
- ☰ 2.5. Common Tensor Pitfalls and Debugging
- ☰ 2.6. Saving and Loading Tensors
-
- ☰ 3. Neural Network Building Blocks
- ☰ 3.1. Linear Layers and Affine Transforms
- ☰ 3.2. Activation Functions
- ☰ 3.3. Loss Functions
- ☰ 3.4. Optimizers and Learning Rates
- ☰ 3.5. Regularization Basics
- ☰ 3.6. Weight Initialization Concepts
-
- ☰ 4. First Models End to End
- ☰ 4.1. A Simple Regression Model
- ☰ 4.2. A Simple Classification Model
- ☰ 4.3. Data Splitting and Validation Sets
- ☰ 4.4. Training Loop Structure
- ☰ 4.5. Tracking Loss and Metrics
- ☰ 4.6. Basic Overfitting and Underfitting Checks
-
- ☰ 5. Data Handling with PyTorch
- ☰ 5.1. Datasets and DataLoaders
- ☰ 5.2. Transforms and Preprocessing
- ☰ 5.3. Collate Functions and Batching
- ☰ 5.4. Shuffling, Sampling, and Class Imbalance
- ☰ 5.5. Working with Tabular, Text, and Image Data
- ☰ 5.6. Data Pipeline Performance Basics
-
- ☰ 6. Training Loops in Practice
- ☰ 6.1. Forward Pass and Backward Pass
- ☰ 6.2. Gradient Zeroing and Accumulation
- ☰ 6.3. Device Placement and Mixed Precision Basics
- ☰ 6.4. Learning Rate Schedules
- ☰ 6.5. Gradient Clipping
- ☰ 6.6. Logging with TensorBoard
-
- ☰ 7. Debugging and Model Troubleshooting
- ☰ 7.1. Verifying Shapes and NaNs
- ☰ 7.2. Checking Gradients and Vanishing Gradients
- ☰ 7.3. Sanity Checks and Overfit a Small Batch
- ☰ 7.4. Common Causes of Training Instability
- ☰ 7.5. Interpreting Learning Curves
- ☰ 7.6. Practical Tips for Faster Iteration
-
- ☰ 8. Feedforward Networks for Structured Data
- ☰ 8.1. Feature Scaling and Normalization
- ☰ 8.2. Embeddings for Categorical Features
- ☰ 8.3. Multi-Layer Perceptrons in PyTorch
- ☰ 8.4. Choosing Hidden Sizes and Depth
- ☰ 8.5. Evaluation Metrics for Classification and Regression
- ☰ 8.6. Model Export and Inference Basics
-
- ☰ 9. Convolutional Neural Networks for Images
- ☰ 9.1. Convolutions, Padding, and Stride
- ☰ 9.2. Pooling and Feature Maps
- ☰ 9.3. Building a Basic CNN in PyTorch
- ☰ 9.4. Data Augmentation for Vision
- ☰ 9.5. Transfer Learning with Pretrained Models
- ☰ 9.6. Fine-Tuning and Freezing Layers
-
- ☰ 10. Recurrent and Sequence Models
- ☰ 10.1. Sequences, Tokens, and Padding
- ☰ 10.2. RNNs, LSTMs, and GRUs
- ☰ 10.3. Handling Variable-Length Sequences
- ☰ 10.4. Teacher Forcing and Sequence Losses
- ☰ 10.5. Basic Text Classification Pipeline
- ☰ 10.6. Simple Sequence-to-Sequence Overview
-
- ☰ 11. Attention and Transformers Fundamentals
- ☰ 11.1. Why Attention Helps
- ☰ 11.2. Self-Attention Intuition
- ☰ 11.3. Transformer Building Blocks
- ☰ 11.4. Using Transformer Models in PyTorch
- ☰ 11.5. Fine-Tuning for Text Classification
- ☰ 11.6. Managing Tokenizers and Batching
-
- ☰ 12. Model Evaluation and Experiment Management
- ☰ 12.1. Train, Validation, and Test Best Practices
- ☰ 12.2. Cross-Validation Basics
- ☰ 12.3. Calibration and Thresholding
- ☰ 12.4. Error Analysis and Confusion Matrices
- ☰ 12.5. Experiment Tracking Concepts
- ☰ 12.6. Comparing Models Fairly
-
- ☰ 13. Saving, Loading, and Deployment Readiness
- ☰ 13.1. Saving Model Weights and Full Checkpoints
- ☰ 13.2. Loading for Inference and Resuming Training
- ☰ 13.3. TorchScript and Tracing Basics
- ☰ 13.4. Exporting to ONNX Overview
- ☰ 13.5. Batch Inference and Latency Considerations
- ☰ 13.6. Reproducible Packaging of Code and Weights
-
- ☰ 14. Performance and Scaling Basics
- ☰ 14.1. Profiling and Bottleneck Identification
- ☰ 14.2. DataLoader Optimization
- ☰ 14.3. GPU Utilization and Memory Management
- ☰ 14.4. Gradient Accumulation for Large Batches
- ☰ 14.5. Distributed Training Concepts Overview
- ☰ 14.6. Practical Guidelines for Faster Training
-
- ☰ 15. Best Practices and Next Steps
- ☰ 15.1. Choosing Hyperparameters Systematically
- ☰ 15.2. Regularization and Generalization Toolkit
- ☰ 15.3. Responsible Use and Dataset Considerations
- ☰ 15.4. Reading PyTorch Documentation Effectively
- ☰ 15.5. Suggested Projects for Beginners
- ☰ 15.6. Pathways to Advanced Topics
-
Comments
Please login to add a comment.
Don't have an account? Register now!