Table of Contents
What This Chapter Covers
Neural networks in PyTorch are built by combining a small set of reusable parts. This chapter frames those parts at a high level so you can recognize them in code and understand how they fit together, without diving into the full details of each component. In the next chapters, each building block will be introduced and used concretely.
The Core Pattern: Layers plus Nonlinearities
A typical neural network is a function that maps inputs to outputs by applying several transformations in sequence. The most common pattern is to alternate a learnable transformation, called a layer, with a fixed nonlinear transformation, called an activation function. The layer gives the model adjustable parameters, and the activation function gives the model the ability to represent complex relationships that cannot be expressed by a single linear mapping.
You can think of the model as composing functions, where each stage takes a tensor and returns a tensor, gradually transforming raw features into something that matches your target.
Parameters, Loss, and Optimization as a Loop
Training is the process of choosing parameter values that make the model’s outputs match the desired targets. Conceptually, you need three ingredients: a loss function that scores how wrong the model is, an optimizer that updates parameters to reduce that loss, and a training loop that repeats this process over data.
The loss is a single scalar that the training algorithm tries to minimize. PyTorch uses automatic differentiation to compute gradients of that scalar with respect to each parameter. The optimizer then applies an update rule using those gradients.
Training in deep learning is almost always formulated as minimizing a loss. If your loss does not decrease, your model is not learning, regardless of how complex it is.
What a “Module” Means in PyTorch
In PyTorch, most neural network components are implemented as subclasses of torch.nn.Module. A module is a Python object that can hold parameters, define a forward computation, and be composed with other modules. A whole model is typically a module that contains submodules, such as layers and activations, and defines how an input flows through them.
This gives a consistent mental model: individual building blocks are modules, and assembling a network is usually assembling modules into a larger module.
Learnable Layers versus Fixed Operations
Some operations have learnable parameters, such as weights and biases, and these parameters are updated during training. Other operations are fixed, such as most activation functions and many shape manipulations. Both kinds are essential: learnable parts provide flexibility, and fixed parts provide structure and stability.
When you read model code, it helps to identify which parts introduce parameters and which parts are purely functional transformations.
The Most Common Building Blocks You Will Use
In beginner friendly PyTorch models, you will repeatedly encounter a small set of components. Linear layers are the standard choice for tabular and vector like data. Convolutions are the standard choice for images. Recurrent layers and attention based blocks are used for sequences like text. Regardless of the architecture, activations, losses, optimizers, and initialization choices show up everywhere.
This chapter is the umbrella for those pieces. The upcoming sections will cover how each one works, when to choose it, and how to use it correctly in PyTorch.
A Minimal Mathematical View
At a very high level, a common supervised learning setup is to choose parameters $\theta$ that minimize an average loss over examples:
$$
\min_{\theta} \; \frac{1}{N}\sum_{i=1}^{N} \mathcal{L}\big(f_{\theta}(x_i), y_i\big)
$$
Here $f_{\theta}$ is your neural network, $\mathcal{L}$ is your loss function, $x_i$ are inputs, and $y_i$ are targets. The specific form of $f_{\theta}$ depends on which layers and activations you stack, and the specific form of $\mathcal{L}$ depends on the task, such as regression or classification.
Keep the contract clear: the model produces predictions, the loss compares predictions to targets, and the optimizer updates only the model parameters, not the data and not the targets.
How to Think About Design Choices
When you build a network, you make a series of practical choices: which layer types match your data, which activation functions keep learning stable, which loss matches your target format, which optimizer and learning rate produce steady progress, which regularization prevents overfitting, and how to initialize weights so training starts smoothly.
You do not need to memorize every option upfront. What matters at this stage is recognizing that these choices are separable knobs. Changing one knob, such as the optimizer, does not require rewriting the entire model, and PyTorch’s module based design makes these swaps straightforward.
Where This Fits in the Course
The next chapters under this heading will turn these ideas into concrete PyTorch code and rules of thumb. You will learn how linear layers implement affine transforms, how activations introduce nonlinearity, how losses are chosen based on targets, how optimizers and learning rates control updates, how regularization reduces overfitting, and how initialization influences early training behavior.