KAHIBARO Bookshelves
Discord Login Register

3. Neural Network Building Blocks

What This Chapter Covers

Neural networks in PyTorch are built by combining a small set of reusable parts. This chapter frames those parts at a high level so you can recognize them in code and understand how they fit together, without diving into the full details of each component. In the next chapters, each building block will be introduced and used concretely.

The Core Pattern: Layers plus Nonlinearities

A typical neural network is a function that maps inputs to outputs by applying several transformations in sequence. The most common pattern is to alternate a learnable transformation, called a layer, with a fixed nonlinear transformation, called an activation function. The layer gives the model adjustable parameters, and the activation function gives the model the ability to represent complex relationships that cannot be expressed by a single linear mapping.

You can think of the model as composing functions, where each stage takes a tensor and returns a tensor, gradually transforming raw features into something that matches your target.

Parameters, Loss, and Optimization as a Loop

Training is the process of choosing parameter values that make the model’s outputs match the desired targets. Conceptually, you need three ingredients: a loss function that scores how wrong the model is, an optimizer that updates parameters to reduce that loss, and a training loop that repeats this process over data.

The loss is a single scalar that the training algorithm tries to minimize. PyTorch uses automatic differentiation to compute gradients of that scalar with respect to each parameter. The optimizer then applies an update rule using those gradients.

Training in deep learning is almost always formulated as minimizing a loss. If your loss does not decrease, your model is not learning, regardless of how complex it is.

What a “Module” Means in PyTorch

In PyTorch, most neural network components are implemented as subclasses of torch.nn.Module. A module is a Python object that can hold parameters, define a forward computation, and be composed with other modules. A whole model is typically a module that contains submodules, such as layers and activations, and defines how an input flows through them.

This gives a consistent mental model: individual building blocks are modules, and assembling a network is usually assembling modules into a larger module.

Learnable Layers versus Fixed Operations

Some operations have learnable parameters, such as weights and biases, and these parameters are updated during training. Other operations are fixed, such as most activation functions and many shape manipulations. Both kinds are essential: learnable parts provide flexibility, and fixed parts provide structure and stability.

When you read model code, it helps to identify which parts introduce parameters and which parts are purely functional transformations.

The Most Common Building Blocks You Will Use

In beginner friendly PyTorch models, you will repeatedly encounter a small set of components. Linear layers are the standard choice for tabular and vector like data. Convolutions are the standard choice for images. Recurrent layers and attention based blocks are used for sequences like text. Regardless of the architecture, activations, losses, optimizers, and initialization choices show up everywhere.

This chapter is the umbrella for those pieces. The upcoming sections will cover how each one works, when to choose it, and how to use it correctly in PyTorch.

A Minimal Mathematical View

At a very high level, a common supervised learning setup is to choose parameters $\theta$ that minimize an average loss over examples:
$$
\min_{\theta} \; \frac{1}{N}\sum_{i=1}^{N} \mathcal{L}\big(f_{\theta}(x_i), y_i\big)
$$
Here $f_{\theta}$ is your neural network, $\mathcal{L}$ is your loss function, $x_i$ are inputs, and $y_i$ are targets. The specific form of $f_{\theta}$ depends on which layers and activations you stack, and the specific form of $\mathcal{L}$ depends on the task, such as regression or classification.

Keep the contract clear: the model produces predictions, the loss compares predictions to targets, and the optimizer updates only the model parameters, not the data and not the targets.

How to Think About Design Choices

When you build a network, you make a series of practical choices: which layer types match your data, which activation functions keep learning stable, which loss matches your target format, which optimizer and learning rate produce steady progress, which regularization prevents overfitting, and how to initialize weights so training starts smoothly.

You do not need to memorize every option upfront. What matters at this stage is recognizing that these choices are separable knobs. Changing one knob, such as the optimizer, does not require rewriting the entire model, and PyTorch’s module based design makes these swaps straightforward.

Where This Fits in the Course

The next chapters under this heading will turn these ideas into concrete PyTorch code and rules of thumb. You will learn how linear layers implement affine transforms, how activations introduce nonlinearity, how losses are chosen based on targets, how optimizers and learning rates control updates, how regularization reduces overfitting, and how initialization influences early training behavior.

Views: 0

Comments

Please login to add a comment.

Don't have an account? Register now!

Where to Move

Move chapter:

  • ☰ 1. Orientation and Learning Setup
    • ☰ 1.1. Course Goals and How to Use This Outline
      • ☰ 1.2. What You Need to Know Before Starting
        • ☰ 1.3. Installing Python and PyTorch
          • ☰ 1.4. Choosing CPU vs CUDA vs Apple Silicon
            • ☰ 1.5. Using Jupyter, VS Code, and Scripts
              • ☰ 1.6. Reproducibility, Seeds, and Determinism
              • ☰ 2. Python and PyTorch Fundamentals
                • ☰ 2.1. Tensors and Basic Operations
                  • ☰ 2.2. Tensor Shapes, Dtypes, and Devices
                    • ☰ 2.3. Indexing, Slicing, and Broadcasting
                      • ☰ 2.4. Autograd and Computational Graphs
                        • ☰ 2.5. Common Tensor Pitfalls and Debugging
                          • ☰ 2.6. Saving and Loading Tensors
                          • ☰ 3. Neural Network Building Blocks
                            • ☰ 3.1. Linear Layers and Affine Transforms
                              • ☰ 3.2. Activation Functions
                                • ☰ 3.3. Loss Functions
                                  • ☰ 3.4. Optimizers and Learning Rates
                                    • ☰ 3.5. Regularization Basics
                                      • ☰ 3.6. Weight Initialization Concepts
                                      • ☰ 4. First Models End to End
                                        • ☰ 4.1. A Simple Regression Model
                                          • ☰ 4.2. A Simple Classification Model
                                            • ☰ 4.3. Data Splitting and Validation Sets
                                              • ☰ 4.4. Training Loop Structure
                                                • ☰ 4.5. Tracking Loss and Metrics
                                                  • ☰ 4.6. Basic Overfitting and Underfitting Checks
                                                  • ☰ 5. Data Handling with PyTorch
                                                    • ☰ 5.1. Datasets and DataLoaders
                                                      • ☰ 5.2. Transforms and Preprocessing
                                                        • ☰ 5.3. Collate Functions and Batching
                                                          • ☰ 5.4. Shuffling, Sampling, and Class Imbalance
                                                            • ☰ 5.5. Working with Tabular, Text, and Image Data
                                                              • ☰ 5.6. Data Pipeline Performance Basics
                                                              • ☰ 6. Training Loops in Practice
                                                                • ☰ 6.1. Forward Pass and Backward Pass
                                                                  • ☰ 6.2. Gradient Zeroing and Accumulation
                                                                    • ☰ 6.3. Device Placement and Mixed Precision Basics
                                                                      • ☰ 6.4. Learning Rate Schedules
                                                                        • ☰ 6.5. Gradient Clipping
                                                                          • ☰ 6.6. Logging with TensorBoard
                                                                          • ☰ 7. Debugging and Model Troubleshooting
                                                                            • ☰ 7.1. Verifying Shapes and NaNs
                                                                              • ☰ 7.2. Checking Gradients and Vanishing Gradients
                                                                                • ☰ 7.3. Sanity Checks and Overfit a Small Batch
                                                                                  • ☰ 7.4. Common Causes of Training Instability
                                                                                    • ☰ 7.5. Interpreting Learning Curves
                                                                                      • ☰ 7.6. Practical Tips for Faster Iteration
                                                                                      • ☰ 8. Feedforward Networks for Structured Data
                                                                                        • ☰ 8.1. Feature Scaling and Normalization
                                                                                          • ☰ 8.2. Embeddings for Categorical Features
                                                                                            • ☰ 8.3. Multi-Layer Perceptrons in PyTorch
                                                                                              • ☰ 8.4. Choosing Hidden Sizes and Depth
                                                                                                • ☰ 8.5. Evaluation Metrics for Classification and Regression
                                                                                                  • ☰ 8.6. Model Export and Inference Basics
                                                                                                  • ☰ 9. Convolutional Neural Networks for Images
                                                                                                    • ☰ 9.1. Convolutions, Padding, and Stride
                                                                                                      • ☰ 9.2. Pooling and Feature Maps
                                                                                                        • ☰ 9.3. Building a Basic CNN in PyTorch
                                                                                                          • ☰ 9.4. Data Augmentation for Vision
                                                                                                            • ☰ 9.5. Transfer Learning with Pretrained Models
                                                                                                              • ☰ 9.6. Fine-Tuning and Freezing Layers
                                                                                                              • ☰ 10. Recurrent and Sequence Models
                                                                                                                • ☰ 10.1. Sequences, Tokens, and Padding
                                                                                                                  • ☰ 10.2. RNNs, LSTMs, and GRUs
                                                                                                                    • ☰ 10.3. Handling Variable-Length Sequences
                                                                                                                      • ☰ 10.4. Teacher Forcing and Sequence Losses
                                                                                                                        • ☰ 10.5. Basic Text Classification Pipeline
                                                                                                                          • ☰ 10.6. Simple Sequence-to-Sequence Overview
                                                                                                                          • ☰ 11. Attention and Transformers Fundamentals
                                                                                                                            • ☰ 11.1. Why Attention Helps
                                                                                                                              • ☰ 11.2. Self-Attention Intuition
                                                                                                                                • ☰ 11.3. Transformer Building Blocks
                                                                                                                                  • ☰ 11.4. Using Transformer Models in PyTorch
                                                                                                                                    • ☰ 11.5. Fine-Tuning for Text Classification
                                                                                                                                      • ☰ 11.6. Managing Tokenizers and Batching
                                                                                                                                      • ☰ 12. Model Evaluation and Experiment Management
                                                                                                                                        • ☰ 12.1. Train, Validation, and Test Best Practices
                                                                                                                                          • ☰ 12.2. Cross-Validation Basics
                                                                                                                                            • ☰ 12.3. Calibration and Thresholding
                                                                                                                                              • ☰ 12.4. Error Analysis and Confusion Matrices
                                                                                                                                                • ☰ 12.5. Experiment Tracking Concepts
                                                                                                                                                  • ☰ 12.6. Comparing Models Fairly
                                                                                                                                                  • ☰ 13. Saving, Loading, and Deployment Readiness
                                                                                                                                                    • ☰ 13.1. Saving Model Weights and Full Checkpoints
                                                                                                                                                      • ☰ 13.2. Loading for Inference and Resuming Training
                                                                                                                                                        • ☰ 13.3. TorchScript and Tracing Basics
                                                                                                                                                          • ☰ 13.4. Exporting to ONNX Overview
                                                                                                                                                            • ☰ 13.5. Batch Inference and Latency Considerations
                                                                                                                                                              • ☰ 13.6. Reproducible Packaging of Code and Weights
                                                                                                                                                              • ☰ 14. Performance and Scaling Basics
                                                                                                                                                                • ☰ 14.1. Profiling and Bottleneck Identification
                                                                                                                                                                  • ☰ 14.2. DataLoader Optimization
                                                                                                                                                                    • ☰ 14.3. GPU Utilization and Memory Management
                                                                                                                                                                      • ☰ 14.4. Gradient Accumulation for Large Batches
                                                                                                                                                                        • ☰ 14.5. Distributed Training Concepts Overview
                                                                                                                                                                          • ☰ 14.6. Practical Guidelines for Faster Training
                                                                                                                                                                          • ☰ 15. Best Practices and Next Steps
                                                                                                                                                                            • ☰ 15.1. Choosing Hyperparameters Systematically
                                                                                                                                                                              • ☰ 15.2. Regularization and Generalization Toolkit
                                                                                                                                                                                • ☰ 15.3. Responsible Use and Dataset Considerations
                                                                                                                                                                                  • ☰ 15.4. Reading PyTorch Documentation Effectively
                                                                                                                                                                                    • ☰ 15.5. Suggested Projects for Beginners
                                                                                                                                                                                      • ☰ 15.6. Pathways to Advanced Topics