KAHIBARO Bookshelves
Discord Login Register

8.3. Multi-Layer Perceptrons in PyTorch

What an MLP Looks Like in PyTorch

A multi layer perceptron, often called an MLP, is the standard neural network for structured or tabular inputs where each example is a fixed length feature vector. In PyTorch, an MLP is typically built as a stack of nn.Linear layers with nonlinear activations in between. The overall mapping looks like repeated blocks of an affine transform plus an activation, for example $h_1 = \phi(W_1 x + b_1)$, then $h_2 = \phi(W_2 h_1 + b_2)$, and so on, ending with an output layer that matches your task.

The key implementation idea in PyTorch is that you define the layers in __init__, then connect them in forward. You can write this explicitly, or you can use nn.Sequential for simple feedforward chains.

A Minimal MLP Module

A clear beginner friendly pattern is to implement an MLP as a subclass of nn.Module with a configurable input size, hidden size, and output size.

python
import torch
import torch.nn as nn
class MLP(nn.Module):
    def __init__(self, in_features: int, hidden_features: int, out_features: int):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(in_features, hidden_features),
            nn.ReLU(),
            nn.Linear(hidden_features, out_features),
        )
    def forward(self, x):
        return self.net(x)

This structure is enough for many regression problems, and for classification problems where you will apply the correct loss function that expects raw logits.

Do not apply softmax inside your model when you plan to use nn.CrossEntropyLoss. CrossEntropyLoss expects raw, unnormalized logits of shape (batch, num_classes).

Choosing the Output Layer for Common Tasks

For regression with a single continuous target, the last layer usually has out_features=1 and you return the raw value. Your model output shape will typically be (batch, 1) or (batch,) depending on whether you squeeze it later.

For binary classification, a common choice is an output of size 1 that represents a logit. Pair it with nn.BCEWithLogitsLoss, which expects raw logits. Another common choice is two outputs, one per class, paired with nn.CrossEntropyLoss. Both approaches work, but you should stay consistent with your loss and label format.

Use nn.BCEWithLogitsLoss with a single output logit per example and targets as floats shaped like the logits. Do not add a sigmoid in the model if you use BCEWithLogitsLoss.

Adding Depth, Dropout, and Normalization in an MLP

A practical MLP often uses more than one hidden layer. You can extend the stack by repeating Linear plus activation multiple times. Dropout is commonly inserted after activations to reduce overfitting. If you use normalization for MLPs, nn.BatchNorm1d or nn.LayerNorm are typical choices, placed around the linear and activation parts depending on your preference.

A straightforward deeper MLP might look like this.

python
class DeepMLP(nn.Module):
    def __init__(self, in_features, hidden_sizes, out_features, p_drop=0.1):
        super().__init__()
        layers = []
        prev = in_features
        for h in hidden_sizes:
            layers.append(nn.Linear(prev, h))
            layers.append(nn.ReLU())
            layers.append(nn.Dropout(p_drop))
            prev = h
        layers.append(nn.Linear(prev, out_features))
        self.net = nn.Sequential(*layers)
    def forward(self, x):
        return self.net(x)

This pattern scales well, because hidden_sizes can be a list like [256, 128, 64], and you can quickly change capacity without rewriting forward.

nn.BatchNorm1d expects input shaped like (batch, features) for tabular data. If you accidentally pass (features,) because you lost the batch dimension, BatchNorm will behave incorrectly or error.

Managing Shapes in the Forward Pass

For structured data, you generally want your input tensor x to have shape (batch_size, num_features). If you are using a DataLoader, each batch should already be in that format.

The output shapes you should aim for are consistent and predictable. For multi class classification with $C$ classes, produce logits of shape (batch_size, C). For a single regression output, produce (batch_size, 1) and keep it that way until the loss computation, where you can reshape targets to match.

A common silent bug is mixing (batch,) and (batch, 1) between predictions and targets, which can trigger unwanted broadcasting and a misleadingly small loss. Make prediction and target shapes match exactly.

A Practical Forward Method With Optional Feature Concatenation

In structured problems, you sometimes combine different feature groups, for example numeric features and learned embeddings for categorical features. The MLP typically sees one concatenated vector per example. In PyTorch, concatenation is done with torch.cat along the feature dimension.

python
def forward(self, x_num, x_cat_emb):
    x = torch.cat([x_num, x_cat_emb], dim=1)  # (batch, num_total_features)
    return self.net(x)

This forward style is common when your model has multiple input branches that merge before the MLP trunk. The important point is that the final tensor handed to the first Linear must match the in_features you declared.

When to Use nn.Sequential vs Explicit Layers

nn.Sequential is ideal when your model is a simple chain of operations. If you need skip connections, multiple inputs, concatenations, or conditional behavior, writing explicit layers like self.fc1, self.fc2 and then calling them in forward is clearer and more flexible.

Even if you use nn.Sequential, remember that the model’s parameters are still registered properly as long as the modules are assigned to an attribute of your nn.Module, such as self.net.

A Small End to End Example of Calling an MLP

Once defined, an MLP is used like any other nn.Module. The example below shows shapes only, not a full training loop.

python
batch_size = 32
num_features = 10
num_classes = 3
model = MLP(in_features=num_features, hidden_features=64, out_features=num_classes)
x = torch.randn(batch_size, num_features)
logits = model(x)
print(logits.shape)  # torch.Size([32, 3])

At this point you would pass logits to your chosen loss function, and the rest of training follows the standard training loop structure covered elsewhere.

Views: 0

Comments

Please login to add a comment.

Don't have an account? Register now!

Where to Move

Move chapter:

  • ☰ 1. Orientation and Learning Setup
    • ☰ 1.1. Course Goals and How to Use This Outline
      • ☰ 1.2. What You Need to Know Before Starting
        • ☰ 1.3. Installing Python and PyTorch
          • ☰ 1.4. Choosing CPU vs CUDA vs Apple Silicon
            • ☰ 1.5. Using Jupyter, VS Code, and Scripts
              • ☰ 1.6. Reproducibility, Seeds, and Determinism
              • ☰ 2. Python and PyTorch Fundamentals
                • ☰ 2.1. Tensors and Basic Operations
                  • ☰ 2.2. Tensor Shapes, Dtypes, and Devices
                    • ☰ 2.3. Indexing, Slicing, and Broadcasting
                      • ☰ 2.4. Autograd and Computational Graphs
                        • ☰ 2.5. Common Tensor Pitfalls and Debugging
                          • ☰ 2.6. Saving and Loading Tensors
                          • ☰ 3. Neural Network Building Blocks
                            • ☰ 3.1. Linear Layers and Affine Transforms
                              • ☰ 3.2. Activation Functions
                                • ☰ 3.3. Loss Functions
                                  • ☰ 3.4. Optimizers and Learning Rates
                                    • ☰ 3.5. Regularization Basics
                                      • ☰ 3.6. Weight Initialization Concepts
                                      • ☰ 4. First Models End to End
                                        • ☰ 4.1. A Simple Regression Model
                                          • ☰ 4.2. A Simple Classification Model
                                            • ☰ 4.3. Data Splitting and Validation Sets
                                              • ☰ 4.4. Training Loop Structure
                                                • ☰ 4.5. Tracking Loss and Metrics
                                                  • ☰ 4.6. Basic Overfitting and Underfitting Checks
                                                  • ☰ 5. Data Handling with PyTorch
                                                    • ☰ 5.1. Datasets and DataLoaders
                                                      • ☰ 5.2. Transforms and Preprocessing
                                                        • ☰ 5.3. Collate Functions and Batching
                                                          • ☰ 5.4. Shuffling, Sampling, and Class Imbalance
                                                            • ☰ 5.5. Working with Tabular, Text, and Image Data
                                                              • ☰ 5.6. Data Pipeline Performance Basics
                                                              • ☰ 6. Training Loops in Practice
                                                                • ☰ 6.1. Forward Pass and Backward Pass
                                                                  • ☰ 6.2. Gradient Zeroing and Accumulation
                                                                    • ☰ 6.3. Device Placement and Mixed Precision Basics
                                                                      • ☰ 6.4. Learning Rate Schedules
                                                                        • ☰ 6.5. Gradient Clipping
                                                                          • ☰ 6.6. Logging with TensorBoard
                                                                          • ☰ 7. Debugging and Model Troubleshooting
                                                                            • ☰ 7.1. Verifying Shapes and NaNs
                                                                              • ☰ 7.2. Checking Gradients and Vanishing Gradients
                                                                                • ☰ 7.3. Sanity Checks and Overfit a Small Batch
                                                                                  • ☰ 7.4. Common Causes of Training Instability
                                                                                    • ☰ 7.5. Interpreting Learning Curves
                                                                                      • ☰ 7.6. Practical Tips for Faster Iteration
                                                                                      • ☰ 8. Feedforward Networks for Structured Data
                                                                                        • ☰ 8.1. Feature Scaling and Normalization
                                                                                          • ☰ 8.2. Embeddings for Categorical Features
                                                                                            • ☰ 8.3. Multi-Layer Perceptrons in PyTorch
                                                                                              • ☰ 8.4. Choosing Hidden Sizes and Depth
                                                                                                • ☰ 8.5. Evaluation Metrics for Classification and Regression
                                                                                                  • ☰ 8.6. Model Export and Inference Basics
                                                                                                  • ☰ 9. Convolutional Neural Networks for Images
                                                                                                    • ☰ 9.1. Convolutions, Padding, and Stride
                                                                                                      • ☰ 9.2. Pooling and Feature Maps
                                                                                                        • ☰ 9.3. Building a Basic CNN in PyTorch
                                                                                                          • ☰ 9.4. Data Augmentation for Vision
                                                                                                            • ☰ 9.5. Transfer Learning with Pretrained Models
                                                                                                              • ☰ 9.6. Fine-Tuning and Freezing Layers
                                                                                                              • ☰ 10. Recurrent and Sequence Models
                                                                                                                • ☰ 10.1. Sequences, Tokens, and Padding
                                                                                                                  • ☰ 10.2. RNNs, LSTMs, and GRUs
                                                                                                                    • ☰ 10.3. Handling Variable-Length Sequences
                                                                                                                      • ☰ 10.4. Teacher Forcing and Sequence Losses
                                                                                                                        • ☰ 10.5. Basic Text Classification Pipeline
                                                                                                                          • ☰ 10.6. Simple Sequence-to-Sequence Overview
                                                                                                                          • ☰ 11. Attention and Transformers Fundamentals
                                                                                                                            • ☰ 11.1. Why Attention Helps
                                                                                                                              • ☰ 11.2. Self-Attention Intuition
                                                                                                                                • ☰ 11.3. Transformer Building Blocks
                                                                                                                                  • ☰ 11.4. Using Transformer Models in PyTorch
                                                                                                                                    • ☰ 11.5. Fine-Tuning for Text Classification
                                                                                                                                      • ☰ 11.6. Managing Tokenizers and Batching
                                                                                                                                      • ☰ 12. Model Evaluation and Experiment Management
                                                                                                                                        • ☰ 12.1. Train, Validation, and Test Best Practices
                                                                                                                                          • ☰ 12.2. Cross-Validation Basics
                                                                                                                                            • ☰ 12.3. Calibration and Thresholding
                                                                                                                                              • ☰ 12.4. Error Analysis and Confusion Matrices
                                                                                                                                                • ☰ 12.5. Experiment Tracking Concepts
                                                                                                                                                  • ☰ 12.6. Comparing Models Fairly
                                                                                                                                                  • ☰ 13. Saving, Loading, and Deployment Readiness
                                                                                                                                                    • ☰ 13.1. Saving Model Weights and Full Checkpoints
                                                                                                                                                      • ☰ 13.2. Loading for Inference and Resuming Training
                                                                                                                                                        • ☰ 13.3. TorchScript and Tracing Basics
                                                                                                                                                          • ☰ 13.4. Exporting to ONNX Overview
                                                                                                                                                            • ☰ 13.5. Batch Inference and Latency Considerations
                                                                                                                                                              • ☰ 13.6. Reproducible Packaging of Code and Weights
                                                                                                                                                              • ☰ 14. Performance and Scaling Basics
                                                                                                                                                                • ☰ 14.1. Profiling and Bottleneck Identification
                                                                                                                                                                  • ☰ 14.2. DataLoader Optimization
                                                                                                                                                                    • ☰ 14.3. GPU Utilization and Memory Management
                                                                                                                                                                      • ☰ 14.4. Gradient Accumulation for Large Batches
                                                                                                                                                                        • ☰ 14.5. Distributed Training Concepts Overview
                                                                                                                                                                          • ☰ 14.6. Practical Guidelines for Faster Training
                                                                                                                                                                          • ☰ 15. Best Practices and Next Steps
                                                                                                                                                                            • ☰ 15.1. Choosing Hyperparameters Systematically
                                                                                                                                                                              • ☰ 15.2. Regularization and Generalization Toolkit
                                                                                                                                                                                • ☰ 15.3. Responsible Use and Dataset Considerations
                                                                                                                                                                                  • ☰ 15.4. Reading PyTorch Documentation Effectively
                                                                                                                                                                                    • ☰ 15.5. Suggested Projects for Beginners
                                                                                                                                                                                      • ☰ 15.6. Pathways to Advanced Topics