Table of Contents
What an MLP Looks Like in PyTorch
A multi layer perceptron, often called an MLP, is the standard neural network for structured or tabular inputs where each example is a fixed length feature vector. In PyTorch, an MLP is typically built as a stack of nn.Linear layers with nonlinear activations in between. The overall mapping looks like repeated blocks of an affine transform plus an activation, for example $h_1 = \phi(W_1 x + b_1)$, then $h_2 = \phi(W_2 h_1 + b_2)$, and so on, ending with an output layer that matches your task.
The key implementation idea in PyTorch is that you define the layers in __init__, then connect them in forward. You can write this explicitly, or you can use nn.Sequential for simple feedforward chains.
A Minimal MLP Module
A clear beginner friendly pattern is to implement an MLP as a subclass of nn.Module with a configurable input size, hidden size, and output size.
import torch
import torch.nn as nn
class MLP(nn.Module):
def __init__(self, in_features: int, hidden_features: int, out_features: int):
super().__init__()
self.net = nn.Sequential(
nn.Linear(in_features, hidden_features),
nn.ReLU(),
nn.Linear(hidden_features, out_features),
)
def forward(self, x):
return self.net(x)This structure is enough for many regression problems, and for classification problems where you will apply the correct loss function that expects raw logits.
Do not apply softmax inside your model when you plan to use nn.CrossEntropyLoss. CrossEntropyLoss expects raw, unnormalized logits of shape (batch, num_classes).
Choosing the Output Layer for Common Tasks
For regression with a single continuous target, the last layer usually has out_features=1 and you return the raw value. Your model output shape will typically be (batch, 1) or (batch,) depending on whether you squeeze it later.
For binary classification, a common choice is an output of size 1 that represents a logit. Pair it with nn.BCEWithLogitsLoss, which expects raw logits. Another common choice is two outputs, one per class, paired with nn.CrossEntropyLoss. Both approaches work, but you should stay consistent with your loss and label format.
Use nn.BCEWithLogitsLoss with a single output logit per example and targets as floats shaped like the logits. Do not add a sigmoid in the model if you use BCEWithLogitsLoss.
Adding Depth, Dropout, and Normalization in an MLP
A practical MLP often uses more than one hidden layer. You can extend the stack by repeating Linear plus activation multiple times. Dropout is commonly inserted after activations to reduce overfitting. If you use normalization for MLPs, nn.BatchNorm1d or nn.LayerNorm are typical choices, placed around the linear and activation parts depending on your preference.
A straightforward deeper MLP might look like this.
class DeepMLP(nn.Module):
def __init__(self, in_features, hidden_sizes, out_features, p_drop=0.1):
super().__init__()
layers = []
prev = in_features
for h in hidden_sizes:
layers.append(nn.Linear(prev, h))
layers.append(nn.ReLU())
layers.append(nn.Dropout(p_drop))
prev = h
layers.append(nn.Linear(prev, out_features))
self.net = nn.Sequential(*layers)
def forward(self, x):
return self.net(x)
This pattern scales well, because hidden_sizes can be a list like [256, 128, 64], and you can quickly change capacity without rewriting forward.
nn.BatchNorm1d expects input shaped like (batch, features) for tabular data. If you accidentally pass (features,) because you lost the batch dimension, BatchNorm will behave incorrectly or error.
Managing Shapes in the Forward Pass
For structured data, you generally want your input tensor x to have shape (batch_size, num_features). If you are using a DataLoader, each batch should already be in that format.
The output shapes you should aim for are consistent and predictable. For multi class classification with $C$ classes, produce logits of shape (batch_size, C). For a single regression output, produce (batch_size, 1) and keep it that way until the loss computation, where you can reshape targets to match.
A common silent bug is mixing (batch,) and (batch, 1) between predictions and targets, which can trigger unwanted broadcasting and a misleadingly small loss. Make prediction and target shapes match exactly.
A Practical Forward Method With Optional Feature Concatenation
In structured problems, you sometimes combine different feature groups, for example numeric features and learned embeddings for categorical features. The MLP typically sees one concatenated vector per example. In PyTorch, concatenation is done with torch.cat along the feature dimension.
def forward(self, x_num, x_cat_emb):
x = torch.cat([x_num, x_cat_emb], dim=1) # (batch, num_total_features)
return self.net(x)
This forward style is common when your model has multiple input branches that merge before the MLP trunk. The important point is that the final tensor handed to the first Linear must match the in_features you declared.
When to Use nn.Sequential vs Explicit Layers
nn.Sequential is ideal when your model is a simple chain of operations. If you need skip connections, multiple inputs, concatenations, or conditional behavior, writing explicit layers like self.fc1, self.fc2 and then calling them in forward is clearer and more flexible.
Even if you use nn.Sequential, remember that the model’s parameters are still registered properly as long as the modules are assigned to an attribute of your nn.Module, such as self.net.
A Small End to End Example of Calling an MLP
Once defined, an MLP is used like any other nn.Module. The example below shows shapes only, not a full training loop.
batch_size = 32
num_features = 10
num_classes = 3
model = MLP(in_features=num_features, hidden_features=64, out_features=num_classes)
x = torch.randn(batch_size, num_features)
logits = model(x)
print(logits.shape) # torch.Size([32, 3])
At this point you would pass logits to your chosen loss function, and the rest of training follows the standard training loop structure covered elsewhere.