Table of Contents
The core idea of recurrent layers
Recurrent neural networks process a sequence one step at a time while carrying forward a hidden state that summarizes what has been seen so far. If your input is a sequence of vectors $x_1, x_2, \dots, x_T$, a basic RNN updates a hidden state $h_t$ at each step and can optionally produce an output at each step. Conceptually, you can think of it as reusing the same small neural network across time, with the hidden state acting as memory.
A simple formulation is:
$$h_t = \tanh(W_{xh} x_t + W_{hh} h_{t-1} + b_h)$$
The reuse of the same parameters across all time steps makes RNNs naturally suited to variable length sequences, but the repeated multiplication through time makes training harder as sequences get long, which motivates LSTMs and GRUs.
Vanilla RNNs in PyTorch
PyTorch provides torch.nn.RNN for a standard recurrent layer. It expects inputs in one of two layouts, either sequence first or batch first. By default it is sequence first, shaped (seq_len, batch, input_size). If you set batch_first=True, it becomes (batch, seq_len, input_size). The output is a sequence of hidden states and the final hidden state.
A minimal example looks like this:
import torch
import torch.nn as nn
rnn = nn.RNN(input_size=16, hidden_size=32, num_layers=1, batch_first=True)
x = torch.randn(8, 20, 16) # batch=8, seq_len=20, features=16
y, h_n = rnn(x)
print(y.shape) # (8, 20, 32) output at each time step
print(h_n.shape) # (1, 8, 32) final hidden for each layer
If you set num_layers>1, PyTorch stacks recurrent layers. The returned h_n then has shape (num_layers * num_directions, batch, hidden_size). If you set bidirectional=True, you get two directions, forward and backward, which doubles the last dimension of y because outputs from both directions are concatenated.
A very common bug is mixing up the shapes returned by recurrent modules. output contains a hidden state for every time step, while h_n contains only the final hidden state for each layer and direction. They are not interchangeable.
Why LSTMs and GRUs exist
Basic RNNs struggle with long range dependencies because gradients can vanish or explode when backpropagated through many time steps. LSTMs and GRUs add gating mechanisms that control what information is stored, forgotten, and exposed at each step. These gates create paths for gradients that are easier to optimize, which usually makes LSTMs and GRUs the default choice for many sequence problems when you are not using Transformers.
You do not need to manually implement the gates to use them in PyTorch. You mainly need to understand their inputs and outputs, and how their state is represented.
LSTMs in PyTorch
torch.nn.LSTM is similar to nn.RNN, but it maintains two pieces of state per layer and direction, the hidden state $h_t$ and the cell state $c_t$. The module returns output and a tuple (h_n, c_n).
import torch
import torch.nn as nn
lstm = nn.LSTM(input_size=16, hidden_size=32, num_layers=2, batch_first=True)
x = torch.randn(8, 20, 16)
y, (h_n, c_n) = lstm(x)
print(y.shape) # (8, 20, 32) because not bidirectional
print(h_n.shape) # (2, 8, 32) num_layers, batch, hidden
print(c_n.shape) # (2, 8, 32)
If you use bidirectional=True, then y becomes (batch, seq_len, 2 hidden_size) and h_n and c_n become (num_layers 2, batch, hidden_size).
For an LSTM, the state is a pair (h, c). If you only carry h forward and drop c when doing stateful processing, you are not continuing the same sequence memory and your results can silently degrade.
GRUs in PyTorch
torch.nn.GRU is a gated alternative that uses a single state tensor, similar to an RNN in terms of returned state shape, but with gating internally. It often trains faster than an LSTM and can perform similarly depending on the task.
import torch
import torch.nn as nn
gru = nn.GRU(input_size=16, hidden_size=32, num_layers=1, batch_first=True)
x = torch.randn(8, 20, 16)
y, h_n = gru(x)
print(y.shape) # (8, 20, 32)
print(h_n.shape) # (1, 8, 32)Choosing between RNN, LSTM, and GRU
Vanilla RNNs are mostly useful as a learning tool or for very short sequences because they are simplest but least robust for long range dependencies. LSTMs are a strong default when you expect longer dependencies or when you want a well established baseline. GRUs are a strong default when you want something simpler than LSTMs with fewer parameters and often similar performance.
In practice for beginners, prefer GRU or LSTM unless you have a specific reason to use a vanilla RNN.
Common configuration options that matter
The hidden_size is the size of the recurrent state and strongly affects capacity and speed. num_layers increases depth by stacking recurrent layers. dropout applies dropout between layers when num_layers>1. bidirectional processes sequences in both directions and is often helpful for tasks where the full sequence is available at once, such as text classification, but it is not suitable for strict online or streaming scenarios where future tokens are unknown.
PyTorch recurrent dropout only applies between stacked layers, not across time steps, and it is ignored when num_layers=1.
Using the outputs correctly for downstream heads
Many tasks use either the full sequence of outputs or a single summary vector. If you need one vector for classification, you often take the last time step output for a unidirectional model, or you combine forward and backward outputs for a bidirectional model. Alternatively, you can use h_n as the per layer final state. The correct choice depends on how you framed your problem, but you should be consistent about which tensor you treat as the sequence representation.
If you later add an MLP head, remember that output has an extra time dimension, so classification per sequence usually requires selecting or pooling over time before passing into nn.Linear.
Initial states and stateful processing
All three modules accept an optional initial state. If you do not provide one, PyTorch initializes it to zeros. Providing an initial state is useful for stateful processing, for example when you process a long stream in chunks, or when you want to warm start from a learned or external state.
For nn.RNN and nn.GRU, you pass h_0 shaped (num_layers * num_directions, batch, hidden_size). For nn.LSTM, you pass (h_0, c_0) with the same shapes.
When carrying state across batches or chunks, detach it from the graph each iteration if you do not intend to backpropagate through the entire history, otherwise your computation graph and memory usage can grow without bound.
A small end to end module example
A common pattern is a recurrent encoder followed by a linear layer. This example uses the final hidden state from a GRU for sequence level classification.
import torch
import torch.nn as nn
class GRUClassifier(nn.Module):
def __init__(self, input_size, hidden_size, num_classes):
super().__init__()
self.gru = nn.GRU(input_size=input_size, hidden_size=hidden_size, batch_first=True)
self.fc = nn.Linear(hidden_size, num_classes)
def forward(self, x):
output, h_n = self.gru(x) # h_n: (1, batch, hidden)
h_last = h_n[-1] # (batch, hidden)
logits = self.fc(h_last) # (batch, num_classes)
return logits
x = torch.randn(8, 20, 16)
model = GRUClassifier(16, 32, 5)
logits = model(x)
print(logits.shape) # (8, 5)
This pattern generalizes to LSTMs by taking h_n[-1] from the returned (h_n, c_n) tuple.
Practical shape rules to memorize
With batch_first=True, inputs are (batch, seq_len, input_size) and output is (batch, seq_len, hidden_size num_directions). The final state tensor h_n is always (num_layers num_directions, batch, hidden_size) regardless of batch_first.
These shape conventions are the main source of confusion when you begin using RNNs, LSTMs, and GRUs. If you keep them straight, the rest is mostly standard PyTorch module wiring.