Table of Contents
Feature maps as the “working representation” of an image
When you pass an image through a convolutional layer, the output is usually not another RGB image. Instead, it becomes a stack of learned channels called feature maps. Each feature map is a 2D grid where each location stores a number that summarizes what the convolutional filters detected around that location in the input. Early in a CNN, feature maps often respond to simple patterns such as edges or textures. Deeper in the network, feature maps tend to represent more abstract patterns, because they are built on top of earlier feature maps rather than raw pixels.
A useful way to think about this is that the spatial dimensions, height and width, keep track of where something is in the image, while the channel dimension keeps track of what was detected. If you start with an image tensor shaped like $(N, C, H, W)$, a convolution typically changes the channel count from $C$ to some new value $C_{out}$, and may also change $H$ and $W$ depending on padding and stride. The result is a new tensor shaped like $(N, C_{out}, H', W')$, which is a batch of feature map stacks.
A feature map tensor in PyTorch image models almost always follows the $NCHW$ convention: batch, channels, height, width. Confusing this order is one of the most common sources of shape errors.
Why pooling exists
Pooling is a family of operations that reduce the spatial resolution of feature maps, usually by summarizing small local neighborhoods. The primary reason to pool is to progressively compress spatial information so that deeper layers can operate on coarser, more semantic representations. This reduces compute and memory, increases the effective receptive field, and can make the model less sensitive to small shifts in the input.
Pooling does not learn weights. It is a fixed rule applied independently per channel, so it keeps the number of channels the same while reducing height and width. If the input has shape $(N, C, H, W)$, typical pooling produces $(N, C, H_{out}, W_{out})$.
Max pooling and average pooling
The two most common types are max pooling and average pooling.
Max pooling takes the maximum value in each window. It tends to preserve the strongest activation, which often corresponds to the most confident detection of a pattern. Average pooling computes the mean value in each window, which tends to smooth activations and keep more background context.
In PyTorch you will most often use torch.nn.MaxPool2d or torch.nn.AvgPool2d. The key settings are kernel_size and stride. A very common choice is kernel_size=2 and stride=2, which halves the height and width.
Pooling is applied per channel. It reduces spatial dimensions but does not mix information across channels.
Output size intuition and a practical rule of thumb
Pooling slides a window over the feature map. If you use a $2 \times 2$ window with stride 2, you are summarizing each non overlapping block of $2 \times 2$ values into one value. That is why spatial size often shrinks by a factor of 2 at each pooling step.
More generally, the output size follows the same kind of logic as convolutions. For each spatial dimension, a common formula is
$$
H_{out} = \left\lfloor \frac{H + 2p - k}{s} \right\rfloor + 1,
\quad
W_{out} = \left\lfloor \frac{W + 2p - k}{s} \right\rfloor + 1,
$$
where $k$ is the kernel size, $s$ is the stride, and $p$ is the padding used by the pooling layer.
If your model suddenly fails with a shape mismatch in a later layer, check every operation that changes $H$ and $W$. Pooling and strided convolutions are the usual causes.
Pooling vs strided convolution
Downsampling can be done with pooling or by using stride greater than 1 in a convolution. Pooling is simple and parameter free. Strided convolution learns how to downsample because its filters are learned, but it adds parameters and compute. Many modern CNNs reduce reliance on pooling and instead use strided convolutions, but pooling is still common and is an easy default for beginner architectures.
For this chapter, the key takeaway is that both pooling and stride reduce spatial resolution, which changes the shape of your feature maps and affects how much local detail remains available to later layers.
Global pooling and why it is used
A special case is global pooling, where you pool across the entire spatial extent so that each channel collapses to a single number. If your feature maps are $(N, C, H, W)$, global average pooling produces $(N, C, 1, 1)$, often reshaped to $(N, C)$ before a final classifier.
Global average pooling is popular because it removes the need for a large fully connected layer over flattened spatial features, which can drastically reduce parameters. It also ties the final decision to the presence of features anywhere in the image, which is often a good bias for classification.
Flattening a large $(C, H, W)$ tensor into a fully connected layer can create a huge number of parameters. Global pooling is a common way to avoid that parameter explosion.
What pooling does to information and invariances
Pooling trades detail for robustness. By summarizing local neighborhoods, it becomes harder for the network to represent exact pixel level positions, but easier to recognize patterns that move slightly. This is useful in many vision tasks, but it can be harmful when precise localization matters, such as segmentation or keypoint detection. In those tasks, architectures often downsample more carefully and then use upsampling or skip connections to recover spatial detail.
A good mental model is that as you go deeper and repeatedly downsample, feature maps become smaller spatially and richer in channels. The network increasingly represents what is present rather than exactly where it is.
Seeing pooling and feature maps in PyTorch shapes
In practice you will recognize pooling effects by watching shapes change. Suppose you have feature maps of shape $(N, 32, 64, 64)$. Applying MaxPool2d(2, 2) yields $(N, 32, 32, 32). Applying it again yields $(N, 32, 16, 16). Channels remain 32 until you apply another convolution that changes them.
This is why many CNN designs follow a pattern where spatial size decreases in stages while channels increase, such as $64 \times 64 \to 32 \times 32 \to 16 \times 16$, paired with channel growth like $32 \to 64 \to 128$. The exact numbers are a design choice, but the feature map and pooling logic stays the same.
Practical debugging note: pooling and odd sizes
If your input sizes are odd, pooling with stride 2 can produce outputs that do not halve cleanly, because of floor rounding. This is not wrong, but it can surprise you when you later expect a specific size.
When using repeated downsampling, prefer input sizes that stay divisible by $2$ across the number of downsampling steps you apply, or explicitly control padding so shapes remain predictable.