Table of Contents
Why shuffling matters
Training usually assumes that each mini batch is a small, roughly representative snapshot of the whole dataset. If your data arrives in an ordered pattern, for example all class 0 samples first and then all class 1 samples, the model will see long stretches of one label and can learn unstable or biased updates. Shuffling breaks this order so that consecutive batches are less correlated, which tends to make optimization smoother and metrics more trustworthy during training.
In PyTorch, shuffling is typically controlled at the DataLoader level. When you pass shuffle=True, the DataLoader will generate a different random ordering of indices each epoch, then fetch samples in that order. This is the simplest and most common choice for training. For evaluation, you typically set shuffle=False so that outputs line up with the dataset order, which makes debugging and reproducibility easier.
Use shuffle=True for training in most cases. Use shuffle=False for validation and test so results are deterministic and aligned with the dataset ordering.
Shuffling vs sampling, who decides the order
There are two related mechanisms in PyTorch that control which items are drawn and in what order. Shuffling is a special case of sampling where you draw every index exactly once per epoch but in random order. Sampling is more general and can include drawing with replacement, drawing only a subset, or drawing with custom probabilities.
In practice, you choose one of these approaches for a DataLoader. You either set shuffle=True and let PyTorch use an internal random sampler, or you provide a sampler= or batch_sampler=. If you pass a sampler, you do not also use shuffle=True, because the sampler already defines the order of indices.
Do not set shuffle=True and sampler=... at the same time. The sampler controls ordering, so shuffling is redundant and disallowed.
Common sampler choices you will actually use
The default behavior when shuffle=False is essentially sequential sampling, meaning indices go from 0 to len(dataset)-1. This is fine for evaluation and sometimes for training only if you have already randomized the dataset order offline.
A random sampler produces a random permutation of indices for each epoch, which is what shuffle=True gives you. This is good for most training runs.
A subset style sampler is useful when you intentionally train on a slice of data, for example a quick experiment on 10 percent of the dataset, or when doing train validation splits by index. The key idea is that the sampler yields indices from a list you define, so the DataLoader only fetches those items.
Class imbalance, what it is and why it breaks accuracy
Class imbalance means that some classes appear much more often than others. In a binary dataset with 95 percent negatives and 5 percent positives, a model that always predicts negative will get 95 percent accuracy while being useless for the positive class. The main training issue is that mini batches will contain few minority examples, gradients will be dominated by the majority class, and the model may not learn decision boundaries that serve minority classes.
Before changing code, it helps to quantify imbalance by counting labels, then decide whether you need to change the sampling strategy, the loss, the evaluation metrics, or all three. Metrics are handled elsewhere in the course, so here the focus is on sampling and data order.
Handling imbalance with weighted sampling
A practical way to counter imbalance is to sample minority class examples more often so that batches are more balanced. In PyTorch this is commonly done with torch.utils.data.WeightedRandomSampler. You assign a weight to each sample, then the sampler draws indices with probability proportional to those weights. If minority samples get higher weights, they appear more frequently during training.
A common weight choice is to make each class contribute roughly equally. If class $c$ has count $n_c$, a simple rule is to give each sample in class $c$ weight $w_c = \frac{1}{n_c}$. Then the per sample weight is w[label]. This does not change the dataset itself, it changes how often items are drawn.
One important detail is replacement. If you want an epoch to still have len(dataset) draws while oversampling minorities, you usually sample with replacement. That means some examples will repeat within an epoch, which is expected.
With WeightedRandomSampler, you usually sample with replacement when oversampling minority classes. Repeated samples per epoch are normal and not a bug.
Oversampling vs undersampling tradeoffs
Oversampling increases the frequency of minority examples. This helps the model see enough minority patterns, but it can increase overfitting on the minority class because the same examples are reused more often.
Undersampling reduces the number of majority examples used per epoch. This can help balance but throws away data, which can hurt performance when the majority class contains useful variability.
Weighted sampling is an oversampling approach without explicitly duplicating data on disk. Undersampling can be implemented by building a subset of indices for the majority class and using a sampler over that subset.
Batch level balance is not guaranteed
Weighted sampling balances the marginal distribution over many draws, but it does not guarantee that every mini batch is perfectly balanced. You might still get an all majority batch occasionally, especially with small batch sizes. If you need stricter control, you can use a custom batch_sampler that constructs batches with fixed counts per class, but that is more advanced and dataset specific.
The practical beginner takeaway is that increasing batch size can reduce batch to batch variance in class proportions, and weighted sampling improves things without requiring custom batching logic.
Shuffling, randomness, and reproducibility with DataLoader workers
When you use multiple DataLoader workers, randomness can become confusing because there are several random number generators involved. Shuffling and samplers rely on a random generator to produce index orders. If you care about repeating the exact same order across runs, you typically provide a torch.Generator with a fixed seed to the DataLoader or sampler, and you make sure each worker has a deterministic seed initialization.
This course covers seeds and determinism elsewhere, so the key point here is scope. If you see different training results across runs, the data order can be one of the causes, especially when using multiple workers.
If training is not reproducible, check data ordering first. Shuffling and sampling randomness can change results even when the model code is unchanged.
Practical recipes
Training on a balanced dataset
If classes are roughly balanced, use DataLoader(dataset, batch_size=..., shuffle=True) for training, and shuffle=False for validation and test. This gives you the simplest, most stable baseline.
Training on an imbalanced dataset
If you see that minority classes are rarely present in batches, use a weighted sampler during training. Compute per sample weights from labels, pass them to WeightedRandomSampler, and pass that sampler to the DataLoader with shuffle=False since the sampler already defines randomness. Keep validation and test unweighted and unshuffled so your metrics reflect the true distribution you will face.
Quick sanity checks
After you set up shuffling or weighted sampling, verify what your batches actually contain by printing a few batches of labels and counting class occurrences. This tiny check catches common mistakes like using weights per class instead of weights per sample, or accidentally applying weighted sampling to validation data.