Table of Contents
What “device” means in practice
In PyTorch, the device is where your tensors live and where computations run. For most beginners, the choice comes down to CPU or GPU. GPUs come in different ecosystems, most commonly NVIDIA CUDA on Windows and Linux, and Apple Silicon GPUs on macOS using the Metal Performance Shaders backend, usually called MPS. Your device choice affects training speed, available operations, numerical behavior, and how you set up your code.
CPU: the default that always works
CPU is the simplest option because it is always available and tends to be the most predictable. It is a good fit for learning, for small models, for debugging shape issues, and for environments where you need maximum compatibility. CPU performance can be surprisingly decent for small datasets and small networks, but it becomes slow when you scale up data size, model size, or image and transformer workloads.
If you are unsure, start on CPU and only move to GPU when you feel the training loop is correct and you are bottlenecked by speed.
CUDA (NVIDIA GPU): the standard for deep learning speed
CUDA is the most mature and widely supported GPU backend in PyTorch. If you have an NVIDIA GPU with enough memory, CUDA usually provides the biggest speedup, the best tooling, and the broadest operator support. CUDA is especially valuable for convolutional networks, transformers, and large batch training.
You should prefer CUDA when you are training anything nontrivial and you have access to a suitable NVIDIA GPU. In practice, CUDA is also the easiest path to mixed precision training later, and it tends to match what many production and research environments use.
:::danger :::
Rule: CUDA only works with NVIDIA GPUs. Installing CUDA software does not make a non NVIDIA GPU work.
Apple Silicon (MPS): macOS GPU acceleration with tradeoffs
On Apple Silicon Macs, PyTorch can use the GPU through the MPS backend. MPS can offer large speedups over CPU for many workloads, and it is often the best choice on modern Mac laptops and desktops when you cannot use CUDA.
At the same time, MPS is less mature than CUDA. Some operations may be slower than expected, behave differently in edge cases, or not be implemented for the GPU path. When that happens, you may need to adjust the model, change a particular operation, or temporarily run that part on CPU. For beginners, the key idea is that MPS can be great for training speed, but you should expect occasional compatibility bumps.
:::danger :::
Rule: If you hit an MPS error for an operation, first try updating PyTorch, then consider switching that experiment to CPU to confirm your code is correct. Do not assume your model is wrong just because MPS fails.
How to decide: a practical checklist
Choose CPU if you are early in the course, you are debugging, your model is small, or you want maximum reliability across machines. Choose CUDA if you have an NVIDIA GPU and you care about training speed, especially for CNNs and transformers. Choose MPS if you are on Apple Silicon and want GPU acceleration, and you can tolerate the possibility that some features lag behind CUDA.
Memory is often the deciding factor more than raw speed. If your GPU has limited VRAM, you may need smaller batch sizes, smaller models, or you may even prefer CPU for certain experiments.
How to detect and select the device in code
A common pattern is to pick the best available device at runtime, then move both the model and tensors to that device.
For CUDA, you typically check torch.cuda.is_available(). For Apple Silicon, you check torch.backends.mps.is_available().
:::danger ::>
Rule: All tensors involved in a computation must be on the same device. If your model is on GPU and your input batch is on CPU, you will get a device mismatch error.
In practice, you will create a device variable once, then use model.to(device) and tensor.to(device) consistently. Later chapters will show this integrated into a full training loop, so the important point here is the decision logic, not the full training code.
Expected speed and workflow differences
On CPU, iteration time is often dominated by computation once models get larger, and you rarely worry about device transfers. On GPU, iteration time can be dominated by data loading and host to device transfer if your input pipeline is slow. This is why GPU training often pushes you to care about DataLoader performance later.
On CUDA, you may notice that errors sometimes appear “later” than the line that caused them because GPU execution is asynchronous. MPS can have similar behavior. This can make debugging feel different from CPU, where execution is mostly synchronous. When debugging correctness, CPU is often the easiest baseline.
Numerical differences and determinism expectations
Even when your code is correct, results can differ slightly between CPU, CUDA, and MPS because floating point math can be implemented differently. You should expect small differences in loss curves and final metrics, especially when using GPU acceleration. These differences are usually normal. Reproducibility settings are handled in another chapter, but it is important to know that identical results across devices are not guaranteed.
:::danger ::>
Statement: Small numeric differences across devices are normal. Focus on whether training is stable and improving, not on exact matching loss values.
Recommended path for this course
If you have CUDA, use it once your basic tensor operations and training loop are working, because it will make experiments faster and more enjoyable. If you have Apple Silicon, try MPS for the same reason, and fall back to CPU if you encounter unsupported operations. If you have neither, CPU is perfectly fine for learning the fundamentals, and you can still train meaningful models with smaller datasets and architectures.
Quick self check before moving on
You should be able to answer: which device you will use on your machine, how you would check its availability, and what it means to keep the model and data on the same device. This is enough to proceed into editor and workflow choices in the next chapter.