2.2. Tensor Shapes, Dtypes, and Devices
Table of Contents
Why shapes, dtypes, and devices matter
In PyTorch, every tensor has a shape, a dtype, and a device. Most beginner bugs come from mixing these up, such as multiplying tensors with incompatible shapes, feeding a model inputs with the wrong dtype, or accidentally keeping some tensors on CPU while the model is on GPU. Getting comfortable reading and controlling these three properties makes everything else in deep learning much smoother.
Tensor shapes
A tensor’s shape tells you how many dimensions it has and how large each dimension is. You can inspect it with x.shape or x.size(). A single number like torch.tensor(3.0) has shape torch.Size([]), which is a scalar. A vector like torch.randn(5) has shape torch.Size([5]). A matrix like torch.randn(2, 3) has shape torch.Size([2, 3]). For images, you commonly see 4D tensors, often shaped like (batch, channels, height, width) in PyTorch.
Many operations require specific shape relationships. Matrix multiplication, for example, follows linear algebra rules: if A is shape (m, n) and B is shape (n, p), then A @ B is shape (m, p). For elementwise operations like A + B, the shapes must match or be broadcastable, which is covered in the broadcasting chapter.
You will often reshape tensors. The most common tools are reshape, view, unsqueeze, squeeze, and permute. reshape changes the shape while keeping the same number of elements, and it can return a copy if needed. view is similar but requires the tensor to be contiguous in memory. unsqueeze adds a dimension of size 1, and squeeze removes dimensions of size 1. permute reorders dimensions, which is common when switching between different layout conventions.
A reshape must preserve the number of elements. If x.numel() changes, the reshape is invalid.
view can fail on non contiguous tensors, such as after permute. In that case, use reshape or call x.contiguous() before view.
A quick way to sanity check shape logic is to print intermediate shapes during development. When a runtime error happens, PyTorch error messages usually include the conflicting shapes. Treat those numbers as clues about which dimension you accidentally swapped or forgot to include, such as a missing batch dimension.
Dtypes in PyTorch
A tensor’s dtype controls how its values are stored and computed. You can inspect it with x.dtype. Common dtypes include torch.float32, torch.float64, torch.float16, torch.bfloat16, and integer types like torch.int64.
Deep learning models almost always use floating point types for inputs and weights, typically float32. Many PyTorch layers and losses expect floating point inputs. Class labels for classification are commonly stored as integer types, often int64, because loss functions like cross entropy treat labels as class indices.
You can set dtype when creating tensors, such as torch.tensor([1, 2], dtype=torch.float32), or convert later with x.to(torch.float32) or x.float(). Conversions create a new tensor, so if you convert frequently in a loop, it can slow things down.
Do not confuse integer and floating point tensors. If you perform operations that require gradients, your tensors must be floating point, because integer tensors cannot require gradients in PyTorch.
Many classification losses expect labels to be integer class indices, typically torch.int64, not one hot float vectors, unless the loss explicitly says so.
Numerical precision also matters. float64 is more precise but slower and uses more memory, and is not usually necessary for neural networks. Mixed precision uses float16 or bfloat16 for speed, mostly on GPUs, but it needs care and is introduced later in the training practice section.
Devices in PyTorch
A tensor’s device tells you where it lives, such as CPU or GPU. You can inspect it with x.device. A CPU tensor has a device like cpu. A CUDA GPU tensor has a device like cuda:0. On Apple Silicon, you may see mps:0 when using the MPS backend.
You move tensors and models to a device with .to(device). A common pattern is to create a device once, then move everything consistently.
When you do an operation, all participating tensors must be on the same device. If one tensor is on CPU and another is on GPU, PyTorch will raise an error instead of silently copying, which is good because silent copies would hide performance problems.
All tensors involved in an operation must be on the same device. This includes model parameters, inputs, targets, and any tensors you create inside the training loop.
A common bug is creating a new tensor on CPU by default inside a loop, such as torch.zeros(...), while your model is on GPU. Prefer creating tensors on the same device, for example by using device=device or by using existing tensors as a reference, such as torch.zeros_like(x).
Another subtlety is that .to(device) returns a tensor on the new device, it does not modify the original tensor in place. The same is true for models, where model.to(device) modifies the module parameters in place, but when moving ordinary tensors you typically write x = x.to(device).
Putting them together in a consistent workflow
In practice, you want your data tensors to have the right shape for the layer you are calling, use floating point dtype for model inputs, and live on the same device as the model. Labels should be the dtype expected by your loss. The simplest way to stay consistent is to choose a device at the start, move the model to that device once, and in each iteration move the batch tensors to that device before the forward pass. If you need to create new tensors during computation, create them on the correct device and with a compatible dtype.
Quick inspection and debugging checklist
When something fails, check x.shape, x.dtype, and x.device for the inputs, the targets, and one or two model parameters. If shapes look wrong, look for missing batch dimensions, swapped axes, or incorrect reshapes. If dtypes look wrong, confirm that your inputs are floating point and your labels match the loss expectations. If devices mismatch, ensure everything has been moved to the same device and that tensors created inside the loop are also created there.
Views: 0
Where to Move
Move chapter:
- ☰ 1. Orientation and Learning Setup
- ☰ 1.1. Course Goals and How to Use This Outline
- ☰ 1.2. What You Need to Know Before Starting
- ☰ 1.3. Installing Python and PyTorch
- ☰ 1.4. Choosing CPU vs CUDA vs Apple Silicon
- ☰ 1.5. Using Jupyter, VS Code, and Scripts
- ☰ 1.6. Reproducibility, Seeds, and Determinism
-
- ☰ 2. Python and PyTorch Fundamentals
- ☰ 2.1. Tensors and Basic Operations
- ☰ 2.2. Tensor Shapes, Dtypes, and Devices
- ☰ 2.3. Indexing, Slicing, and Broadcasting
- ☰ 2.4. Autograd and Computational Graphs
- ☰ 2.5. Common Tensor Pitfalls and Debugging
- ☰ 2.6. Saving and Loading Tensors
-
- ☰ 3. Neural Network Building Blocks
- ☰ 3.1. Linear Layers and Affine Transforms
- ☰ 3.2. Activation Functions
- ☰ 3.3. Loss Functions
- ☰ 3.4. Optimizers and Learning Rates
- ☰ 3.5. Regularization Basics
- ☰ 3.6. Weight Initialization Concepts
-
- ☰ 4. First Models End to End
- ☰ 4.1. A Simple Regression Model
- ☰ 4.2. A Simple Classification Model
- ☰ 4.3. Data Splitting and Validation Sets
- ☰ 4.4. Training Loop Structure
- ☰ 4.5. Tracking Loss and Metrics
- ☰ 4.6. Basic Overfitting and Underfitting Checks
-
- ☰ 5. Data Handling with PyTorch
- ☰ 5.1. Datasets and DataLoaders
- ☰ 5.2. Transforms and Preprocessing
- ☰ 5.3. Collate Functions and Batching
- ☰ 5.4. Shuffling, Sampling, and Class Imbalance
- ☰ 5.5. Working with Tabular, Text, and Image Data
- ☰ 5.6. Data Pipeline Performance Basics
-
- ☰ 6. Training Loops in Practice
- ☰ 6.1. Forward Pass and Backward Pass
- ☰ 6.2. Gradient Zeroing and Accumulation
- ☰ 6.3. Device Placement and Mixed Precision Basics
- ☰ 6.4. Learning Rate Schedules
- ☰ 6.5. Gradient Clipping
- ☰ 6.6. Logging with TensorBoard
-
- ☰ 7. Debugging and Model Troubleshooting
- ☰ 7.1. Verifying Shapes and NaNs
- ☰ 7.2. Checking Gradients and Vanishing Gradients
- ☰ 7.3. Sanity Checks and Overfit a Small Batch
- ☰ 7.4. Common Causes of Training Instability
- ☰ 7.5. Interpreting Learning Curves
- ☰ 7.6. Practical Tips for Faster Iteration
-
- ☰ 8. Feedforward Networks for Structured Data
- ☰ 8.1. Feature Scaling and Normalization
- ☰ 8.2. Embeddings for Categorical Features
- ☰ 8.3. Multi-Layer Perceptrons in PyTorch
- ☰ 8.4. Choosing Hidden Sizes and Depth
- ☰ 8.5. Evaluation Metrics for Classification and Regression
- ☰ 8.6. Model Export and Inference Basics
-
- ☰ 9. Convolutional Neural Networks for Images
- ☰ 9.1. Convolutions, Padding, and Stride
- ☰ 9.2. Pooling and Feature Maps
- ☰ 9.3. Building a Basic CNN in PyTorch
- ☰ 9.4. Data Augmentation for Vision
- ☰ 9.5. Transfer Learning with Pretrained Models
- ☰ 9.6. Fine-Tuning and Freezing Layers
-
- ☰ 10. Recurrent and Sequence Models
- ☰ 10.1. Sequences, Tokens, and Padding
- ☰ 10.2. RNNs, LSTMs, and GRUs
- ☰ 10.3. Handling Variable-Length Sequences
- ☰ 10.4. Teacher Forcing and Sequence Losses
- ☰ 10.5. Basic Text Classification Pipeline
- ☰ 10.6. Simple Sequence-to-Sequence Overview
-
- ☰ 11. Attention and Transformers Fundamentals
- ☰ 11.1. Why Attention Helps
- ☰ 11.2. Self-Attention Intuition
- ☰ 11.3. Transformer Building Blocks
- ☰ 11.4. Using Transformer Models in PyTorch
- ☰ 11.5. Fine-Tuning for Text Classification
- ☰ 11.6. Managing Tokenizers and Batching
-
- ☰ 12. Model Evaluation and Experiment Management
- ☰ 12.1. Train, Validation, and Test Best Practices
- ☰ 12.2. Cross-Validation Basics
- ☰ 12.3. Calibration and Thresholding
- ☰ 12.4. Error Analysis and Confusion Matrices
- ☰ 12.5. Experiment Tracking Concepts
- ☰ 12.6. Comparing Models Fairly
-
- ☰ 13. Saving, Loading, and Deployment Readiness
- ☰ 13.1. Saving Model Weights and Full Checkpoints
- ☰ 13.2. Loading for Inference and Resuming Training
- ☰ 13.3. TorchScript and Tracing Basics
- ☰ 13.4. Exporting to ONNX Overview
- ☰ 13.5. Batch Inference and Latency Considerations
- ☰ 13.6. Reproducible Packaging of Code and Weights
-
- ☰ 14. Performance and Scaling Basics
- ☰ 14.1. Profiling and Bottleneck Identification
- ☰ 14.2. DataLoader Optimization
- ☰ 14.3. GPU Utilization and Memory Management
- ☰ 14.4. Gradient Accumulation for Large Batches
- ☰ 14.5. Distributed Training Concepts Overview
- ☰ 14.6. Practical Guidelines for Faster Training
-
- ☰ 15. Best Practices and Next Steps
- ☰ 15.1. Choosing Hyperparameters Systematically
- ☰ 15.2. Regularization and Generalization Toolkit
- ☰ 15.3. Responsible Use and Dataset Considerations
- ☰ 15.4. Reading PyTorch Documentation Effectively
- ☰ 15.5. Suggested Projects for Beginners
- ☰ 15.6. Pathways to Advanced Topics
-
Comments
Please login to add a comment.
Don't have an account? Register now!