2.1. Tensors and Basic Operations
Table of Contents
What a Tensor Is in PyTorch
A tensor is PyTorch’s main data structure for representing numbers. You can think of it as a generalization of a scalar, a vector, and a matrix into one object that can have any number of dimensions. In deep learning, tensors hold inputs, model parameters, intermediate activations, and outputs.
In code, tensors are created with functions in torch, for example torch.tensor, torch.zeros, and torch.randn. A tensor contains values and also carries metadata such as its shape, data type, and device, but those details are covered in later chapters.
Rule: In PyTorch, almost everything you compute in a model ends up being tensor operations. Getting comfortable creating tensors and applying basic math operations is essential.
Creating Tensors
The most direct way is to create a tensor from Python data.
import torch
x = torch.tensor([1.0, 2.0, 3.0])
M = torch.tensor([[1, 2],
[3, 4]])You will often create tensors with common initializations.
a = torch.zeros(3, 4) # all zeros
b = torch.ones(2, 2) # all ones
c = torch.full((2, 3), 7) # constant value
d = torch.arange(0, 10) # 0,1,2,...,9
e = torch.linspace(0, 1, 5) # evenly spaced pointsRandom initialization is common in machine learning.
r1 = torch.rand(2, 3) # uniform in [0, 1)
r2 = torch.randn(2, 3) # standard normal N(0, 1)
When you know you want the new tensor to match an existing tensor’s “settings”, you can use *_like functions.
y = torch.zeros_like(x)
z = torch.randn_like(x)Basic Arithmetic Operations
PyTorch supports elementwise arithmetic that looks like normal Python math. If two tensors have compatible shapes, operations are applied element by element.
x = torch.tensor([1.0, 2.0, 3.0])
y = torch.tensor([10.0, 20.0, 30.0])
x_plus_y = x + y
x_times_y = x * y
x_over_y = x / y
neg_x = -xYou can also mix tensors with Python scalars.
u = x + 5
v = x * 0.1
w = x ** 2Many common math functions are available.
t = torch.tensor([1.0, 4.0, 9.0])
root = torch.sqrt(t)
logt = torch.log(t)
expt = torch.exp(x)
absx = torch.abs(torch.tensor([-2.0, 3.0]))
Rule: Most operators like +, -, , /, and * are elementwise for tensors. This is different from traditional matrix multiplication.
Reductions and Simple Statistics
Reductions collapse one or more dimensions into a single value or a smaller tensor. The most common are sum, mean, min, and max.
A = torch.tensor([[1.0, 2.0],
[3.0, 4.0]])
total = A.sum()
avg = A.mean()
mn = A.min()
mx = A.max()
You can reduce along a specific dimension using dim. For a 2D tensor, dim=0 reduces down the rows (per column), and dim=1 reduces across the columns (per row).
col_sums = A.sum(dim=0) # shape (2,)
row_sums = A.sum(dim=1) # shape (2,)Some reductions return both values and indices.
vals, idx = A.max(dim=1)Matrix Multiplication Versus Elementwise Multiplication
Elementwise multiplication uses *. Matrix multiplication uses @ or torch.matmul.
A = torch.tensor([[1.0, 2.0],
[3.0, 4.0]])
B = torch.tensor([[5.0, 6.0],
[7.0, 8.0]])
elem = A * B
mat = A @ B
For vectors, dot products can be done with torch.dot.
p = torch.tensor([1.0, 2.0, 3.0])
q = torch.tensor([4.0, 5.0, 6.0])
dot = torch.dot(p, q)
Important: A * B is elementwise, A @ B is matrix multiplication. Confusing these is a common beginner error.
Reshaping and Rearranging Basics
Deep learning code often reshapes tensors to match what a layer expects. The most common operations are reshape, view, and transpose.
x = torch.arange(12) # shape (12,)
X = x.reshape(3, 4) # shape (3, 4)
view is similar, but may require that the underlying memory layout is compatible.
Xv = x.view(3, 4)
Transposing swaps dimensions. For 2D, .T is handy.
Xt = X.T
More generally, use transpose(dim0, dim1).
Xt2 = X.transpose(0, 1)Stacking, Concatenation, and Splitting
You will often combine tensors into larger ones. torch.cat concatenates along an existing dimension. torch.stack creates a new dimension.
a = torch.tensor([1.0, 2.0])
b = torch.tensor([3.0, 4.0])
cat0 = torch.cat([a, b], dim=0) # shape (4,)
stack0 = torch.stack([a, b], dim=0) # shape (2, 2)
stack1 = torch.stack([a, b], dim=1) # shape (2, 2)
You can split tensors with torch.split or torch.chunk.
x = torch.arange(10)
parts = torch.split(x, 3) # tuples of sizes 3,3,3,1
chunks = torch.chunk(x, 2) # two roughly equal chunksIn Place Operations and Why They Matter
Many tensor operations have an in place version that ends with an underscore, such as add_, mul_, and zero_. These modify the tensor directly.
x = torch.tensor([1.0, 2.0, 3.0])
x.add_(10) # x is now [11, 12, 13]In place operations can save memory, but they can also cause issues later when you start using autograd for training neural networks. It is fine to learn the syntax now, and you will learn when to avoid in place ops in the autograd chapter.
Rule: Functions ending with _ modify tensors in place. Use them carefully, especially once gradients are involved.
Converting Between PyTorch and Python or NumPy
You can extract a Python number from a single element tensor using .item().
loss = torch.tensor(2.5)
value = loss.item()
You can convert a tensor to a NumPy array with .numpy().
x = torch.tensor([1.0, 2.0, 3.0])
arr = x.numpy()
You can create a tensor from a NumPy array with torch.from_numpy.
import numpy as np
arr = np.array([1.0, 2.0, 3.0], dtype=np.float32)
x = torch.from_numpy(arr)These conversions are useful for logging, plotting, and interoperability, but they should not be overused inside performance critical training loops.
A Small Practice Example
The following example uses only basic operations to compute a simple linear expression $y = 2x + 1$ and then computes the mean.
import torch
x = torch.linspace(0, 4, 5) # [0,1,2,3,4]
y = 2 * x + 1 # [1,3,5,7,9]
m = y.mean()
print(x)
print(y)
print(m.item())This is the kind of tensor manipulation you will use constantly when building models, preparing data, and computing losses.
Views: 0
Where to Move
Move chapter:
- ☰ 1. Orientation and Learning Setup
- ☰ 1.1. Course Goals and How to Use This Outline
- ☰ 1.2. What You Need to Know Before Starting
- ☰ 1.3. Installing Python and PyTorch
- ☰ 1.4. Choosing CPU vs CUDA vs Apple Silicon
- ☰ 1.5. Using Jupyter, VS Code, and Scripts
- ☰ 1.6. Reproducibility, Seeds, and Determinism
-
- ☰ 2. Python and PyTorch Fundamentals
- ☰ 2.1. Tensors and Basic Operations
- ☰ 2.2. Tensor Shapes, Dtypes, and Devices
- ☰ 2.3. Indexing, Slicing, and Broadcasting
- ☰ 2.4. Autograd and Computational Graphs
- ☰ 2.5. Common Tensor Pitfalls and Debugging
- ☰ 2.6. Saving and Loading Tensors
-
- ☰ 3. Neural Network Building Blocks
- ☰ 3.1. Linear Layers and Affine Transforms
- ☰ 3.2. Activation Functions
- ☰ 3.3. Loss Functions
- ☰ 3.4. Optimizers and Learning Rates
- ☰ 3.5. Regularization Basics
- ☰ 3.6. Weight Initialization Concepts
-
- ☰ 4. First Models End to End
- ☰ 4.1. A Simple Regression Model
- ☰ 4.2. A Simple Classification Model
- ☰ 4.3. Data Splitting and Validation Sets
- ☰ 4.4. Training Loop Structure
- ☰ 4.5. Tracking Loss and Metrics
- ☰ 4.6. Basic Overfitting and Underfitting Checks
-
- ☰ 5. Data Handling with PyTorch
- ☰ 5.1. Datasets and DataLoaders
- ☰ 5.2. Transforms and Preprocessing
- ☰ 5.3. Collate Functions and Batching
- ☰ 5.4. Shuffling, Sampling, and Class Imbalance
- ☰ 5.5. Working with Tabular, Text, and Image Data
- ☰ 5.6. Data Pipeline Performance Basics
-
- ☰ 6. Training Loops in Practice
- ☰ 6.1. Forward Pass and Backward Pass
- ☰ 6.2. Gradient Zeroing and Accumulation
- ☰ 6.3. Device Placement and Mixed Precision Basics
- ☰ 6.4. Learning Rate Schedules
- ☰ 6.5. Gradient Clipping
- ☰ 6.6. Logging with TensorBoard
-
- ☰ 7. Debugging and Model Troubleshooting
- ☰ 7.1. Verifying Shapes and NaNs
- ☰ 7.2. Checking Gradients and Vanishing Gradients
- ☰ 7.3. Sanity Checks and Overfit a Small Batch
- ☰ 7.4. Common Causes of Training Instability
- ☰ 7.5. Interpreting Learning Curves
- ☰ 7.6. Practical Tips for Faster Iteration
-
- ☰ 8. Feedforward Networks for Structured Data
- ☰ 8.1. Feature Scaling and Normalization
- ☰ 8.2. Embeddings for Categorical Features
- ☰ 8.3. Multi-Layer Perceptrons in PyTorch
- ☰ 8.4. Choosing Hidden Sizes and Depth
- ☰ 8.5. Evaluation Metrics for Classification and Regression
- ☰ 8.6. Model Export and Inference Basics
-
- ☰ 9. Convolutional Neural Networks for Images
- ☰ 9.1. Convolutions, Padding, and Stride
- ☰ 9.2. Pooling and Feature Maps
- ☰ 9.3. Building a Basic CNN in PyTorch
- ☰ 9.4. Data Augmentation for Vision
- ☰ 9.5. Transfer Learning with Pretrained Models
- ☰ 9.6. Fine-Tuning and Freezing Layers
-
- ☰ 10. Recurrent and Sequence Models
- ☰ 10.1. Sequences, Tokens, and Padding
- ☰ 10.2. RNNs, LSTMs, and GRUs
- ☰ 10.3. Handling Variable-Length Sequences
- ☰ 10.4. Teacher Forcing and Sequence Losses
- ☰ 10.5. Basic Text Classification Pipeline
- ☰ 10.6. Simple Sequence-to-Sequence Overview
-
- ☰ 11. Attention and Transformers Fundamentals
- ☰ 11.1. Why Attention Helps
- ☰ 11.2. Self-Attention Intuition
- ☰ 11.3. Transformer Building Blocks
- ☰ 11.4. Using Transformer Models in PyTorch
- ☰ 11.5. Fine-Tuning for Text Classification
- ☰ 11.6. Managing Tokenizers and Batching
-
- ☰ 12. Model Evaluation and Experiment Management
- ☰ 12.1. Train, Validation, and Test Best Practices
- ☰ 12.2. Cross-Validation Basics
- ☰ 12.3. Calibration and Thresholding
- ☰ 12.4. Error Analysis and Confusion Matrices
- ☰ 12.5. Experiment Tracking Concepts
- ☰ 12.6. Comparing Models Fairly
-
- ☰ 13. Saving, Loading, and Deployment Readiness
- ☰ 13.1. Saving Model Weights and Full Checkpoints
- ☰ 13.2. Loading for Inference and Resuming Training
- ☰ 13.3. TorchScript and Tracing Basics
- ☰ 13.4. Exporting to ONNX Overview
- ☰ 13.5. Batch Inference and Latency Considerations
- ☰ 13.6. Reproducible Packaging of Code and Weights
-
- ☰ 14. Performance and Scaling Basics
- ☰ 14.1. Profiling and Bottleneck Identification
- ☰ 14.2. DataLoader Optimization
- ☰ 14.3. GPU Utilization and Memory Management
- ☰ 14.4. Gradient Accumulation for Large Batches
- ☰ 14.5. Distributed Training Concepts Overview
- ☰ 14.6. Practical Guidelines for Faster Training
-
- ☰ 15. Best Practices and Next Steps
- ☰ 15.1. Choosing Hyperparameters Systematically
- ☰ 15.2. Regularization and Generalization Toolkit
- ☰ 15.3. Responsible Use and Dataset Considerations
- ☰ 15.4. Reading PyTorch Documentation Effectively
- ☰ 15.5. Suggested Projects for Beginners
- ☰ 15.6. Pathways to Advanced Topics
-
Comments
Please login to add a comment.
Don't have an account? Register now!