Table of Contents
What a Tensor Is in PyTorch
A tensor is PyTorch’s main data structure for representing numbers. You can think of it as a generalization of a scalar, a vector, and a matrix into one object that can have any number of dimensions. In deep learning, tensors hold inputs, model parameters, intermediate activations, and outputs.
In code, tensors are created with functions in torch, for example torch.tensor, torch.zeros, and torch.randn. A tensor contains values and also carries metadata such as its shape, data type, and device, but those details are covered in later chapters.
Rule: In PyTorch, almost everything you compute in a model ends up being tensor operations. Getting comfortable creating tensors and applying basic math operations is essential.
Creating Tensors
The most direct way is to create a tensor from Python data.
import torch
x = torch.tensor([1.0, 2.0, 3.0])
M = torch.tensor([[1, 2],
[3, 4]])You will often create tensors with common initializations.
a = torch.zeros(3, 4) # all zeros
b = torch.ones(2, 2) # all ones
c = torch.full((2, 3), 7) # constant value
d = torch.arange(0, 10) # 0,1,2,...,9
e = torch.linspace(0, 1, 5) # evenly spaced pointsRandom initialization is common in machine learning.
r1 = torch.rand(2, 3) # uniform in [0, 1)
r2 = torch.randn(2, 3) # standard normal N(0, 1)
When you know you want the new tensor to match an existing tensor’s “settings”, you can use *_like functions.
y = torch.zeros_like(x)
z = torch.randn_like(x)Basic Arithmetic Operations
PyTorch supports elementwise arithmetic that looks like normal Python math. If two tensors have compatible shapes, operations are applied element by element.
x = torch.tensor([1.0, 2.0, 3.0])
y = torch.tensor([10.0, 20.0, 30.0])
x_plus_y = x + y
x_times_y = x * y
x_over_y = x / y
neg_x = -xYou can also mix tensors with Python scalars.
u = x + 5
v = x * 0.1
w = x ** 2Many common math functions are available.
t = torch.tensor([1.0, 4.0, 9.0])
root = torch.sqrt(t)
logt = torch.log(t)
expt = torch.exp(x)
absx = torch.abs(torch.tensor([-2.0, 3.0]))
Rule: Most operators like +, -, , /, and * are elementwise for tensors. This is different from traditional matrix multiplication.
Reductions and Simple Statistics
Reductions collapse one or more dimensions into a single value or a smaller tensor. The most common are sum, mean, min, and max.
A = torch.tensor([[1.0, 2.0],
[3.0, 4.0]])
total = A.sum()
avg = A.mean()
mn = A.min()
mx = A.max()
You can reduce along a specific dimension using dim. For a 2D tensor, dim=0 reduces down the rows (per column), and dim=1 reduces across the columns (per row).
col_sums = A.sum(dim=0) # shape (2,)
row_sums = A.sum(dim=1) # shape (2,)Some reductions return both values and indices.
vals, idx = A.max(dim=1)Matrix Multiplication Versus Elementwise Multiplication
Elementwise multiplication uses *. Matrix multiplication uses @ or torch.matmul.
A = torch.tensor([[1.0, 2.0],
[3.0, 4.0]])
B = torch.tensor([[5.0, 6.0],
[7.0, 8.0]])
elem = A * B
mat = A @ B
For vectors, dot products can be done with torch.dot.
p = torch.tensor([1.0, 2.0, 3.0])
q = torch.tensor([4.0, 5.0, 6.0])
dot = torch.dot(p, q)
Important: A * B is elementwise, A @ B is matrix multiplication. Confusing these is a common beginner error.
Reshaping and Rearranging Basics
Deep learning code often reshapes tensors to match what a layer expects. The most common operations are reshape, view, and transpose.
x = torch.arange(12) # shape (12,)
X = x.reshape(3, 4) # shape (3, 4)
view is similar, but may require that the underlying memory layout is compatible.
Xv = x.view(3, 4)
Transposing swaps dimensions. For 2D, .T is handy.
Xt = X.T
More generally, use transpose(dim0, dim1).
Xt2 = X.transpose(0, 1)Stacking, Concatenation, and Splitting
You will often combine tensors into larger ones. torch.cat concatenates along an existing dimension. torch.stack creates a new dimension.
a = torch.tensor([1.0, 2.0])
b = torch.tensor([3.0, 4.0])
cat0 = torch.cat([a, b], dim=0) # shape (4,)
stack0 = torch.stack([a, b], dim=0) # shape (2, 2)
stack1 = torch.stack([a, b], dim=1) # shape (2, 2)
You can split tensors with torch.split or torch.chunk.
x = torch.arange(10)
parts = torch.split(x, 3) # tuples of sizes 3,3,3,1
chunks = torch.chunk(x, 2) # two roughly equal chunksIn Place Operations and Why They Matter
Many tensor operations have an in place version that ends with an underscore, such as add_, mul_, and zero_. These modify the tensor directly.
x = torch.tensor([1.0, 2.0, 3.0])
x.add_(10) # x is now [11, 12, 13]In place operations can save memory, but they can also cause issues later when you start using autograd for training neural networks. It is fine to learn the syntax now, and you will learn when to avoid in place ops in the autograd chapter.
Rule: Functions ending with _ modify tensors in place. Use them carefully, especially once gradients are involved.
Converting Between PyTorch and Python or NumPy
You can extract a Python number from a single element tensor using .item().
loss = torch.tensor(2.5)
value = loss.item()
You can convert a tensor to a NumPy array with .numpy().
x = torch.tensor([1.0, 2.0, 3.0])
arr = x.numpy()
You can create a tensor from a NumPy array with torch.from_numpy.
import numpy as np
arr = np.array([1.0, 2.0, 3.0], dtype=np.float32)
x = torch.from_numpy(arr)These conversions are useful for logging, plotting, and interoperability, but they should not be overused inside performance critical training loops.
A Small Practice Example
The following example uses only basic operations to compute a simple linear expression $y = 2x + 1$ and then computes the mean.
import torch
x = torch.linspace(0, 4, 5) # [0,1,2,3,4]
y = 2 * x + 1 # [1,3,5,7,9]
m = y.mean()
print(x)
print(y)
print(m.item())This is the kind of tensor manipulation you will use constantly when building models, preparing data, and computing losses.