6.6. Logging with TensorBoard
Table of Contents
What TensorBoard Gives You During Training
TensorBoard is a lightweight dashboard for viewing training progress while your model runs. In a beginner friendly PyTorch workflow, it is mainly used to track scalars like training loss, validation loss, accuracy, and learning rate over time. It can also display parameter histograms, gradient histograms, images, and embeddings, but the most common and most useful starting point is scalar logging, because it immediately answers whether training is improving and whether it is overfitting.
The typical pattern is simple: you create a log writer, you write values to it at regular steps, you close it at the end, then you launch TensorBoard pointing at the log directory. The only thing you must decide is what you consider a step. Most projects use either a global step that increments every optimization update, or an epoch based step that increments once per epoch. Global step is usually more informative because it aligns directly with gradient updates.
Installing and Importing the Writer
In PyTorch, TensorBoard logging is provided through torch.utils.tensorboard. If you installed PyTorch normally, you often already have what you need. If you see an import error, installing the tensorboard package usually resolves it.
In code, the entry point is SummaryWriter, which creates an event file inside a directory you choose.
from torch.utils.tensorboard import SummaryWriter
writer = SummaryWriter(log_dir="runs/exp1")
The log_dir should be unique per experiment run, otherwise multiple runs will mix together and your plots become confusing. A common approach is to include a timestamp or a short run name that encodes key hyperparameters.
Do not reuse the same log_dir across different experiments unless you explicitly want TensorBoard to merge them. Use a fresh directory per run to keep comparisons clean.
Logging Scalars: Loss, Metrics, and Learning Rate
The core method is writer.add_scalar(tag, value, global_step). The tag is a string that defines the series name, the value is a Python number, and global_step is the x axis.
A common convention is to namespace tags so related plots group together, for example Loss/train, Loss/val, Accuracy/train, Accuracy/val.
global_step = 0
for epoch in range(num_epochs):
model.train()
for xb, yb in train_loader:
optimizer.zero_grad()
preds = model(xb)
loss = criterion(preds, yb)
loss.backward()
optimizer.step()
writer.add_scalar("Loss/train", loss.item(), global_step)
lr = optimizer.param_groups[0]["lr"]
writer.add_scalar("LR", lr, global_step)
global_step += 1
Validation is typically logged once per epoch, because running validation every batch can be expensive and noisy. In that case, you still log against a step value, either the current global_step or the epoch number. Using global_step makes training and validation curves line up on the same horizontal scale.
model.eval()
val_loss = 0.0
correct = 0
count = 0
with torch.no_grad():
for xb, yb in val_loader:
preds = model(xb)
loss = criterion(preds, yb)
val_loss += loss.item() * xb.size(0)
# example for classification
predicted = preds.argmax(dim=1)
correct += (predicted == yb).sum().item()
count += xb.size(0)
val_loss /= count
val_acc = correct / count
writer.add_scalar("Loss/val", val_loss, global_step)
writer.add_scalar("Accuracy/val", val_acc, global_step)
Always log Python numbers, not tensors that still require gradients. Use loss.item() or float(value) so logging does not accidentally keep parts of the computation graph alive and waste memory.
Choosing a Step Convention That Stays Consistent
TensorBoard plots are only as readable as your step definition. If you log training per batch and validation per epoch, you should still use a single consistent step axis. A practical rule is to use global optimization step for everything, because it is monotonic and comparable across runs even if you change batch size or number of batches.
If you prefer epoch based logging, then log both training and validation using epoch as the step, which means you should aggregate training metrics across the whole epoch before writing them.
Do not mix different meanings of global_step under the same tag. If Loss/train uses batch steps, then keep it that way for the whole run.
Logging Histograms: Weights and Gradients
Once scalar curves look reasonable, histograms can help you diagnose issues like exploding weights or dead activations. You can log parameter values with add_histogram. Doing this every step is expensive, so doing it every few epochs is common.
if epoch % 5 == 0:
for name, param in model.named_parameters():
writer.add_histogram(f"Weights/{name}", param.detach().cpu().numpy(), global_step)
if param.grad is not None:
writer.add_histogram(f"Grads/{name}", param.grad.detach().cpu().numpy(), global_step)Histograms are most useful when you know what you are looking for. For example, gradients that become extremely concentrated near zero across layers can hint at vanishing gradients, and gradients with very large magnitude spikes can hint at instability.
Logging Images: Predictions and Inputs (Optional but Powerful)
For vision tasks, logging a small batch of input images and model outputs can catch data pipeline mistakes quickly, such as wrong normalization or swapped channels. TensorBoard accepts images in CHW format per image and values typically in $[0, 1]$ or $[0, 255]$ depending on how you log.
A simple pattern is to log a fixed batch from the validation set every few epochs so you can visually compare changes over time.
import torchvision
# xb: (N, C, H, W)
grid = torchvision.utils.make_grid(xb[:16], nrow=4, normalize=True)
writer.add_image("Images/val_batch", grid, global_step)If you also want to log predicted labels, you usually include them in the tag name or write a small text summary separately. Keep it minimal at first, since images alone already catch many issues.
Logging Text: Short Summaries and Config
TensorBoard can also store text. This is useful for recording hyperparameters, dataset version, or a short run description.
writer.add_text("Run/notes", "Baseline MLP, lr=1e-3, batch=128", 0)
If you want structured hyperparameter logging, PyTorch also supports add_hparams, which creates a compact table view of hyperparameters and final metrics. This is best used when you have a consistent set of hyperparameters across many runs.
Where to Put Logging in a Clean Training Loop
A clean separation is to compute metrics in your training and validation code, then log them immediately after you have the numbers. Training loss is often logged per step, while epoch level summaries like mean training loss, mean validation loss, and validation accuracy are logged once per epoch.
At the end of training, always close the writer so all events flush to disk.
writer.close()If you stop a run abruptly, some recent events may not be written to disk yet. Closing the writer at the end, and occasionally flushing during very long runs, reduces the chance of losing logs.
Launching TensorBoard and Viewing Runs
TensorBoard reads event files from a directory. If your logs are in runs/, you run TensorBoard pointing there and open the provided local URL in your browser.
tensorboard --logdir runs
Each subdirectory under runs/ appears as a separate run. Naming those directories well makes comparison easy, especially when you overlay training curves from multiple runs.
Minimal Checklist for Reliable TensorBoard Logging
Use a unique log directory per experiment, use a consistent step definition, log only plain numbers for scalars, and remember to close the writer so data is flushed to disk.
Views: 0
Where to Move
Move chapter:
- ☰ 1. Orientation and Learning Setup
- ☰ 1.1. Course Goals and How to Use This Outline
- ☰ 1.2. What You Need to Know Before Starting
- ☰ 1.3. Installing Python and PyTorch
- ☰ 1.4. Choosing CPU vs CUDA vs Apple Silicon
- ☰ 1.5. Using Jupyter, VS Code, and Scripts
- ☰ 1.6. Reproducibility, Seeds, and Determinism
-
- ☰ 2. Python and PyTorch Fundamentals
- ☰ 2.1. Tensors and Basic Operations
- ☰ 2.2. Tensor Shapes, Dtypes, and Devices
- ☰ 2.3. Indexing, Slicing, and Broadcasting
- ☰ 2.4. Autograd and Computational Graphs
- ☰ 2.5. Common Tensor Pitfalls and Debugging
- ☰ 2.6. Saving and Loading Tensors
-
- ☰ 3. Neural Network Building Blocks
- ☰ 3.1. Linear Layers and Affine Transforms
- ☰ 3.2. Activation Functions
- ☰ 3.3. Loss Functions
- ☰ 3.4. Optimizers and Learning Rates
- ☰ 3.5. Regularization Basics
- ☰ 3.6. Weight Initialization Concepts
-
- ☰ 4. First Models End to End
- ☰ 4.1. A Simple Regression Model
- ☰ 4.2. A Simple Classification Model
- ☰ 4.3. Data Splitting and Validation Sets
- ☰ 4.4. Training Loop Structure
- ☰ 4.5. Tracking Loss and Metrics
- ☰ 4.6. Basic Overfitting and Underfitting Checks
-
- ☰ 5. Data Handling with PyTorch
- ☰ 5.1. Datasets and DataLoaders
- ☰ 5.2. Transforms and Preprocessing
- ☰ 5.3. Collate Functions and Batching
- ☰ 5.4. Shuffling, Sampling, and Class Imbalance
- ☰ 5.5. Working with Tabular, Text, and Image Data
- ☰ 5.6. Data Pipeline Performance Basics
-
- ☰ 6. Training Loops in Practice
- ☰ 6.1. Forward Pass and Backward Pass
- ☰ 6.2. Gradient Zeroing and Accumulation
- ☰ 6.3. Device Placement and Mixed Precision Basics
- ☰ 6.4. Learning Rate Schedules
- ☰ 6.5. Gradient Clipping
- ☰ 6.6. Logging with TensorBoard
-
- ☰ 7. Debugging and Model Troubleshooting
- ☰ 7.1. Verifying Shapes and NaNs
- ☰ 7.2. Checking Gradients and Vanishing Gradients
- ☰ 7.3. Sanity Checks and Overfit a Small Batch
- ☰ 7.4. Common Causes of Training Instability
- ☰ 7.5. Interpreting Learning Curves
- ☰ 7.6. Practical Tips for Faster Iteration
-
- ☰ 8. Feedforward Networks for Structured Data
- ☰ 8.1. Feature Scaling and Normalization
- ☰ 8.2. Embeddings for Categorical Features
- ☰ 8.3. Multi-Layer Perceptrons in PyTorch
- ☰ 8.4. Choosing Hidden Sizes and Depth
- ☰ 8.5. Evaluation Metrics for Classification and Regression
- ☰ 8.6. Model Export and Inference Basics
-
- ☰ 9. Convolutional Neural Networks for Images
- ☰ 9.1. Convolutions, Padding, and Stride
- ☰ 9.2. Pooling and Feature Maps
- ☰ 9.3. Building a Basic CNN in PyTorch
- ☰ 9.4. Data Augmentation for Vision
- ☰ 9.5. Transfer Learning with Pretrained Models
- ☰ 9.6. Fine-Tuning and Freezing Layers
-
- ☰ 10. Recurrent and Sequence Models
- ☰ 10.1. Sequences, Tokens, and Padding
- ☰ 10.2. RNNs, LSTMs, and GRUs
- ☰ 10.3. Handling Variable-Length Sequences
- ☰ 10.4. Teacher Forcing and Sequence Losses
- ☰ 10.5. Basic Text Classification Pipeline
- ☰ 10.6. Simple Sequence-to-Sequence Overview
-
- ☰ 11. Attention and Transformers Fundamentals
- ☰ 11.1. Why Attention Helps
- ☰ 11.2. Self-Attention Intuition
- ☰ 11.3. Transformer Building Blocks
- ☰ 11.4. Using Transformer Models in PyTorch
- ☰ 11.5. Fine-Tuning for Text Classification
- ☰ 11.6. Managing Tokenizers and Batching
-
- ☰ 12. Model Evaluation and Experiment Management
- ☰ 12.1. Train, Validation, and Test Best Practices
- ☰ 12.2. Cross-Validation Basics
- ☰ 12.3. Calibration and Thresholding
- ☰ 12.4. Error Analysis and Confusion Matrices
- ☰ 12.5. Experiment Tracking Concepts
- ☰ 12.6. Comparing Models Fairly
-
- ☰ 13. Saving, Loading, and Deployment Readiness
- ☰ 13.1. Saving Model Weights and Full Checkpoints
- ☰ 13.2. Loading for Inference and Resuming Training
- ☰ 13.3. TorchScript and Tracing Basics
- ☰ 13.4. Exporting to ONNX Overview
- ☰ 13.5. Batch Inference and Latency Considerations
- ☰ 13.6. Reproducible Packaging of Code and Weights
-
- ☰ 14. Performance and Scaling Basics
- ☰ 14.1. Profiling and Bottleneck Identification
- ☰ 14.2. DataLoader Optimization
- ☰ 14.3. GPU Utilization and Memory Management
- ☰ 14.4. Gradient Accumulation for Large Batches
- ☰ 14.5. Distributed Training Concepts Overview
- ☰ 14.6. Practical Guidelines for Faster Training
-
- ☰ 15. Best Practices and Next Steps
- ☰ 15.1. Choosing Hyperparameters Systematically
- ☰ 15.2. Regularization and Generalization Toolkit
- ☰ 15.3. Responsible Use and Dataset Considerations
- ☰ 15.4. Reading PyTorch Documentation Effectively
- ☰ 15.5. Suggested Projects for Beginners
- ☰ 15.6. Pathways to Advanced Topics
-
Comments
Please login to add a comment.
Don't have an account? Register now!