8. Feedforward Networks for Structured Data
Table of Contents
What Structured Data Means in This Course
Structured data is the kind of data that fits naturally into rows and columns, like spreadsheets and SQL tables. Each row is one example, and each column is a feature such as age, income, country, number of purchases, or a category code. In this section of the course, the default model you will reach for is a feedforward neural network, often called a multi layer perceptron. It expects each example to be represented as a fixed length vector, so the central job is turning your table into a clean numeric tensor of shape $(N, D)$, where $N$ is the number of rows in a batch and $D$ is the number of input features after preprocessing.
A key distinction in tabular data is between numeric features and categorical features. Numeric features can often be used directly after scaling. Categorical features should not be naively converted to arbitrary integers and treated as numeric magnitudes, they need a representation that preserves category identity without implying order.
The Standard Pipeline: From Table to Tensor
A typical structured data pipeline for deep learning has four conceptual steps. First, you decide what the target is and which columns are inputs. Second, you split columns into numeric and categorical groups. Third, you preprocess numeric columns and encode categorical columns, usually with embeddings. Fourth, you concatenate everything into a single vector per row and feed it into an MLP.
The result is that your model input is a tensor built from two sources. One source is a float tensor for numeric features, typically torch.float32. The other source is one or more integer tensors for categorical features, typically torch.int64, which are used to index embedding tables. After embedding lookup, those become float vectors too, and you concatenate them with the numeric floats.
What Changes Compared to “Basic Models”
Earlier in the course you trained simple models end to end. With structured data, the training loop is usually the same, but the input construction is more deliberate. The most common beginner mistake is to focus on the neural network and ignore data representation. For tabular problems, good representation and consistent preprocessing often matter as much as the architecture.
A feedforward network for structured data assumes each feature position means the same thing in every example. If your preprocessing changes between training and evaluation, your model will silently break.
Input and Output Shapes You Should Expect
For a batch of $N$ examples, numeric features typically form a tensor $X_{\text{num}} \in \mathbb{R}^{N \times D_{\text{num}}}$. Each categorical column $k$ forms an integer tensor $x^{(k)}_{\text{cat}} \in \{0,\dots,C_k-1\}^{N}$, where $C_k$ is the number of categories for that column after you build a vocabulary.
An embedding layer maps each categorical column into a dense vector. If column $k$ uses embedding size $E_k$, then after embedding you get $X^{(k)}_{\text{emb}} \in \mathbb{R}^{N \times E_k}$. Concatenation produces a single design matrix
$$
X = \big[ X_{\text{num}}, X^{(1)}_{\text{emb}}, \dots, X^{(K)}_{\text{emb}} \big] \in \mathbb{R}^{N \times D},
$$
where $D = D_{\text{num}} + \sum_{k=1}^K E_k$.
The model output shape depends on the task. For regression it is often $(N, 1)$ or $(N,)$. For binary classification it is commonly logits of shape $(N, 1)$ or $(N,)$. For multiclass classification it is logits of shape $(N, C)$.
The Main Modeling Choice in This Section
The defining modeling idea here is that categorical variables are usually represented with embeddings rather than one hot vectors when there are many categories, and numeric variables are typically standardized before the network sees them. Once everything is turned into a float vector per row, an MLP can learn nonlinear interactions between features.
The rest of this section will break down the specific tools that make this pipeline work in PyTorch, including scaling numeric columns, embedding categorical columns, constructing an MLP, choosing reasonable hidden sizes, and evaluating tabular tasks with appropriate metrics.
Views: 0
Where to Move
Move chapter:
- ☰ 1. Orientation and Learning Setup
- ☰ 1.1. Course Goals and How to Use This Outline
- ☰ 1.2. What You Need to Know Before Starting
- ☰ 1.3. Installing Python and PyTorch
- ☰ 1.4. Choosing CPU vs CUDA vs Apple Silicon
- ☰ 1.5. Using Jupyter, VS Code, and Scripts
- ☰ 1.6. Reproducibility, Seeds, and Determinism
-
- ☰ 2. Python and PyTorch Fundamentals
- ☰ 2.1. Tensors and Basic Operations
- ☰ 2.2. Tensor Shapes, Dtypes, and Devices
- ☰ 2.3. Indexing, Slicing, and Broadcasting
- ☰ 2.4. Autograd and Computational Graphs
- ☰ 2.5. Common Tensor Pitfalls and Debugging
- ☰ 2.6. Saving and Loading Tensors
-
- ☰ 3. Neural Network Building Blocks
- ☰ 3.1. Linear Layers and Affine Transforms
- ☰ 3.2. Activation Functions
- ☰ 3.3. Loss Functions
- ☰ 3.4. Optimizers and Learning Rates
- ☰ 3.5. Regularization Basics
- ☰ 3.6. Weight Initialization Concepts
-
- ☰ 4. First Models End to End
- ☰ 4.1. A Simple Regression Model
- ☰ 4.2. A Simple Classification Model
- ☰ 4.3. Data Splitting and Validation Sets
- ☰ 4.4. Training Loop Structure
- ☰ 4.5. Tracking Loss and Metrics
- ☰ 4.6. Basic Overfitting and Underfitting Checks
-
- ☰ 5. Data Handling with PyTorch
- ☰ 5.1. Datasets and DataLoaders
- ☰ 5.2. Transforms and Preprocessing
- ☰ 5.3. Collate Functions and Batching
- ☰ 5.4. Shuffling, Sampling, and Class Imbalance
- ☰ 5.5. Working with Tabular, Text, and Image Data
- ☰ 5.6. Data Pipeline Performance Basics
-
- ☰ 6. Training Loops in Practice
- ☰ 6.1. Forward Pass and Backward Pass
- ☰ 6.2. Gradient Zeroing and Accumulation
- ☰ 6.3. Device Placement and Mixed Precision Basics
- ☰ 6.4. Learning Rate Schedules
- ☰ 6.5. Gradient Clipping
- ☰ 6.6. Logging with TensorBoard
-
- ☰ 7. Debugging and Model Troubleshooting
- ☰ 7.1. Verifying Shapes and NaNs
- ☰ 7.2. Checking Gradients and Vanishing Gradients
- ☰ 7.3. Sanity Checks and Overfit a Small Batch
- ☰ 7.4. Common Causes of Training Instability
- ☰ 7.5. Interpreting Learning Curves
- ☰ 7.6. Practical Tips for Faster Iteration
-
- ☰ 8. Feedforward Networks for Structured Data
- ☰ 8.1. Feature Scaling and Normalization
- ☰ 8.2. Embeddings for Categorical Features
- ☰ 8.3. Multi-Layer Perceptrons in PyTorch
- ☰ 8.4. Choosing Hidden Sizes and Depth
- ☰ 8.5. Evaluation Metrics for Classification and Regression
- ☰ 8.6. Model Export and Inference Basics
-
- ☰ 9. Convolutional Neural Networks for Images
- ☰ 9.1. Convolutions, Padding, and Stride
- ☰ 9.2. Pooling and Feature Maps
- ☰ 9.3. Building a Basic CNN in PyTorch
- ☰ 9.4. Data Augmentation for Vision
- ☰ 9.5. Transfer Learning with Pretrained Models
- ☰ 9.6. Fine-Tuning and Freezing Layers
-
- ☰ 10. Recurrent and Sequence Models
- ☰ 10.1. Sequences, Tokens, and Padding
- ☰ 10.2. RNNs, LSTMs, and GRUs
- ☰ 10.3. Handling Variable-Length Sequences
- ☰ 10.4. Teacher Forcing and Sequence Losses
- ☰ 10.5. Basic Text Classification Pipeline
- ☰ 10.6. Simple Sequence-to-Sequence Overview
-
- ☰ 11. Attention and Transformers Fundamentals
- ☰ 11.1. Why Attention Helps
- ☰ 11.2. Self-Attention Intuition
- ☰ 11.3. Transformer Building Blocks
- ☰ 11.4. Using Transformer Models in PyTorch
- ☰ 11.5. Fine-Tuning for Text Classification
- ☰ 11.6. Managing Tokenizers and Batching
-
- ☰ 12. Model Evaluation and Experiment Management
- ☰ 12.1. Train, Validation, and Test Best Practices
- ☰ 12.2. Cross-Validation Basics
- ☰ 12.3. Calibration and Thresholding
- ☰ 12.4. Error Analysis and Confusion Matrices
- ☰ 12.5. Experiment Tracking Concepts
- ☰ 12.6. Comparing Models Fairly
-
- ☰ 13. Saving, Loading, and Deployment Readiness
- ☰ 13.1. Saving Model Weights and Full Checkpoints
- ☰ 13.2. Loading for Inference and Resuming Training
- ☰ 13.3. TorchScript and Tracing Basics
- ☰ 13.4. Exporting to ONNX Overview
- ☰ 13.5. Batch Inference and Latency Considerations
- ☰ 13.6. Reproducible Packaging of Code and Weights
-
- ☰ 14. Performance and Scaling Basics
- ☰ 14.1. Profiling and Bottleneck Identification
- ☰ 14.2. DataLoader Optimization
- ☰ 14.3. GPU Utilization and Memory Management
- ☰ 14.4. Gradient Accumulation for Large Batches
- ☰ 14.5. Distributed Training Concepts Overview
- ☰ 14.6. Practical Guidelines for Faster Training
-
- ☰ 15. Best Practices and Next Steps
- ☰ 15.1. Choosing Hyperparameters Systematically
- ☰ 15.2. Regularization and Generalization Toolkit
- ☰ 15.3. Responsible Use and Dataset Considerations
- ☰ 15.4. Reading PyTorch Documentation Effectively
- ☰ 15.5. Suggested Projects for Beginners
- ☰ 15.6. Pathways to Advanced Topics
-
Comments
Please login to add a comment.
Don't have an account? Register now!