KAHIBARO Bookshelves
Discord Login Register

16.4. Simple Data Cleaning Techniques

Why Data Cleaning Matters

Before any meaningful analysis, your data must be in a usable state. Simple data cleaning aims to fix common problems such as missing values, impossible values, duplicated rows, and inconsistent formats. In MATLAB you rarely change the original file directly. Instead you typically import the data into a table or arrays, clean these in memory, then save a cleaned version.

This chapter focuses on straightforward, practical techniques you can apply to small and medium sized datasets using core MATLAB functions.

Inspecting Data for Problems

The first step of cleaning is to look at your data. Once you have your data in a table T, basic inspection usually includes:

Use head(T) and tail(T) to inspect the first and last few rows. This helps you spot obvious issues such as headers being read as data, extra rows, or strange values at the end.

Use summary(T) to get a quick overview of each variable. For numeric variables, summary reports minimum, maximum, and counts of missing values. For categorical and string variables, it shows categories and counts.

Use size(T) to check how many rows and columns you have, and T.Properties.VariableNames to inspect column names.

A small example:

T = readtable("data.csv");
head(T)
summary(T)
size(T)
T.Properties.VariableNames

These inspection steps guide what cleaning steps you apply next.

Handling Missing Values

Missing data appears in different ways depending on type and how the file was imported. Common representations are NaN for numeric data, "" or <missing> for strings and categorical arrays, and empty cells in tables.

For simple cleaning, you typically either remove rows with missing values or fill them with replacement values.

To find missing entries in a numeric vector x you can use:

idxMissing = isnan(x);

To find missing values in a table you can use ismissing. It returns a logical array with true where values are missing:

M = ismissing(T);

You can then remove any row that has at least one missing value:

T(any(ismissing(T), 2), :) = [];

Sometimes it is better to fill missing values rather than drop rows. For numeric columns, use fillmissing. For example, to replace missing entries in variable Temperature with the mean of the nonmissing values:

T.Temperature = fillmissing(T.Temperature, "movmean", 5);

or with a constant:

T.Temperature = fillmissing(T.Temperature, "constant", 0);

For timelike sequences, fillmissing can interpolate or use neighboring values. The key idea is that you can choose a strategy that fits your analysis, such as constant replacement, interpolation, or local averaging, without rewriting low level loops.

Detecting and Removing Duplicates

Duplicate rows can bias your results. MATLAB provides simple tools to detect and remove them.

To detect duplicate rows in a table:

idxDup = duplicated(T);

idxDup is true for rows that have already appeared earlier. To keep only the first occurrence of each unique row, remove duplicates:

T(idxDup, :) = [];

If you only care about duplicates based on specific columns, use those columns instead:

[~, idxUnique] = unique(T(:, ["ID", "Date"]), "rows");
T = T(idxUnique, :);

Here unique keeps a single row for each combination of ID and Date, and you select those rows from the original table.

Fixing Inconsistent Text and Categories

Text data, such as city names or categories, can be inconsistent. Simple problems include differences in case, extra spaces, or slightly different spellings that refer to the same concept.

Assume a table variable T.City holds city names as strings. To trim unwanted leading and trailing spaces use:

T.City = strtrim(T.City);

To make everything lower case or upper case:

T.City = lower(T.City);
% or
T.City = upper(T.City);

If some values should be corrected or normalized, for example "NY", "New York", and "N.Y." that should all be "New York", you can use string replacement:

T.City = replace(T.City, "NY", "New York");
T.City = replace(T.City, "N.Y.", "New York");

For string arrays or character vectors, regexprep can handle more complicated patterns, but for simple cleaning, straightforward replace is often enough.

When working with categorical variables, you may want to merge several categories into one. Suppose T.Outcome is categorical and contains "ok", "OK", "Ok", and "failure". First standardize the text, then convert to categorical:

s = string(T.Outcome);
s = upper(strtrim(s));
T.Outcome = categorical(s);

Now all "ok" variants become one category "OK".

Simple Outlier Detection and Handling

Outliers are values that look unusually large or small compared to the rest of the data. Simple methods to detect them use summary statistics such as the mean and standard deviation or quantiles.

For a numeric vector x, you can compute the mean and standard deviation:

m = mean(x, "omitnan");
s = std(x, "omitnan");

A basic rule is to mark values more than 3 standard deviations away from the mean:

idxOutlier = abs(x - m) > 3*s;

You can inspect these outliers, remove them, or replace them:

xClean = x;
xClean(idxOutlier) = NaN;

After that, you can either drop rows with missing values or fill them.

An alternative uses quantiles. Compute the 1st and 99th percentiles:

q = quantile(x, [0.01 0.99]);
idxOutlier = x < q(1) | x > q(2);

Again you can decide how to handle those values. These rules are simple and not suitable for every situation, but they are easy to apply and often helpful for an initial pass.

Correcting Obvious Invalid Values

Sometimes you know that a variable must be within a realistic range. For example, age should not be negative, and percentages should be between 0 and 100.

Assume a numeric column T.Age. You can mark impossible values as missing:

idxInvalid = T.Age < 0 | T.Age > 120;
T.Age(idxInvalid) = NaN;

Similarly for a percentage variable T.Rate:

idxInvalid = T.Rate < 0 | T.Rate > 100;
T.Rate(idxInvalid) = NaN;

You can combine this with fillmissing or row removal. The key is to use logical expressions to select values that violate your assumptions and then either set them to missing or correct them if you know the proper value.

Cleaning and Standardizing Date and Time Data

Date and time information often comes as strings in different formats. For simple cleaning, your tasks are to convert them to MATLAB datetime, fix obvious invalid dates, and ensure they all use the same format and time zone.

Suppose T.Date is currently a string column with dates like "2025-01-15" or "01/15/2025". Convert them to datetime by telling MATLAB the input format:

T.Date = datetime(T.Date, "InputFormat", "uuuu-MM-dd");

If the data uses a different format:

T.Date = datetime(T.Date, "InputFormat", "MM/dd/uuuu");

If some entries have invalid values, such as 2025-02-30, they become NaT which is the missing value for datetimes. You can find and handle them with ismissing:

idxMissingDates = ismissing(T.Date);

Once the dates are converted, you can standardize the display format if you like:

T.Date.Format = "dd-MMM-uuuu";

If time zones matter, you can set or convert them, but for simple cleaning it is often enough to make sure all dates are valid and in a consistent representation.

Putting the Steps Together

In practice you will combine these simple techniques to clean a dataset. For example, a basic cleaning function for a small table might:

Read the raw file into a table. Trim and standardize text fields. Convert date strings to datetime. Mark invalid numeric ranges as missing. Fill or drop missing values. Remove duplicate rows.

A short illustration:

T = readtable("raw_data.csv");
% Standardize city names
T.City = strtrim(lower(string(T.City)));
% Convert date
T.Date = datetime(T.Date, "InputFormat", "MM/dd/uuuu");
% Fix invalid ages
idxInvalidAge = T.Age < 0 | T.Age > 120;
T.Age(idxInvalidAge) = NaN;
% Remove rows with any missing values
T(any(ismissing(T), 2), :) = [];
% Remove duplicate rows
T(duplicated(T), :) = [];
writetable(T, "clean_data.csv");

This is not a universal recipe, but it shows how the simple tools described in this chapter can be combined into a straightforward cleaning workflow.

Key points to remember:
Use head, tail, and summary to inspect data before cleaning.
Find missing values with ismissing or isnan, then either remove or fill them.
Remove duplicates with duplicated or keep unique rows with unique.
Standardize text using strtrim, lower, upper, and replace.
Detect obvious invalid numeric values using logical conditions and set them to NaN.
Convert date strings to datetime with the correct InputFormat and handle NaT.
Always work on a copy or save a cleaned version instead of overwriting raw data.

Views: 0

Comments

Please login to add a comment.

Don't have an account? Register now!

Where to Move

Move chapter:

  • ☰ 1. Getting Started with MATLAB
    • ☰ 1.1. What Is MATLAB and What Is It Used For
      • ☰ 1.2. Installing MATLAB and Setting Up the Environment
        • ☰ 1.3. Understanding the MATLAB Desktop Interface
          • ☰ 1.4. Command Window, Workspace, and Current Folder
            • ☰ 1.5. Basic Workflow and Running Simple Commands
              • ☰ 1.6. Getting Help and Documentation
              • ☰ 2. MATLAB Language Basics
                • ☰ 2.1. MATLAB Syntax and Expressions
                  • ☰ 2.2. Variables and Assignment
                    • ☰ 2.3. Data Types and Type Conversion
                      • ☰ 2.4. Vectors and Matrices Fundamentals
                        • ☰ 2.5. Creating and Indexing Arrays
                          • ☰ 2.6. Basic Arithmetic and Elementwise Operations
                            • ☰ 2.7. Using Built-In Constants and Functions
                            • ☰ 3. Working with Variables and the Workspace
                              • ☰ 3.1. Creating, Viewing, and Editing Variables
                                • ☰ 3.2. Workspace Browser and Variable Inspector
                                  • ☰ 3.3. Saving and Loading Data Files
                                    • ☰ 3.4. MAT-Files and Importing External Data
                                      • ☰ 3.5. Clearing Variables and Managing the Workspace
                                      • ☰ 4. Vectors, Matrices, and Arrays
                                        • ☰ 4.1. Row and Column Vectors
                                          • ☰ 4.2. Matrix Creation and Concatenation
                                            • ☰ 4.3. Indexing, Slicing, and Logical Indexing
                                              • ☰ 4.4. Matrix Operations vs Elementwise Operations
                                                • ☰ 4.5. Reshaping and Rearranging Arrays
                                                  • ☰ 4.6. Special Matrices and Functions
                                                  • ☰ 5. Basic Programming Constructs
                                                    • ☰ 5.1. Scripts vs Functions
                                                      • ☰ 5.2. Control Flow with if, elseif, else
                                                        • ☰ 5.3. Loops with for and while
                                                          • ☰ 5.4. break, continue, and return
                                                            • ☰ 5.5. Logical Expressions and Relational Operators
                                                              • ☰ 5.6. Best Practices for Clear and Readable Code
                                                              • ☰ 6. Functions in MATLAB
                                                                • ☰ 6.1. Creating Simple Functions
                                                                  • ☰ 6.2. Function Inputs and Outputs
                                                                    • ☰ 6.3. Function Files and the Path
                                                                      • ☰ 6.4. Local vs Global Variables
                                                                        • ☰ 6.5. Anonymous Functions
                                                                          • ☰ 6.6. Organizing Code into Multiple Functions
                                                                          • ☰ 7. Working with Numbers and Math
                                                                            • ☰ 7.1. Scalars and Vectorized Computations
                                                                              • ☰ 7.2. Basic Mathematical Functions
                                                                                • ☰ 7.3. Trigonometric and Exponential Functions
                                                                                  • ☰ 7.4. Complex Numbers in MATLAB
                                                                                    • ☰ 7.5. Rounding, Modulo, and Special Values
                                                                                      • ☰ 7.6. Numerical Precision and Floating Point Considerations
                                                                                      • ☰ 8. Strings, Characters, and Categorical Data
                                                                                        • ☰ 8.1. Character Arrays and String Arrays
                                                                                          • ☰ 8.2. Creating and Manipulating Strings
                                                                                            • ☰ 8.3. Converting Between Data Types and Strings
                                                                                              • ☰ 8.4. Comparing and Searching Strings
                                                                                                • ☰ 8.5. Categorical Arrays Basics
                                                                                                • ☰ 9. Tables, Structures, and Cell Arrays
                                                                                                  • ☰ 9.1. Introduction to Tables
                                                                                                    • ☰ 9.2. Creating Tables and Accessing Data
                                                                                                      • ☰ 9.3. Modifying and Joining Tables
                                                                                                        • ☰ 9.4. Structures and Nested Structures
                                                                                                          • ☰ 9.5. Cell Arrays and Cell Indexing
                                                                                                            • ☰ 9.6. Choosing the Right Container Type
                                                                                                            • ☰ 10. Importing and Exporting Data
                                                                                                              • ☰ 10.1. Reading Text and CSV Files
                                                                                                                • ☰ 10.2. Importing Excel Spreadsheets
                                                                                                                  • ☰ 10.3. Importing Data with the Import Tool
                                                                                                                    • ☰ 10.4. Exporting Data to Files
                                                                                                                      • ☰ 10.5. Working with Image and Audio Files
                                                                                                                        • ☰ 10.6. Basic File Management from MATLAB
                                                                                                                        • ☰ 11. Plotting and Visualization
                                                                                                                          • ☰ 11.1. Introduction to 2D Plotting
                                                                                                                            • ☰ 11.2. Customizing Plots and Axes
                                                                                                                              • ☰ 11.3. Multiple Plots in One Figure
                                                                                                                                • ☰ 11.4. Labels, Titles, Legends, and Annotations
                                                                                                                                  • ☰ 11.5. Plotting with Different Line and Marker Styles
                                                                                                                                    • ☰ 11.6. Saving and Exporting Figures
                                                                                                                                    • ☰ 12. Advanced Plotting for Beginners
                                                                                                                                      • ☰ 12.1. Subplots and Tiled Layouts
                                                                                                                                        • ☰ 12.2. Bar, Scatter, and Stem Plots
                                                                                                                                          • ☰ 12.3. Histograms and Density Plots
                                                                                                                                            • ☰ 12.4. 3D Plots and Surface Plots
                                                                                                                                              • ☰ 12.5. Using plottools and Interactive Editing
                                                                                                                                                • ☰ 12.6. Basics of Handle Graphics
                                                                                                                                                • ☰ 13. Working with Scripts and Live Scripts
                                                                                                                                                  • ☰ 13.1. Creating and Running Script Files
                                                                                                                                                    • ☰ 13.2. Sectioning Code and Running Sections
                                                                                                                                                      • ☰ 13.3. Live Scripts and Rich Text
                                                                                                                                                        • ☰ 13.4. Adding Equations, Images, and Outputs
                                                                                                                                                          • ☰ 13.5. Sharing and Exporting Live Scripts
                                                                                                                                                          • ☰ 14. Debugging and Code Quality
                                                                                                                                                            • ☰ 14.1. Finding and Fixing Syntax Errors
                                                                                                                                                              • ☰ 14.2. Setting Breakpoints and Stepping Through Code
                                                                                                                                                                • ☰ 14.3. Inspecting Variable Values During Execution
                                                                                                                                                                  • ☰ 14.4. Using the Editor Debugging Tools
                                                                                                                                                                    • ☰ 14.5. Using Code Analyzer and Warnings
                                                                                                                                                                      • ☰ 14.6. Writing Clean and Maintainable MATLAB Code
                                                                                                                                                                      • ☰ 15. Basic Linear Algebra with MATLAB
                                                                                                                                                                        • ☰ 15.1. Vectors and Norms
                                                                                                                                                                          • ☰ 15.2. Matrix Addition and Multiplication
                                                                                                                                                                            • ☰ 15.3. Solving Linear Systems with Backslash
                                                                                                                                                                              • ☰ 15.4. Inverses, Determinants, and Rank
                                                                                                                                                                                • ☰ 15.5. Eigenvalues and Eigenvectors
                                                                                                                                                                                  • ☰ 15.6. Using Linear Algebra Functions Safely
                                                                                                                                                                                  • ☰ 16. Data Analysis and Statistics Fundamentals
                                                                                                                                                                                    • ☰ 16.1. Descriptive Statistics and Summary Measures
                                                                                                                                                                                      • ☰ 16.2. Sorting, Filtering, and Grouping Data
                                                                                                                                                                                        • ☰ 16.3. Basic Correlation and Regression
                                                                                                                                                                                          • ☰ 16.4. Simple Data Cleaning Techniques
                                                                                                                                                                                            • ☰ 16.5. Basic Time Series Handling
                                                                                                                                                                                            • ☰ 17. Working with Time and Dates
                                                                                                                                                                                              • ☰ 17.1. Representing Dates and Times
                                                                                                                                                                                                • ☰ 17.2. Creating and Formatting datetime Arrays
                                                                                                                                                                                                  • ☰ 17.3. Date Calculations and Differences
                                                                                                                                                                                                    • ☰ 17.4. Plotting Data over Time
                                                                                                                                                                                                    • ☰ 18. Introduction to Signal and Image Handling
                                                                                                                                                                                                      • ☰ 18.1. Basic Signal Representation in MATLAB
                                                                                                                                                                                                        • ☰ 18.2. Simple Filtering and Transformations
                                                                                                                                                                                                          • ☰ 18.3. Reading, Displaying, and Writing Images
                                                                                                                                                                                                            • ☰ 18.4. Basic Image Transformations
                                                                                                                                                                                                            • ☰ 19. Automation and Scripting Tasks
                                                                                                                                                                                                              • ☰ 19.1. Batch Processing of Files
                                                                                                                                                                                                                • ☰ 19.2. Writing Reusable Utility Scripts
                                                                                                                                                                                                                  • ☰ 19.3. Using Relative Paths and the MATLAB Path
                                                                                                                                                                                                                    • ☰ 19.4. Simple User Interaction with input and disp
                                                                                                                                                                                                                    • ☰ 20. Introduction to Toolboxes
                                                                                                                                                                                                                      • ☰ 20.1. What Are Toolboxes and How to Use Them
                                                                                                                                                                                                                        • ☰ 20.2. Exploring Installed Toolboxes
                                                                                                                                                                                                                          • ☰ 20.3. Finding Functions in Toolboxes
                                                                                                                                                                                                                            • ☰ 20.4. Add-Ons and MATLAB File Exchange
                                                                                                                                                                                                                            • ☰ 21. Best Practices and Next Steps
                                                                                                                                                                                                                              • ☰ 21.1. Organizing Projects and Folders
                                                                                                                                                                                                                                • ☰ 21.2. Version Control Basics with MATLAB
                                                                                                                                                                                                                                  • ☰ 21.3. Performance Tips for Beginners
                                                                                                                                                                                                                                    • ☰ 21.4. Learning Resources and Documentation Path
                                                                                                                                                                                                                                      • ☰ 21.5. Suggested Practice Projects for Beginners