15.3. Data Cleaning

Removing invalid events

In real experimental datasets, some events cannot be trusted. They may come from detector malfunctions, incomplete readout, corrupted files, or known periods of bad data taking. Data cleaning is the step where you remove, or at least flag, such events before doing any physics analysis.

In ROOT this usually happens either in an event loop over a TTree or in an RDataFrame pipeline. The logic is the same in both cases: for each event, you apply a series of checks, and only if all checks pass do you keep the event or use it to fill histograms.

Typical reasons to reject an event include missing detector information, non-physical values, known bad run numbers, or explicit quality flags provided by the experiment. Many experiments store event quality in dedicated branches such as isGoodEvent or trigger decision bits. If such branches exist, it is usually best to rely on them first and then add your own analysis specific checks.

In a traditional TTree loop, the basic pattern is to skip invalid events early in the loop. For example, you test the quality flags at the top of the event loop and use continue to move to the next entry if a problem is found. This keeps your analysis code for valid events simpler and easier to read, because you do not need to repeat the same checks everywhere.

In RDataFrame, the same idea is implemented with Filter. Each Filter keeps only events that satisfy a user defined condition. Multiple filters can be chained to gradually remove more and more problematic data, which also makes the logic of your cleaning steps explicit and reproducible. You can name filters or document them with comments so that later you know exactly why certain events were removed.

Whenever you remove events, it is important to understand the impact. If you remove too many, you may lose statistical power or introduce bias. If you remove too few, you may keep corrupted or non-physical events that can distort histograms and fits. For this reason, a common practice is to monitor how many events are rejected by each cleaning criterion, for example by counting them or filling control histograms for removed events. Comparing cleaned and uncleaned distributions can reveal whether your cuts are doing what you expect.

Finally, data cleaning decisions should be recorded, for example in comments in your macro, in a configuration file, or in an analysis note. Cleaned datasets are often reused for multiple analyses, so you want to be able to reconstruct exactly which event selection and quality requirements were applied, and why.

Range checks

Range checks are a specific and very common form of data cleaning. They ensure that values of physical quantities lie in regions that are physically meaningful and technically sensible. For example, an energy deposit cannot be negative, a time stamp must lie within the run duration, and an ADC count should not exceed the hardware limits of the digitizer.

Range checks are usually simple comparisons in C++, such as rejecting events when a branch value falls outside a predefined minimum and maximum. What matters is how you choose those limits. There are three main sources of information. First, detector or hardware specifications provide absolute technical limits. Second, physics knowledge provides realistic expectations for signal and background. Third, exploratory plots such as quick histograms of raw variables help you see the actual distribution of your data, including unexpected tails and outliers.

You typically apply range checks as early as possible in your analysis. For example, right after reading a TTree entry, you may check that energy > 0 and energy < Emax, or that a time variable lies between a start and stop time stored in the run conditions. If a variable has clear non-physical values used as error codes, such as -999 or 1e9, you either reject such events entirely or treat those values specially as missing data.

In RDataFrame, range checks translate directly into filters. You can write a filter expression like "E > 0 && E < 1000" to keep only events where the energy is within a certain range. Multiple variables can be checked together, for example requiring that all detectors in an array report values inside acceptable intervals. This can catch partial detector failures in single events.

Range checks can also be relaxed control tools. You might start with broad ranges that only remove wildly unphysical values and later refine them to define cleaner signal and control regions. It is important to distinguish between cleaning cuts, which remove clearly invalid data, and physics selection cuts, which define a sample for a particular measurement. Cleaning cuts should ideally not depend strongly on the physics you are trying to measure, otherwise you risk bias.

Monitoring the effect of each range check is essential. You can maintain counters for how many events failed each individual check. You can also plot distributions before and after applying a range cut. If a cut sharply removes a large central part of a distribution rather than only pathological tails, you may have chosen limits that are too tight or misinterpreted the units or calibration of the variable.

Missing values

Real experimental data often contain missing or incomplete information. A detector may have been temporarily off, some channels may not have recorded data for a particular event, or a reconstruction algorithm may have failed to compute a quantity. In ROOT based analyses, missing values typically appear in three ways.

First, a branch might not exist at all in some files or runs. Second, a branch may exist but contain sentinel values that indicate missing data such as -999, -1, or very large numbers. Third, a variable may be stored as a container like std::vector, and a missing measurement for one channel might show up as a shorter vector than expected.

Your cleaning strategy for missing values depends on their cause and their frequency. If a critical variable is missing, and you cannot compute your observable without it, the simplest option is to reject that event from your main analysis. In practice this means checking for the sentinel value and skipping such entries. If you use RDataFrame, you would implement this as a filter that removes events with the sentinel. When entire runs lack a crucial branch, you may decide to exclude those runs from your dataset entirely, or treat them separately.

In some cases, you may want to keep events with missing values, but treat them differently. For example, if a timing measurement is occasionally missing but not essential to your primary observable, you might still use those events for energy spectra, but not for time dependent analyses. This leads naturally to the idea of defining multiple datasets or flags: one cleaned sample for analyses that need all variables present, and a more inclusive sample for analyses that only use a subset.

Another option is to replace missing values with some default or imputed value, but in physics analyses this must be done with significant care. Naive imputation with means or zeros can distort distributions and bias fits. In most high energy or nuclear physics workflows, the standard choice is to avoid imputation for core physics quantities and instead remove or specially flag events where essential measurements are missing.

When missingness is not random, it can bias results even if you simply remove affected events. For example, if a detector fails more often at high rates, you may preferentially lose high energy or high multiplicity events. To diagnose this, you can compare distributions for events with and without missing values. ROOT makes this easy because you can fill separate histograms depending on whether a variable is present, or draw distributions of another branch for events where a given branch has sentinel values.

From a technical perspective, you should document how missing values are encoded for each branch in your TTree. If your conversion from text or external formats into ROOT files is under your control, you can choose consistent sentinel values and even store additional boolean branches like hasEnergy or isValidTrack that make it easier to write clear and explicit cleaning code later.

Views: 12