KAHIBARO Bookshelves
Discord Login Register

17.4. Chi-Square Tests

Comparing distributions

Chi square tests in ROOT are used to quantify how similar two binned distributions are, or how well a model matches a histogram. In this chapter the focus is on the practical use of chi square within ROOT, not on the full statistical theory.

When you compare distributions in ROOT you usually start from histograms. Suppose you have a histogram filled with data, and either another histogram filled with simulated or reference data, or a function that represents a theoretical model. The chi square statistic measures the squared differences between “observed” and “expected” counts, scaled by their uncertainties, bin by bin.

For two histograms, a very common task is to check if they can be considered to come from the same underlying distribution. In ROOT this is typically done with the Chi2Test method of TH1. The simplest usage compares two histograms h1 and h2:

cpp
Double_t pvalue = h1->Chi2Test(h2);

By default, Chi2Test returns a p value. This value is the probability, under the assumption that the two histograms are statistically compatible, of obtaining a chi square as large as or larger than the one computed from the histograms. Values of the p value close to 1 indicate good agreement. Very small p values indicate that the histograms are unlikely to represent the same distribution.

You can change what Chi2Test returns and how it behaves with options. A very common option is "CHI2" to get the chi square statistic itself instead of the p value:

cpp
Double_t chi2 = h1->Chi2Test(h2, "CHI2");

Another important option is "UU" to use unweighted histograms and the correct Poisson uncertainties, and "NORM" if you want ROOT to normalize the histograms to the same integral before computing the chi square. Combining options is done by concatenating characters in the option string, for example "CHI2 UU NORM". Always read the ROOT reference for the exact meaning of each option, because it controls how uncertainties and normalizations are handled.

The chi square comparison depends on the binning. Bins that have very low statistics can make the chi square test unreliable. Before using Chi2Test, you should ensure that the two histograms have the same number of bins and the same bin edges. ROOT does not automatically rebin them. You can check consistency by comparing GetNbinsX, GetXaxis()->GetXmin, and GetXaxis()->GetXmax, and if needed, use Rebin or recreate histograms with the same binning.

ROOT also provides access to more detailed output from Chi2Test. For example, you can get both the chi square and the number of degrees of freedom in one call:

cpp
Int_t ndf = 0;
Double_t chi2 = h1->Chi2Test(h2, "CHI2 NORM", &ndf);

You can then construct the reduced chi square, which is the chi square divided by the number of degrees of freedom, using this result. The reduced chi square is often used to summarize the overall agreement between two distributions.

When comparing a histogram with a reference distribution that has no statistical uncertainty, for example a fixed template with very high statistics, you should reflect that in the way you compute uncertainties. Some options in Chi2Test are designed to handle the case where only one histogram has statistical fluctuations. Checking which histogram is treated as reference and which as data is crucial, because it changes the denominator in each bin of the chi square sum.

It is also important to remember that the chi square test is meaningful only when the bin contents represent independent counts and when expected counts in each bin are not too small. If many bins have very few entries, you should consider merging bins or using a different test such as a likelihood ratio test, which is not the topic of this chapter.

Chi square comparison rule
For two histograms with bin contents $n_i$ and $m_i$ and bin uncertainties $\sigma_i$, a typical chi square statistic has the form
$$
\chi^2 = \sum_i \frac{(n_i - m_i)^2}{\sigma_i^2},
$$
with degrees of freedom approximately equal to the number of bins used in the sum, possibly minus constraints such as normalization.

In ROOT, Chi2Test encapsulates this computation and returns either the p value or the chi square, together with the degrees of freedom if requested. You should interpret the numerical result in the context of your physics problem and chosen significance level, not as an automatic pass or fail.

Evaluating models

Evaluating how well a model describes data is another central use of chi square in ROOT. Here the “model” is usually a TF1 function that has been fitted to a histogram. In that case, the chi square is computed during the fit, and ROOT stores and reports it along with the fit parameters and their uncertainties.

When you fit a histogram h with a function f using Fit, for example:

cpp
h->Fit("f");

ROOT by default uses a chi square minimization algorithm for binned data, unless you request a different fit method. The minimization finds the parameter values that produce the smallest chi square. After the fit, you can retrieve the fitted function and examine its chi square and related quantities.

The TF1 returned by the fit can be obtained with GetFunction:

cpp
TF1 *fitfunc = h->GetFunction("f");
Double_t chi2      = fitfunc->GetChisquare();
Int_t    ndf       = fitfunc->GetNDF();
Double_t chi2_red  = chi2 / ndf;

The chi square value, together with the number of degrees of freedom, is your main diagnostic for the overall quality of the fit. The reduced chi square, chi2 divided by ndf, is often used as a single summary number. Values near 1 usually indicate that the model describes the data within the expected statistical fluctuations, assuming that uncertainties are correctly estimated. Values much larger than 1 suggest that the model does not capture the data structure or that uncertainties are underestimated. Values much less than 1 can indicate that uncertainties are overestimated, that data points are correlated, or that there are other issues in the error model.

You can also compute a p value for the fit using the chi square and degrees of freedom. ROOT provides the TMath::Prob function, which returns the probability of obtaining a chi square at least as large as the observed one if the model is correct:

cpp
Double_t pvalue = TMath::Prob(chi2, ndf);

This p value can be used to quantify the goodness of fit in a probabilistic way. It is analogous to the p value from Chi2Test for two histograms, but here the comparison is between data and the fitted model. A very small p value indicates that the model is unlikely to be consistent with the data under the assumed uncertainties.

Goodness of fit from chi square
Given a fitted model with chi square $\chi^2$ and degrees of freedom $\nu$,

  1. The reduced chi square is
    $$
    \chi^2_{\text{red}} = \frac{\chi^2}{\nu}.
    $$
  2. The goodness of fit p value is
    $$
    p = P(\chi^2_{\nu} \ge \chi^2) = \text{TMath::Prob}(\chi^2, \nu) \text{ in ROOT}.
    $$

When you evaluate models with chi square in ROOT, you should keep in mind how errors are assigned to bins. For histograms filled with unweighted counts, the natural bin uncertainty is the square root of the bin content. ROOT uses this by default for chi square fits. If you have weighted events, or if you want to specify custom uncertainties, you can use Sumw2 on the histogram so that ROOT stores the sum of squared weights and uses it in error calculations. For very low statistics, a chi square fit is often not reliable, and an unbinned or likelihood based approach is usually preferred.

The choice of fit range also affects the chi square and goodness of fit. If you specify a narrower range in the Fit call, for example:

cpp
h->Fit("f", "", "", xlow, xhigh);

then only bins in that range contribute to the chi square and degrees of freedom. This allows you to focus on the region of interest, such as a peak, but it also changes the interpretation of the goodness of fit. You should always report the fit range when you quote chi square and p values.

Model evaluation is not limited to a single fit. You can fit the same histogram with several different models, for example a Gaussian and a Gaussian plus polynomial background, and compare their chi square values and p values. Models with more free parameters can usually achieve smaller chi square, so it is important to consider the number of degrees of freedom. You should compare reduced chi square values rather than raw chi square values, and you should be cautious when adding many parameters just to force chi square close to 1 without physical motivation.

Finally, chi square is a global measure. It tells you about the average agreement across all bins, but it does not show where the model fails. To understand the detailed behavior, you should inspect residuals, that is the bin by bin differences between data and model, which are covered in a separate chapter. Combined with the global chi square, residual plots give a much more complete picture of how well your model describes the data.

Views: 0

Comments

Please login to add a comment.

Don't have an account? Register now!

Where to Move

Move chapter:

  • ☰ 1. Introduction to ROOT
    • ☰ 1.1. What Is ROOT?
      • ☰ 1.2. Installing ROOT
        • ☰ 1.3. Starting ROOT
          • ☰ 1.4. Your First ROOT Session
          • ☰ 2. C++ Basics for ROOT
            • ☰ 2.1. Why ROOT Uses C++
              • ☰ 2.2. Variables and Data Types
                • ☰ 2.3. Operators
                  • ☰ 2.4. Conditional Statements
                    • ☰ 2.5. Loops
                      • ☰ 2.6. Functions
                        • ☰ 2.7. Arrays and Vectors
                        • ☰ 3. ROOT Macros
                          • ☰ 3.1. What Is a ROOT Macro?
                            • ☰ 3.2. Running ROOT Macros
                              • ☰ 3.3. Functions Inside ROOT Macros
                                • ☰ 3.4. Compiling ROOT Macros
                                  • ☰ 3.5. Organizing a ROOT Project
                                  • ☰ 4. ROOT Objects and the ROOT Object Model
                                    • ☰ 4.1. Introduction to ROOT Classes
                                      • ☰ 4.2. Object Names and Titles
                                        • ☰ 4.3. ROOT Collections
                                          • ☰ 4.4. Inspecting ROOT Objects
                                          • ☰ 5. Histograms
                                            • ☰ 5.1. Introduction to Histograms
                                              • ☰ 5.2. Creating a 1D Histogram
                                                • ☰ 5.3. Filling Histograms
                                                  • ☰ 5.4. Drawing Histograms
                                                    • ☰ 5.5. Histogram Properties
                                                      • ☰ 5.6. Accessing Histogram Bins
                                                        • ☰ 5.7. Histogram Errors
                                                          • ☰ 5.8. Normalizing Histograms
                                                            • ☰ 5.9. Combining Histograms
                                                            • ☰ 6. Two-Dimensional Histograms
                                                              • ☰ 6.1. Creating 2D Histograms
                                                                • ☰ 6.2. Filling 2D Histograms
                                                                  • ☰ 6.3. Drawing 2D Histograms
                                                                    • ☰ 6.4. Projections
                                                                      • ☰ 6.5. Profiles
                                                                      • ☰ 7. Graphs
                                                                        • ☰ 7.1. Introduction to TGraph
                                                                          • ☰ 7.2. Creating a TGraph
                                                                            • ☰ 7.3. Graph Styling
                                                                              • ☰ 7.4. TGraphErrors
                                                                                • ☰ 7.5. TGraphAsymmErrors
                                                                                  • ☰ 7.6. MultiGraph
                                                                                  • ☰ 8. Canvases and Plotting
                                                                                    • ☰ 8.1. TCanvas
                                                                                      • ☰ 8.2. Dividing a Canvas
                                                                                        • ☰ 8.3. Drawing Multiple Objects
                                                                                          • ☰ 8.4. Legends
                                                                                            • ☰ 8.5. Text and Annotations
                                                                                              • ☰ 8.6. Logarithmic Axes
                                                                                                • ☰ 8.7. Saving Plots
                                                                                                • ☰ 9. Styling ROOT Plots
                                                                                                  • ☰ 9.1. ROOT Plot Styles
                                                                                                    • ☰ 9.2. Axis Configuration
                                                                                                      • ☰ 9.3. Line Styles
                                                                                                        • ☰ 9.4. Marker Styles
                                                                                                          • ☰ 9.5. Publication-Quality Figures
                                                                                                          • ☰ 10. Mathematical Functions
                                                                                                            • ☰ 10.1. Introduction to TF1
                                                                                                              • ☰ 10.2. Built-in Functions
                                                                                                                • ☰ 10.3. Custom Functions
                                                                                                                  • ☰ 10.4. Evaluating Functions
                                                                                                                    • ☰ 10.5. Function Integrals
                                                                                                                    • ☰ 11. Curve Fitting
                                                                                                                      • ☰ 11.1. Introduction to Fitting
                                                                                                                        • ☰ 11.2. Gaussian Fits
                                                                                                                          • ☰ 11.3. Polynomial Fits
                                                                                                                            • ☰ 11.4. Custom Fit Functions
                                                                                                                              • ☰ 11.5. Fit Parameters
                                                                                                                                • ☰ 11.6. Goodness of Fit
                                                                                                                                  • ☰ 11.7. Fit Residuals
                                                                                                                                    • ☰ 11.8. Fitting Graphs
                                                                                                                                    • ☰ 12. ROOT Files
                                                                                                                                      • ☰ 12.1. Introduction to ROOT Files
                                                                                                                                        • ☰ 12.2. Creating ROOT Files
                                                                                                                                          • ☰ 12.3. Writing Objects
                                                                                                                                            • ☰ 12.4. Reading ROOT Files
                                                                                                                                              • ☰ 12.5. Exploring ROOT Files
                                                                                                                                                • ☰ 12.6. Directories Inside ROOT Files
                                                                                                                                                • ☰ 13. TTrees
                                                                                                                                                  • ☰ 13.1. Introduction to TTrees
                                                                                                                                                    • ☰ 13.2. Creating a TTree
                                                                                                                                                      • ☰ 13.3. Filling a TTree
                                                                                                                                                        • ☰ 13.4. Writing a TTree to a File
                                                                                                                                                          • ☰ 13.5. Reading a TTree
                                                                                                                                                            • ☰ 13.6. Inspecting TTrees
                                                                                                                                                              • ☰ 13.7. Drawing Data from TTrees
                                                                                                                                                                • ☰ 13.8. Applying Selection Cuts
                                                                                                                                                                  • ☰ 13.9. Branch Addresses
                                                                                                                                                                  • ☰ 14. Event-Based Data Analysis
                                                                                                                                                                    • ☰ 14.1. Understanding Event Data
                                                                                                                                                                      • ☰ 14.2. Event Loops
                                                                                                                                                                        • ☰ 14.3. Event Selection
                                                                                                                                                                          • ☰ 14.4. Derived Quantities
                                                                                                                                                                            • ☰ 14.5. Analysis Workflow
                                                                                                                                                                            • ☰ 15. Working with Real Experimental Data
                                                                                                                                                                              • ☰ 15.1. Importing Text Data
                                                                                                                                                                                • ☰ 15.2. Converting Data to ROOT
                                                                                                                                                                                  • ☰ 15.3. Data Cleaning
                                                                                                                                                                                    • ☰ 15.4. Data Calibration
                                                                                                                                                                                      • ☰ 15.5. Signal and Background
                                                                                                                                                                                      • ☰ 16. Random Numbers and Simulations
                                                                                                                                                                                        • ☰ 16.1. Random Number Generation
                                                                                                                                                                                          • ☰ 16.2. Common Distributions
                                                                                                                                                                                            • ☰ 16.3. Simple Monte Carlo Simulation
                                                                                                                                                                                              • ☰ 16.4. Detector Resolution Simulation
                                                                                                                                                                                                • ☰ 16.5. Efficiency Simulation
                                                                                                                                                                                                • ☰ 17. Statistical Analysis with ROOT
                                                                                                                                                                                                  • ☰ 17.1. Basic Statistics
                                                                                                                                                                                                    • ☰ 17.2. Probability Distributions
                                                                                                                                                                                                      • ☰ 17.3. Statistical Uncertainties
                                                                                                                                                                                                        • ☰ 17.4. Chi-Square Tests
                                                                                                                                                                                                          • ☰ 17.5. Confidence Intervals
                                                                                                                                                                                                          • ☰ 18. Modern ROOT with RDataFrame
                                                                                                                                                                                                            • ☰ 18.1. Introduction to RDataFrame
                                                                                                                                                                                                              • ☰ 18.2. Loading Data
                                                                                                                                                                                                                • ☰ 18.3. Filtering Data
                                                                                                                                                                                                                  • ☰ 18.4. Defining New Columns
                                                                                                                                                                                                                    • ☰ 18.5. Creating Histograms
                                                                                                                                                                                                                      • ☰ 18.6. Statistical Operations
                                                                                                                                                                                                                        • ☰ 18.7. Saving Results
                                                                                                                                                                                                                          • ☰ 18.8. Multithreading
                                                                                                                                                                                                                          • ☰ 19. ROOT and Python
                                                                                                                                                                                                                            • ☰ 19.1. Introduction to PyROOT
                                                                                                                                                                                                                              • ☰ 19.2. ROOT Objects in Python
                                                                                                                                                                                                                                • ☰ 19.3. Reading ROOT Files with Python
                                                                                                                                                                                                                                  • ☰ 19.4. PyROOT and NumPy
                                                                                                                                                                                                                                    • ☰ 19.5. PyROOT and Matplotlib
                                                                                                                                                                                                                                      • ☰ 19.6. Choosing Between C++ and Python
                                                                                                                                                                                                                                      • ☰ 20. Advanced TTree Techniques
                                                                                                                                                                                                                                        • ☰ 20.1. Trees with Multiple Branches
                                                                                                                                                                                                                                          • ☰ 20.2. Storing std::vector Objects
                                                                                                                                                                                                                                            • ☰ 20.3. TChain
                                                                                                                                                                                                                                              • ☰ 20.4. Friends
                                                                                                                                                                                                                                                • ☰ 20.5. Efficient TTree Analysis
                                                                                                                                                                                                                                                • ☰ 21. ROOT for Particle and Nuclear Physics
                                                                                                                                                                                                                                                  • ☰ 21.1. Four-Vectors
                                                                                                                                                                                                                                                    • ☰ 21.2. Particle Kinematics
                                                                                                                                                                                                                                                      • ☰ 21.3. Invariant Mass
                                                                                                                                                                                                                                                        • ☰ 21.4. Angular Distributions
                                                                                                                                                                                                                                                          • ☰ 21.5. Detector Spectra
                                                                                                                                                                                                                                                          • ☰ 22. Debugging and Common ROOT Problems
                                                                                                                                                                                                                                                            • ☰ 22.1. ROOT Object Ownership
                                                                                                                                                                                                                                                              • ☰ 22.2. Null Pointers
                                                                                                                                                                                                                                                                • ☰ 22.3. Segmentation Faults
                                                                                                                                                                                                                                                                  • ☰ 22.4. Missing Histograms
                                                                                                                                                                                                                                                                    • ☰ 22.5. ROOT File Problems
                                                                                                                                                                                                                                                                      • ☰ 22.6. TTree Branch Problems
                                                                                                                                                                                                                                                                        • ☰ 22.7. Histogram Range Problems
                                                                                                                                                                                                                                                                          • ☰ 22.8. Fit Failures
                                                                                                                                                                                                                                                                            • ☰ 22.9. Debugging ROOT Macros
                                                                                                                                                                                                                                                                            • ☰ 23. Writing Better ROOT Analysis Code
                                                                                                                                                                                                                                                                              • ☰ 23.1. Separating Analysis and Plotting
                                                                                                                                                                                                                                                                                • ☰ 23.2. Using Functions
                                                                                                                                                                                                                                                                                  • ☰ 23.3. Using Header Files
                                                                                                                                                                                                                                                                                    • ☰ 23.4. Naming Conventions
                                                                                                                                                                                                                                                                                      • ☰ 23.5. Avoiding Hard-Coded Values
                                                                                                                                                                                                                                                                                        • ☰ 23.6. Configuration Parameters
                                                                                                                                                                                                                                                                                          • ☰ 23.7. Reproducible Analysis
                                                                                                                                                                                                                                                                                            • ☰ 23.8. Version Control with Git
                                                                                                                                                                                                                                                                                            • ☰ 24. Final Project
                                                                                                                                                                                                                                                                                              • ☰ 24.1. Project Overview
                                                                                                                                                                                                                                                                                                • ☰ 24.2. Analyze a complete scientific dataset using ROOT.
                                                                                                                                                                                                                                                                                                  • ☰ 24.3. Import the Data
                                                                                                                                                                                                                                                                                                    • ☰ 24.4. Explore the Dataset
                                                                                                                                                                                                                                                                                                      • ☰ 24.5. Apply Data Selection
                                                                                                                                                                                                                                                                                                        • ☰ 24.6. Create Histograms
                                                                                                                                                                                                                                                                                                          • ☰ 24.7. Fit the Data
                                                                                                                                                                                                                                                                                                            • ☰ 24.8. Calculate Physical Quantities
                                                                                                                                                                                                                                                                                                              • ☰ 24.9. Estimate Statistical Uncertainties
                                                                                                                                                                                                                                                                                                                • ☰ 24.10. Create Publication-Quality Figures
                                                                                                                                                                                                                                                                                                                  • ☰ 24.11. Save the Analysis Results
                                                                                                                                                                                                                                                                                                                    • ☰ 24.12. Write a Short Analysis Report
                                                                                                                                                                                                                                                                                                                    • ☰ 25. Appendices
                                                                                                                                                                                                                                                                                                                      • ☰ 25.1. A. ROOT Command Cheat Sheet
                                                                                                                                                                                                                                                                                                                        • ☰ 25.2. B. ROOT Histogram Classes
                                                                                                                                                                                                                                                                                                                          • ☰ 25.3. C. ROOT Graph Classes
                                                                                                                                                                                                                                                                                                                            • ☰ 25.4. D. Common TTree Commands
                                                                                                                                                                                                                                                                                                                              • ☰ 25.5. E. Common Drawing Options
                                                                                                                                                                                                                                                                                                                                • ☰ 25.6. F. Common Fit Functions
                                                                                                                                                                                                                                                                                                                                  • ☰ 25.7. G. ROOT C++ Cheat Sheet
                                                                                                                                                                                                                                                                                                                                    • ☰ 25.8. H. PyROOT Cheat Sheet
                                                                                                                                                                                                                                                                                                                                      • ☰ 25.9. I. Common ROOT Errors and Solutions
                                                                                                                                                                                                                                                                                                                                        • ☰ 25.10. J. Recommended ROOT Analysis Workflow