KAHIBARO Bookshelves
Discord Login Register

11.6. Goodness of Fit

Chi-square

In ROOT, the standard way to quantify how well a model describes data is the chi square, written $\chi^2$. For binned data such as histograms, $\chi^2$ compares the observed counts in each bin with the counts predicted by the fit function, and weights the difference by the expected statistical uncertainty.

For a histogram with bins indexed by $i$, observed counts $O_i$ and predicted counts $E_i$, the usual definition is
$$
\chi^2 = \sum_{i} \frac{(O_i - E_i)^2}{\sigma_i^2},
$$
where $\sigma_i$ is the uncertainty on $O_i$. For Poisson statistics and unit weights, a common choice is $\sigma_i = \sqrt{O_i}$, possibly replaced by $\sqrt{E_i}$ in some conventions. ROOT will handle these technical details internally when you use standard fitting tools.

A large chi square value means that, on average, the data points lie far from the model prediction compared to their uncertainties, while a small value means the data points lie close to the model within their error bars.

In ROOT, when you fit a histogram or graph with a TF1 function using TH1::Fit, TGraph::Fit or similar, the fit result carries a chi square value. You can access it programmatically through the fit result object or through the fitted function. For example, if you fit a histogram h with a function "gaus" and save the result:

cpp
TFitResultPtr r = h->Fit("gaus", "S");  // "S" returns a fit result
double chi2 = r->Chi2();

You can also get it from the TF1 that was used in the fit, assuming f is that function:

cpp
double chi2 = f->GetChisquare();

Both give the same quantity, the total chi square of the fit over the range and bins used.

The chi square itself grows with the number of data points, so by itself it is not very informative. A $\chi^2$ of 50 could be either excellent or terrible, depending on whether there were 10 or 500 bins in the fit. To interpret it, you need to relate it to the number of degrees of freedom and look at the ratio, the reduced chi square, which is discussed later in this chapter.

Degrees of freedom

The number of degrees of freedom, usually written $\nu$ or ndf in ROOT, tells you how many independent pieces of information you have after accounting for the parameters that were adjusted in the fit. For a simple least squares fit to $N_{\text{points}}$ data points with a model that has $N_{\text{par}}$ free parameters, the standard definition is
$$
\nu = N_{\text{points}} - N_{\text{par}}.
$$

For binned histogram fits, $N_{\text{points}}$ is typically the number of bins used in the fit, not the number of events in the histogram. The number of free parameters is the number actually varied during the fit, which excludes parameters that you have fixed or tied to other parameters.

Degrees of freedom:
$$\nu = N_{\text{data points}} - N_{\text{fitted parameters}}.$$
ROOT usually calls this quantity ndf.

When you perform a fit with ROOT, it automatically determines ndf from the number of bins or points it is fitting and the number of free parameters in the function. You can inspect it via the fit result or the fitted function.

With a fit result pointer r:

cpp
int ndf = r->Ndf();

or from the TF1 itself:

cpp
int ndf = f->GetNDF();

Interpreting ndf is essential when judging goodness of fit. Typical expectations for a well specified model with correct uncertainties are:

  1. The chi square should be of order $\nu$.
  2. The quantity $\chi^2 / \nu$ should be of order 1.

Because $\chi^2$ fluctuates, especially for small $\nu$, these are only guidelines. To be more quantitative, you can compute the chi square probability, that is the probability of obtaining a chi square as large or larger than the one observed assuming the model is correct. This is available as

cpp
double prob = r->Prob();          // from the fit result
double prob2 = f->GetProb();      // from the TF1

A very small probability (for example less than $10^{-3}$) often indicates that the model is not adequate, the uncertainties are underestimated, or there are issues in the data. A very large probability close to 1 can also be a sign that uncertainties are overestimated.

Reduced chi-square

The reduced chi square, written $\chi^2_{\text{red}}$, is defined as the chi square divided by the number of degrees of freedom:
$$
\chi^2_{\text{red}} = \frac{\chi^2}{\nu}.
$$

This quantity is convenient because, for a good fit with properly estimated Gaussian uncertainties, you expect $\chi^2_{\text{red}}$ to be close to 1. This expectation is independent of the absolute number of data points. This is why fitting tools and analysis reports commonly quote $\chi^2_{\text{red}}$ when summarizing goodness of fit.

In ROOT, you typically compute the reduced chi square yourself using the values returned by the fit, for example:

cpp
TFitResultPtr r = h->Fit("gaus", "S");
double chi2 = r->Chi2();
int ndf = r->Ndf();
double chi2_red = chi2 / ndf;

or, using the TF1:

cpp
double chi2 = f->GetChisquare();
int ndf = f->GetNDF();
double chi2_red = chi2 / ndf;

You can also calculate and print it directly in your macros when producing fit diagnostics.

Reduced chi square:
$$\chi^2_{\text{red}} = \frac{\chi^2}{\nu}.$$
Typical interpretation:

  • $\chi^2_{\text{red}} \approx 1$: model and uncertainties broadly consistent with data.
  • $\chi^2_{\text{red}} \gg 1$: model does not describe data within the quoted errors or uncertainties are too small.
  • $\chi^2_{\text{red}} \ll 1$: uncertainties probably too large or data points correlated in a way not accounted for.

The table below summarizes common qualitative interpretations.

$\chi^2_{\text{red}}$ valueQualitative interpretation
around 1Fit consistent with data and uncertainties
from about 2 to 5Possible tension, check model and error estimates
much larger than 5Likely poor model, outliers, or underestimated errors
much smaller than 1Likely overestimated errors or strong correlations

When you inspect a fit visually in a ROOT canvas, comparing the curve with the data points or histogram, you should also look at the reduced chi square and the fit probability. A fit that looks reasonable by eye but has a very large $\chi^2_{\text{red}}$ is a warning sign that either the model is missing structure or the uncertainties are misestimated. Conversely, if $\chi^2_{\text{red}}$ is very small, then even a model that looks perfect may not be telling you much, because the data errors might be so large that many very different models could also be acceptable.

In practical ROOT analysis, a useful workflow is to save the chi square, degrees of freedom, reduced chi square and fit probability along with the fitted parameter values. This lets you later compare different models or different fit ranges on the same footing, and helps you decide which modeling choice provides the best description of your data in a statistically meaningful way.

Views: 0

Comments

Please login to add a comment.

Don't have an account? Register now!

Where to Move

Move chapter:

  • ☰ 1. Introduction to ROOT
    • ☰ 1.1. What Is ROOT?
      • ☰ 1.2. Installing ROOT
        • ☰ 1.3. Starting ROOT
          • ☰ 1.4. Your First ROOT Session
          • ☰ 2. C++ Basics for ROOT
            • ☰ 2.1. Why ROOT Uses C++
              • ☰ 2.2. Variables and Data Types
                • ☰ 2.3. Operators
                  • ☰ 2.4. Conditional Statements
                    • ☰ 2.5. Loops
                      • ☰ 2.6. Functions
                        • ☰ 2.7. Arrays and Vectors
                        • ☰ 3. ROOT Macros
                          • ☰ 3.1. What Is a ROOT Macro?
                            • ☰ 3.2. Running ROOT Macros
                              • ☰ 3.3. Functions Inside ROOT Macros
                                • ☰ 3.4. Compiling ROOT Macros
                                  • ☰ 3.5. Organizing a ROOT Project
                                  • ☰ 4. ROOT Objects and the ROOT Object Model
                                    • ☰ 4.1. Introduction to ROOT Classes
                                      • ☰ 4.2. Object Names and Titles
                                        • ☰ 4.3. ROOT Collections
                                          • ☰ 4.4. Inspecting ROOT Objects
                                          • ☰ 5. Histograms
                                            • ☰ 5.1. Introduction to Histograms
                                              • ☰ 5.2. Creating a 1D Histogram
                                                • ☰ 5.3. Filling Histograms
                                                  • ☰ 5.4. Drawing Histograms
                                                    • ☰ 5.5. Histogram Properties
                                                      • ☰ 5.6. Accessing Histogram Bins
                                                        • ☰ 5.7. Histogram Errors
                                                          • ☰ 5.8. Normalizing Histograms
                                                            • ☰ 5.9. Combining Histograms
                                                            • ☰ 6. Two-Dimensional Histograms
                                                              • ☰ 6.1. Creating 2D Histograms
                                                                • ☰ 6.2. Filling 2D Histograms
                                                                  • ☰ 6.3. Drawing 2D Histograms
                                                                    • ☰ 6.4. Projections
                                                                      • ☰ 6.5. Profiles
                                                                      • ☰ 7. Graphs
                                                                        • ☰ 7.1. Introduction to TGraph
                                                                          • ☰ 7.2. Creating a TGraph
                                                                            • ☰ 7.3. Graph Styling
                                                                              • ☰ 7.4. TGraphErrors
                                                                                • ☰ 7.5. TGraphAsymmErrors
                                                                                  • ☰ 7.6. MultiGraph
                                                                                  • ☰ 8. Canvases and Plotting
                                                                                    • ☰ 8.1. TCanvas
                                                                                      • ☰ 8.2. Dividing a Canvas
                                                                                        • ☰ 8.3. Drawing Multiple Objects
                                                                                          • ☰ 8.4. Legends
                                                                                            • ☰ 8.5. Text and Annotations
                                                                                              • ☰ 8.6. Logarithmic Axes
                                                                                                • ☰ 8.7. Saving Plots
                                                                                                • ☰ 9. Styling ROOT Plots
                                                                                                  • ☰ 9.1. ROOT Plot Styles
                                                                                                    • ☰ 9.2. Axis Configuration
                                                                                                      • ☰ 9.3. Line Styles
                                                                                                        • ☰ 9.4. Marker Styles
                                                                                                          • ☰ 9.5. Publication-Quality Figures
                                                                                                          • ☰ 10. Mathematical Functions
                                                                                                            • ☰ 10.1. Introduction to TF1
                                                                                                              • ☰ 10.2. Built-in Functions
                                                                                                                • ☰ 10.3. Custom Functions
                                                                                                                  • ☰ 10.4. Evaluating Functions
                                                                                                                    • ☰ 10.5. Function Integrals
                                                                                                                    • ☰ 11. Curve Fitting
                                                                                                                      • ☰ 11.1. Introduction to Fitting
                                                                                                                        • ☰ 11.2. Gaussian Fits
                                                                                                                          • ☰ 11.3. Polynomial Fits
                                                                                                                            • ☰ 11.4. Custom Fit Functions
                                                                                                                              • ☰ 11.5. Fit Parameters
                                                                                                                                • ☰ 11.6. Goodness of Fit
                                                                                                                                  • ☰ 11.7. Fit Residuals
                                                                                                                                    • ☰ 11.8. Fitting Graphs
                                                                                                                                    • ☰ 12. ROOT Files
                                                                                                                                      • ☰ 12.1. Introduction to ROOT Files
                                                                                                                                        • ☰ 12.2. Creating ROOT Files
                                                                                                                                          • ☰ 12.3. Writing Objects
                                                                                                                                            • ☰ 12.4. Reading ROOT Files
                                                                                                                                              • ☰ 12.5. Exploring ROOT Files
                                                                                                                                                • ☰ 12.6. Directories Inside ROOT Files
                                                                                                                                                • ☰ 13. TTrees
                                                                                                                                                  • ☰ 13.1. Introduction to TTrees
                                                                                                                                                    • ☰ 13.2. Creating a TTree
                                                                                                                                                      • ☰ 13.3. Filling a TTree
                                                                                                                                                        • ☰ 13.4. Writing a TTree to a File
                                                                                                                                                          • ☰ 13.5. Reading a TTree
                                                                                                                                                            • ☰ 13.6. Inspecting TTrees
                                                                                                                                                              • ☰ 13.7. Drawing Data from TTrees
                                                                                                                                                                • ☰ 13.8. Applying Selection Cuts
                                                                                                                                                                  • ☰ 13.9. Branch Addresses
                                                                                                                                                                  • ☰ 14. Event-Based Data Analysis
                                                                                                                                                                    • ☰ 14.1. Understanding Event Data
                                                                                                                                                                      • ☰ 14.2. Event Loops
                                                                                                                                                                        • ☰ 14.3. Event Selection
                                                                                                                                                                          • ☰ 14.4. Derived Quantities
                                                                                                                                                                            • ☰ 14.5. Analysis Workflow
                                                                                                                                                                            • ☰ 15. Working with Real Experimental Data
                                                                                                                                                                              • ☰ 15.1. Importing Text Data
                                                                                                                                                                                • ☰ 15.2. Converting Data to ROOT
                                                                                                                                                                                  • ☰ 15.3. Data Cleaning
                                                                                                                                                                                    • ☰ 15.4. Data Calibration
                                                                                                                                                                                      • ☰ 15.5. Signal and Background
                                                                                                                                                                                      • ☰ 16. Random Numbers and Simulations
                                                                                                                                                                                        • ☰ 16.1. Random Number Generation
                                                                                                                                                                                          • ☰ 16.2. Common Distributions
                                                                                                                                                                                            • ☰ 16.3. Simple Monte Carlo Simulation
                                                                                                                                                                                              • ☰ 16.4. Detector Resolution Simulation
                                                                                                                                                                                                • ☰ 16.5. Efficiency Simulation
                                                                                                                                                                                                • ☰ 17. Statistical Analysis with ROOT
                                                                                                                                                                                                  • ☰ 17.1. Basic Statistics
                                                                                                                                                                                                    • ☰ 17.2. Probability Distributions
                                                                                                                                                                                                      • ☰ 17.3. Statistical Uncertainties
                                                                                                                                                                                                        • ☰ 17.4. Chi-Square Tests
                                                                                                                                                                                                          • ☰ 17.5. Confidence Intervals
                                                                                                                                                                                                          • ☰ 18. Modern ROOT with RDataFrame
                                                                                                                                                                                                            • ☰ 18.1. Introduction to RDataFrame
                                                                                                                                                                                                              • ☰ 18.2. Loading Data
                                                                                                                                                                                                                • ☰ 18.3. Filtering Data
                                                                                                                                                                                                                  • ☰ 18.4. Defining New Columns
                                                                                                                                                                                                                    • ☰ 18.5. Creating Histograms
                                                                                                                                                                                                                      • ☰ 18.6. Statistical Operations
                                                                                                                                                                                                                        • ☰ 18.7. Saving Results
                                                                                                                                                                                                                          • ☰ 18.8. Multithreading
                                                                                                                                                                                                                          • ☰ 19. ROOT and Python
                                                                                                                                                                                                                            • ☰ 19.1. Introduction to PyROOT
                                                                                                                                                                                                                              • ☰ 19.2. ROOT Objects in Python
                                                                                                                                                                                                                                • ☰ 19.3. Reading ROOT Files with Python
                                                                                                                                                                                                                                  • ☰ 19.4. PyROOT and NumPy
                                                                                                                                                                                                                                    • ☰ 19.5. PyROOT and Matplotlib
                                                                                                                                                                                                                                      • ☰ 19.6. Choosing Between C++ and Python
                                                                                                                                                                                                                                      • ☰ 20. Advanced TTree Techniques
                                                                                                                                                                                                                                        • ☰ 20.1. Trees with Multiple Branches
                                                                                                                                                                                                                                          • ☰ 20.2. Storing std::vector Objects
                                                                                                                                                                                                                                            • ☰ 20.3. TChain
                                                                                                                                                                                                                                              • ☰ 20.4. Friends
                                                                                                                                                                                                                                                • ☰ 20.5. Efficient TTree Analysis
                                                                                                                                                                                                                                                • ☰ 21. ROOT for Particle and Nuclear Physics
                                                                                                                                                                                                                                                  • ☰ 21.1. Four-Vectors
                                                                                                                                                                                                                                                    • ☰ 21.2. Particle Kinematics
                                                                                                                                                                                                                                                      • ☰ 21.3. Invariant Mass
                                                                                                                                                                                                                                                        • ☰ 21.4. Angular Distributions
                                                                                                                                                                                                                                                          • ☰ 21.5. Detector Spectra
                                                                                                                                                                                                                                                          • ☰ 22. Debugging and Common ROOT Problems
                                                                                                                                                                                                                                                            • ☰ 22.1. ROOT Object Ownership
                                                                                                                                                                                                                                                              • ☰ 22.2. Null Pointers
                                                                                                                                                                                                                                                                • ☰ 22.3. Segmentation Faults
                                                                                                                                                                                                                                                                  • ☰ 22.4. Missing Histograms
                                                                                                                                                                                                                                                                    • ☰ 22.5. ROOT File Problems
                                                                                                                                                                                                                                                                      • ☰ 22.6. TTree Branch Problems
                                                                                                                                                                                                                                                                        • ☰ 22.7. Histogram Range Problems
                                                                                                                                                                                                                                                                          • ☰ 22.8. Fit Failures
                                                                                                                                                                                                                                                                            • ☰ 22.9. Debugging ROOT Macros
                                                                                                                                                                                                                                                                            • ☰ 23. Writing Better ROOT Analysis Code
                                                                                                                                                                                                                                                                              • ☰ 23.1. Separating Analysis and Plotting
                                                                                                                                                                                                                                                                                • ☰ 23.2. Using Functions
                                                                                                                                                                                                                                                                                  • ☰ 23.3. Using Header Files
                                                                                                                                                                                                                                                                                    • ☰ 23.4. Naming Conventions
                                                                                                                                                                                                                                                                                      • ☰ 23.5. Avoiding Hard-Coded Values
                                                                                                                                                                                                                                                                                        • ☰ 23.6. Configuration Parameters
                                                                                                                                                                                                                                                                                          • ☰ 23.7. Reproducible Analysis
                                                                                                                                                                                                                                                                                            • ☰ 23.8. Version Control with Git
                                                                                                                                                                                                                                                                                            • ☰ 24. Final Project
                                                                                                                                                                                                                                                                                              • ☰ 24.1. Project Overview
                                                                                                                                                                                                                                                                                                • ☰ 24.2. Analyze a complete scientific dataset using ROOT.
                                                                                                                                                                                                                                                                                                  • ☰ 24.3. Import the Data
                                                                                                                                                                                                                                                                                                    • ☰ 24.4. Explore the Dataset
                                                                                                                                                                                                                                                                                                      • ☰ 24.5. Apply Data Selection
                                                                                                                                                                                                                                                                                                        • ☰ 24.6. Create Histograms
                                                                                                                                                                                                                                                                                                          • ☰ 24.7. Fit the Data
                                                                                                                                                                                                                                                                                                            • ☰ 24.8. Calculate Physical Quantities
                                                                                                                                                                                                                                                                                                              • ☰ 24.9. Estimate Statistical Uncertainties
                                                                                                                                                                                                                                                                                                                • ☰ 24.10. Create Publication-Quality Figures
                                                                                                                                                                                                                                                                                                                  • ☰ 24.11. Save the Analysis Results
                                                                                                                                                                                                                                                                                                                    • ☰ 24.12. Write a Short Analysis Report
                                                                                                                                                                                                                                                                                                                    • ☰ 25. Appendices
                                                                                                                                                                                                                                                                                                                      • ☰ 25.1. A. ROOT Command Cheat Sheet
                                                                                                                                                                                                                                                                                                                        • ☰ 25.2. B. ROOT Histogram Classes
                                                                                                                                                                                                                                                                                                                          • ☰ 25.3. C. ROOT Graph Classes
                                                                                                                                                                                                                                                                                                                            • ☰ 25.4. D. Common TTree Commands
                                                                                                                                                                                                                                                                                                                              • ☰ 25.5. E. Common Drawing Options
                                                                                                                                                                                                                                                                                                                                • ☰ 25.6. F. Common Fit Functions
                                                                                                                                                                                                                                                                                                                                  • ☰ 25.7. G. ROOT C++ Cheat Sheet
                                                                                                                                                                                                                                                                                                                                    • ☰ 25.8. H. PyROOT Cheat Sheet
                                                                                                                                                                                                                                                                                                                                      • ☰ 25.9. I. Common ROOT Errors and Solutions
                                                                                                                                                                                                                                                                                                                                        • ☰ 25.10. J. Recommended ROOT Analysis Workflow