11.6. Goodness of Fit

Table of Contents

Chi-square

In ROOT, the standard way to quantify how well a model describes data is the chi square, written $\chi^2$. For binned data such as histograms, $\chi^2$ compares the observed counts in each bin with the counts predicted by the fit function, and weights the difference by the expected statistical uncertainty.

For a histogram with bins indexed by $i$, observed counts $O_i$ and predicted counts $E_i$, the usual definition is
$$
\chi^2 = \sum_{i} \frac{(O_i - E_i)^2}{\sigma_i^2},
$$
where $\sigma_i$ is the uncertainty on $O_i$. For Poisson statistics and unit weights, a common choice is $\sigma_i = \sqrt{O_i}$, possibly replaced by $\sqrt{E_i}$ in some conventions. ROOT will handle these technical details internally when you use standard fitting tools.

A large chi square value means that, on average, the data points lie far from the model prediction compared to their uncertainties, while a small value means the data points lie close to the model within their error bars.

In ROOT, when you fit a histogram or graph with a TF1 function using TH1::Fit, TGraph::Fit or similar, the fit result carries a chi square value. You can access it programmatically through the fit result object or through the fitted function. For example, if you fit a histogram h with a function "gaus" and save the result:

cpp

TFitResultPtr r = h->Fit("gaus", "S");  // "S" returns a fit result
double chi2 = r->Chi2();

You can also get it from the TF1 that was used in the fit, assuming f is that function:

cpp

double chi2 = f->GetChisquare();

Both give the same quantity, the total chi square of the fit over the range and bins used.

The chi square itself grows with the number of data points, so by itself it is not very informative. A $\chi^2$ of 50 could be either excellent or terrible, depending on whether there were 10 or 500 bins in the fit. To interpret it, you need to relate it to the number of degrees of freedom and look at the ratio, the reduced chi square, which is discussed later in this chapter.

Degrees of freedom

The number of degrees of freedom, usually written $\nu$ or ndf in ROOT, tells you how many independent pieces of information you have after accounting for the parameters that were adjusted in the fit. For a simple least squares fit to $N_{\text{points}}$ data points with a model that has $N_{\text{par}}$ free parameters, the standard definition is
$$
\nu = N_{\text{points}} - N_{\text{par}}.
$$

For binned histogram fits, $N_{\text{points}}$ is typically the number of bins used in the fit, not the number of events in the histogram. The number of free parameters is the number actually varied during the fit, which excludes parameters that you have fixed or tied to other parameters.

Degrees of freedom:
$$\nu = N_{\text{data points}} - N_{\text{fitted parameters}}.$$
ROOT usually calls this quantity ndf.

When you perform a fit with ROOT, it automatically determines ndf from the number of bins or points it is fitting and the number of free parameters in the function. You can inspect it via the fit result or the fitted function.

With a fit result pointer r:

cpp

int ndf = r->Ndf();

or from the TF1 itself:

cpp

int ndf = f->GetNDF();

Interpreting ndf is essential when judging goodness of fit. Typical expectations for a well specified model with correct uncertainties are:

The chi square should be of order $\nu$.
The quantity $\chi^2 / \nu$ should be of order 1.

Because $\chi^2$ fluctuates, especially for small $\nu$, these are only guidelines. To be more quantitative, you can compute the chi square probability, that is the probability of obtaining a chi square as large or larger than the one observed assuming the model is correct. This is available as

cpp

double prob = r->Prob();          // from the fit result
double prob2 = f->GetProb();      // from the TF1

A very small probability (for example less than $10^{-3}$) often indicates that the model is not adequate, the uncertainties are underestimated, or there are issues in the data. A very large probability close to 1 can also be a sign that uncertainties are overestimated.

Reduced chi-square

The reduced chi square, written $\chi^2_{\text{red}}$, is defined as the chi square divided by the number of degrees of freedom:
$$
\chi^2_{\text{red}} = \frac{\chi^2}{\nu}.
$$

This quantity is convenient because, for a good fit with properly estimated Gaussian uncertainties, you expect $\chi^2_{\text{red}}$ to be close to 1. This expectation is independent of the absolute number of data points. This is why fitting tools and analysis reports commonly quote $\chi^2_{\text{red}}$ when summarizing goodness of fit.

In ROOT, you typically compute the reduced chi square yourself using the values returned by the fit, for example:

cpp

TFitResultPtr r = h->Fit("gaus", "S");
double chi2 = r->Chi2();
int ndf = r->Ndf();
double chi2_red = chi2 / ndf;

or, using the TF1:

cpp

double chi2 = f->GetChisquare();
int ndf = f->GetNDF();
double chi2_red = chi2 / ndf;

You can also calculate and print it directly in your macros when producing fit diagnostics.

Reduced chi square:
$$\chi^2_{\text{red}} = \frac{\chi^2}{\nu}.$$
Typical interpretation:

$\chi^2_{\text{red}} \approx 1$: model and uncertainties broadly consistent with data.
$\chi^2_{\text{red}} \gg 1$: model does not describe data within the quoted errors or uncertainties are too small.
$\chi^2_{\text{red}} \ll 1$: uncertainties probably too large or data points correlated in a way not accounted for.

The table below summarizes common qualitative interpretations.

$\chi^2_{\text{red}}$ value	Qualitative interpretation
around 1	Fit consistent with data and uncertainties
from about 2 to 5	Possible tension, check model and error estimates
much larger than 5	Likely poor model, outliers, or underestimated errors
much smaller than 1	Likely overestimated errors or strong correlations

When you inspect a fit visually in a ROOT canvas, comparing the curve with the data points or histogram, you should also look at the reduced chi square and the fit probability. A fit that looks reasonable by eye but has a very large $\chi^2_{\text{red}}$ is a warning sign that either the model is missing structure or the uncertainties are misestimated. Conversely, if $\chi^2_{\text{red}}$ is very small, then even a model that looks perfect may not be telling you much, because the data errors might be so large that many very different models could also be acceptable.

In practical ROOT analysis, a useful workflow is to save the chi square, degrees of freedom, reduced chi square and fit probability along with the fitted parameter values. This lets you later compare different models or different fit ranges on the same footing, and helps you decide which modeling choice provides the best description of your data in a statistically meaningful way.

11.6. Goodness of Fit

Chi-square

Degrees of freedom

Reduced chi-square

Comments

Where to Move