17.4. Chi-Square Tests

Comparing distributions

Chi square tests in ROOT are used to quantify how similar two binned distributions are, or how well a model matches a histogram. In this chapter the focus is on the practical use of chi square within ROOT, not on the full statistical theory.

When you compare distributions in ROOT you usually start from histograms. Suppose you have a histogram filled with data, and either another histogram filled with simulated or reference data, or a function that represents a theoretical model. The chi square statistic measures the squared differences between “observed” and “expected” counts, scaled by their uncertainties, bin by bin.

For two histograms, a very common task is to check if they can be considered to come from the same underlying distribution. In ROOT this is typically done with the Chi2Test method of TH1. The simplest usage compares two histograms h1 and h2:

Double_t pvalue = h1->Chi2Test(h2);

By default, Chi2Test returns a p value. This value is the probability, under the assumption that the two histograms are statistically compatible, of obtaining a chi square as large as or larger than the one computed from the histograms. Values of the p value close to 1 indicate good agreement. Very small p values indicate that the histograms are unlikely to represent the same distribution.

You can change what Chi2Test returns and how it behaves with options. A very common option is "CHI2" to get the chi square statistic itself instead of the p value:

Double_t chi2 = h1->Chi2Test(h2, "CHI2");

Another important option is "UU" to use unweighted histograms and the correct Poisson uncertainties, and "NORM" if you want ROOT to normalize the histograms to the same integral before computing the chi square. Combining options is done by concatenating characters in the option string, for example "CHI2 UU NORM". Always read the ROOT reference for the exact meaning of each option, because it controls how uncertainties and normalizations are handled.

The chi square comparison depends on the binning. Bins that have very low statistics can make the chi square test unreliable. Before using Chi2Test, you should ensure that the two histograms have the same number of bins and the same bin edges. ROOT does not automatically rebin them. You can check consistency by comparing GetNbinsX, GetXaxis()->GetXmin, and GetXaxis()->GetXmax, and if needed, use Rebin or recreate histograms with the same binning.

ROOT also provides access to more detailed output from Chi2Test. For example, you can get both the chi square and the number of degrees of freedom in one call:

Int_t ndf = 0;
Double_t chi2 = h1->Chi2Test(h2, "CHI2 NORM", &ndf);

You can then construct the reduced chi square, which is the chi square divided by the number of degrees of freedom, using this result. The reduced chi square is often used to summarize the overall agreement between two distributions.

When comparing a histogram with a reference distribution that has no statistical uncertainty, for example a fixed template with very high statistics, you should reflect that in the way you compute uncertainties. Some options in Chi2Test are designed to handle the case where only one histogram has statistical fluctuations. Checking which histogram is treated as reference and which as data is crucial, because it changes the denominator in each bin of the chi square sum.

It is also important to remember that the chi square test is meaningful only when the bin contents represent independent counts and when expected counts in each bin are not too small. If many bins have very few entries, you should consider merging bins or using a different test such as a likelihood ratio test, which is not the topic of this chapter.

In ROOT, Chi2Test encapsulates this computation and returns either the p value or the chi square, together with the degrees of freedom if requested. You should interpret the numerical result in the context of your physics problem and chosen significance level, not as an automatic pass or fail.

Evaluating models

Evaluating how well a model describes data is another central use of chi square in ROOT. Here the “model” is usually a TF1 function that has been fitted to a histogram. In that case, the chi square is computed during the fit, and ROOT stores and reports it along with the fit parameters and their uncertainties.

When you fit a histogram h with a function f using Fit, for example:

h->Fit("f");

ROOT by default uses a chi square minimization algorithm for binned data, unless you request a different fit method. The minimization finds the parameter values that produce the smallest chi square. After the fit, you can retrieve the fitted function and examine its chi square and related quantities.

The TF1 returned by the fit can be obtained with GetFunction:

TF1 *fitfunc = h->GetFunction("f");
Double_t chi2      = fitfunc->GetChisquare();
Int_t    ndf       = fitfunc->GetNDF();
Double_t chi2_red  = chi2 / ndf;

The chi square value, together with the number of degrees of freedom, is your main diagnostic for the overall quality of the fit. The reduced chi square, chi2 divided by ndf, is often used as a single summary number. Values near 1 usually indicate that the model describes the data within the expected statistical fluctuations, assuming that uncertainties are correctly estimated. Values much larger than 1 suggest that the model does not capture the data structure or that uncertainties are underestimated. Values much less than 1 can indicate that uncertainties are overestimated, that data points are correlated, or that there are other issues in the error model.

You can also compute a p value for the fit using the chi square and degrees of freedom. ROOT provides the TMath::Prob function, which returns the probability of obtaining a chi square at least as large as the observed one if the model is correct:

Double_t pvalue = TMath::Prob(chi2, ndf);

This p value can be used to quantify the goodness of fit in a probabilistic way. It is analogous to the p value from Chi2Test for two histograms, but here the comparison is between data and the fitted model. A very small p value indicates that the model is unlikely to be consistent with the data under the assumed uncertainties.

When you evaluate models with chi square in ROOT, you should keep in mind how errors are assigned to bins. For histograms filled with unweighted counts, the natural bin uncertainty is the square root of the bin content. ROOT uses this by default for chi square fits. If you have weighted events, or if you want to specify custom uncertainties, you can use Sumw2 on the histogram so that ROOT stores the sum of squared weights and uses it in error calculations. For very low statistics, a chi square fit is often not reliable, and an unbinned or likelihood based approach is usually preferred.

The choice of fit range also affects the chi square and goodness of fit. If you specify a narrower range in the Fit call, for example:

h->Fit("f", "", "", xlow, xhigh);

then only bins in that range contribute to the chi square and degrees of freedom. This allows you to focus on the region of interest, such as a peak, but it also changes the interpretation of the goodness of fit. You should always report the fit range when you quote chi square and p values.

Model evaluation is not limited to a single fit. You can fit the same histogram with several different models, for example a Gaussian and a Gaussian plus polynomial background, and compare their chi square values and p values. Models with more free parameters can usually achieve smaller chi square, so it is important to consider the number of degrees of freedom. You should compare reduced chi square values rather than raw chi square values, and you should be cautious when adding many parameters just to force chi square close to 1 without physical motivation.

Finally, chi square is a global measure. It tells you about the average agreement across all bins, but it does not show where the model fails. To understand the detailed behavior, you should inspect residuals, that is the bin by bin differences between data and model, which are covered in a separate chapter. Combined with the global chi square, residual plots give a much more complete picture of how well your model describes the data.

Views: 13