17.5. Confidence Intervals

Statistical interpretation

A confidence interval in ROOT is a way to express the uncertainty on a quantity as a range of values instead of a single best estimate. You usually obtain these intervals after a fit, or when you compute properties of a distribution, and you want to know how precisely you have determined some parameter.

In frequentist statistics, which ROOT fitting tools follow by default, a confidence interval has a specific interpretation. Suppose you construct a 68 percent confidence interval for a parameter, for example a mean, using a fixed procedure. If you repeated the whole experiment and the analysis many times, then about 68 percent of those intervals would contain the true value of the parameter. The confidence level, written as CL, is the percentage used to define the interval, for example 68 percent, 90 percent, 95 percent, or 99 percent.

It is important to notice what you cannot say in this interpretation. For a single data set and a single confidence interval, you do not say that there is a 68 percent probability that the true value is inside the interval. Instead, you say that you used a procedure that produces intervals which cover the true value with probability 68 percent if repeated many times. ROOT tools adopt this frequentist view when they compute and report parameter errors and related intervals.

By default, many ROOT tools, such as TH1::Fit or TGraph::Fit, give you parameter estimates with uncertainties that correspond approximately to 68 percent confidence intervals. This is because they are based on the shape of the chi square function or log likelihood function around the best fit value and use a Gaussian approximation. Under that approximation, a 1 sigma interval corresponds to a 68.3 percent confidence level. When you ask ROOT to report errors or to scan a parameter, it typically relies on this approximate Gaussian relationship.

Different choices of confidence level correspond to different widths of the interval. For a Gaussian distribution of a parameter estimate, a 68.3 percent interval corresponds to approximately plus or minus 1 standard deviation, 95.4 percent corresponds to plus or minus 2 standard deviations, and 99.7 percent corresponds to plus or minus 3 standard deviations. In ROOT analyses that rely on chi square or likelihood fits, the usual 1 sigma errors are used, and you can scale them or recompute them if you require another confidence level.

There are two common kinds of intervals you will meet in ROOT based analyses. One is an interval on a fitted parameter, such as the mean of a Gaussian peak or the slope of a line. The other is an interval for a count or rate, such as the mean of a Poisson process describing the number of observed events. For both cases, ROOT provides tools that can compute confidence intervals or produce objects that already encode the relevant uncertainties, such as TGraphErrors, TGraphAsymmErrors, or fit result objects.

You will also encounter one sided and two sided confidence intervals. A two sided interval gives both a lower and an upper bound around the estimate. A one sided interval gives only an upper bound or only a lower bound, which is useful for setting limits, for example when you want to say that a cross section is less than some value at 95 percent confidence level. ROOT libraries that deal with statistical intervals, for example RooStats and some Math functions, can construct both kinds of intervals using profile likelihood or other methods.

In practice, the connection between confidence intervals and chi square or likelihood is central in ROOT. If a model is a good description of the data and the parameter estimate is approximately Gaussian, then moving the parameter until the chi square increases by 1 from its minimum gives a 68 percent confidence interval for that parameter in one dimension. The same principle applies to likelihood, where a change in minus twice the log likelihood by 1 corresponds to a 68 percent interval for one parameter.

ROOT also allows Bayesian style interpretations of intervals in dedicated libraries, where you use a posterior probability distribution and define a credible interval that contains a chosen fraction of the posterior probability. However, the core fitting classes that you use in basic ROOT analysis, such as TFitResult from standard histogram fits, follow the frequentist convention described earlier.

Parameter uncertainties

Parameter uncertainties in ROOT appear whenever you fit a function to data or estimate a quantity such as a mean or a rate. These uncertainties are directly related to confidence intervals. The fitted value is the best estimate of the true parameter, and the error is a measure of how wide the confidence interval is around this best estimate.

In a simple histogram fit with TH1::Fit, ROOT finds the parameter values that minimize the chi square between the data and the model, or maximize the likelihood if you use a likelihood fit. At the minimum, ROOT evaluates the curvature of the chi square or log likelihood function with respect to each parameter. From this curvature it constructs the covariance matrix. The diagonal elements of this matrix give the variances of the parameters. The square root of each variance is the standard error, which ROOT prints as the parameter uncertainty.

For a parameter $p_i$, the covariance matrix element $V_{ii}$ is the variance, and the standard uncertainty is
$$\sigma_i = \sqrt{V_{ii}}.$$
Under the Gaussian approximation, the interval
$$p_i \pm \sigma_i$$
is interpreted as a 68 percent confidence interval for $p_i$.

The off diagonal elements of the covariance matrix, $V_{ij}$, describe correlations between parameters. These correlations affect the joint confidence region for several parameters at once. The correlation coefficient between parameters $i$ and $j$ is
$$\rho_{ij} = \frac{V_{ij}}{\sqrt{V_{ii} V_{jj}}}.$$
In ROOT, strong correlations appear in the correlation matrix printed by the fit. When parameters are highly correlated, it becomes harder to interpret each parameter uncertainty in isolation, and you may need to look at the full covariance matrix or plot contours of constant chi square in the plane of two parameters.

In many ROOT workflows, you will encounter symmetric parameter uncertainties first. These are the plus or minus values printed in fit outputs. However, in cases with low statistics or non Gaussian likelihood shapes, the true confidence interval can be asymmetric. ROOT has tools, for example profile likelihood scans and TGraphAsymmErrors, that can represent such asymmetric intervals. In a profile likelihood approach, for each value of a parameter of interest, you reoptimize all other nuisance parameters and compute the likelihood or chi square. The shape of this profile curve provides a more accurate interval, which can be asymmetric around the best fit.

You will also work with parameter uncertainties when you propagate errors to derived quantities. Suppose you fit a Gaussian peak and obtain a mean $\mu$ and sigma $\sigma$, each with its own uncertainty and perhaps a correlation between them. If you compute a derived quantity $f(\mu,\sigma)$, its uncertainty can be approximated by standard error propagation using the covariance matrix,
$$\sigma_f^2 \approx \sum_i \sum_j \frac{\partial f}{\partial p_i} \frac{\partial f}{\partial p_j} V_{ij},$$
where $p_i$ are the parameters. ROOT provides functions and classes in its Math and statistical libraries that can help with this propagation once you have extracted the parameter values and the covariance matrix from a fit result.

Parameter uncertainties also guide your choice of model and the precision of your data. If errors on parameters are very large compared to the parameter values, your model may be too flexible for the amount of data you have, or the data sample may be too small to constrain the parameters effectively. Conversely, very small errors can reveal tensions between the data and model when you look at the goodness of fit statistics, such as chi square per degree of freedom.

Finally, parameter uncertainties are the starting point for more advanced statistical procedures in ROOT, such as building confidence bands around fitted functions, making predictions with uncertainty envelopes, and constructing significance measures for signal parameters. In all these tasks you rely on the same core ingredients provided by ROOT fits, namely the parameter estimates, their standard errors, and the covariance matrix that describes how they vary together.

Views: 11