KAHIBARO Bookshelves
Discord Login Register

17.5. Confidence Intervals

Statistical interpretation

A confidence interval in ROOT is a way to express the uncertainty on a quantity as a range of values instead of a single best estimate. You usually obtain these intervals after a fit, or when you compute properties of a distribution, and you want to know how precisely you have determined some parameter.

In frequentist statistics, which ROOT fitting tools follow by default, a confidence interval has a specific interpretation. Suppose you construct a 68 percent confidence interval for a parameter, for example a mean, using a fixed procedure. If you repeated the whole experiment and the analysis many times, then about 68 percent of those intervals would contain the true value of the parameter. The confidence level, written as CL, is the percentage used to define the interval, for example 68 percent, 90 percent, 95 percent, or 99 percent.

It is important to notice what you cannot say in this interpretation. For a single data set and a single confidence interval, you do not say that there is a 68 percent probability that the true value is inside the interval. Instead, you say that you used a procedure that produces intervals which cover the true value with probability 68 percent if repeated many times. ROOT tools adopt this frequentist view when they compute and report parameter errors and related intervals.

By default, many ROOT tools, such as TH1::Fit or TGraph::Fit, give you parameter estimates with uncertainties that correspond approximately to 68 percent confidence intervals. This is because they are based on the shape of the chi square function or log likelihood function around the best fit value and use a Gaussian approximation. Under that approximation, a 1 sigma interval corresponds to a 68.3 percent confidence level. When you ask ROOT to report errors or to scan a parameter, it typically relies on this approximate Gaussian relationship.

A 1 sigma uncertainty reported by ROOT corresponds approximately to a 68 percent confidence interval under Gaussian assumptions. It does not mean that there is a 68 percent probability that the true value lies in that particular interval. It means that the procedure would cover the true value in about 68 percent of repeated experiments.

Different choices of confidence level correspond to different widths of the interval. For a Gaussian distribution of a parameter estimate, a 68.3 percent interval corresponds to approximately plus or minus 1 standard deviation, 95.4 percent corresponds to plus or minus 2 standard deviations, and 99.7 percent corresponds to plus or minus 3 standard deviations. In ROOT analyses that rely on chi square or likelihood fits, the usual 1 sigma errors are used, and you can scale them or recompute them if you require another confidence level.

There are two common kinds of intervals you will meet in ROOT based analyses. One is an interval on a fitted parameter, such as the mean of a Gaussian peak or the slope of a line. The other is an interval for a count or rate, such as the mean of a Poisson process describing the number of observed events. For both cases, ROOT provides tools that can compute confidence intervals or produce objects that already encode the relevant uncertainties, such as TGraphErrors, TGraphAsymmErrors, or fit result objects.

You will also encounter one sided and two sided confidence intervals. A two sided interval gives both a lower and an upper bound around the estimate. A one sided interval gives only an upper bound or only a lower bound, which is useful for setting limits, for example when you want to say that a cross section is less than some value at 95 percent confidence level. ROOT libraries that deal with statistical intervals, for example RooStats and some Math functions, can construct both kinds of intervals using profile likelihood or other methods.

In practice, the connection between confidence intervals and chi square or likelihood is central in ROOT. If a model is a good description of the data and the parameter estimate is approximately Gaussian, then moving the parameter until the chi square increases by 1 from its minimum gives a 68 percent confidence interval for that parameter in one dimension. The same principle applies to likelihood, where a change in minus twice the log likelihood by 1 corresponds to a 68 percent interval for one parameter.

For a single parameter and a well behaved likelihood, the standard 68 percent confidence interval is given by all parameter values for which
$$-2\ln L(\theta) \le -2\ln L(\hat{\theta}) + 1,$$
or, in a chi square fit,
$$\chi^2(\theta) \le \chi^2_{\min} + 1.$$
This rule underlies the usual 1 sigma errors reported by ROOT fits.

ROOT also allows Bayesian style interpretations of intervals in dedicated libraries, where you use a posterior probability distribution and define a credible interval that contains a chosen fraction of the posterior probability. However, the core fitting classes that you use in basic ROOT analysis, such as TFitResult from standard histogram fits, follow the frequentist convention described earlier.

Parameter uncertainties

Parameter uncertainties in ROOT appear whenever you fit a function to data or estimate a quantity such as a mean or a rate. These uncertainties are directly related to confidence intervals. The fitted value is the best estimate of the true parameter, and the error is a measure of how wide the confidence interval is around this best estimate.

In a simple histogram fit with TH1::Fit, ROOT finds the parameter values that minimize the chi square between the data and the model, or maximize the likelihood if you use a likelihood fit. At the minimum, ROOT evaluates the curvature of the chi square or log likelihood function with respect to each parameter. From this curvature it constructs the covariance matrix. The diagonal elements of this matrix give the variances of the parameters. The square root of each variance is the standard error, which ROOT prints as the parameter uncertainty.

For a parameter $p_i$, the covariance matrix element $V_{ii}$ is the variance, and the standard uncertainty is
$$\sigma_i = \sqrt{V_{ii}}.$$
Under the Gaussian approximation, the interval
$$p_i \pm \sigma_i$$
is interpreted as a 68 percent confidence interval for $p_i$.

The standard parameter uncertainty that ROOT reports,
$$\sigma_i = \sqrt{V_{ii}},$$
is a 1 sigma error. It corresponds to an approximate 68 percent confidence interval for that parameter when the chi square or log likelihood is well behaved and the Gaussian approximation is valid.

The off diagonal elements of the covariance matrix, $V_{ij}$, describe correlations between parameters. These correlations affect the joint confidence region for several parameters at once. The correlation coefficient between parameters $i$ and $j$ is
$$\rho_{ij} = \frac{V_{ij}}{\sqrt{V_{ii} V_{jj}}}.$$
In ROOT, strong correlations appear in the correlation matrix printed by the fit. When parameters are highly correlated, it becomes harder to interpret each parameter uncertainty in isolation, and you may need to look at the full covariance matrix or plot contours of constant chi square in the plane of two parameters.

In many ROOT workflows, you will encounter symmetric parameter uncertainties first. These are the plus or minus values printed in fit outputs. However, in cases with low statistics or non Gaussian likelihood shapes, the true confidence interval can be asymmetric. ROOT has tools, for example profile likelihood scans and TGraphAsymmErrors, that can represent such asymmetric intervals. In a profile likelihood approach, for each value of a parameter of interest, you reoptimize all other nuisance parameters and compute the likelihood or chi square. The shape of this profile curve provides a more accurate interval, which can be asymmetric around the best fit.

You will also work with parameter uncertainties when you propagate errors to derived quantities. Suppose you fit a Gaussian peak and obtain a mean $\mu$ and sigma $\sigma$, each with its own uncertainty and perhaps a correlation between them. If you compute a derived quantity $f(\mu,\sigma)$, its uncertainty can be approximated by standard error propagation using the covariance matrix,
$$\sigma_f^2 \approx \sum_i \sum_j \frac{\partial f}{\partial p_i} \frac{\partial f}{\partial p_j} V_{ij},$$
where $p_i$ are the parameters. ROOT provides functions and classes in its Math and statistical libraries that can help with this propagation once you have extracted the parameter values and the covariance matrix from a fit result.

Parameter uncertainties also guide your choice of model and the precision of your data. If errors on parameters are very large compared to the parameter values, your model may be too flexible for the amount of data you have, or the data sample may be too small to constrain the parameters effectively. Conversely, very small errors can reveal tensions between the data and model when you look at the goodness of fit statistics, such as chi square per degree of freedom.

Finally, parameter uncertainties are the starting point for more advanced statistical procedures in ROOT, such as building confidence bands around fitted functions, making predictions with uncertainty envelopes, and constructing significance measures for signal parameters. In all these tasks you rely on the same core ingredients provided by ROOT fits, namely the parameter estimates, their standard errors, and the covariance matrix that describes how they vary together.

Views: 0

Comments

Please login to add a comment.

Don't have an account? Register now!

Where to Move

Move chapter:

  • ☰ 1. Introduction to ROOT
    • ☰ 1.1. What Is ROOT?
      • ☰ 1.2. Installing ROOT
        • ☰ 1.3. Starting ROOT
          • ☰ 1.4. Your First ROOT Session
          • ☰ 2. C++ Basics for ROOT
            • ☰ 2.1. Why ROOT Uses C++
              • ☰ 2.2. Variables and Data Types
                • ☰ 2.3. Operators
                  • ☰ 2.4. Conditional Statements
                    • ☰ 2.5. Loops
                      • ☰ 2.6. Functions
                        • ☰ 2.7. Arrays and Vectors
                        • ☰ 3. ROOT Macros
                          • ☰ 3.1. What Is a ROOT Macro?
                            • ☰ 3.2. Running ROOT Macros
                              • ☰ 3.3. Functions Inside ROOT Macros
                                • ☰ 3.4. Compiling ROOT Macros
                                  • ☰ 3.5. Organizing a ROOT Project
                                  • ☰ 4. ROOT Objects and the ROOT Object Model
                                    • ☰ 4.1. Introduction to ROOT Classes
                                      • ☰ 4.2. Object Names and Titles
                                        • ☰ 4.3. ROOT Collections
                                          • ☰ 4.4. Inspecting ROOT Objects
                                          • ☰ 5. Histograms
                                            • ☰ 5.1. Introduction to Histograms
                                              • ☰ 5.2. Creating a 1D Histogram
                                                • ☰ 5.3. Filling Histograms
                                                  • ☰ 5.4. Drawing Histograms
                                                    • ☰ 5.5. Histogram Properties
                                                      • ☰ 5.6. Accessing Histogram Bins
                                                        • ☰ 5.7. Histogram Errors
                                                          • ☰ 5.8. Normalizing Histograms
                                                            • ☰ 5.9. Combining Histograms
                                                            • ☰ 6. Two-Dimensional Histograms
                                                              • ☰ 6.1. Creating 2D Histograms
                                                                • ☰ 6.2. Filling 2D Histograms
                                                                  • ☰ 6.3. Drawing 2D Histograms
                                                                    • ☰ 6.4. Projections
                                                                      • ☰ 6.5. Profiles
                                                                      • ☰ 7. Graphs
                                                                        • ☰ 7.1. Introduction to TGraph
                                                                          • ☰ 7.2. Creating a TGraph
                                                                            • ☰ 7.3. Graph Styling
                                                                              • ☰ 7.4. TGraphErrors
                                                                                • ☰ 7.5. TGraphAsymmErrors
                                                                                  • ☰ 7.6. MultiGraph
                                                                                  • ☰ 8. Canvases and Plotting
                                                                                    • ☰ 8.1. TCanvas
                                                                                      • ☰ 8.2. Dividing a Canvas
                                                                                        • ☰ 8.3. Drawing Multiple Objects
                                                                                          • ☰ 8.4. Legends
                                                                                            • ☰ 8.5. Text and Annotations
                                                                                              • ☰ 8.6. Logarithmic Axes
                                                                                                • ☰ 8.7. Saving Plots
                                                                                                • ☰ 9. Styling ROOT Plots
                                                                                                  • ☰ 9.1. ROOT Plot Styles
                                                                                                    • ☰ 9.2. Axis Configuration
                                                                                                      • ☰ 9.3. Line Styles
                                                                                                        • ☰ 9.4. Marker Styles
                                                                                                          • ☰ 9.5. Publication-Quality Figures
                                                                                                          • ☰ 10. Mathematical Functions
                                                                                                            • ☰ 10.1. Introduction to TF1
                                                                                                              • ☰ 10.2. Built-in Functions
                                                                                                                • ☰ 10.3. Custom Functions
                                                                                                                  • ☰ 10.4. Evaluating Functions
                                                                                                                    • ☰ 10.5. Function Integrals
                                                                                                                    • ☰ 11. Curve Fitting
                                                                                                                      • ☰ 11.1. Introduction to Fitting
                                                                                                                        • ☰ 11.2. Gaussian Fits
                                                                                                                          • ☰ 11.3. Polynomial Fits
                                                                                                                            • ☰ 11.4. Custom Fit Functions
                                                                                                                              • ☰ 11.5. Fit Parameters
                                                                                                                                • ☰ 11.6. Goodness of Fit
                                                                                                                                  • ☰ 11.7. Fit Residuals
                                                                                                                                    • ☰ 11.8. Fitting Graphs
                                                                                                                                    • ☰ 12. ROOT Files
                                                                                                                                      • ☰ 12.1. Introduction to ROOT Files
                                                                                                                                        • ☰ 12.2. Creating ROOT Files
                                                                                                                                          • ☰ 12.3. Writing Objects
                                                                                                                                            • ☰ 12.4. Reading ROOT Files
                                                                                                                                              • ☰ 12.5. Exploring ROOT Files
                                                                                                                                                • ☰ 12.6. Directories Inside ROOT Files
                                                                                                                                                • ☰ 13. TTrees
                                                                                                                                                  • ☰ 13.1. Introduction to TTrees
                                                                                                                                                    • ☰ 13.2. Creating a TTree
                                                                                                                                                      • ☰ 13.3. Filling a TTree
                                                                                                                                                        • ☰ 13.4. Writing a TTree to a File
                                                                                                                                                          • ☰ 13.5. Reading a TTree
                                                                                                                                                            • ☰ 13.6. Inspecting TTrees
                                                                                                                                                              • ☰ 13.7. Drawing Data from TTrees
                                                                                                                                                                • ☰ 13.8. Applying Selection Cuts
                                                                                                                                                                  • ☰ 13.9. Branch Addresses
                                                                                                                                                                  • ☰ 14. Event-Based Data Analysis
                                                                                                                                                                    • ☰ 14.1. Understanding Event Data
                                                                                                                                                                      • ☰ 14.2. Event Loops
                                                                                                                                                                        • ☰ 14.3. Event Selection
                                                                                                                                                                          • ☰ 14.4. Derived Quantities
                                                                                                                                                                            • ☰ 14.5. Analysis Workflow
                                                                                                                                                                            • ☰ 15. Working with Real Experimental Data
                                                                                                                                                                              • ☰ 15.1. Importing Text Data
                                                                                                                                                                                • ☰ 15.2. Converting Data to ROOT
                                                                                                                                                                                  • ☰ 15.3. Data Cleaning
                                                                                                                                                                                    • ☰ 15.4. Data Calibration
                                                                                                                                                                                      • ☰ 15.5. Signal and Background
                                                                                                                                                                                      • ☰ 16. Random Numbers and Simulations
                                                                                                                                                                                        • ☰ 16.1. Random Number Generation
                                                                                                                                                                                          • ☰ 16.2. Common Distributions
                                                                                                                                                                                            • ☰ 16.3. Simple Monte Carlo Simulation
                                                                                                                                                                                              • ☰ 16.4. Detector Resolution Simulation
                                                                                                                                                                                                • ☰ 16.5. Efficiency Simulation
                                                                                                                                                                                                • ☰ 17. Statistical Analysis with ROOT
                                                                                                                                                                                                  • ☰ 17.1. Basic Statistics
                                                                                                                                                                                                    • ☰ 17.2. Probability Distributions
                                                                                                                                                                                                      • ☰ 17.3. Statistical Uncertainties
                                                                                                                                                                                                        • ☰ 17.4. Chi-Square Tests
                                                                                                                                                                                                          • ☰ 17.5. Confidence Intervals
                                                                                                                                                                                                          • ☰ 18. Modern ROOT with RDataFrame
                                                                                                                                                                                                            • ☰ 18.1. Introduction to RDataFrame
                                                                                                                                                                                                              • ☰ 18.2. Loading Data
                                                                                                                                                                                                                • ☰ 18.3. Filtering Data
                                                                                                                                                                                                                  • ☰ 18.4. Defining New Columns
                                                                                                                                                                                                                    • ☰ 18.5. Creating Histograms
                                                                                                                                                                                                                      • ☰ 18.6. Statistical Operations
                                                                                                                                                                                                                        • ☰ 18.7. Saving Results
                                                                                                                                                                                                                          • ☰ 18.8. Multithreading
                                                                                                                                                                                                                          • ☰ 19. ROOT and Python
                                                                                                                                                                                                                            • ☰ 19.1. Introduction to PyROOT
                                                                                                                                                                                                                              • ☰ 19.2. ROOT Objects in Python
                                                                                                                                                                                                                                • ☰ 19.3. Reading ROOT Files with Python
                                                                                                                                                                                                                                  • ☰ 19.4. PyROOT and NumPy
                                                                                                                                                                                                                                    • ☰ 19.5. PyROOT and Matplotlib
                                                                                                                                                                                                                                      • ☰ 19.6. Choosing Between C++ and Python
                                                                                                                                                                                                                                      • ☰ 20. Advanced TTree Techniques
                                                                                                                                                                                                                                        • ☰ 20.1. Trees with Multiple Branches
                                                                                                                                                                                                                                          • ☰ 20.2. Storing std::vector Objects
                                                                                                                                                                                                                                            • ☰ 20.3. TChain
                                                                                                                                                                                                                                              • ☰ 20.4. Friends
                                                                                                                                                                                                                                                • ☰ 20.5. Efficient TTree Analysis
                                                                                                                                                                                                                                                • ☰ 21. ROOT for Particle and Nuclear Physics
                                                                                                                                                                                                                                                  • ☰ 21.1. Four-Vectors
                                                                                                                                                                                                                                                    • ☰ 21.2. Particle Kinematics
                                                                                                                                                                                                                                                      • ☰ 21.3. Invariant Mass
                                                                                                                                                                                                                                                        • ☰ 21.4. Angular Distributions
                                                                                                                                                                                                                                                          • ☰ 21.5. Detector Spectra
                                                                                                                                                                                                                                                          • ☰ 22. Debugging and Common ROOT Problems
                                                                                                                                                                                                                                                            • ☰ 22.1. ROOT Object Ownership
                                                                                                                                                                                                                                                              • ☰ 22.2. Null Pointers
                                                                                                                                                                                                                                                                • ☰ 22.3. Segmentation Faults
                                                                                                                                                                                                                                                                  • ☰ 22.4. Missing Histograms
                                                                                                                                                                                                                                                                    • ☰ 22.5. ROOT File Problems
                                                                                                                                                                                                                                                                      • ☰ 22.6. TTree Branch Problems
                                                                                                                                                                                                                                                                        • ☰ 22.7. Histogram Range Problems
                                                                                                                                                                                                                                                                          • ☰ 22.8. Fit Failures
                                                                                                                                                                                                                                                                            • ☰ 22.9. Debugging ROOT Macros
                                                                                                                                                                                                                                                                            • ☰ 23. Writing Better ROOT Analysis Code
                                                                                                                                                                                                                                                                              • ☰ 23.1. Separating Analysis and Plotting
                                                                                                                                                                                                                                                                                • ☰ 23.2. Using Functions
                                                                                                                                                                                                                                                                                  • ☰ 23.3. Using Header Files
                                                                                                                                                                                                                                                                                    • ☰ 23.4. Naming Conventions
                                                                                                                                                                                                                                                                                      • ☰ 23.5. Avoiding Hard-Coded Values
                                                                                                                                                                                                                                                                                        • ☰ 23.6. Configuration Parameters
                                                                                                                                                                                                                                                                                          • ☰ 23.7. Reproducible Analysis
                                                                                                                                                                                                                                                                                            • ☰ 23.8. Version Control with Git
                                                                                                                                                                                                                                                                                            • ☰ 24. Final Project
                                                                                                                                                                                                                                                                                              • ☰ 24.1. Project Overview
                                                                                                                                                                                                                                                                                                • ☰ 24.2. Analyze a complete scientific dataset using ROOT.
                                                                                                                                                                                                                                                                                                  • ☰ 24.3. Import the Data
                                                                                                                                                                                                                                                                                                    • ☰ 24.4. Explore the Dataset
                                                                                                                                                                                                                                                                                                      • ☰ 24.5. Apply Data Selection
                                                                                                                                                                                                                                                                                                        • ☰ 24.6. Create Histograms
                                                                                                                                                                                                                                                                                                          • ☰ 24.7. Fit the Data
                                                                                                                                                                                                                                                                                                            • ☰ 24.8. Calculate Physical Quantities
                                                                                                                                                                                                                                                                                                              • ☰ 24.9. Estimate Statistical Uncertainties
                                                                                                                                                                                                                                                                                                                • ☰ 24.10. Create Publication-Quality Figures
                                                                                                                                                                                                                                                                                                                  • ☰ 24.11. Save the Analysis Results
                                                                                                                                                                                                                                                                                                                    • ☰ 24.12. Write a Short Analysis Report
                                                                                                                                                                                                                                                                                                                    • ☰ 25. Appendices
                                                                                                                                                                                                                                                                                                                      • ☰ 25.1. A. ROOT Command Cheat Sheet
                                                                                                                                                                                                                                                                                                                        • ☰ 25.2. B. ROOT Histogram Classes
                                                                                                                                                                                                                                                                                                                          • ☰ 25.3. C. ROOT Graph Classes
                                                                                                                                                                                                                                                                                                                            • ☰ 25.4. D. Common TTree Commands
                                                                                                                                                                                                                                                                                                                              • ☰ 25.5. E. Common Drawing Options
                                                                                                                                                                                                                                                                                                                                • ☰ 25.6. F. Common Fit Functions
                                                                                                                                                                                                                                                                                                                                  • ☰ 25.7. G. ROOT C++ Cheat Sheet
                                                                                                                                                                                                                                                                                                                                    • ☰ 25.8. H. PyROOT Cheat Sheet
                                                                                                                                                                                                                                                                                                                                      • ☰ 25.9. I. Common ROOT Errors and Solutions
                                                                                                                                                                                                                                                                                                                                        • ☰ 25.10. J. Recommended ROOT Analysis Workflow