Why Compare with Reference Data

In any Geant4 simulation, numerical results by themselves do not tell you whether the physics is implemented correctly or whether the geometry and settings reflect reality. You need an external point of comparison. Reference data provide that anchor.

Reference data can come from measurements, from recommended evaluated databases, or from well tested simulations and analytical models. By comparing your Geant4 results with these references you test whether your choices of geometry, physics list, materials, and source description produce realistic outcomes.

Good comparison is not simply overlaying two curves. It involves choosing appropriate observables, understanding the expected level of agreement, and treating statistical and systematic uncertainties consistently.

Types of Reference Data

Reference data cover different levels of detail and come from various sources.

Experimental data are the most common target. These include measured dose distributions in phantoms, detector responses, efficiency curves, energy spectra, or count rates. Journals and technical reports often publish these, sometimes in tabular form, sometimes only as plots.

Evaluated nuclear and particle data are compiled and reviewed by expert groups. For example, cross section libraries, stopping power tables, and attenuation coefficients. These are useful when you want to validate a specific aspect of the physics, such as proton ranges or gamma attenuation.

Standards and codes of practice provide reference quantities and recommended values. In medical physics, standards organizations publish reference depth dose curves or beam quality indices. In shielding studies, standard build up factors or attenuation data are available.

You can also use analytical models or textbook formulas as reference, particularly for simple geometries. For example, exponential attenuation in a slab, range-energy relations in water, or Bethe-Bloch stopping power in a limited energy range.

Finally, other well established Monte Carlo codes can act as a reference when experimental data are not yet available. In that case you are performing code-to-code comparison rather than full physical validation, but it still helps to detect major inconsistencies.

Choosing Appropriate Reference Data

You should choose reference data that match both your use case and the level of detail of your simulation. This requires careful reading of the original data source.

First, match the geometry and materials as closely as possible. If a beam attenuation measurement was done with a narrow pencil beam in a specific material grade and exact thicknesses, your simulated setup should reproduce these details. Differences in composition, density, or geometry can easily dominate any disagreement.

Second, match the beam and source characteristics. Energy spectrum, angular distribution, beam size, and field shape all affect observables such as dose or count rates. If the reference used a realistic spectrum but you simulate a monoenergetic source, you will not obtain a meaningful comparison.

Third, consider the detector or scoring quantity. Experimental results usually include the response of a specific detector, not the raw physical quantity in the material. For example, a scintillator may have an energy dependent efficiency and threshold. Decide whether you will model the detector response explicitly or whether the publication provides a conversion between detector signal and physical quantity, such as absorbed dose.

Fourth, check the quality of the reference. Look for stated uncertainties, calibration methods, and any corrections applied. Prefer data from peer reviewed papers, well known databases, or official standards.

Finally, ensure that the reference covers the same range of energies, angles, or depths that are relevant for your simulation. Using data outside their intended range often leads to spurious disagreements.

Preparing Simulation Results for Comparison

Raw Geant4 output often has much higher detail and lower noise than the reference data, but it is not automatically in the same form. You must process the simulated quantities to mirror the way the reference was obtained.

The first step is to define observables that match the reference. If the experimental paper provides depth dose curves averaged over finite voxels, define scoring regions in your geometry with the same size and position and score dose per region. If a detector efficiency curve is reported as counts per incident particle, simulate that ratio rather than only deposited energy.

Next, apply the same units and normalization. Reference dose may be in Gy per incident particle, Gy per monitor unit, or cGy per delivered charge. Convert your scoring output to the same units and define the same normalization, for example per primary particle, per fluence, or per integrated charge. Geant4’s unit system does not enforce the same conventions that experimentalists use, so this step requires explicit conversion.

You should also mimic the experimental resolution and binning. If the reference provides histograms with specific energy bin widths, rebin your simulated spectra accordingly. When a detector has finite spatial or energy resolution, you may need to smear your simulated data with appropriate Gaussian functions so that comparisons are made between similarly blurred quantities, not between an ideal and a measured response.

Another important step is to reproduce selection cuts and acceptance. Experiments often apply thresholds, timing windows, or angular cuts. Implement the same cuts when analyzing simulation output, for example by discarding hits below a threshold or only considering coincidences within a given time window.

A very simple but easily overlooked element is coordinate and axis definition. Ensure that depth is measured from the same reference plane, angles use the same convention, and that any flips or rotations between coordinate systems are properly accounted for.

Quantitative Comparison Methods

Once both simulated and reference data are in compatible forms, use quantitative metrics to assess agreement. Visual inspection is useful for a first impression, but it is not sufficient.

A basic approach is to compute the relative difference in each bin or at each measurement point. If $R_i$ is the reference value and $S_i$ is the simulated value for point $i$, then the relative difference is
$$
\delta_i = \frac{S_i - R_i}{R_i}.
$$
You can then summarize these by their mean, root mean square, or maximum.

When both reference and simulation have estimated uncertainties, you can compute a normalized residual or pull:
$$
z_i = \frac{S_i - R_i}{\sqrt{\sigma_{S,i}^2 + \sigma_{R,i}^2}},
$$
where $\sigma_{S,i}$ and $\sigma_{R,i}$ are the uncertainties on simulation and reference for point $i$. Values of $z_i$ around 1 in magnitude typically indicate agreement within combined uncertainties.

Chi square tests are a common choice when comparing binned distributions. The standard chi square statistic is
$$
\chi^2 = \sum_i \frac{(S_i - R_i)^2}{\sigma_{i}^2},
$$
with an appropriate choice of $\sigma_i$ and number of degrees of freedom. This allows you to derive a $p$ value, although interpreting small deviations can be subtle when systematic uncertainties dominate.

In medical physics and dose comparison, the gamma index is often used. It combines dose difference and distance to agreement in a single metric. For each point in the reference distribution, the gamma index searches nearby points in the simulation and measures whether both the local dose difference and spatial distance are within preselected criteria, such as 3 percent and 3 mm. Implementations of this method are available in external analysis tools.

For detection efficiencies or probabilities, you may summarize agreement by comparing integral quantities. For example, total detection efficiency integrated over an energy range, or the position or area under specific peaks in energy spectra.

Handling Uncertainties

Any comparison must take into account uncertainties from both simulation and reference data. Ignoring them can lead you to overinterpret small differences or overlook significant discrepancies.

Simulation results have statistical uncertainties because they are obtained by Monte Carlo sampling. For simple counts or tallies, the relative statistical uncertainty decreases roughly as $1/\sqrt{N}$, where $N$ is the number of primary events. When you record histograms or voxel doses, Geant4 analysis tools can accumulate not only sums but also sums of squares, so that statistical uncertainties for each bin can be derived. Ensure that your scoring code correctly tracks these quantities.

Systematic uncertainties in the simulation arise from imperfect knowledge of geometry, material compositions, source parameters, and physics models. For example, a 1 percent error in material density can shift ranges and doses by a similar amount. You can estimate some of these by varying input parameters within reasonable bounds and observing how outputs change.

Experimental reference data come with their own statistical and systematic uncertainties. These are usually reported in the original publications. Statistical errors are often small if many events were measured. Systematic components, such as calibration uncertainties, beam alignment, or detector modeling, can be larger and sometimes dominate.

When combining uncertainties from simulation and experiment, treat independent contributions in quadrature. For example, the combined uncertainty in a bin is often taken as
$$
\sigma_{\text{combined}} = \sqrt{\sigma_{S}^2 + \sigma_{R}^2},
$$
where $\sigma_{S}$ is the simulation uncertainty and $\sigma_{R}$ is the reference uncertainty.

Make sure that your simulation statistics are sufficiently high that Monte Carlo noise does not obscure the comparison. If the reference data have a 2 percent uncertainty, target simulation statistical errors well below that level in the relevant bins.

Interpreting Discrepancies

When you find differences between simulation and reference data, your task is to understand their origin. Not every discrepancy implies a problem. Some differences are expected given model limitations or input uncertainties.

First, check if the pattern of disagreement is systematic. For example, a constant offset across all depths could indicate a normalization issue, such as an incorrect number of primaries or an error in unit conversion. In contrast, discrepancies that grow with depth might suggest an issue in material density or stopping power.

Second, examine regions where the physics models are known to be less accurate. Certain physics lists have limited validity ranges or approximations that are acceptable for one application but not another. The Geant4 documentation and publications often discuss these limitations for specific physics processes or energy ranges.

Third, consider geometrical or material modeling errors. Small shifts in layer thickness, misalignments, or incorrect compositions can have significant effects, especially near steep gradients such as distal edges in depth dose curves or sharp shielding cutoffs. Visualizing the geometry and reviewing material definitions can reveal such issues.

You should also revisit the source description and scoring definitions. A mismatch between the experimental beam profile and the simulated source, or between the experimental acceptance and the simulation scoring region, often explains localized differences.

If after thorough checking you still see significant, unexplained discrepancies, you may need to explore alternative physics lists, adjust production cuts, or consult the Geant4 release notes to see if particular processes have known issues.

Documenting Validation Results

For your simulation to be credible and reusable, you should carefully document how you performed the comparison with reference data and what you concluded.

This documentation should describe the reference data source, including full citations, the specific quantities used, and any digitization or transformations applied. It should also specify how the simulation was configured to match the experimental or reference setup, for example which geometry elements were modeled, which materials and densities were used, and how the source and detector were represented.

You should clearly outline the observables that were compared, the analysis steps taken to convert raw simulation and reference data into comparable forms, and the units and normalization conventions used. If you applied any smearing, thresholds, or acceptance cuts, describe them and justify their values.

For the comparison itself, include quantitative metrics such as relative differences, chi square values, or gamma passing rates, along with graphs that overlay simulation and reference data. State the statistical and systematic uncertainties assumed for both simulation and reference, and note the number of events simulated.

Most importantly, summarize the level of agreement you achieved and any remaining discrepancies. Identify the parameter ranges where the simulation is validated and where it is not, and record any decisions you made based on the comparison, such as selecting one physics list over another.

Careful documentation ensures that others, and your future self, can understand how reliable the simulation is for its intended use, and it provides a baseline for further improvements or for adapting the simulation to new scenarios.

Views: 8