Quantifying Uncertainty and Variability

Uncertainty, variability, and errors are terms that are often used when discussing ¹⁴C dating and intercomparisons. Uncertainty quantification is the natural place to start the discussion and that begins with the laboratory quoted error, which is calculated based on a number of considerations, but in essence represents the uncertainty in the measurement. This is quantified based on the variability we observe and is interpreted as the variability we would expect to observe if we were able to repeat the measurement. This also accounts for variability in standards, backgrounds and other laboratory processes. Typically, in many laboratories, this will be based in part on the observed standard deviation from a set of measurements (variation) made on an in-house well-characterized material (see Naysmith et al. Reference Naysmith, Scott, Dunbar and Cook2019, this volume). In general terms, uncertainty, variability and error also relate to precision (precise results show small variation) and to the concept of repeatability where “repeatability of a measurement is the closeness of agreement between successive measurements carried out under the same conditions e.g., same location, same person, same measurement procedure.” Another key property is that of reproducibility which refers to “the closeness of agreement between the results of experiments conducted under changed conditions, including different laboratories and instruments” (Taylor and Kuyatt Reference Taylor and Kuyatt2001).

While these descriptions focus on “the variation in the final ¹⁴C results,” we can also consider how to decompose this variation into the contributions from the component stages of providing a ¹⁴C date through designing certain aspects into the intercomparison or in-house. This is an attribution process that allows a laboratory to complete an accounting of where the variability is coming from and which sources contribute most. For an individual laboratory, the pretreatment chemistry used, the graphitization process, the measurement procedure (background and standards used), the stability of the AMS, and indeed the human aspects, all will make a contribution to the overall variability. How can we estimate, and quantify those uncertainties (i.e. attribute and quantify the variability due to these factors)? It is true that a full decomposition of the variation in a set of results is best achieved within the laboratory (Scott et al. Reference Scott, Cook, Naysmith and Staff2019, this volume) using a designed experiment approach, but it is also possible to quantify some components of within and between laboratory variability in an intercomparison, if designed appropriately. In the intercomparison designs we have developed, this includes replication at different stages in the measurement process (e.g. duplicate samples), a hierarchical approach including both a pretreated and untreated material, and repeat of materials over time, providing linking samples to provide the temporal aspects. To achieve these designs, we need large quantities of sample material which must be shown to be homogeneous (see FIRI homogeneity testing [Scott Reference Scott2003]). For the user, who may be interested in a compendium of dates e.g. for a specific site, they may be using results from multiple laboratories (focussing thus on reproducibility), so that the individual laboratory is also a potential source of variability.

Within laboratory aspects, included in the intercomparison design, have included a hierarchical approach (e.g. the three stages of ICS and FIRI/VIRI) where we designed a series of stages looking at bulk pretreatments, duplicate samples, and different pretreatment methods where variability may be introduced. Thus, in stage 2 of the ICS, we provided homogenized pretreated samples (in duplicate) and in stage 3, the raw material, while in FIRI, we again introduced duplicate samples. We have used two streams of humic samples (and wood) that appear in several studies (see Table 1; Scott et al. Reference Scott, Naysmith and Cook2018, Reference Scott, Cook, Naysmith and Staff2019).

Table 1 Bone samples used in the intercomparisons.

All of this argues for the need to have a sufficient amount of well homogenized materials as part of the design, to ensure that there is sufficient to provide the laboratories with duplicate samples of raw and pretreated materials. Materials we have used include a bulk grain sample from a single year of growth, a bulk tree ring sample (single and multiple rings; see Scott et al. Reference Scott, Cook, Naysmith and Staff2019), including both a cellulose sample but also non-pretreated wood, a bulk humic acid and a raw peat.

Benchmarking

A second significant benefit is participation in the intercomparison gives a laboratory an opportunity to benchmark their own operation and while this is a snapshot in time of performance, this should be considered as supplementing internal laboratory QC procedures. Benchmarking allows the laboratory to compare its performance to an external standard, in this case typically the ¹⁴C community. Benchmarking also assists in identifying processes that represent best practices. When results are considered as a whole, then laboratories can identify areas of strength or weakness, and can demonstrate the validity of the results (NIST 2015).

Our more recent work (Scott et al. Reference Scott2003, Reference Scott, Naysmith and Cook2018) has provided z-scores as a means of benchmarking. The z-score is calculated relative to the sample consensus value and incorporates random and systematic uncertainties where

(1) $${\rm{z \hbox{-} score}} = \,\,\left( {{{\rm{x}}_{\rm{m}}}-{{\rm{x}}_{\rm{A}}}} \right){\rm{/}}{\sigma _{\rm{p}}}$$

where x_m is the reported result, x_A is the assigned or true value for the material, and σ_p is the target value for standard deviation. This latter value is often set a-priori to reflect the precision needed for a specific application field.

And interpretation of the z-scores follows typically as

• |z-score|≤ 2 satisfactory

• 2< |z-score|<3 warning

• |z-score|≥ 3 action

An example of z-scores is given in Figure 1(a), showing the z-score for sample D in SIRI and Figure 1(b) for a specific lab across all samples.

Figure 1 (a) Z-scores for a single sample (D) in SIRI, with warning and action values identified with solid and dashed lines. (b) Z-scores for an individual laboratory with warning and action values identified with solid and dashed lines.

Reference Materials

One key benefit which participation in an intercomparison brings to laboratories and which also provides wider benefits to the community (and one which is not time limited) is access to well characterized samples that are typical of the routinely dated materials and span the applied ¹⁴C time-scale, and ultimately become what the community recognize as reference materials.

“A reference material is sufficiently homogeneous and stable with respect to one or more specified properties, which has been established to be fit for its intended use in a measurement process. Uses may include the calibration of a measurement system, assessment of a measurement procedure, assigning values to other materials, and quality control” (NIST 2018). Our consensus values provide both a specified activity/age but also the associated uncertainty. The procedures now used to calculate the consensus values have changed from the early studies since, with the predominance of AMS laboratories, we have introduced a linear mixed model approach (Scott et al. Reference Scott, Cook and Naysmith2017, Reference Scott, Naysmith and Cook2018) which appropriately accommodates shared sources of uncertainty (such as more than one result being reported, or a single AMS facility being used by several laboratories, etc). In SIRI, unlike earlier studies, we have used a random effects model, this model allows us to include key information on the multiple measurements reported on the same material by each laboratory, which is increasingly relevant since in SIRI, the vast majority of laboratories were AMS.

Historically, we used a robust estimation approach that also takes account of the laboratory quoted error (Rozanski et al. Reference Rozanski, Stichler, Gonfiantini, Scott, Beukens, Kromer and van der Plicht1992). Examples of some of our current archive of reference materials include: near background bone, background wood, background doublespar, humic acid and barley mash. The values of such reference materials to the community include (1) to help develop new methods of analysis, and (2) to calibrate measurements, institute quality control, and determine performance characteristics. At the end of each study, we have published the consensus values and their uncertainties, and also made known when we have archived material that can be made available.

To Quantify Reproducibility, and Comparability

The fourth benefit is similarly related to benchmarking but explores this in a more specific way, e.g. identifying and quantifying pretreatment effects, different technologies, or sensitivities to effects of background and modern standards. The focus here is on the individual laboratory, but also on the community. In this benefit, we have examined laboratory offsets, identified any significant (statistical) differences that can be attributed to background standards, modern reference standards and pretreatment effects. Laboratory offsets, relative to the consensus values, are expressed as the average laboratory difference from the consensus profile. Figure 2 shows the distribution of laboratory offsets estimated in SIRI (with a mean of –1.5 yr).

Figure 2 Histogram of the distribution of offsets in yr (BP) in SIRI.

Additionally to check measurement uncertainties (where we have not used duplicates), we have used a zeta score defined as:

(2) $${\rm{zeta \hbox{-} score = (}}{{\rm{x}}_{\rm{m}}}{\rm{-}}{{\rm{x}}_{\rm{A}}}{\rm{)}}\;{\rm{/ }}\sqrt {{\rm{(}}\sigma _{\rm{p}}^2{\rm{ + }}\sigma _{\rm{a}}^2{\rm{)}}} $$

This now incorporates the uncertainty on the reference values σ_a (all other terms are as defined in Equation 1). Interpretation of the zeta-scores is similar to z-scores, and from them, it is possible to evaluate a reduced χ² (Steele and Douglas Reference Steele and Douglas2006) to quantify any additional variability above that expected given the quoted errors. Visualization is often also shown in a distribution plot such as that illustrated in Figure 3. The shape in such a plot allows us to identify the underlying distribution as well as the spread (or variability).

Figure 3 Distribution plot of age ±2 sigma for sample D (FIRI).

Bone Studies

Substudies have included emphasis on bone samples: apart from the main intercomparisons, we also organized a small cremated bone intercomparison (Naysmith et al. Reference Naysmith, Scott, Cook, Heinemeier, van der Plicht, van Strydonck, Ramsey, Grootes and Freeman2007) at the same time as we were running Stage 2 of VIRI. Overall, 8 bone samples have been used, with VIRI stage 2 including only bone samples (see Table 1). Not every laboratory routinely measures bone and there are considerable differences in the pretreatment procedures used. There is also interest in the quality of the bone sample, so that we were also able to study the C/N ratio and pretreatment procedure differences.

Background and Near Background Samples

We have investigated and included a number of both organic and inorganic background and near background samples over the years identified in Table 2. The role that the laboratory background plays in the result and the uncertainty is important (especially in older samples), and this is one area of laboratory practice where there remains considerable divergence in which materials are used and how the background is calculated. We observed instances where the material we provided was better than the in-house background material and also the very clear issues that still exist in how backgrounds are calculated and reported.

Table 2 Background samples used in the intercomparisons.

For the background samples, we introduced two approaches to summarize the results; In FIRI and VIRI we used a Kaplan-Meier (KM) approach (Scott Reference Scott2003; Dudley et al. Reference Dudley, Wickham and Coombs2016) to estimate the age, dealing with censored age reporting, while in SIRI, we introduced the concept of the limit of background (LoB) (Armbruster and Pry Reference Armbruster and Pry2008; Scott et al. Reference Scott, Cook and Naysmith2017, Reference Scott, Naysmith and Cook2018). Censored ages mean that the age is reported simply in the form “> BP”, and this type of measurement is commonly observed in survival or reliability analysis, where the KM method was first developed, The KM method is a non-parametric method of estimation and the KM survival estimator have been used to estimate the “mean” age (or activity) of the sample whereas in the LoB, we have adopted a different estimation model reflecting the different reporting protocols used by laboratories.