2.1.1. What Risk Factors to Model?

Perhaps the single biggest modelling choice to be made by a company is simply what risks to model. As briefly explained above, a model can only hope to be a simplified representation of the reality it intends to model. A philosophical way of thinking about it is that any model of the universe needs to be at least as large as the universe itself, and in fact in order for us to project it into the future faster than real time, it needs to be even larger.

A key constraint on the number of risk factors to include in a model is availability of resources. Human resources, computer resources and time are required in most modelling exercises and these have to be used in the most efficient manner. To that end, we need to choose the most appropriate aspects to model. Suppose we could simplify the modelling problem considerably, by converting the problem into a simple choice regarding the number of “risk factors” to model. It can be shown that the number of modelling choices faced by a company is simply enormous! For example, section 3.1 details a case study to highlight the enormity of this decision, where over a million inputs to the balance sheet are summarised by as few as 100 risk factors.

Needless to say, choosing the risk factors to model is an extremely important exercise and one that should not be taken lightly. Also important is that once the modelling choices have been made is how to allow for the risk factors that are not modelled. This is addressed in the “grossing-up techniques” commentary in section 3.3.

2.1.2. Choice of Overall Framework

This is quite possibly the second most significant judgement to be made within a (capital) modelling context, although it is not always appreciated as such. Of course, the materiality of the different choices depends on the particular problem at hand. We try to illustrate this in the context of aggregation methodology using a very simple case study with two risks (described as two products of equal size, A and B):

• • Each risk akin to a simple product with a guaranteed £100 m liability, in which not all the risks are hedgeable. There is a residual 1-in-200 risk that the assets (and capital) would lose half their value. The extra capital required at time 0 such that the product has a 99.5% probability of meeting its guarantees at time 1 is (an extra) £200 m (£100 m for each product).

• • The two products are assumed to be uncorrelated.

Let us now consider two commonly used methods of calculating the aggregate capital requirement:

1. 1. Using an “external correlation matrix” approach, the answer is relatively simple. The aggregate capital requirement can be calculated as $$$\sqrt {Capital_{A}^{2} + Capital_{B}^{2} } $$$ , which is £141.6 m in total and £70.8 m per product.

2. 2. Using an alternative approach of undertaking a Monte Carlo simulation of the losses of the two products assuming the distribution of the losses are lognormal (again, a popular choice amongst practitioners) with the same 1-in-200 individual capital requirements. This produces a different aggregate capital requirement of £121 m and £60.5 m post-diversification capital requirement for each product.

This simple example of the impact of the choice of the aggregation framework shows that the aggregate capital requirement can differ by 20% of the liabilities. This example is not special in any sense, in that the two products could represent pretty much any two risk factors.

One can appreciate that there are multiple choices to be made in simply aggregating different risk factors. This is further compounded by choices that need to be made in relation to exotic copula structures, non-linearities, etc.

Finally, this example only touches on a very specific aspect of framework choice; there are many other implicit judgements necessitated by trying to create a simplified representation of the real world. There is a long list of other judgements that need to be made, examples of which include:

• • using heavy models versus lite (proxy) models;

• • if proxy model, choice of proxy model;

• • what measure is used to estimate capital, for example, VaR, tail VaR, etc;

• • granularity of assets, model points;

• • use of instantaneous stress approximation (time 0, time 1 or other);

• • holistic model of the business versus detailed product-specific models aggregated;

• • treatment of new business;

• • fungibility of capital;

• • measure of correlation used.

2.1.3. Choice of Model Components

The choice of model for each risk is an important aspect of modelling. A previous paper, an extract of which is showed below, from this working party (Frankland et al., Reference Frankland, Smith, Wilkins, Varnell, Holtham, Bifis, Eshun and Dullaway2009) showed that fitting different models to the same historical data can have a range of different results, even when using relatively large amounts of data (Chart 1).

Chart 1 Quantile-Quantile plot of UK equity returns, 1969–2008, with various fitted models

Chart 2 shows the 0.5^th percentile estimated using different models for the UK and Denmark. Interestingly, the same extreme potential model error is also true for Denmark (which also has relatively large amounts of historical data), but in this case the models that result in the most and least extreme values are completely different.

Chart 2 0.5th percentile estimated using different models based on UK and Danish equity data

Another example of model risk is described in section 4.2 and the reader is also referred to a recent paper by Richards, Currie and Ritchie, and the subsequent discussion.

2.1.4. Choices Inherent within the Calibration

There are other, more commonly acknowledged calibration choices that should be noted in the same context. For example, calibration of a model is required to fit historical data to a 1-in-200 event, and one of the decisions is the choice of data period to use. Very simply, if we are using x years of data, then the model would accept the worst event within this data window as being a 1 in x event. Thus, the choice of data window makes an important contribution to the results, and it is important to note that even picking any available data set comes with a default assumption regarding the data period.

This context results in an obvious place where one may want to exercise judgement, that is, one may want to impose some views on the extremity of actual events observed. An obvious example can be constructed by using very short data series that included the recent credit crisis, which would naively overestimate the resulting extreme percentile calculated by assuming such an extreme event that would occur regularly within such a short time period. Of course, it goes without saying that the reverse would also have been true when looking at short-term data before the recent financial crisis.

Another example of judgement related to calibration is the choice of method to estimate the parameters of a model. The two approaches that are commonly used are maximum likelihood estimation and the method of moments (described further in section 4.1.4). In calibration exercises where data are limited, these different approaches can lead to very different results.

2.1.5. Judgement Inherent within Underlying Data

Before the application of judgement to the data, it is useful to consider the nature of the raw data used to calibrate the model. The data might not themselves be accurate, being based on estimates, or might contain a systemic effect that would influence our interpretation of the data and perhaps the calibration.

A recent example of data that is based on estimates, which was perhaps not generally appreciated at the time, is the ONS mortality data for older ages in England and Wales. The results of the 2011 Census revealed that there were 30,000 fewer lives aged 90 years and above than expected. In absolute terms, a difference of 30,000 lives is not a large change. However, in relative terms, it represents a reduction of around 15%. Between census years, the exposure to risk is an estimated figure. The funnel of doubt around the estimate increases as the time since the last census increases. The ONS data, being based on a large credible data set, are the source of most actuarial work on longevity improvements, including the projection model developed by the continuous mortality investigation. One implication of the recent census data is that estimates of the rate of improvement of mortality at high ages might have been significantly overstated.

An example of a systematic effect that may be present in data, and should be considered before using any data to calibrate an internal model, would be the derivation process behind a complex market index. As an illustration, Markit publishes a report on its iBoxx EUR Benchmark IndexFootnote ¹ that documents multiple changes to the basis of preparation. This index may be considered a suitable starting point to construct a model of future EUR bond spread behaviour, but such an analysis should consider the effect of the various changes. Of course, the report from Markit contains extensive details of the past index changes, but an equivalent level of detail might not be available for other potential data sources.

2.1.6. Choice of Parameters and Expert Overlay

Irrespective of the model we have chosen to use, we would need to supply it with suitable parameters. This can be done by using some statistical methods of choosing the best parameters, acknowledging the parameter uncertainty inherent from fitting to limited data. Section 4.1.1 looks at possible ways of quantifying the risk capital where parameters are uncertain.

In addition to genuine parameter uncertainty (assuming that past data are truly reflective of the future), we may also have uncertainty as the future conditions may be different to historic conditions (taking interest rates, for example). In this case, management still needs to make some judgements on the choice of parameters.

This judgement on model parameters can be explicit or implicit. For example, at the most explicit level, one may override the 1-in-200 stress itself by superimposing the views of investment experts. In many cases, the paucity of data and the changing economic landscape makes this a regular part of the capital calculation exercise. Alternatively, judgement can be more implicit in the structure of the model. For example, conditional on the form of the model, one may have prior views on certain parameters (e.g. we may have a prior view on the volatility parameter of a lognormal distribution).

However, it should be recognised that any judgement (even though necessitated because of poor data and changing environment) is ultimately subjective. One advantage of explicit judgement (over implicit judgement) is that it is extremely transparent and openly recognises that models and data can only go so far in terms of predicting future distributions.

For either explicit or implicit judgement, we need to recognise that, although one may have a better base assumption, parameter uncertainty (section 4.1.1) still needs to be taken into account. In addition, one should aim to follow a good process when coming up with the parameters, and some observations on current practice within the industry are discussed in the next section.

2.2.1. Regulatory Background

The Solvency II Directive does not contain any direct references to expert judgement, but its use is anticipated in the Level 2 technical standards:

Expert judgement

Article 4, implementing measures, states that:

Level 3, Guideline 55 requires that:

CEIOPS issued a consultative paper on internal models (CP56) in 2009 that, inter alia, set out its views at that time on expert judgement, commencing that “CEIOPS recognises that in a great many cases expert judgement comes into play in internal model design, operation and validation”. Further references from the CP are set out in the Appendix to this policy.

It is likely that the use of expert judgement will be subject to Level 3 guidance. EIOPA prepared an early draft guidance paper on expert judgement, which concentrated on the need for proper documentation, validation and governance of assumptions derived by expert judgement. It also identified the need for clear communication between users and providers of expert judgement to avoid misunderstandings. However, with the ongoing delays to the Solvency II project, it is not clear when this particular guidance will be open to public consultation.

In a letter to firms on IMAP progress in July 2012 (Adams, Reference Adams2012), the FSA identified expert judgement as an area for additional commentary. This recognised expert judgement as important and necessary in many aspects of internal models but identified areas for improvement that include the same areas of documentation, validation and governance. This feedback indicates that firms are finding it challenging to integrate expert judgement, with its inevitable uncertainties, into internal model processes where the emphasis is on a high standard of justification and documentation.

Against this background, a wide range of practices have developed. We comment below on a few of these, based on the experience of members of the working party and a survey made by the Solvency & Capital Management Research Group (Michael et al., Reference Michael, Roger and Peter2012) covering 15 UK life firms. Our comments are biased towards risk calibration in internal models, although, as we note below, expert judgement may apply in many other areas.

2.2.2. Scope of Expert Judgement

Expert judgement can be applied in many areas of actuarial work and not limited to deciding on the distribution of risks. Examples of other areas where expert judgement is commonly used include management actions applied in extreme scenarios, asset valuation in illiquid markets and the use of approximations. Expert judgement applies to standard formula calculations and internal models. Firms differ on which areas are within the scope of expert judgement and in fact defining that scope is not straightforward. There is almost no area of a capital model that does not involve some subjective aspects, and thus it is difficult to determine where the boundary should lie.

Defining when expert judgement should be used is also not easy. Sometimes there seems to be an assumption that data analysis and expert judgement are mutually exclusive, usually with the expectation that expert judgement is used only when data are insufficient to be relied on alone. In practice, it seems more likely that decisions will combine both, but with more weight on expert judgement as the quantity or quality of data diminishes.

In the area of risk distributions, expert judgement is commonly applied to the choice of data, calibrated parameters of risk models and correlations. It is also commonly used to adjust the tails of the distribution that are not considered sufficiently extreme, or conversely, are considered too extreme.

A more fundamental divergence between firms is whether the choice of overall risk model is considered to be subject to expert judgement or not. In section 2.1, we have illustrated how model risk can be as significant as other modelling decisions, if not more so; however, as we noted above, it seems that some firms limit the scope of expert judgement to the choices made once the model is selected.

2.2.3. Expert Judgement Policies

Most firms have developed a framework for expert judgement and this is usually embodied in an expert judgement policy. This is commonly a standalone policy covering all applications of expert judgement, but may instead be embedded in the documentation of those parts of the internal model where judgement is applied. Typically, the policies cover the high-level principles and the governance process, such as the various levels of review and signoff needed for the approval of specific expert judgements. Other contents vary widely and there are some significant differences in the scope and application of expert judgement.

2.2.4. Who are the Experts?

Arguably, the single most important aspect of expert judgement is simply identifying suitable and relevant experts. Policies may indicate what criteria are used to establish whether someone qualifies as an expert, but often there is no formal process to identify them in advance. Instead the choice might be left to the modeller's discretion, which allows for some flexibility, but one may need to be careful to ensure that a wide enough range of views is sought. Another important aspect is for the experts to be sufficiently independent of both the calibration process and the capital calculation process. The importance of this is likely underestimated, as are the risks of expert judgement influencing the capital outcomes.

Occasionally more than one expert may express a view, which in turn leads to differences of opinion that need to be merged into a single modelling decision. Thus, in addition to model risk, we now have to consider “expert risk”. Some firms get round this by using a technical committee, and therefore effectively (or explicitly) the committee acts as the “expert” and the committee process resolves diverse views into a single decision. However, committees are only one way to aggregate the opinions of several experts and may not be the most effective, and this aspect has been better researched in other fields. SeeFootnote ³ Ouchi (Reference Ouchi2004) for a literature review of expert judgement research. A large part of this research covers elicitation and aggregation of expert judgement, which so far do not seem to have been widely considered in economic capital applications (other than possibly operational risk).

Elicitation is the process of gathering expert judgement. Experts are human and can be subject to various biases or herding. In a group or committee, the most senior, most confident or simply most biased (e.g. not independent) experts may overrule other opinions. Various structured approaches to elicitation have been developed to reduce the impact of these biases (Cooke & Goossens, Reference Cooke and Goossens1999) give a detailed procedure guide. As an example, one of the first such approaches was the Delphi technique, in which experts give their initial opinions individually and these are then shared anonymously with the group. The experts can then revise their opinions and the process is repeated until a consensus is reached.

To summarise, we need to be conscious about careful choice of experts, ensuring independence of experts, and the aggregation of expert judgement from more than one expert. There is scope for further research in these areas to see whether a more structured approach to expert judgement could address some of these perceived weaknesses.

2.2.5. Validation of Expert Judgement

Expert judgement in an internal model is subject to validation requirements. Expert judgement, by its nature, is not the output of a mechanical process that can be tested in a simple manner. Where expert judgement has been used and is of material consequence to capital requirements, it is good practice to try and validate the judgement.

Some examples of potential methods that can be used to validate judgement are given below:

• • The most common approach is a review and challenge process, which involves discussing the judgement with the expert, with the aim of better understanding the rationale and applicability to the current problem.

• • Discussion by an independent panel (e.g. a technical committee) may provide a useful avenue for discussion and debate.

• • Sensitivity analysis, to establish the importance of the expert judgement.

• • Comparison against any relevant external information such as industry benchmarks and other regulatory information (e.g. internal model stress against the standard formula stresses). Care must be taken to avoid systemic risk via “herd behaviour”.

• • Backtesting has also been proposed, although this may be problematic as judgement is often used in situations where data are sparse by definition (i.e. opining on an extreme event).

2.2.6. Case Study

We illustrate the importance of the validation of expert judgement and independence of experts by considering the views of different experts (at different points in history) on the maximum possible human lifetime for different countries. In 2002, Jim Oeppen and James W. Vaupel published a paper looking at life expectancy, and how proposed limits on life expectancy have consistently been exceeded by reality (Chart 3).

Chart 3 Male life expectancy at birth for a series of countries as a function of the year of publication

The markers on Chart 3 show male life expectancy at birth for a series of countries, expressed as a function of the year of mortality table publication. The fitted trend is shown as the thick black ascending line. The horizontal bars correspond to expert assertions of biological upper bounds on life expectancy. The left-hand end of each bar represents the date of the assertion. The intersection of each horizontal line with the ascending trend indicates the point at which experience refuted the claimed upper bound. Where there is no intersection, and the horizontal bar lies to the right of the fitted trend, this indicates that the asserted upper bound was already contrary to published mortality tables at the date the assertions were made.

It is not surprising that even experts get life expectancy wrong. Longevity is a subtle and complex area, with its fair share of historic data, data errors, competing models and uncertain impact of future social trends. What is surprising is that experts have appeared to systematically underestimate future lifespans rather than a mixture of underestimating and overestimating, which we would have intuitively expected.

Oeppen and Vaupel give one possible explanation for the consistent underestimates that “They give politicians license to postpone painful adjustments to social security and medical-care systems”. One might say that in the market for theories, there is a demand from politicians for short lifespan theories. There might be a similar demand from insurers or pension funds who write annuities. There is no similar demand for theories of long lifespans, as the pensioners who might benefit from better capitalised annuity writers are seldom sufficiently well organised to commission research to support their case. As pointed out by Smith & Thomas (Reference Smith and Thomas2002), a skewed demand for experts may bias the theories that the market supplies.

3.2.1. PCA

This is a statistical method that transforms data from a high dimension space into a space of reduced dimension. A subset of new features from the original features in the data is performed by means of functional mapping techniques that aim to keep as many features as possible of the original data set. PCA is one of the most commonly used feature extraction techniques.

PCA uses an orthogonal transformation to convert a set of observations of correlated variables into a set of orthogonal principal components. The number of principal components is less than or equal to the number of original variables. PCA is a heavily researched statistical technique and already has a lot of applications in finance. For detailed examples in the context of interest rate modelling, refer to Lazzari & Wong (Reference Lazzari and Wong2012).

Deciding which factors to retain in the model can be based on the following techniques:

1. a) The eigenvalue-one criterion: also known as the Kaiser criterion, components are retained that have an eigenvalue greater than or equal to 1, that is, the retained variable is accounting for a greater amount of the variance than that contributed by one variable.

2. b) Proportion of variance retained: retain a component if it accounts for a threshold percentage, for example, 5% or 10% of the total variance, where the proportion of variance retained is given by: eigenvalue for component of interest/Σ (eigenvalues of correlation matrix).

Non-linear dimension reduction techniques extend these ideas based on non-linear mappings. Again, the principle is the same, maximising the variance in the resulting data set relative to the variance in the original data set.

The single biggest criticism of PCA in finance is that it is a pure statistical technique and ignores any economic or real-world causalities. Thus, it needs to be used and interpreted very carefully to avoid data mining and spurious accuracy. For example, you do not want to inadvertently have one of the important risk factors of your model be the size of Brazilian coffee beans, simply because the time series was included in the set of regression variables and provided an almost perfect fit. Although this is an extreme example that is obviously nonsensical, the lesson is that pure statistical models would be blind to this. More importantly, there may be many other significantly more subtle areas where the models are inadvertently subject to data mining.

Another criticism is that pure PCA factors are simply a linear combination of many variables and would lack intuition, as well as being subject to changes over time that are hard to explain.

The main mitigant of inadvertent data mining is for there to be expert human intervention at key stages of the process, choice of regression inputs through to sense checking and validation of the output results. Some of these are discussed in the next section.

3.2.2. Direct Choice of Principal Factors

This is another technique that is also commonly used alongside the PCA approach. This method directly chooses what are thought to be the principal factors so as to arrive directly at a fitted model, without recourse to any dimension reduction technique being applied. As such, this approach relies hugely on the skill or otherwise of the modeller in picking the appropriate modelling random variables that best satisfy the “eigenvalue-one” criterion and/or the “maximising the proportion of variance retained” criterion discussed above.

Although this approach could be very efficient in the presence of a skilled modeller, it suffers from the same drawbacks as per other aspects of judgement. In addition, the subjective element reduces the transparency of the choice components if not properly documented.

3.2.3. Hybrid Methods

In practice, neither a purely statistical nor a purely judgemental method is used. The method often adopted is a combination of the direct choice of principal factors and statistical techniques. Although this may often be done implicitly, it may be useful to be aware of the judgement and statistical steps explicitly, so as to avoid the worst pitfalls of each method.

Also important is the need to be aware of areas where selection of risk factors can cause potential problems. Some possible pitfalls are:

1. a) The model may have been subject to too wide a range of explanatory variables, resulting in inadvertent data mining.

2. b) The most important factors may have been wrongly selected or not selected at all, giving spurious and misleading results.

3. c) Presence of correlated variables results in a risk of a more intuitive/direct explanatory variable killed for another less intuitive/useful variable.

4. d) If the factors selected mechanically have not been validated, there is a risk that some inappropriate factors have been inadvertently included.

5. e) The weighting of the different factors may change over time. Factors that may have had an influence in the past may dissipate over time, whereas the opposite may be true of other factors.

6. f) Exogenous factors, for example, a change in external regulatory or commercial environment may change the nature of the problem and one needs to remember to revisit the choice of random variables in the selection process (i.e. there may be some “emerging” risk factors).

3.3.1. R ² Adjustment

Following dimension reduction, a computation of the R ² value can be compared between the original data set and the data set following dimension reduction. As noted above, the R ² value in the dimensionally reduced data set will be an overestimate based on number of degrees of freedom used in the fitted variables, rather than the degrees of freedom used in the fitting process. A possible approximate grossing-up factor may be undertaken as follows:

$$\frac{{{\rm{(Adjusted) }} \,{{R}^{\rm{2}}} \ {\rm{value}} \ {\rm{in}} \ {\rm{dimension - reduced}} \ {\rm{data}} \ {\rm{set}}}}{{{{R}^{\rm{2}}} \ {\rm{value}} \ {\rm{in}} \ {\rm{original}} \ {\rm{data}} \ {\rm{set}}}}$$

This only allows for the PCA element. A suitable margin for prudence should also be incorporated, to allow for any potential missed features in both the original data set and the dimensionally reduced data set that could affect results significantly in the future.

3.3.2. R Eigenvalue Grossing-up Factor

Compute the eigenvalue correlation matrix for both the components in the dimensionally reduced model and the original model. The grossing-up factor for the model then becomes:

$$\frac{{\rSigma ({\rm{eigenvalues}} \ {\rm{of}} \ {\rm{correlation}} \ {\rm{matrix}} \ {\rm{of}} \ {\rm{original}} \ {\rm{model}})}}{{\rSigma {\rm{(eigenvalues}} \ {\rm{of}} \ {\rm{correlation}} \ {\rm{matrix}} \ {\rm{of}} \ {\rm{dimensionally}} \ {\rm{reduced}} \ {\rm{model)}}}}$$

Again a suitable prudence margin should be allowed for risk selection errors in the original model.

3.4.1. Stress and Scenario Testing: an Example

In this section, the results of a case study to assess the impact of the use of different copulas models are investigated.

This case study assesses the impact of prior beliefs, especially when there are not sufficient data to be conclusive about the assumption being made. It involves assessing the impact on aggregate capital when different types of copulas are used to aggregate individual capital requirements.

The copulas used in these case studies are the copulas that lend themselves easily to aggregate more than two risks. As such copulas such as Gumbel, Clayton and Frank are not considered.Footnote ⁴

In the first case, the marginal distributions are all assumed to be normally distributed as shown in Table 2. Another case study is presented later in which different marginal distributions are assumed. We assume that we have the following risks with the distributions as specified in Table 2.

Table 2 Marginal risk factor distribution

The risks are assumed to be correlated as per Table 3.

Table 3 Correlation between risk factors

The aggregate capitals based on the following copulas are assessed:

1. a. correlation matrix;

2. b. normal;

3. c. Student T with different degrees of freedom.

The results of using different copulas are set out in Table 4.

Table 4 Capital required by type of copula

It is worth noting that when the marginals are all elliptically distributed (e.g. normal, Student T) the results of a normal copula and correlation matrix should be very similar, with any difference arising because of an inadequate number of simulations used in the Monte Carlo simulation.

Another sensitivity test was undertaken in which the marginal distributions were not all normally distributed. The distributions used are set out in Table 5. The parameters and the distributions were selected such that the capital of the marginal distributions was the same as that of the normal distributions above.

Table 5 Sensitivity - the effect of some risk factors not being normally distributed

¹Note that the mean and standard deviation do not refer to lognormal distribution, but to its logarithm.

The results of this sensitivity are shown in Table 6.

Table 6 Sensitivity - capital required when some risk drivers are not normally distributed

The sensitivity testing as described in the case study above can be used to assess the range of values that the aggregate capital can assume, given the uncertainty about the appropriate dependency approach to use. It is important to note that this covers just one aspect of sensitivity testing, and that this would ideally need to be repeated for all of the key decisions in the capital calculation framework.

4.1.1. Parameter Uncertainty in Location Scale Families

We consider an example where the underlying model is of a known shape, but where the location and scale of the distribution are subject to uncertainty. This is usually the case in practice.

We then consider three possible definitions of a percentile where parameters are uncertain (Table 7).

Table 7 Possible percentile definitions when there is parameter uncertainty

It is not always clear in practice, even for statutory purposes such as computing the solvency capital requirement, which, if any, of these definitions applies.

4.1.2. Comparison of Statistical Definitions to Actuarial Best Estimates

We contrast probability definitions with actuarial concepts of best estimates, as discussed, for example by Jones et al. (Reference Jones, Copeman, Gibson, Line, Lowe, Martin, Matthews and Powell2006):

The actuarial best estimate definitions refer to a lack of deliberate bias. This suggests that provided the actuary did not intend to introduce bias, a “best estimate” results. If there turns out to be a bias in a statistical sense, this does not disqualify an actuarial best estimate provided the bias is unintentional. This highlights the difference between actuarial best estimates defined in terms of intention and statistically unbiased estimates defined in a mechanical way.

Our statistical definitions refer to probabilities that can in principle be tested in controlled simulation experiments, and indeed we will conduct such experiments in the example that follows. As defined above, any best estimate is a point estimate. However, there is an interval of plausible outcomes around this estimate. There are different statistical definitions for such an interval. The experiment that follows considers two of these intervals (described in section 4.1.1):

• • substitution: β-quantile;

• • confidence interval with probability γ containing the β-quantile;

• • prediction interval containing the unseen observation with probability β.

What the experiment shows is that the properties of a probability distribution do not necessarily determine a unique construction for an interval. There might be (and indeed, there are) several ways to construct intervals satisfying the definitions.

4.1.3. Five Example Distributions

We now show some example calculations of these intervals based on five distribution families. In each case, we define a standard version of a random variable X; the other distributions in the family are the distributions of sX + m where m can take any value and s > 0.

Our five distributions include fatter- and thinner-tailed examples, and include asymmetric distributions (Table 8).

Table 8 The five example distributions and their properties

Our families consistent of cumulative distribution functions $$$F\left( {\frac{{x\,{\rm{ - }}\,m}}{s}} \right)$$$ and density $$$\frac{1}{s}f\left( {\frac{{x\,{\rm{ - }}\,m}}{s}} \right)$$$ , where F and f are taken from a (known) row of Table 8.

Chart 6 shows these probability density functions. We have chosen values of m and s for each family to make the distributions as close as possible to each other.

Chart 6 Probability density functions of the five example distributions

4.1.4. Methods of Estimating Parameters

We consider four ways of estimating model parameters given a random sample of historic data (Table 9).

Table 9 Four methods of estimating model parameters

4.1.5. Interval Calculation

We now provide algorithms for calculating the different types of intervals (Table 10).

Table 10 Algorithm for calculating the different types of interval

When talking about probabilities, care needs to be taken with respect of the set over which averages are calculated. The frequentist methods (method of moments, probability weighted moments, maximum likelihood) measure probabilities over alternative data sets. The Bayesian statements average over possible alternative parameter sets but not over any data sets, besides the one that actually emerged. A 99.5% interval under a frequentist approach is not the same as a 99.5% Bayesian interval.

We use Monte Carlo methods to calculate confidence and prediction intervals. We show these in the case of 20 data points, using the method of moments, based on the 99.5%-ile. These measures differ only because the data are limited; as the data increase these are all consistent estimators of the “true” percentile. None of these is *the* right answer; the different numbers simply answer different questions (Chart 7).

Chart 7 Illustration of various interval definitions for the five example distributions

The interval size (expressed as a number of sample standard deviations) varies from one distribution to another. Note that, although the T4 distribution has the fattest tails in an asymptotic sense (it has a power law tail, whereas the others are exponential), the Gumbel produces the largest confidence intervals. This is because the asymptotic point at which the T4 distribution becomes fatter lies way beyond the 99.5%-ile in which we are interested.

We also consider how the intervals vary by the number of observations. The prediction intervals are shown below. We can see that the prediction interval requires a smaller number of standard deviations as the sample size increases, because a better supply of data reduces the errors in parameter estimates (Chart 8).

Chart 8 Prediction Interval variation by sample size for the five example distribution

4.2.1. Example of Model Risk

A recent paper by Richards, Currie and Ritchie compares seven different models of longevity. They compare the value of a life annuity with a 70-year-old man, limited to 35 years and discounted at 3%. The paper constructs a probability distribution forecast for this annuity in 1 year's time using different models of mortality improvements. We reproduce table 5 from that paper below (Table 11).

Table 11 Comparison of average annuity value and percentile estimate by different longevity models

.

Please refer to the paper by Richards et al. (2014) for more details about the mortality models. We might hope for a degree of consensus between the different modelling approaches, but these figures show the opposite. Indeed, a future annuity value of 12.5 lies below the mean for two of the models, but above the 99.5%-ile for two other models (http://www.actuaries.org.uk/sites/all/files/documents/pdf/value-risk-framework-longevity-risk-printversionupdated20121109.pdf).

4.2.2. Another Example: Gauss and Gumbel Distributions

Let us return to the set of five probability distributions described in section 4.1.3. The distribution functions are sufficiently close that the curves are difficult to distinguish by eye. However, we can separate them by subtracting one of the cumulative distribution functions, for example, the logistic. Chart 9 shows the CDF for each distribution minus the logistic CDF.

Chart 9 Illustrating the difference in the CDF of the five example distributions, measured relative to the logistic

Although these distributions have different shapes, the distribution functions differ nowhere by more than 4%. This suggests that it will be difficult to separate the distributions using statistical tests based on small sample sizes.

Goodness-of-fit tests such as Kolmogorov–Smirnov and Anderson–Darling have low power when data are scarce. Table 12 shows the power of KS and AD tests with 20 data points. The chance of rejecting an incorrect model is often only marginally better than the chance of rejecting the correct model, and in a few cases the correct model is more likely rejected than an incorrect one.

Table 12 Probability of Rejecting a Model Fitted Using the Method of Moments Tested Using a Kolmogorov–Smirnov (or Anderson–Darling) Statistic Based on 95% Confidence and 20 Observations

Even with samples of 200 or more, it is common not to reject any of our five models. This implies that model risk remains relevant for applications including scenario generators, longevity forecasts or estimates of reserve variability in general insurance.

4.3.1. Robust Statistics and Ambiguity Sets

Robust statistics is the study of techniques that can be justified across a range of possible models rather than a single model. It can be used to help derive prediction intervals that are robust to the choice of distribution.

Suppose using the method of moments we wanted to construct a prediction interval of the form $$$\left( {{\rm{ - }}\infty, \hat{\mu }\, + \,y\hat{\sigma }} \right)$$$ valid across a class of distributions (this set is known as the ambiguity set). We cannot achieve exact α-coverage for all distributions. We can, however, achieve at least α-coverage for a class of distributions simply by taking the largest y across that class. For example, we might specify that the methodology should cover at least a proportion α of observations X _{n + 1} for an ambiguity class, including Gaussian and logistic distributions. In this case, our chart in section 4.1.4 shows that the logistic distribution produces the highest y, and therefore the prediction interval is defined based on the logistic distribution, knowing this is conservative for other distributions in the specification of the ambiguity class.

Robustness for a given ambiguity class says nothing about models outside the class. We might construct prediction intervals robust across uniform, normal and logistic distributions, but these intervals would not be valid for Gumbel distributions. They would also not be valid if other assumptions are violated – if the observations are not independent or are drawn from more than one distribution.

The size of the ambiguity set is a key determinant of prediction interval size. A bigger ambiguity set means a harsher test of coverage and larger prediction intervals. It also means more wastage, in the sense that if the true distribution is one of the more benign from the ambiguity set, then the prediction interval will contain a higher than intended proportion of future observations.

Nassim Taleb (2007) has popularised the notion of “Black Swan” events, that is, unpredictable events that arise with no prior warning (and by definition unpredictable with models that existed before the event happening). Models popular in the mid-2000s, many based on Gaussian distributions, were criticised for producing too few Black Swan events and therefore understating risks. Black Swan events also highlight the classic philosophical “problem of induction” that inference from past events can only work to the extent that the same model applies in the past as in the future. We could define a “Black Swan” model as one where one set of formulas applied until yesterday, and tomorrow another unrelated set of formulas takes over. There can be no robust statistical procedure that works for such models as the past data are of no value for future inference. It may potentially be the case that the world has fundamentally and unrecognisably changed, but one also needs to be extremely careful that less likely models are not rejected early on in the decision-making process in favour of the most plausible model without allowing for model risk. One conclusion is that any claims of robustness with respect to a statistical procedure should include a careful description of the ambiguity set against which the procedure is robust.

5.1.1. Spanning Error

The method of proxy functions assumes that the true assets and liability functions are of the chosen form, or can be approximated sufficiently accurately by functions of that form. This assumption could fail severely, for example, if:

• • Assets and liabilities depend on inputs that have been excluded in the selection of risk factors to model, through the dimension reduction process discussed previously. Thus, the behaviour of assets and liabilities are not sufficiently explained by the retained risk factor inputs.

• • The true function has a “cliff-edge” (i.e. a discontinuity), whereas all the basis functions are continuous. More subtly, the true function could have a kink (i.e. point where it is not differentiable), whereas the basis functions are assumed to be continuously differentiable.

• • The assets and liabilities have “sink holes”, that is, regions of parameter values where assets collapse or liabilities explode, whereas none of the basis functions exhibit this behaviour.

• • The asymptotic behaviour of the liabilities is simply different from the functions chosen. This is not an uncommon occurrence, because polynomials (apart from constants) tend to ±∞, whereas guaranteed liabilities would tend to 0 when they are out of the money and ∞ when in the money.

These are all examples of “spanning failure”, where the true assets and liabilities are not in the linear span of a proposed set of basis functions. The “spanning error” is any mis-statement in the required capital that arises from spanning failure. We set out some approaches that can be used to validate proxy models through case studies in the sections that follow.

5.1.2. Selected Test Models

We construct test valuation formulas based on three risky asset classes: government and corporate bonds, equities and a risk-free cash asset. We consider three lines of business: regular premium term assurance, a life annuity and a single premium-guaranteed equity bond. We use simplified policy models so that exact valuation is possible with closed formulas, based on nine risk drivers: risk free rate, equity price, equity volatility, corporate bond spread, liquidity premium, mortality (term and annuity) and lapses (term and guaranteed equity).

This is a simplified set of risk drivers for ease of calculation. We use a single discount rate for risk-free cash flows of all maturities. The equity volatility is required for valuing the guarantee according to option pricing theory. We assume that annuities cash flow will be discounted at a discount rate that includes a return premium for holding illiquid assets. This is assumed to be assessed with respect to the observed yield spread on a class of illiquid asset subject to credit rating criteria.

Our firm also holds corporate bonds. The change in value of these bonds is driven by changes in bond spreads, but in this case we need to follow the fate of a fixed portfolio of bonds rather than tracking typical spreads for a particular grade. This distinction implies that a portfolio of bonds cannot perfectly hedge the liquidity premium assessed in relation to a particular grade.

The formulae for the assets and liabilities described in the table overleaf are detailed in Appendix B.

5.1.3. Curve Fitting Methodology – Annuity Case Study

To illustrate the issues involved, we start by approximating the annuity examples, using polynomials in the discount rate (which we denote by r) and in the maximum lifetime (denoted by t). There are several different ways to build up polynomials of increasing order. We have considered the following four ways (Table 13).

Table 13 Possible ways to construct polynomials for curve fitting

5.1.3.1. Least Squares Results

We calculated least squares estimates using a 101-step grid for each of the two risk drivers, that is, a 101 × 101 grid covering max lifetime between 0 and 20 years combined with discount rates between 0% and 25%. The Chart 10 below shows the minimised root mean square error (RMSE) as a function of parameter count (maximum 36) for each of our fit families.

Chart 10 Root mean square error against parameter count by type of fitting polynomial

Chart 10 shows how the fit improves as the number of coefficients increases. The results show that the separable formula, with or without the xy terms, quickly gets stuck. Adding higher order polynomials in just one variable has a negligible impact on the RMSE. Instead, it is necessary to add sufficiently rich joint powers alongside the univariate powers to achieve convergence.

The chief advantages of the least squares method include:

• • It is known to have a unique solution provided the number of sample points exceeds the number of fitted coefficients.

• • Finding that unique involves the relatively straightforward solution of simultaneous linear equations. In addition to being readily soluble, it is also easy to demonstrate that a proposed solution satisfies the relevant equations and thus verify that the solution is optimal from a least square perspective.

• • Beyond our particular example, where the liability is defined as the average of Monte Carlo simulations, there may be computational efficiencies by the use of least squares Monte Carlo techniques.

The chief limitation of least squares techniques is the difficulty of converting a RMSE into a bound in the accuracy of an estimated percentile. The problem is that the RMSE is an average of the errors; the error in a percentile estimate may be much larger because not all points have an equal impact on the percentile. Large percentile errors arise if the estimate is sensitive to the fit-for-risk driver values where the fitted proxy polynomial diverges most from the true function.

5.1.3.2. Regression Diagnostics

Statistical regression packages usually provide additional output, including R ² statistics, parameter significance tests, confidence intervals and information criteria such as those of Akaike. We might hope these could provide a scientific basis for including or excluding terms in a polynomial. The derivation of these tools relies on various assumptions listed below (Table 14).

Table 14 Assumptions implicit in regression diagnostic tools

The standard statistical assumptions clearly do not apply in our example where the fitting is not even random and the underlying annuity function is not a polynomial. Such statistical tools overstate the effective number of observations by assuming that each new observation contains an independent model observation, whereas in our curve fit new observations are still from the same deterministic curve. The statistically derived intervals are spectacularly too narrow compared with the true error.

5.1.4. Curve Fitting Methodology: Combined Model

We now develop tools for fitting curves to the actuarial formulas in a combined model of all risks. This can help us to evaluate the extent to which observed RMSE for each asset and liability type may translate into errors in estimated percentiles.

We fit curves to a series of stress tests. The stress tests are constructed using median values for each risk driver, as well as high (α-quantile) and low (1−α-quantile) values. For example, if α = 0.995 then we would calibrate to the 0.5%-ile, median (50%-ile) and 99.5%-ile for each risk driver.

We look at all combinations of high, median and low for each risk driver. This implies that with nine risk drivers, we may have to examine 3⁹ = 19,683 stress tests. Thankfully, this is an overestimate, as our example assets and liabilities depend individually on at most four risk drivers. The maximum number of stresses is then 3⁴ = 81 risk drivers, in the case of the guaranteed equity bond. The minimum number of stresses is three, for the risk-free bond and the equity, which depend on only one risk driver.

Having constructed the stress tests, we then fit a polynomial. In each case, we use ordinary least squares to fit the formula. We consider three fits based on the polynomials considered in the annuity example:

• • Linear fit: we estimate assets or liabilities as a constant term plus linear terms in each risk driver.

• • Separable quadratic fit: we estimate assets or liabilities as a constant term, plus linear and squared terms in each risk driver. This formula is still separable, that is, basic own funds is a sum of functions, each of which depends only on one risk driver. There is no mechanism for the level of one risk driver to affect the sensitivity of net assets to another driver.

• • Quadratic fit with cross terms: assets and liabilities are a constant term, plus linear and squared terms in each risk driver, and also products of risk drivers taken two at a time.

We can consider extending these fits to cubic and higher order polynomials. However, we are unable to investigate such fits in our simple example, at least in the one-dimensional cases (equities and government bonds), because we have chosen to evaluate only three fitting points that are insufficient to estimate polynomials of higher than quadratic order.

5.1.5. Risk Driver Distribution

For demonstration purposes we assume that risk drivers have shifted lognormal distributions. We construct these using formulas of the form:

$${\rm{riskdriver}}\, = \,{\rm{median}}\, + b\,\frac{{{{e}^{cZ}} \ {\rm{ - }}\,1}}{c}$$

where b is a positive coefficient, whereas c is positive or negative. If c = 0 we replace the fraction $$$\frac{{{{e}^{cZ}} \ {\rm{ - }}\,1}}{c}$$$ by its limiting value Z. This is readily calibrated to quantiles. For example, suppose for some k, the Φ(−k) quantile is median −α and the Φ(k) quantile is median + β, we have to solve the equations:

$$\matrix { b\frac{{{{e}^{{\rm{ - }}ck}} \ {\rm{ - }}\,1}}{c}\, = \,{\rm{ - }}\alpha \cr b\frac{{{{e}^{ck}} \ {\rm{ - }}\,1}}{c}\, = \,\beta $$

From this, it immediately follows that

$$\matrix { {{e}^{{\rm{ - }}ck}} \, = \,\frac{{{{e}^{{\rm{ - }}ck}} \ {\rm{ - }}\,1}}{{1\,{\rm{ - }} \ {{e}^{ck}} }}\, = \,\frac{\alpha }{\beta } \cr c\, = \,\frac{1}{k}\ln \left( {\frac{\beta }{\alpha }} \right) \cr b\, = \,\frac{{c.\alpha \beta }}{{\beta \,{\rm{ - }}\,\alpha }} $$

There is a special limiting case when α = β, where c = 0 and b = α/k.

Within the firm's internal models, there is no general rule relating the opening (t = 0) risk drive values to their future distribution. However, for simplicity in the current curve fitting tests, we assume that the opening values are equal to the median.

5.1.6. Some Simulated Results

Chart 11 shows a scatter of the basic own funds (for the whole company) based on fitted polynomials compared with the “true” result based on the exact formula, for 1,000 simulations. In a capital calculation, we are interested principally in the bottom left-hand side, which corresponds to low values of own funds.

Chart 11 Actual BOF against BOF calculated by proxy formula by type of fitting polynomial

Overall, the quadratic with cross terms provides the best overall fit, which is not surprising as it has the largest parameter count. However, the deviations between the exact and fitted formula are substantial in places, and furthermore appear to show significant biases – for example, the linear regression overestimates BOF more often than it underestimates BOF.

This observation may at first seem odd; if the coefficients are estimated by regression, then surely the average residual should be 0. To resolve this paradox, we need to note that there are many different ways to calculate an average. The regression is an average over a small number of selected stress points, whereas our simulated BOF produces an average over a realistic distribution of risk drivers. We would not expect these averages to coincide. Indeed, biases in fits are commonly observed whenever the scenarios used for fitting a proxy model are not the same as those used for capital calculations.

Further investigations involving the elimination of negative reserves have revealed that discontinuities or kinks in the formulas need to be modelled as an additional layer. They cannot be well modelled using polynomial functions, even of very high order.

5.2.1. Market-Consistent Valuation and Monte Carlo Sampling

A market-consistent valuation of a complex, path-dependent liability will typically involve running a number of risk-neutral scenarios, from an appropriately calibrated economic scenario generator through a cashflow model. The mean of the guarantee and option cost, across all the simulations, is the market-consistent value of the guarantees. Furthermore, we can use the standard deviation of the individual scenario costs to calculate a confidence interval for the mean (i.e. the market-consistent cost of the guarantees and options in this instance) using the independent and identically distributed property and the central limit theorem. It is worth noting that estimation of the confidence interval as shown below applies under some strict conditions, such as the variable in question being normally distributed and the suitably large number of scenarios:

$${\rm{95\%, }} \ {\rm{two}} \ {\rm{sided,}} \ {\rm{confidence}} \ {\rm{interval}} \ {\rm{for}} \ {\rm{mean}} \ {\rm{ = }}\,\hat{\mu }\, \pm \,{\rm{1}}{\rm{.96}}\frac{\sigma }{{\sqrt N }}$$

Various variance reduction techniques can be applied to make the valuation converge more rapidly – antithetic sampling, control variates, stratified sampling, importance sampling and low-discrepancy numbers (Sobol, Niederreiter, etc.). When using these techniques, the assumption of independent and identically distributed scenario outputs may not be valid.

Occasionally, situations arise, associated with sampling error, which can trap the unwary. For example, consider 1,000 observations from a lognormal distribution whose parameter (the standard deviation of the log) is β. Each observation is of the form exp(β.Z) for some Z ≅ N(0,1). When β is large (and hence the distribution is most skewed) the interval

$$\left( {\hat{\mu } \ {\rm{ - }}\,1.96\frac{{\hat{\sigma }}}{{\sqrt N }},\;\hat{\mu } \ {\rm{ + }}\,1.96\frac{{\hat{\sigma }}}{{\sqrt N }}} \right)$$

is dominated by the term exp(β.Z _max) where Z _max is the largest observation. However, we know that the true mean is exp(β ²/2) and that for large β it is pretty certain that the true mean lies outside the confidence interval. In this case, it would be much better to recognise the data as lognormal and take logs of the data and estimate the parameters using analytical formula. The reason for this is that the above formula for determining confidence interval is based on the assumption that the variable of interest is normally distributed. Thus, when the distribution followed by the variable is highly skewed, then the above formula would not be applicable.

In cases such as this it seems useful to have some room to override the standard approach – the example described here is relevant to out-of-the-money guarantee costs.

5.2.2. Quantile Estimates and Monte Carlo Sampling

It is useful to start by considering the base case of an internal model using real-world simulations of risk drivers and loss functions. It is desirable to know the sampling error in the SCR estimate that is obtained from an ordered series of losses from a run of the internal model.

At a naïve and qualitative level, the degree of sampling error can be assessed by examining the distribution properties of the ranked internal model output. If the losses around the 99.5% simulation are “smoothly distributed”, tightly grouped and lie within the same range, then the sampling error will be small. Equivalently, we can look at the distribution of the 99.5%-ile simulation from a repeated sampling of the internal model with different random number sequences (but with all other parameters left unchanged). Obviously, we will need to recheck if these distribution properties apply for each recalibration of the model.

The estimation of quantiles can be considered at a more mathematical level. First, defining a notation for a quantile as Q_α (b.X) where:

• • α is a probability between 0 and 1;

• • X is a random vector, which might represent profits from different business units or risk types;

• • b is a vector of multipliers, reflecting a firm's exposure to each business or risk type.

The given data take the form of n observations of a random vector X, assumed to be a random sample from some distribution. For historic data, the assumption of a random sample may be open to challenge. However, in some applications, we are seeking to estimate percentiles from data that have been generated from Monte Carlo simulation; in that case, we can be more confident that the observations truly are independent samples from a common distribution.

The estimation of a quantile is often an intermediate step in a longer calculation. For example, a firm may wish to understand the sensitivity of a percentile to the selected risk exposures, which means calculating the sensitivities of the quantile Q_α to the exposure vector b. There may be some form of optimisation involved, for example, finding multipliers to maximise expected profit subject to constraints on financial strength that might be expressed in terms of extreme quantiles.

In the simplest case, we can directly estimate the quantile from the empirical distribution function – here we refer to this approach of quantile estimation as direct percentile estimation.

5.2.2.1. Direct Percentile Estimation

To ease notation, we consider the scalar case. Suppose then we have n independent scalar observations X ₁, X ₂, .., X_n .

We can sort this into increasing order, denoting the sorted variables with subscripts in parentheses: X ₍₁₎ ≤ X ₍₂₎ ≤ … ≤ X _{( n )}.

The empirical cumulative distribution function is given by

$$\hat{F}(k)\, = \,\frac{1}{N}\mathop{\sum}\limits_{i\, = \,1}^N {\chi \{ {{X}_i}\,\leq \,k\} } \, = \,\frac{1}{N}\mathop{\sum}\limits_{i\, = \,1}^N {\chi \{ {{X}_{(i)}}\,\leq \,k\} } $$

This is the average of a series of step functions and is easily seen to be an unbiased estimate of the true cumulative distribution function. The corresponding right continuous inverse is (for 0 ≤ α < 1):

$${{\hat{F}}^{{\rm{ - }}1}} (\alpha )\, = \,X\left( {1\, + \,\left\lfloor {N\alpha } \right\rfloor } \right)$$

where X(i) denotes the i ^th sample value of X, in increasing order. This provides the most basic estimate of sample quantiles.

5.2.2.2. Direct Percentiles: Exact Distribution

We can construct the cumulative distribution function of the sorted observations X _{( r )} in the important special case where the true distribution function F(x) is continuous.

The exact distribution of the r ^th observation is a transformation of the β distribution (this is readily proved by induction):

$${\rm{Prob}}\left\{ {X(r)\,\leq \,x} \right\}\, = \,\frac{{\rGamma (n\, + \,1)}}{{\rGamma (r)\rGamma (n\,{\rm{ - }}\,r\, + \,1)}}\mathop{\int}\limits_0^{F(x)} {{{u}^{r\,{\rm{ - }}\,1}} {{{(1\,{\rm{ - }}\,u)}}^{n\,{\rm{ - }}\,r}} du} $$

This enables us to construct prediction intervals for specific percentiles; for example, to construct a 95% prediction interval for the 75%-ile, we need to find integers 1 ≤ r ≤ s ≤ n such that

$${ &amp;#x0026; {\rm{Prob}}\left\{ {X(r)\,\leq \,{{F}^{{\rm{ - }}1}} (0.75)\,\leq \,X(s)} \right\} \, = \,\frac{{\rGamma (n\, + \,1)}}{{\rGamma (r\, + \,1)\rGamma (n\,{\rm{ - }}\,r\, + \,1)}}\mathop{\int}\limits_0^{0.75} {{{u}^{r\,{\rm{ - }}\,1}} {{{(1\,{\rm{ - }}\,u)}}^{n\,{\rm{ - }}\,r}} du} \cr\ \ \ \qquad\qquad {\rm{ - }}\,\frac{{\rGamma (n\, + \,1)}}{{\rGamma (s\, + \,1)\rGamma (n\,{\rm{ - }}\,s\, + \,1)}}\mathop{\int}\limits_0^{0.75} {{{u}^{s\,{\rm{ - }}\,1}} {{{(1\,{\rm{ - }}\,u)}}^{n\,{\rm{ - }}\,s}} du} \, = \,0.95 \cr}$$

The confidence interval is then [X(r), X(s)]; the size of the interval follows from the claim above.

If F is continuous, we can restate the result as saying F(X _{( r )}) has a β (r, n + 1−r) distribution. As the mean of this distribution is r/(n + 1), the observation X _{( r )} is sometimes taken to be an estimator of the r/(n + 1)-quantile. Our inverse CDF method, however, allows X _{( r )} to estimate any quantile between (r – 1)/n and r/n, a range which of course includes r/(n + 1).

It is worth noting that in most cases, the solution for r and s, as described above, are not unique. A unique solution can be found by applying some constraints such as the interval being symmetrical around the percentile of interest.

5.2.2.3. Direct Percentiles: Approximate Distribution

In practice it is usual to appeal to the central limit theorem and derive a 100. β confidence interval for the α-quantile from the ordered X(i) (where i goes from 1 to n) is given by an interval (X(n.α−Y), X(n.α + Y)) where

$$Y\, = \,\sqrt {n.\alpha (1\,{\rm{ - }}\,\alpha )} {{\rPhi }^{{\rm{ - }}1}} \left( {\frac{{1\, + \,\beta }}{2}} \right)$$

This formula makes no assumptions about the distribution of the X(i). This formula also does not depend on the number of risk drivers modelled in the internal model (the dimensionality of the internal model).

Applying this formula to a 95% confidence interval for a 99.5% quantile, where n = 10,000, an interval of (X(36), X(64)) is obtained. Of course, the actual interval that this gives will depend on the distribution of the X(i).

Repeating the above calculation but using n = 1,000,000 the interval (X(4,861), X(5,139)) is obtained. As mentioned above, the actual interval that this gives will depend on the distribution of the X(i), but the 100× increase in the internal model output has only led to a 10× reduction in the confidence interval (strictly speaking this is a confidence interval for the SCR from the Monte Carlo error sampling process – there may well be many other errors that are also present in the SCR estimate).

These calculations so far have made no assumptions about the true CDF F(x). If the distribution is known to be of a certain form, then parametric approaches may be preferable. It is instructive to consider the special case where F(x) is normally distributed.

5.2.2.4. Alternative Percentile Estimates: Normal Distribution

Suppose that the X _{( i )} are known to be drawn from a normal distribution, whose CDF is

$$F(x)\, = \,\rPhi \left( {\frac{{x\,{\rm{ - }}\,\mu }}{\sigma }} \right)$$

Here, Φ is the standard normal CDF. We might then choose to estimate the α-quantile by substituting estimated parameters into an exact formula, of the form

$${{\hat{Q}}_\alpha }\, = \,\hat{\mu }\, + \,\hat{\sigma }\rPhi {{\,}^{{\rm{ - }}1}} (\alpha )$$

Here, the standard estimates are given as

$$\matrix { \hat{\mu }\, = \,\frac{1}{n}\mathop{\sum}\limits_{r\, = \,1}^n {{{X}_r}} \cr {{{\hat{\sigma }}}^2} \, = \,\frac{1}{{n\,{\rm{ - }}\,1}}\mathop{\sum}\limits_{r\, = \,1}^n {{{{\left( {{{X}_r} \ {\rm{ - }}\,\hat{\mu }} \right)}}^2} } $$

These expressions are of course invariant under permutations of the X_r . In particular, we can write the estimated quantile in terms of the sorted observations:

$${{\hat{Q}}_\alpha }\, = \,\mathop{\sum}\limits_{r\, = \,1}^n {\left( {\frac{1}{n}\, + \,\frac{{{{\rPhi }^{{\rm{ - }}1}} (\alpha )}}{{n\,{\rm{ - }}\,1}}\frac{{{{X}_{(r)}} \ {\rm{ - }}\,\hat{\mu }}}{{\hat{\sigma }}}} \right){{X}_{(r)}}} $$

We can interpret this as a weighted average of the ranked observations, with (for α > 1/2) higher weights given to the largest observation. In this case, the weights depend on the observations themselves; however, we could approximate the weights by their expected values (or, more accurately, approximate Φ(X _{( r )}) by its expected value based on estimated parameters) to obtain a new estimated quantile:

$${{\hat{Q}}_\alpha }\, \approx \,\mathop{\sum}\limits_{r\, = \,1}^n {\left( {\frac{1}{n}\, + \,\frac{{{{\rPhi }^{{\rm{ - }}1}} (\alpha )}}{{n\,{\rm{ - }}\,1}}{{\rPhi }^{{\rm{ - }}1}} \left( {\frac{r}{{n\, + \,1}}} \right)} \right){{X}_{(r)}}} $$

This is now an example of an L-estimate, that is, an estimated quantile that is a linear combination of the sample quantiles, in which the weights add to 1.

5.2.3. Bias and Sampling Error in Smoothed Percentiles

A sample quantile uses only one observation out of a set of n. The other observations are not totally correlated with the selected observation. Taking an average of nearby quantiles would therefore have a smaller variance than a single quantile. However, this would be at the expense of introducing bias, that is, averaging with nearby percentiles may on average (over many random samples of size n) produce underestimates or overestimates of the desired quantile.

The sign of any bias, positive or negative, depends on the shape of the true CDF. Where the shape of the true CDF is known, we can use that knowledge to construct clever weights that reduce variability without introducing bias.

There is another reason to use smooth percentiles for applications involving sensitivities or optimisation. The inverse empirical CDF is a step function, and thus we cannot calculate a meaningful gradient. This renders sensitivities meaningless and creates multiple local maxima and minima to thwart any optimisation routine.

A smoothed version, however, could be differentiable, provided that the weights vary smoothly as a function of the desired probability α. A smoothed version is also naturally easier to optimise. The extent of smoothing should be chosen carefully, to eliminate spurious local optima without smoothing out the true optimum of the objective function.

There are three common examples of L-estimates for quantiles:

• • logistic Epanechnikov smoothing;

• • the Harrel–Davis method;

• • hypergeometric Smoothing.

We provide more details on Logistic Epanechnikov smoothing in Appendix section B.5.