1.6.2. Scientific approach/critical thinking

We propose to consider expert judgements made as hypotheses, which may be falsified by further information. Related to this are the disciplines and skills of critical thinking that push those asking the questions to provide a clear purpose and implications for the advice and for experts to articulate the bases for their judgements and the underlying data, information, evidence, concepts, ideas and interpretations that underlie their view. Our work is not intended to make expert judgements easy. Indeed, we are proposing a potentially more challenging way of working that requires effort and perseverance. However, we believe that the benefits to the organisation both in terms of justifying the assumptions made and understanding the linkages between the organisation and its environment to be worthwhile. For those interested in learning more about critical analysis, the authors direct you to the following website: http://www.criticalthinking.org/pages/the-national-council-for-excellence-in-critical-thinking/406

1.6.3. Group learning

Within the wider academic discussions on expert judgement, one area of common contention is a view by some that expert judgements can only be made by individuals. In the context of the actuarial work covered in this paper, these judgements clearly need to be owned by organisations. There are particular challenges that we seek to address to avoid any contradiction:

• • Each of the experts needs to be able to explain the basis for their judgement in terms in which their peers can understand.

• • The collection of experts needs to be able to work together in a group in order to come to a combined decision.

• • The judgements made will be re-assessed as part of a regular cycle of actuarial work and therefore need to be recorded in a form that a (potentially different) set of experts can use as a base for their future work.

Underlying our proposals is the concept that knowledge is “constructed” by arranging experience so that it makes sense to individuals. A common judgement can therefore arise through active and constructive social mediation of experiences and ideas. Our processes and templates are designed to facilitate the pooling of experience and ideas through “guided practice” to work steadily and surely towards a decision, sometimes in the absence of consensus.

We also believe that there is value in the annual cycle of dialogue, recording, reflecting and gathering experience within an actuarial team in improving “group knowledge”. For those that are interested in learning more in this area then the authors recommend the study by Nonaka & Takeuchi (Reference Nonaka and Takeuchi1995).

2.2.1. Data and data manipulation

2.2.1.1. Data, both internal and external, forms the basis of the best estimate assumptions. All data are limited, and for some assumptions there may be significant issues in obtaining sufficient relevant data. Nevertheless, where data exists, it is expected by the regulator that this will be taken into account in some way. There are a number of challenges relating to this data that necessitate the use of expert judgement. Some of these relate to the choice of data itself, and some to the way in which the data is used in order to set assumptions.

1. i. Choice of data set (internal): internal data are typically applied, in conjunction with external data, during the setting of demographic assumptions. Experts need to consider:

   1. a The extent to which internal data can be considered reliable and robust – for example, are any errors and limitations that apply to internal data acceptable and is there a sufficient volume of data at an appropriately granular level?

   2. b. The extent to which internal data can be considered relevant – for example, changes to the structure or terms of a product over time may result in historical data for this product having limited relevance. An example of this would be historical lapse experience where surrender charges have been removed.

   3. c. Over what historical period should data be considered? There is a trade-off here between the desirability of using more data and the potential lack of relevance of data from a distant time period. For example, lapse experience is likely to change over time owing to changes in consumer behaviour, media coverage, changes in taxation treatment of alternative investments, etc.

   4. d. Similarly, should data be weighted according to its relative recency? In addition, does the data reflect “one-off” events such as a change of company ownership and should this effect be smoothed/removed?

2. ii. Choice of data set (external): External data are often used to adjust internal data results or, in the case of market assumptions or for demographic assumptions where insufficient internal data exists, used alone to set assumptions. Experts need to consider:

   1. a The extent to which external data can be considered reliable and robust – this is generally straightforward for some data sets (e.g. CMIB, FTSE), but may be less clear cut for other external data such as surveys.

   2. b. The appropriateness of the external data set to the assumptions being modelled. For example, in order to reflect an individual company’s current equity holdings, is a single index appropriate or should a weighted mix of equity data sets be modelled? Which specific index/indices are most appropriate? For some indices (e.g. FTSE 100), the assets forming the index are clear. However, this may be more opaque for other indices (e.g. some property indices).

   3. c. The use of proxy data sets where insufficient external (or internal) data exists. For example, in recent years, alternative assets such as infrastructure bonds have become popular amongst insurance companies, but little historical data exist on their performance.

   4. ad. Similar to internal data, there are considerations about the length of the historical period used, and any weighting applied.

   5. ad. Regardless of the external datasets chosen, there is likely to be residual basis difference as the indices are unlikely to exactly replicate a company’s holdings at all points in time.

   6. ad. The extent to which any outliers should be removed from the data. A small number of outliers could have a significant impact on, for example, a calibration outcome, which may not be considered appropriate. Conversely, removal of outliers may result in a thinner-tailed dataset than is appropriate.

2.2.2. Adjustments to data set analysis

2.2.2.1 Adjustments, involving the use of expert judgement, may be made to the results of the initial data analysis for a number of reasons. For example:

1. i. To allow for estimated future changes in experience (e.g. risk discount rate impact on persistency and medical advances on mortality/morbidity/longevity risks).

2. ii. To reflect differences between the data and the specific nature of the company’s assets and liabilities. For example, adjustments may be made to external mortality tables to allow for the different geographical or social features of a company’s policyholders.

3. iii. To allow for known limitations of the data. For example, survivorship bias is inherent in many indices where failing companies are removed from the indices and replaced. Using indices without adjusting for this may underestimate volatility and price movements. Margins may be added where there is a lack of sufficiently credible data.

4. iv. Estimating the impact of the changing nature of the current existing business as it runs off. Different cohorts of business may exhibit differing propensities to lapse, for example, with-profits policies with valuable guarantees may be “stickier” than otherwise similar unit-linked policies.

5. v. Removal or reduction of the impact from “one-off” events that are reflected in the historical data (e.g. unusual persistency experience following a company merger or demutualisation).

6. vi. Manipulation of the data (e.g. the use of overlapping time periods to increase the size of the data set).

2.2.3. Approximations

2.2.3.1. Approximations are often made during the assumption-setting process. For example, some smaller lines of business may be modelled using the same assumptions as larger lines, without performing a separate experience analysis. The use (and determination of the appropriateness) of an approximation is an expert judgement. In particular, experts must satisfy themselves that the use of such an approximation does not lead to material differences in the results.

2.3.1. General methodology

2.3.1.1. The setting of the methodology for the calibration of stresses and their aggregation will involve considering a number of options. The choice of methodology therefore involves expert judgement, including (but not limited to) the following key considerations:

1. i. Whether to calibrate stresses using 1-year Value at Risk (VaR) or another measure (e.g. run off) that is considered to be more appropriate and no less prudent.

2. ii. Which stresses to model separately. For example:

   1. a credit spread stress may be separated into further sub-stresses such as spread movements, transitions between rating classes, etc.; and

   2. b longevity stress may be separated into trend stress, base stress, etc.

3. iii. The level of granularity at which stresses are determined. For example:

   1. a. equity stress may be separated by currency or broad investment market (e.g. Europe, United States, Asia, emerging markets, etc.) or by industry (e.g. financials/non-financials) or in some other way appropriate to the company’s holdings; and

   2. b lapse stress could vary by product type, country or year of entry.

4. iv. The level at which aggregation is applied. For example, consider a company that has split operational, longevity and credit stresses into a number of sub-stresses. There is a choice to be made about the level at which aggregation is applied. Are these sub-stresses then aggregated to give an overall stress for operational risk (similarly for longevity and credit) before then being aggregated with all other stresses? Or is the aggregation performed in one step allowing for the correlations between all sub-stresses with all other stresses (and sub-stresses) in the model?

2.3.2. Stress calibration

2.3.2.1. Stresses are typically calibrated to a 1-year VaR using a chosen probability distribution function. This gives the likelihood of a particular stress occurring with a particular magnitude (e.g. there is a probability of x% that equity markets will fall by y% over the next year). Key areas of expert judgement for calibrating both economic and non-economic risks are:

1. i. The choice of probability distribution to fit data to. There are often a variety of valid distributions that could be used. A trade-off may be needed between the achievement of a good fit to historical data and practicality of use within the internal model. In reality, a number of different distributions may display adequate goodness of fit, with the ultimate choice requiring further expert judgement.

2. ii. The choice of method used to fit the data to the chosen distribution – for example, maximum likelihood/method of moments.

3. iii. The application of any adjustments to the distribution – for example, to allow for a lack of data in the tails of the distribution, to reduce the impact from significant single outliers in the data points or to allow for other known limitations in the data.

2.3.3. Aggregation

2.3.3.1. There are a number of issues with historical correlation data – for example, correlations changing, even flipping sign, over time and very little useful data giving correlations between demographic risks and other risks. In particular there are a limited number of stress events, and the correlations between the risks considered in an internal model are not always evident during these events (e.g. the data from the credit crunch gives us little information on the correlations between credit spreads and longevity).

2.3.3.2. The choice of aggregation method is also likely to have a significant impact on results, for example, the choice of copula to use.

2.3.3.3. Expert judgement is therefore applied widely during this stage of the modelling. As the assumed correlations are likely to be highly material to the solvency capital requirement (SCR) results, a robust, well-documented process is essential in order to justify the assumptions made.

2.5. Management actions

2.5.1. Assumptions may also be required about the management actions that will be taken in the future. For the best estimate, management actions may have been codified within the realistic balance sheet models and represent an accepted view of what is consistent with “normal practice” for the with-profits fund. Potentially this could be extended to other types of business. Firms may also choose to make assumptions on the actions they would take during and/or following a change in financial conditions to mitigate losses for the purpose of the capital model. Expert judgement may be required to extrapolate historical action (if any) of those used in extreme circumstances.

3.2.1. Expert judgement policy and framework

3.2.1.1. In much the same way that the firm will have policies on its various risks (e.g. market risk, credit risk, operational risk and so on), we believe that it is fundamentally important for the firm to develop an expert judgement policy. This will form the basis on which all expert judgement governance requirements sit and could cover aspects such as:

• • what is meant by expert judgement;

• • when the policy applies and limitations;

• • interaction with any materiality, proportionality and validation policies;

• • requirements of the management board in relation to expert judgement;

• • requirements of executive and operational owners;

• • documentation requirements;

• • reporting requirements, including escalation;

• • requirements on the expert, including defining when an external expert needs to be sought;

• • required review (both internal and external); and

• • required frequency of refresh and review.

3.2.1.2. The expert judgement policy should link clearly to other policies, and the policies should be mutually compatible. Although this is important of all policies, it is particularly important that there is a clear link to, and fit with, the materiality, proportionality and validation policies. For example (and as discussed in section 5), expert judgement does not always lend itself to traditional validation methods.

3.2.1.3. The expert judgement policy should also link clearly to the firm’s risk appetite. In particular, if the firm is able to define its tolerance to expert judgements – both at an aggregate level and at a more granular level (e.g. by main risk types) – then it will be better placed to understand the significance of its expert judgements. This is related to the concept of uncertainty total impact in section 3.2.3.

3.2.1.4. The expert judgement policy, along with the related policies for materiality, proportionality and validation, together with the expert judgement process form the expert judgement framework. We would envisage this framework sitting within the overall risk management framework of the firm.

3.2.2. Plausible range

3.2.2.1. A helpful concept in the context of expert judgement is that of a “plausible range”. The idea is that any expert judgement should lie within a plausible range. By plausible range, we mean that a judgement which lies within the range would be considered plausible by: (1) a single expert using several ways of analysing the problem; and (2) by a range of experts.

3.2.2.2. We would envisage that the range of quantiles defining the plausible range would be set out in the expert judgement policy (e.g. it may be defined as the range between the lower and upper quartile). Each expert would then be asked to provide a range (e.g. the lower and upper quantile) around their central view of the item on which they are providing their advice. So, for example, the expert may be asked to give their best view of the 99.5^th percentile 1-year stress, along with their assessment of the range around that point estimate. These views of the individual experts would then be combined into an overall central view and plausible range of expert judgement by the decision maker. The combined plausible range will provide an indication of the level of uncertainty around the decision, and be used to estimate the sensitivity of the model output metrics. A large plausible range around a particular item with significant impact may also indicate an area of the model which requires additional consideration and prioritisation around whether that plausible range may be reduced – we explore this further in section (REF).

3.2.2.3. Some experts who come from a non-statistical background may initially find the concept of expressing a lower and upper quantile around their central view as intellectually challenging. Cooke & Gossans (Reference Cooke and Gossans1999) suggest that this could be dealt with by providing some probability training to the relevant experts. Where this is not practical, we would suggest asking the experts to initially provide a minimum and maximum around their estimate, and then the person with responsibility for obtaining the expert judgement could attempt to establish during elicitation a plausible range consistent with the quantile range of interest.

3.2.2.4. We recognise that for certain expert judgements, especially those relating to methodology, this plausible range will be difficult to assess and also that the impact of the plausible range on the output metrics may also be challenging to compute in anything other than a broad brush manner. However, we believe that even in such circumstances (including where qualitative rather than quantitative assessments have been made), it does add value to think about expert judgements in this manner as it provides additional insights compared with the alternative of not having any such assessment. We will explore the practicalities of using plausible ranges later in the paper.

3.2.3. Uncertainty total impact

3.2.3.1. A model is likely to contain a number of expert judgements, each of which will have a plausible range, and the choices made within these plausible ranges will impact on the model output metrics of interest. “Impact” could be defined in a number of ways, for example:

1. i. The difference in the output metric at the upper end of the plausible range and the output metric at the lower end of the plausible range.

2. ii. The maximum of the difference in the output metric between the upper end and central view, and lower end and central view.

3.2.3.2. Irrespective of the definition used, aggregating the impacts on the output metrics across all expert judgements will give a sense of the variability of the model output metrics as a result of uncertainty around expert judgements. Although in theory there may be some degree of overlap between some of the judgements and thus their impacts, a simple summing of this total impact is likely to be helpful to consider. We describe this as the expert judgement uncertainty total impact, and it will be by reference to the particular output metric of interest.

The key purpose of this expert judgement uncertainty total impact measure is to give a sense of the variability of the model output metrics as a result of uncertainty around expert judgements. It is not suggesting that all of the expert judgements will be wrong to the maximum extent. This will require careful communication with senior management, regulators, auditors, etc. to ensure they understand the purpose of the assessment. The calculation of this impact helps to demonstrate that the firm has got a good understanding of the level of variability of the model outputs resulting from expert judgement, which also helps senior management to prioritise those areas of expert judgement which would most benefit from additional effort to reduce the variability around those judgements (where this is possible). We explore this further in appendix sections 1.5.7 and 1.5.8.

3.2.3.3. Although on the face of it this notion may jar with the Solvency II requirement for a single number for solvency capital at the 99.5% VaR level over 1 year, we believe that the concept brings an element of realism. This reflects the fact that while the aspiration is that the capital numbers are calibrated at this level of confidence, those outputs depend on many inherent expert judgements, which by definition are uncertain, and thus the capital calculated will have some degree of uncertainty around it.

3.2.4. Uncertainty reduction budget

Decision makers are likely to want to reduce the expert judgement uncertainty total impact over time. To be able to do so, they will need to allocate a budget. We refer to this budget as the uncertainty reduction budget. Such a budget should generally be spent in a manner which targets those expert judgements the plausible ranges for which have the largest impact on the model output metrics coupled with the greatest potential for being reduced. Other factors may also be relevant including the strategic aims of the firm. It should be noted that it may not be possible to reduce the plausible range around certain expert judgements, irrespective of the level of budget available, due to the inherent uncertainty in their nature.

3.2.5. Region/area of the expert judgement

3.2.5.1. To facilitate the efficient allocation of the uncertainty reduction budget across the various aspects of expert judgement, we would suggest that the uncertainty reduction budget should first be allocated to a “region” of expert judgement, for example, “expenses”, “longevity”, “equity”, etc. The uncertainty reduction budget within each “region” should then be allocated to more detailed areas such as “base table” and “improvement factors” within longevity risk, and then to specific elements such as “parameter A” or “modelling approach B”.

3.2.5.2. Again for efficiency, the expert judgements within a region may be bundled together for consideration when going through the proposed process below. This will be important if there are inter-relationships between the expert judgements (e.g. agreeing the allocation of expenses between two funds and calibrating an expense capital model) or if similar techniques are used (e.g. analysis of equity and property risk for a capital model).

3.2.5.3. The efficiency of the process can also be improved by considering common themes or principles which can be used to leverage effort in areas which require a large number of expert judgements. For example, an insurer with a broad range of risks will need a very large number of correlation assumptions, each of which will have its own expert judgement. “Industrialising” this process may be an effective use of the uncertainty reduction budget. “Industrialising” may include having a standard template and standard analysis for the draft brief for the experts (discussed below) and having “implied expert judgements” such as a common ratio between the calibration of market falls and increases in volatility across many different asset classes in a capital model.

3.2.5.4. Also note that the techniques for assessing the materiality of each “region” (e.g. profit and loss attribution) will be different to those at the more detailed level (e.g. sensitivity testing of parameters/methods).

3.5.1. Define terminology

3.5.1.1. It is critical to define terminology clearly. This is especially the case when seeking external expertise, but it is also important when the expertise is sourced internally. This will ensure that everyone involved has a consistent understanding of the terminology, and will also ensure that this consistency is maintained over time when the expert judgement is reviewed in subsequent periods. Clarifying “jargon” may also help to remove barriers to including relevant expertise from different academic and professional fields.

3.5.1.2. Some errors in expert judgement have occurred because the identification of the risk, the data sourced and the modelling approach have been based on subtly different definitions. For example, does “lapses” mean the absolute level of lapses or differences from expected? Is this the “expected” the current valuation assumption, the valuation assumption made in the year of the observation or some other figure? If an increase in lapses is expected following a market fall (sometimes embedded in the asset-liability model), should the data be transformed to include only the lapse experience net of the market-dependent factor? Defining terminology in a clear and unambiguous manner will help reduce the risk of such issues.

3.5.2. Articulate what the expert judgement relates to and why it is required

3.5.2.1. It is useful to start with the basics so that it is clear what the expert judgement relates to and why it is required. A good starting point is to set out which of the following categories the expert judgement falls into:

1. i. methodology;

2. ii. assumptions (including parameters); and

3. iii. approximations.

If the judgement relates to methodology, any practical constraints around that judgement should be considered by those responsible for obtaining the expert judgement. For example, are there any constraints resulting from the form of the input data, or the form of the output data required by other parts of the model? As we will see in subsequent sections, a key part of the elicitation technique will be exploring the experts’ thought processes that should not be constrained by the existing modelling. Although these practical constraints need to be borne in mind, care needs to be taken to avoid unduly restricting the range of possible answers too early.

3.5.2.2. It is also important to establish the context and purpose, and in particular, what model output metrics are of ultimate interest. For example, assessing the impact on profit (e.g. Market Consistent Embedded Value profit), pricing metrics (e.g. new business contribution to embedded value), capital requirements (e.g. internal economic capital) or even the balance sheet as a whole (e.g. the Solvency II balance sheet).

3.5.2.3. As well as identifying the output metrics of interest, it is also useful to be clear on the ultimate purpose of the model output. For example, is it being used for calculating best estimate technical provisions, assessing capital requirements, pricing, assessing a reinsurance programme, etc.?

3.5.2.4. Once the expert judgement category and key model output metrics have been established, it is helpful for the firm to set out a high-level understanding of its exposure. Examples might include, “Financial losses are incurred when:

• • Fewer people die than expected…

• • Interest rates rise…

• • Expenses are higher than expected”.

This can guide the areas in which more certainty is preferable, which can in turn inform the nature and extent of questions to ask the experts. It can also highlight those areas of expert judgement that may be multi-faceted and where judgements may need to be broken down for each facet. Examples of some complications that could occur include:

• • The exposure may be different in different parts of the business. An increase in lapses on non-profit business typically leads to a loss of future profits, whereas an increase in lapses on with-profits reduces the value of options and guarantees.

• • A further level of granularity may be required. For example, when considering interest rate exposures, it may be appropriate to consider different changes in interest rates at different durations. The level of granularity required may also differ between different users of the model, for example, the pricing actuary may want additional factors relevant for pricing that may not be required by the capital actuary.

• • There may be interaction effects. For example, exposure to interest rates may “flip” as the life expectation of annuitants changes.

3.5.2.5. After the nature of the expert judgement has been articulated, it is important to set out the reason why expert judgement is required. The general context may be due to poor data in terms of quality, volume or relevance; or it could be because a risk is being modelled for the first time; or it could be part of a review of risks which are material to the firm or which are highly subjective in their nature (in both cases the expert judgement could provide useful validation or challenge). The specific event which has triggered the expert judgement to be considered could be one of many such as:

• • a scheduled review of an existing judgement;

• • a non-scheduled review of an existing judgement owing to monitored triggers being exceeded, validation queries or audit queries; and

• • more certainty being required.

3.5.3. Establish what was done previously

3.5.3.1. Although in certain circumstances there may not have been an expert judgement made previously for the area in question (e.g. a risk which was not previously modelled), a previous judgement will often exist. Generally, it is important to capture information on this previous expert judgement as it is likely to provide useful input into the current expert judgement. This will include not just the decision made, but also the approach to forming the decision including the key drivers underlying the decision and the information sources used. If the plausible range of the expert judgement was assessed previously, this should also be captured, as well as any assessment of the impact of the plausible range on the output metrics.

3.5.3.2. There is an argument that a scientific approach to analysis should be used under which the analyst should draw their conclusions from the evidence alone without any pre-conceptions. We would challenge this approach. An expert will generally have prior beliefs when undertaking any investigation. Indeed, it could be argued that a benefit of being a member of the actuarial (or any) profession is that its members are informed by the development of knowledge and skills over many years of past practice, and these are likely to influence the individual’s prior beliefs. Surfacing these prior beliefs and making them explicit will highlight those beliefs and facilitate challenge to the implicit assumptions made. The ability to communicate with others (including the introduction of new ideas) is greater if the individual develops a commonality in language and an understanding of what was done previously.

3.5.3.3. Nevertheless, some decision makers may feel that this step unduly restricts the range of possible outcomes that should be considered in forming an expert judgement, particularly if the area is novel or previous precedents are unhelpful or misleading. If this is the case then this element of defining the problem may be skipped in the first iteration of the process.

3.5.3.4. Aspects of prior beliefs will be recorded in the documentation of the expert judgement used in previous years. However, decision makers should be mindful that documents are summaries of historical thinking and may miss the context and the competing factors that were relevant in coming to a decision. In particular, previous decisions may have been informed by tacit knowledge which cannot always be recorded.

3.5.4. Identify potential drivers for change to previous expert judgement

3.5.4.1. Once the previous expert judgement has been captured, it is sensible to consider what drivers exist which could make it appropriate to change the previous judgement. There are a wide range of potential drivers, examples of which include:

• • an additional year of data;

• • updates to the information in previous data sources, for example the correction of errors in historical data or changes in methodology applied retrospectively;

• • identification of potential new information sources (e.g. new research papers);

• • improvement in actuarial and statistical techniques (e.g. techniques which were previously prohibitively expensive now become sufficiently cost-effective to be viable for consideration);

• • changes to the key drivers underlying the previous expert judgement (e.g. legislative changes);

• • identification of new drivers which may potentially be useful in forming the decision;

• • changes to related expert judgements (e.g. longevity trend and mortality trend);

• • the desire for greater precision regarding the expert judgement; and

• • changes in the level of risk or the potential level of risk, for example new business written or through an upcoming transaction.

• • Identification of new experts to contribute to the expert judgement.

The value in systematically considering this element of the process may result in uncovering drivers which would otherwise not have been identified.

3.5.4.2. The information sourced should be used to correct or widen the factual basis on which previous decisions were made, to challenge the prior beliefs or to highlight new ways of thinking about the same problem. This approach is based on previous expert judgements being based on verifiable statements and the information source chosen to be most relevant to these statements. For example, the prior beliefs may include that the insurer has sold annuities to wealthier than an average population, the wealthier already have access to the best medical care and so the scope for further improvements from the expected level is lower than for the general population. What evidence can be used to test the demographic profile of the insured portfolio? Are the wealthier more likely to use private healthcare and is the benefit of private healthcare material? Are future improvements likely to be a result of disseminating the best in the current healthcare or in the pipeline of expensive new medicine unlikely to be available on the NHS? The firm may need to consult their experience analysis, industry and population statistics, along with experts from other fields such as medical underwriters, gerontologists, cardiologists, oncologists, medical scientists, etc.

3.5.5. Prepare initial estimate of plausible range

3.5.5.1. As mentioned in section 3.2.2, we believe that the concept of a plausible range for expert judgements is helpful. At this problem definition stage of the process, we would suggest that the initial estimate should be quick and approximate, rather than being slow and detailed.

3.5.5.2. There are two distinct aspects to the concept of a plausible range: the plausible range of the expert judgement (effectively a measure of the uncertainty around the judgement), and the impact of that plausible range on the model output metrics of interest. We will first consider the plausible range around the expert judgement, and then consider its impact on the model output metrics. For certain expert judgements, especially those relating to methodology, the plausible range is likely to be expressed in qualitative terms rather than in quantitative terms. For example, the list of plausible families includes the normal distribution, the Student’s T distributions, etc.

3.5.5.3. If this is not the first time that the expert judgement is being made, then a good starting point would be to consider the previous expert judgement (as highlighted in section 3.5.3), and in particular the plausible range around that judgement. Assuming that such a plausible range was previously assessed, the next step would be to consider whether there are any factors or new information during the period as the judgement was made which may have changed the plausible range. These factors may increase the uncertainty or decrease it, with a corresponding increase or decrease in the plausible range.

3.5.5.4. If there was no plausible range stated for the previous expert judgement, the documentation around the previous expert judgement should be analysed to try to get a sense for the uncertainty around the judgement. This should help to form a view on an initial plausible range, albeit approximate. We would recommend that this estimate should err on the side of caution, that is an estimated plausible range which is likely to be larger than the true plausible range.

3.5.5.5. The above process should result in a revised initial plausible range along with a brief rationale behind any changes from the previous plausible range.

3.5.5.6. If this is the first time that an expert judgement is being made on the item of interest, then an estimate needs to be made of both the expert judgement and the plausible range around that expert judgement. Within the confines of the requirement for the initial estimate to be quick and approximate, clearly the initial estimate of the plausible range in these circumstances is likely to be large given the uncertainty. In forming the view of the initial plausible range, account should be taken of any relevant information sources identified.

3.5.5.7. Once the initial estimate of the plausible range for the expert judgement has been set out, the impact of this plausible range on the model output metrics of interest should be estimated. So let us say that for a particular expert judgement the initial central estimate is B with an initial estimate of a plausible range from A to C, then we want an estimate of what the output metrics will be at A, B and C. This will give a sense of how material the expert judgement is in relation to the output metrics. Again, speed is the key feature at this stage of the process, which may therefore involve approximations. We recognise that this will often not be straightforward, and that quite broad approximations may be required. Indeed, the impact on the model output metrics may even need to be done in the form of a qualitative assessment (e.g. high–medium–low) rather than a quantitative assessment. However, we feel that it is an important step to understand how significant the expert judgement is in relation to the model outputs. If this has been performed as part of the previous expert judgement, this is likely to be helpful in relation to the estimation for the current expert judgement.

3.5.5.8. Where there are multiple related expert judgements (e.g. the base mortality table and the mortality improvement factors for pricing an annuity portfolio), the initial estimate of the impact may focus on the main factor with the relative importance of the two related judgements being considered further through the process.

3.5.6. Assess potential for reducing plausible range

3.5.6.1. After the initial estimate of the plausible range has been completed, it is useful to consider whether there are any ways of reducing the plausible range, especially for those expert judgements with the most material impact on the model output metrics. For example:

• • Is the engagement of additional experts likely to reduce the plausible range?

• • Would further analysis of the existing data lead to a reduction?

• • Would analysis of new information sources have an impact?

• • Would an alternative methodology help (e.g. a new model)?

3.5.6.2. For each of the potential methods, there should be a qualitative assessment of the likelihood of reducing the plausible range, along with an approximate assessment of costs and timescales (which again may need to be formed qualitatively).

3.5.6.3. Practicalities should also be borne in mind regarding any methodology changes. For example, the absence of an in-house expert on longevity may steer the firm to choose a statistical aggregate mortality method rather than use more subjective judgement to build a cause of death model. It may also be proportional for a firm with a sizeable exposure to use more than one model to inform their choice of parameters.

3.5.6.4. As we explore in subsequent sections of this paper, we are proposing an approach under which successive iterations of the process over a number of years may replace certain aspects of expert judgement with data. For some assumptions, the marginal benefit of using expert judgement to reduce uncertainty diminishes as the process becomes more data orientated. In effect, the knowledge required to determine the assumption becomes embedded in the process used to derive the assumption rather than remaining in the tacit skills of the expert. An example is the process now used to determine implied volatilities in the realistic balance sheet. Although uncertainty remains, it is considered by many firms that it is not practical/feasible to use expert judgement to determine a more precise assumption. For other expert judgements, such an approach may not be appropriate and the firm may wish to build the expertise into the firm over time. For those who are interested in learning more in this area, the authors recommend considering techniques such as “deep smarts” as described in the study by Leonard & Swap (Reference Leonard and Swap2005).

3.5.7. Assess appetite for reducing plausible range

3.5.7.1. The key decision makers should be identified as the success or failure of the expert judgement process and is likely to rest on the ability to communicate clearly with individuals within this group. The appropriate individuals may vary according to the area covered as well as the governance framework. For example, certain individuals may be more comfortable with market risks than demographic risks and the governance group may need to be extended when discussing operational risk.

3.5.7.2. Having assessed the potential for reducing the plausible range, there should be engagement with the identified decision makers to assess their appetite for reducing the plausible range. If the firm has already established the overall expert judgement uncertainty total impact, this should help inform the view as to whether this particular expert judgement merits an investment from the expert judgement uncertainty reduction budget to reduce the plausible range. The cost of capital may also feature in the decision. If so, the decision makers should confirm the practicalities such as:

• • Amount of uncertainty reduction budget available for this particular expert judgement.

• • The timescales over which the revised judgement needs to be made.

• • The availability of the decision makers to provide input into the process before the final judgement being made.

• • What aspects around the judgement are they most concerned with?

• • In a wider context, are they happy for the judgement in this area to simply be “within the pack” of their peers (where relevant), or are they keen to minimise the plausible range?

3.5.7.3. The qualitative impact of the judgement should also be discussed with the decision makers. For example, if the risk management strategy is dependent on the ability to hedge out equity risk, the significance of basis risk would be worth exploring. If the strategy of the firm is based on better underwriting, the granular judgements underlying the underwriting process may be scrutinised more carefully than the financial impact of a change in the valuation assumption within the plausible range alone would suggest.

3.5.8. Prepare overview of need for expert judgement

Based on the information gathered in the previous steps in the process, an overview of the need for expert judgement should be prepared including the timescales over which the work will be required. This overview document will be used in subsequent steps.

3.5.9. Identify personnel involved and their roles

3.5.9.1. To be able to form an expert judgement, suitable experts will need to be identified. This step needs to be mindful of the practical constraints set out by the decision makers such as budget and timescales.

3.5.9.2. It is useful to bear in mind that a more insightful result may be achieved from using a wide range of experts rather than too many in a narrow area, that is breadth and depth of expertise are both important. There is also the problem of self-referencing: a model that appears to be industry standard may actually only be accepted within one school of academia, so it is sensible to ensure there is a sufficient breadth of expertise.

3.5.9.3. If the judgement is not entirely new, then the documentation around the previous expert judgement should be helpful in identifying at least some potential experts. Experts will come from two sources:

1. i. internal experts, that is those individuals who work for the firm; and

2. ii. external experts.

3.5.9.4. A list of internal individuals who may have the necessary expertise to be usefully involved in forming the expert judgement should be drawn up, including the nature of their relevant experience, and an initial view on their relative level of expertise. It is sensible to try to ensure that the breadth of the expertise covers the key business areas that will be impacted by the decision.

3.5.9.5. If it is felt that the internal expertise is insufficient in relation to the importance of the judgement or may not achieve the targeted reduction in the plausible range, a list of external experts should also be identified. Initially the external experts may be by company or by academic institution rather than by individual. We would encourage the firm to try to retain sufficient internal expertise so that it can effectively critique the work of others.

3.5.9.6. The overview of the need for expert judgement prepared in the previous step should then be sent to the identified experts to check their availability, interest and indicative cost where appropriate.

3.5.9.7. Any other roles relating to the expert judgement should also be documented, for example coordinator, decision makers, etc. For certain judgements, it may be that the experts are also the decision makers. Even in this context it is useful to give thought to whether there may be additional experts available internally or externally who could materially enhance the quality of the expert judgement or provide insight and context on the business implications of any judgement. It is also important to consider how to ensure that there is an appropriate level of independence and that expert judgements are subjected to an adequate level of challenge.

3.5.10. Set out the draft brief for the experts

3.5.10.1. A brief for the experts should be drafted based on the information gathered in the previously completed steps. The aim of this is to ensure that all of the experts are clear on exactly what is being asked of them, and to give them an opportunity to ask for aspects to be clarified or modified before the brief being finalised. Having a common format and analysis accompanying the draft brief for the experts will aid the communication of ideas and potentially make the decision making more efficient.

3.5.10.2. The document should build on the overview document prepared in an earlier step. If external expertise is being used, it may be considered appropriate for certain aspects of the information to remain confidential in which case such confidential information should be detailed within an additional document. Care should be taken before deciding to withhold information from experts, as this has the potential to make the expert judgement less robust.

3.5.10.3. Even in the case where the experts are also the decision makers, we believe that the development of a brief is worthwhile as it captures all of the key information around an expert judgement, which is likely to enhance the quality of the expert judgement and is a key stage in the documentation requirements.

3.5.10.4. The information to be included within the draft brief will be dependent on the nature and materiality of the expert judgement as well as the relationship between the experts and the decision makers, but may include aspects such as:

1. i. Definitions of terminology to ensure consistency of interpretation by the experts of the various terms used in the brief.

2. ii. Articulation of what the expert judgement relates to and why it is required, including aspects such as:

   • − whether the judgement relates to a methodology, an assumption or an approximation;

   • − the model output metrics of interest including purpose and any qualitative aspects;

   • − a high-level overview of the firm’s exposure and the strategic importance of the judgement;

   • − the general context of why the judgement is required, for example poor data quality, volume or relevance, etc; and

   • − the specific event which has triggered the expert judgement to be considered, for example scheduled review, non-scheduled review, greater precision required, a previously unmodelled risk, etc.

3. iii. Documentation of what was done previously including aspects such as:

   • − the approach to forming the decision including the information sources used, experts’ views and rationale, key drivers underlying the decision, etc.

   • − the actual decision made, when it was made, the scheduled review date and triggers for non-scheduled review;

   • − a clear logical thought process behind the decision;

   • − any assessment made of the plausible range and its impact on the output metrics; and

   • − any related judgements.

4. iv. Articulation and analysis of any potential drivers for change to previous expert judgement such as:

   • − updates to existing information sources and any new information sources identified;

   • − changes to key drivers underlying previous judgement and identification of potential new drivers; and

   • − a desire for greater precision.

5. v. Initial estimate of plausible range.

6. vi. Potential information sources (if the judgement relates to something new).

7. vii. Practicalities:

   • − an indication of the appetite for reducing the plausible range;

   • − any areas of concern for decision makers;

   • − timescales over which the experts’ views need to be provided;

   • − summary of the proposed elicitation approach including format in which experts’ views will be required and the likelihood of iterations between the experts and the decision makers.

3.5.10.5. If the experts are actuaries, they will need to abide by the relevant TASs, which include a requirement for the actuary to make clear in any report their understanding of the high-level purpose of the advice as well as key assumptions made and inherent limitations. A well-written brief (including clearly set out expectations of the format in which the experts’ views must be given) should facilitate this.

3.5.10.6. Information sources can be wide ranging such as internal data, academic papers, surveys, publications, data from reinsurers, data from industry bodies, data from the actuarial profession, data from consultancies, etc. Each of them has the potential to provide a particular insight or perspective and it is important that they are captured in the brief so that they can be considered during the elicitation and decision-making processes.

3.5.10.7. We believe that it is important to clearly set out the format in which the experts’ written views are expected. This should have a logical structure containing information such as the information sources used (including any additional information sources beyond those set out in the brief); the relative importance attached to each information source and rationale; the thought processes leading to their recommendation; their assessed plausible range including rationale; how long the recommendation is valid for; the triggers for non-scheduled review; and any restrictions on how the judgement should be applied if the scope of the judgement is any different to that set out in the brief.

3.5.11. Clarify and finalise brief

3.5.11.1. Once the draft brief has been issued to the experts, the experts should provide feedback on any areas of the brief that they feel are unclear, or where they believe the proposed approach to forming an expert judgement could be improved. For instance, the experts may not easily be able to answer the question asked but if some changes are made to the nature of the question or the frame of reference, the experts may be able to provide much more useful input. An example of this would be where the experts may be able to describe a relationship between risks in a situation where the correlation matrix approach used by the firm is expressed in terms of correlation between capital requirements which are specific to the firm. Another example would be the situation under which the modelling framework is for unconditional stresses, whereas the expert’s frame of reference is based on what is happening in the current environment. A further example would be where the particular assumption will vary through the business cycle. It is likely that compromises will need to be made: we believe that it is preferable for this to be done explicitly rather than implicitly.

3.5.11.2. The brief should then be updated to try to ensure clarity and to take into account any workable suggestions from the experts. There may need to be more than one iteration before the brief can be finalised.

3.8. Ongoing Monitoring

3.8.1. After a decision has been made, it is important to make sure that there is a robust system in place to monitor the decision. The monitoring should concentrate on the context underlying the judgement and the key triggers that the context has changed (that had been identified during the decision-making process for non-scheduled review). Again, the expert judgement register could help with this process. The monitoring framework should also pick up when expert judgements are approaching their scheduled review date to ensure that they are reviewed in a timely manner to gain comfort that they are still appropriate.

4.1.1. Preliminary assessment of judgement

4.1.1.1. As mortality improvements are a key risk for insurers writing annuity business, it is clear that the judgements around mortality improvement assumptions fall into the expert judgement category.

4.1.2. Defining the problem

4.1.2.1. Solvency II requires the technical provisions to be calculated using the best estimate of liability for its annuity portfolio. In addition, other capital and financial measures for ABC Life also require a best estimate of the annuity liability. To calculate the liability they will require assumptions for the mortality of their portfolio in future years, in addition to other assumptions. The problem is: what will the death rates be in the portfolio in future years?

4.1.3. Define terminology

5.1.3.1. Two key aspects are defined. First, there is a base mortality table which defines the rates of mortality for a particular year. There may be rating factors applied to the base mortality table for specific policies or groups of policies. The second is the annual rates of improvement applied to the base mortality table in order to determine the mortality rate in each year of the projection. The annual improvement rate is defined as the percentage reduction in mortality rate from one year to the next for a given age. These annual rates of improvement will be represented in a table showing the improvement rate, calendar year and age. Again, there may also be different tables used depending on the rating factors of the policy or group of policies.

4.1.3.2. For this example, in order to just consider one situation, we use annual improvement rates for males as the expert judgement that we are considering.

4.1.4. Articulate what the expert judgement relates to and why it is required

4.1.4.1. Area of judgement. The annual rates of improvement relate to experience assumptions.

4.1.4.2. Metrics of interest. As ABC Life is a bulk annuity writer it considers all of the following metrics important:

• • IFRS profit;

• • MCEV profit (including new business contribution);

• • statutory balance sheet;

• • Solvency II balance sheet and capital requirements;

• • ICA; and

• • internal economic capital forecasts.

4.1.4.3. High-level understanding of the firm’s exposure. The high-level description of ABC Life’s exposure is as follows: financial losses are incurred when fewer lives insured die than expected.

4.1.4.4. Areas where judgements may need to be broken down. The pricing team may require more granular assumptions than the financial reporting teams.

4.1.4.5. Trigger for expert judgement. as ABC Life is going to launch a new product, this is the trigger for the expert judgement.

4.1.5. Establish what has been done previously and drivers for change to previous judgement

5.1.5.1. This is the first time that ABC Life has made a judgement in these areas and therefore there are no previous judgements to review.

4.1.6. Prepare initial estimate of plausible range

4.1.6.1. One expert will need to be involved at the preparation of the initial estimate of the plausible range. A model is required to project future mortality improvements. There are a number of options for mortality projection models (Continuous Mortality Investigation (CMI), 2013):

• • the “92” Series and Interim Cohort Projections;

• • Adjusted Interim Cohort Projection;

• • ONS National Population Projections;

• • P-spline projections;

• • Lee-Carter projections; and

• • The CMI Mortality Projections Model.

4.1.6.2. For our initial assessment we will use the CMI model which we understand to be the most common model in use. For this example, we use the CMI_2014 model. Again, other valid options exist; here we simply exemplify the process.

4.1.6.3. The CMI model allows the user to input a different initial rate of improvement to the model other than the standard parameters. ABC Life does not have access to further information regarding initial rates of improvement. Therefore the expert recommends that the initial rates in the core model be used.

4.1.6.4. There is no default rate of improvement set in the CMI model and the user is left to establish one. The expert’s initial estimate is based on benchmarking using the data in Table 1 from publicly available PRA returns and past improvement rates in Figure 2.

Figure 2 Males – observed crude annual mortality improvement rates in England and Wales population (CMI, 2014)

Table 1 Continuous Mortality Investigation Model Long-Term Improvement Rates for Selected Insurers

4.1.6.5. Based on this information, the initial plausible range for males is selected as a central estimate of 2% improvement rate with a 1.5% and 2.5% selected as the 25% and 75% percentiles, respectively.

4.1.6.6. The core CMI model converges to the long-term rate over a period of time using default parameters. The model has the flexibility for users to alter these parameters. The expert recommends that at this stage there are no modifications to the default rates of convergence.

4.1.6.7. The CMI model has the ability to add or subtract a constant rate of improvement. Owing to a lack of additional information available to the company the expert recommends that this is set to 0.

4.1.7. Initial estimate of plausible range

4.1.7.1. For simplicity, we estimate the financial impact of the plausible range by considering a portfolio of 10,000 males aged exactly 65 being paid an annuity of £10,000 annually in advance. We use the best estimate of liabilities as the metric of interest. The valuation is assumed to use a 3% interest rate and 100% of PMCA starting 01 July 2000 and the CMI 2014 model. This gives the results shown in Table 2.

Table 2 Practical Example – Initial Estimate of Plausible Range

4.1.8. Assess the potential for reducing the plausible range

4.1.8.1. As the expert’s initial assessment of the plausible range has been completed, she now considers the potential for reducing the plausible range.

4.1.8.2. The initial estimate of the plausible range has been determined through the use of the CMI model and benchmarking to other insurers assumptions. There are further approaches that could be used and these may reduce the plausible range. These are listed below along with estimates of timescales and costs:

1. iv. Performing further analysis on past population and industry mortality improvements to extrapolate the long-term rates of improvement (10 days, using an internal actuary).

2. v. The use of different data sources including socio-economic specific data (20 days, using an internal actuary).

3. vi. The use of a cause of death model (40 days using an internal actuary).

4. vii. The use of two additional external experts (10 days/expert, £2,000/day cost).

5. viii. Using a different mortality projection model (10 days/model, using an internal actuary).

4.1.9. Assess the appetite for reducing the plausible range

4.1.9.1. As mortality improvements are a key assumption in the business model, the Board of ABC Life want to ensure that they reduce the plausible range as much as practically possible. However, the Board appreciate that they are constrained by a desire to be quick to market with their bulk annuity offering and the limited time that the actuarial team has available. The Board decide on balance that they are willing to devote £50,000 of spend for external experts and 60 days of an internal actuary’s time.

4.1.10. Prepare overview of need for expert judgement

4.1.10.1. The expert judgement on the mortality improvement rates is required as a key assumption in many of the models used in ABC Life. The Board is comfortable with a timescale of 3 months to conduct further analysis.

4.1.11. Identify personnel involved and their roles

4.1.11.1. Management has identified the two external experts that they wish to use; one medical expert and an external actuary. These are in addition to the internal actuary used. Their roles are defined as follows:

1. i. The internal actuary is to explore methods of improving the accuracy of the central estimate and reducing the plausible range.

2. ii. The external actuary is to provide an independent review of the internal actuary’s work, identify additional data sources and provide an additional view of the central estimate and plausible range of the long-term improvement rate.

3. iii. The medical expert is to provide an expert opinion on likely future medical advances and their expected impact on annuitant mortality.

4.1.12. Set out the brief for the experts

4.1.12.1. The brief for the experts is shown in Table 3.

Table 3 Brief for the Experts

4.1.13. Clarify and finalise the brief

4.1.13.1. There would naturally be a step for the experts to finalise the brief and for any clarification required.

4.2.1. Elicitation of expertise

4.2.1.1. The three experts after considering the evidence return with the following estimates of long-term improvement rates for males: 1.75%, 2% and 2.5%. These are accompanied with written justification from the experts for these rates.

4.3.1. Decision making

4.3.1.1. Although the formal decision will be made by the Board, this will be based on the recommendation of the Chief Actuary, who is the primary decision maker.

4.3.1.2. The experts may meet the Chief Actuary to discuss their respective recommendations and this may result in revised recommendations.

4.3.1.3. The Chief Actuary may take a number of approaches in reaching the final decision, for example:

1. i. take an average of the experts’ views;

2. ii. take a weighted average of the experts’ views (with the AFH determining the weights);

3. iii. accepting a single expert’s justification based on the arguments made; or

4. iv. reject the views of the external experts and form an in-house view (although this may be more difficult to justify).

4.3.1.4. Whichever of these approaches is chosen, justification of the final judgement will need to be documented and approved by the Board.

4.4.1. Ongoing monitoring

4.4.1.1. Management would put in place a regular review process for this expert judgement. This could be 1 year from the date of the judgement.

4.4.1.2. There would also be trigger events that would cause the non-scheduled review of the expert judgement. These could include:

• • discovery of a material error in the data used to form the judgement;

• • significant additional data on mortality improvement rates becoming available;

• • a significant change in industry approach to modelling mortality improvement rates;

• • formal guidance from the regulator indicating that the expert judgement is not within a range they consider to be acceptable; and/or

• • a level of new business that is materially above plan and/or new business that is significantly different in terms of the lives insured.

5.3.1. Testing the robustness of the internal model

Robustness includes identifying which expert judgements are most material and how credible an alternative view point could be. We expect that the validation will build on the assessment of materiality of the impact of the plausible range described in the process above. The validator may wish to check that the sensitivity testing is fit for the purpose of validation. They may also wish to examine the evidence to support making a different choice and form an opinion.

5.3.2. Testing results against experience

5.3.2.1. A scientific approach often relies on setting a hypothesis and testing the hypothesis against experience. We propose that expert judgement is considered in the same way. Testing the hypotheses underlying the expert judgement against experience focusses on assumptions made about the external environment, whereas profit and loss attribution tests assumptions on the impact that this experience has on the assets and liabilities within or internal to the insurer. Testing results against experience may be less helpful (without other assumptions) for judgements on methodology or where the focus is on “tail” events.

5.3.2.2. A key judgement by the validator is the granularity of this analysis. This may be at the broad level of the output from a sub-component of a model such as “expenses”, “lapses” or “equity return”. Alternatively, experience analysis may focus on the underlying symptoms or causes (e.g. poor service standard driving lapse experience) and how they are related to the assumptions on which the expert judgements are built.

5.3.3. Profit and loss attribution

5.3.3.1. Profit and loss attribution tests assumptions on the impact that this experience has on the assets and liabilities within or internal to the insurer, whereas testing against experience focusses on assumptions made about the external environment.

5.3.3.2. Profit and loss attribution can inform the conversation between the validator and the firm on what factors are modelled, which are not and why. For example, a model design where, speaking loosely, the correlation between two factors is assumed to be constant in extreme and less extreme scenarios may be a result of an expert’s views underlying an expert judgement, an accepted model limitation based either on a materiality assessment or which has been allowed for elsewhere in the model (e.g. through strengthening some correlation parameters) or it may be something that the firm has placed on the development plan for future iterations of the model. The validator may form a judgement on the specification of the model based on this understanding.

5.3.4. Stress and scenario testing

5.3.4.1. Stress and scenario testing can be considered to be a test of the assumptions made about the external macro environment and the assumptions within the asset-liability model describing how the insurer’s financial position would react in the face of the stress/scenario test. This combines testing of experience and profit and loss attribution using a hypothetical situation rather than one which is necessarily grounded in the past. Stress and scenario testing events do not necessarily need to have a probability attached to them although this may help communication. Stress and scenario testing may also be mild – well within the experts’ frame of reference – or extreme but still plausible. The validators may also review reverse stress testing results (i.e. identifying the scenarios which “break” the business).

5.3.4.2. Development of the scenario from the macro-environmental source of the shock can provide an alternative perspective on the assumptions on the root causes underlying the expert judgement. Comparing a macro-environmental scenario with the expert judgement can reveal inconsistencies in the expert judgements made on two or more different areas of the model. For example, an increase in expenses may be assumed to be partially driven by increases in inflation which in turn is accompanied by increases in long-term gilt rates. The risk of equity falls may be heightened if there is an extreme increase in long-term gilt rates but expenses may have been assumed to be unrelated to equity markets. The source of the inconsistency (which may be removed when a positive-semidefinite adjustment is made) may be in the way the expert(s) think about the problem or in different definitions of the problem. That is not to say that inconsistency in the model should be avoided at all costs, particularly if the expert has not got the frame of reference to provide advice on a conditional/unconditional basis, but it may help understanding and highlight a limitation.

5.3.4.3. Projection of the insurer’s assets and liabilities in the face of the stress/scenario test can provide a test of the assumptions made in comparable areas of expert judgement. For example, one expert’s view/mathematical model may assume that the financial position of the insurer remains static if investment returns are equal to the risk-free rate. A more sophisticated model that includes the impact of expected maturing and new business may show that the financial position will improve over time. The second model may be adjusted to be more like the first by assuming that maturities and new business form a stable population and the results compared.

5.3.5. Benchmarking

5.3.5.1. Benchmarking can be thought of as the introduction of alternative view points into the expert judgement process. Benchmarking may come from high-level statistics such as the median from an industry survey or in the models used by practitioners (such as estimating an implied property volatility parameter from historical experience or from current implied equity volatility observations) and intermediate assumptions (such as NHS funding in longevity projections) used to build expert judgement. The validation team have a role as bringing in ideas from external sources of expert judgements such as academic literature and industry/professional contacts as well as an understanding of the context in which benchmark expert judgements are made.

5.3.6. Simplified models

5.3.6.1. As discussed above, one of the benefits of having an independent validator is the ability to approach the same problem from a fresh perspective. By their nature, expert judgements are made by balancing a number of conflicting ideas and information. Fresh models do not need to be detailed or have the same explanatory power. Simplified models of potential thought processes can add insight to the validation activity.

5.3.7. Manual tracking of some internal model calculations

5.3.7.1. Judgement is often informed by a number of approximate calculations or rules of thumb. A sample of these calculations can be checked for reasonableness.

5.3.8. Peer review

5.3.8.1. Effective validation will also rely on the ability to critique the work of the firm’s experts. The need for critique of the underlying sources of information was highlighted in our proposed expert judgement process in section 3. The validators’ role is to critique the sources of information, logical steps and conclusions. There is a rich amount of literature in approaches and skills of critical appraisal. Brown & Keeley (Reference Brown and Keeley2007) highlight that key steps in critical analysis include:

1. i. Identifying the issue and the conclusion.

2. ii. Identifying the reasons underlying the expert judgements.

3. iii. Where does ambiguity exist?

4. iv. What are the value conflicts and assumptions?

5. v. What are the descriptive assumptions?

6. vi. Are there any fallacies in the reasons?

7. vii. How good is the evidence?

8. viii. Are their rival causes?

9. ix. Are statistics deceptive?

10. ix What significant information is omitted?

11. xi. What other reasonable conclusions are possible?

These steps, and those used by other authors, are explained further in the literature on critical analysis.

5.3.8.2. The validators may wish to consider if expert judgement is appropriate for the methodology or assumptions, or whether a more quantitative approach might be more suitable (or vice versa).

5.3.8.3. Many actuarial assumptions are impacted by social systems. For example, longevity may be related to diet and medical advancement. Equity risk may be related to fear and greed in the market. As well as understanding if experience invalidates the principles underlying the expert judgements, where the expert judgement is based on a frame of reference of current market or macro-environmental conditions, the validator may wish to consider if that understanding of the macro-environmental conditions is still valid.

6.1.1. Expert judgement process guidelines

Firms should define guidelines for the process of development and use of expert judgement. This could cover aspects such as:

1. i. Identifying when and why expert judgement is required.

2. ii. Required practice for eliciting expert judgements in various scenarios, for example, direct consultation with single experts, the use of multiple experts, the use of certain elicitation techniques (e.g. the Nominal Group technique, the Delphi technique, etc.).

3. iii. Requirements in relation to the all-important contextual information – including sufficient background and intended use – to be supplied to the expert to allow him or her to provide judgement effectively.

4. iv. The required approach to internal review, challenge and sign-off.

5. v. The required approach to feeding back the use of the judgement.

6. vi. The required process for reporting and escalation.

A documented and validated policy will help to bring consistency to the use of expert judgement, both across different areas within the firm and over time.

6.1.2. Expert judgement register

6.1.2.1. We believe that an expert judgement register would be very useful tool in the management of expert judgement within a firm. This would effectively be a log of all expert judgements relevant to the particular context, for example, an insurer’s Solvency II internal model.

6.1.2.2. The contents of the register will vary, but may contain aspects such as:

1. i. a unique reference number;

2. ii. a category into which the judgement falls – to be used for grouping and reporting purposes;

3. iii. a description of the judgement;

4. iv. the key drivers underlying the judgement;

5. v. a measure of the materiality of the judgement;

6. vi. a measure of the firm’s tolerance to the particular expert judgement;

7. vii. links to the location of all of the relevant documentation;

8. viii. the names of the experts and their qualifications;

9. ix. the date at which the judgement applies;

10. ix the scheduled review date;

11. xi. triggers and trigger levels, which if breached, require the judgement to be reviewed;

12. xii. the way in which the judgement is used (e.g. accepted, altered, rejected) and reasons;

13. xiii. dates of meetings to feedback to the expert on how his/her judgement was used, and links to meeting notes and papers;

14. xiv. the senior-level executive “sponsor” with ultimate responsibility; and

15. xv. the day-to-day owner.

6.1.2.3. The benefits of a comprehensive register are clear, and include the following:

1. i. it brings together all areas of expert judgement in one place;

2. ii. it allows expert judgements to be summarised, shared and reported;

3. iii. it facilitates consistency across expert judgements in related areas; and

4. iv. for those firms with an articulated expert judgement tolerance, it allows the position against the stated tolerance levels to be assessed and reported on, enabling areas where remedial action is required to be identified, prioritised and agreed.

6.5.2. Keep it fresh

6.5.2.1. Rotate responsibilities within the firm, both at executive and operational owner level. This will help to ensure the firm does not become too entrenched in its position and “blind” to alternatives. Use different experts – or even seek the opinion of multiple experts – where this is possible.

6.5.3. Keep it focussed

6.5.3.1 Keep an eye on the expert judgement plausible ranges. With limited budgets to spend, a measure of materiality should be used to focus efforts on the areas where greatest clarity is needed and to investigate what can be done to elicit that clarity. Expert judgements outside of tolerance would be a key focus.

6.5.4. Be humble and recognise your limitations

6.5.4.1 By its nature, the ‘correct’ answer is unknown – possibly even unknowable – and to think otherwise is potentially very dangerous indeed. While it may seem obvious, it is important to keep this front of mind to ensure that complacency – in the form of passive acceptance of established expert judgements – doesn’t set in.