The Chairman (Mr N. C. Dexter, F.I.A.): Welcome to this discussion on the expert judgement paper from the Life Solvency and Capital Management Working Party.
I have just a few opening comments. Kieran Barnes and I are from the Prudential Regulation Authority (PRA). Therefore, can you please note that any comments we make may not necessarily be the views of our employer.
I thought this paper was helpful. Some people might think that the requirements for expert judgement and Solvency II are too bureaucratic but I do think that this is a useful framework. I have had questions in the past about how to properly document the judgement you have made. We are all professional actuaries. We make judgements all of the time. How do we establish why we made the judgement that we did? How do we compare judgements with those that other people may have made?
Also, how do you prevent groupthink? Groupthink could be a good thing; it could be a bad thing. It is useful to think about it. In particular, the interesting thing about judgements is that they depend on the use to which they are put. It is interesting to be able to explain what are the limitations of any judgements that are made. This paper gives a good framework for giving expert judgement.
We are now going to have a presentation by four members of the Working Party.
Michael Ashcroft joined the Working Party in 2009 and now chairs the Working Party. He started his career at Standard Life, where he worked in a wide variety of roles, including marketing, financial reporting and capital risk management. He joined Scottish Widows just over six months ago, where he is currently Head of Investment, Capital and Analysis.
Roger Austin joined the Working Party in 2007. He is currently a partner at Austin Professional Resourcing, a company which he founded in 2006 to deliver interim actuarial resourcing solutions.
Kieran Barnes is a life actuary working at the PRA. His roles include supervising a number of life assurance companies and providing expert advice on internal model reviews, validation techniques and underwriting risks to the organisation. Kieran joined the FSA in 2005. He played a key role in implementing the ICAS regime.
Stephen Makin qualified as a Fellow in 1999 and CERA in 2011. He joined the Working Party in 2013 and is a consulting life actuary and partner of Hymans Robertson.
Four of the co-authors [out of the eight in the Working Group who authored the paper]: Mr M. R. Ashcroft, F.F.A.; Mr R. Austin, F.F.A.; Mr K. Barnes, F.I.A.; and Mr S. J. Makin, F.F.A. introduced the paper in substantially the same way as in the introduction at the Edinburgh meeting of 8 June 2015, with Mr Barnes covering similar ground to that covered by Peter Scolley at the earlier meeting.
The opener at both meetings was Mr P. J. Heffernan, F.I.A., and he made substantially the same comments at both meetings.
The Chairman then opened the discussion to the floor.
Mr M. G. White, F.I.A.: The word “validation” worries me because to clients and users outside the process, it sounds like proving something is correct, as opposed to a failure to show it is wrong. Would you like to comment about how to handle that?
Mr R. Austin, F.F.A.: One of the ideas behind our proposals on the process was trying to make it clear at each step what the decision process was; what the judgement process was. That then enables each of those steps to be scrutinised appropriately through the validation process.
With certain judgements, there is going be much greater uncertainty than with others. Validating those judgements is going to be more challenging. However, I think going through the process, and splitting it out the way that we propose, facilitates the validation.
Mr M. R. Ashcroft, F.F.A.: I think one of the fundamental things here is that, as with many things, there is no right answer. You are trying to create a sensible estimate that meets the proportionate needs of the business, and this ties in with the materiality concept. The question from the validation perspective is: what process would you have put that through to become comfortable?
This process is meant to give a framework to go through, and to demonstrate some rigour.
I guess it is a terminology specific to the industry. It is not that comfort that gives you certainty, which is the terminology from a dictionary point of view. In the actuarial world, it is giving some level of comfort that the challenge has been gone through. I do not think that there is any certainty.
Mr S. J. Makin, F.F.A.: I would disagree that validation gives you false comfort. But if the word “validation” is seen as being synonymous with “rubber-stamping” then something has gone wrong. An important part about validation, as I see it, is to bring out the limitations of the model. It is not about giving people false comfort that something is much better than it is. Poor validation from my perspective starts from what is good about the model. Good validation starts from what the limitations of this model might be.
So while “validation” may be a bad word, good validation, done properly, should not give false comfort.
Mr Ashcroft: The words “assurance” and “validation” are loaded words and I accept that. Stephen [Makin]’s point is critical: it is not about giving comfort that it is right: it is about making sure you have gone through the process to identify issues, limitations and so on. There is no right answer. I accept your point. As a word that the industry has chosen, it probably is not the best one.
Mr G. C. Woodruff, F.I.A.: Please note that my views are not necessarily those of my employer.
Thank you for the presentation and for the paper. In your model example you had three experts giving views. You talked about the interaction between the experts and the decision-makers. The step I was expecting to see there was the experts being invited to review and challenge each other’s views before the expert views reach the decision-makers. That can often lead to some experts potentially changing their views.
As an example, we may have an actuary forming views based on an analysis of data and a medical expert. It may be that the medical expert makes some arguments or has some views which have not been taken into account in the data analysis. Maybe the actuary should go away and adjust his or her views and recommendations as a result.
Mr K. Barnes, F.I.A.: I think elicitation is a very big subject. You have hit on one method used in order to pool resource and expertise around the company. One of the methods that I have seen in academic literature is called a Delphic method, where the people who get together are experts. They have a discussion and take a vote on what they think is the most plausible answer. People discuss why they come to different conclusions. They vote again. They discuss it. That is a good way of building consensus. You may come to an estimate through different ways.
Mr I. Marshall, F.F.A.: I have a question on the quantification of the uncertainty around expert judgement. How confident are we that we can really understand the level of uncertainty with which we are dealing, especially if we are going to use that as a way to communicate results to users?
If we consider the longevity example, it is quite different if we are going to use it for an assumption in capital modelling. The 99.5th percentile is going to be at a certain level. I think that there is much more uncertainty around that. If we quantify that uncertainty by looking at what is the 25th and 75th percentile of an assumption of the 99.5th percentile, we need a distribution of the tail.
It is an interesting concept. I think actuaries could talk about what that would mean. If we take it to committees or to boards and we start bandying around big numbers about the size of this total uncertainty, it would be quite difficult to communicate.
It is maybe more of a statement, just to say that there is more certainty around the uncertainty on the best estimates than there would potentially be around some of the tail evidence.
Mr Ashcroft: You are right. What we are setting out here is a way of trying to articulate that aspect. Part of this is about communication and putting a framework in place that allows you to have that consistency in communication. When we started this work, everybody was focused on capital only, internal model only. But, it is the link between base and stress that is critical. Yes, by definition, that extreme event is an expert judgement. How you communicate that point is very difficult. We are not going to skirt around that. You have to have some way of articulating why you have comfort that your view is sensible. In an overall capital event, it is a combination of all the things together, which makes the message even more complicated.
Mr Barnes: The idea of the total uncertainty budget is all about giving budgets. Do you invest your time trying to refine the equity assumption? Do you spend it on longevity? Do you spend it on expenses? It was a way to communicate the need to focus our expertise in this region. We were not particular specific whether in this region we are talking about a 99.5 fit or a best estimate or any other measure. It was just a useful communication tool.
Mr Ashcroft: On that point about budget, no matter how much you spend in some of these areas, in terms of effort, time or other resource, you will not get that much more benefit because the uncertainty still remains.
Mr Austin: In certain circumstances, the more experts involved, the wider your plausible range will be.
Mr Ashcroft: Some of the extremes, like black swan events, are also important to try to understand. There are many ways to arrive at the position of articulating this point. But part of the benefit is the debate, and setting it out in terms of the logical thought process that has been gone through. Why you have arrived at this position is probably the biggest benefit of this approach.
Mr Marshall: To follow on from that, I think one of the ways in which it could be communicated would be to use the results of alternative assumptions to show potentially what the range is, but not to call it the 25th to 75th percentile range. Let us say: “these alternative assumptions would lead to these results”.
Mr Ashcroft: Stephen [Makin] touched on stress and scenario testing. If it is not this, what else could it be? How badly wrong can you be?
This all flows back to one of the core things: materiality. It comes back to a proportionate approach. Approach it sensibly, document it, identify the uncertainty and then move on. Something like longevity will always be a huge issue. That is why, as an industry, we focus on it.
Mr Barnes: I think you can accept crude methods not just for assumptions, but also for methodology. We talk about plausible ranges and methodology curves. You have to use approximate definitions; you have to use the most available data in order to do that. It is a good communication tool but do not let it rule the process.
Mr A. B. Pepper, F.I.A: The case of Saville vs Central Capital Limited is especially important. If the policyholder proves he took out a toxic policy, he can sue for redress. He does not need to prove what he was told or not told.
This is likely to increase the numbers of successful complaints and therefore lead to a greatly increased redress bill payable by the insurer.
Mr Austin: I think this situation would fit into our framework in terms of an operational risk for an insurance company. The framework that we are proposing is that it will have plausible ranges for the experts’ views. You will have a number of experts involved and that would open up the fact that there is uncertainty. It might be highly material, as you are indicating, but it would be one of those many risks that an insurer is writing or running. It would come within the framework that way.
Mr Pepper: You talk about “risk”. It is not a risk. Saville is already in the armoury of complainants.
Mrs M. E. Emery, F.I.A: The framework that you have explained makes sense to me. I can see how you go through all of those processes to document an expert judgement. If you went through all of that process, for example with longevity, and then you received feedback from, say, the regulator that, in its view, and based on its benchmarking, it should be something different, how would that feedback fit into that framework?
Mr Ashcroft: There would be the input from the regulator as part of the overall process. With the rigour you have gone through with the logging of evidence, we would expect the expert judgement process would mean that you could have those detailed discussions with the regulator. The thing to remember is that benchmarking is a useful tool. It is a validation tool. But it can go badly wrong if your business is not quite aligned with everybody else’s business, and also someone leads the pack in changing a future view. If you are able to demonstrate that through your judgement process, through demonstrating that your experts are effectively the best you can choose from a materiality point of view for your business, we would expect that that would give you a strong grounding to have a robust discussion with the regulator.
It is about trying to understand, with the expert input, the benchmarking between your company and those other companies. You can then feed in why the expert judgements used in the other companies were very different.
Mrs Emery: Are you saying that you could ask them what their evidence was?
Mr Ashcroft: Yes.
Mr Makin: It is a battle of expert judgements. Longevity is inherently expert judgement based, so you are inevitably pitting your expert judgement against that of the regulator. Going through the expert judgement process, and having to hand over all of the information resulting from it, will increase your chances of reaching a successful outcome in your discussions with the regulator.
Mrs Emery: In terms of the framework, let us say you could not resolve the differences. Do you then just document that as part of the rest of the expert judgement?
Mr Ashcroft: I think the debate you have with the regulator hopefully would find the area where the regulator felt the expert judgement was not robust, or where it effectively identified there may be limitations. That may be because your view is that your portfolio is very different. At least you can articulate some detail of where there is a limitation. So it gives you then, as part of your ongoing cycle, an ability to say “If we are to push back on this, we need to gather more evidence and understand it, and monitor this specific part of the judgement”. It is in this debate where the value comes in. It may not always change the view but at least having the evidence and the data allows you to focus on specific areas of concern. You would document this. Then that is probably an area on which you would focus over the next year.
Mr Barnes: I cannot speak on behalf of the PRA, but I can speak as someone who has experience with models and expert judgement. One of the things that I look for is a hypothesis. We think that, because our company is an annuity writer, these are the lines we need to focus on. This is the most appropriate data. This is the most appropriate methodology to go through. If you have a stack of evidence, it allows us as reviewers to take what we call quantitative indicators, not benchmarks. It is just a starting point for our discussion. It allows us to say that this company is a little different or this is new information. We did not consider it when we did our validation. It is not only validation for you; it is also validation for us. We are continually looking at the new information and what we can learn from other companies.
Mr Ashcroft: We are not trying to say that wrong judgements are being made. Actuaries have been making good judgements since the profession began. Part of this is just about trying to make it easier by having the evidence set out so that it can be communicated appropriately, and that some comfort – not necessarily validation – can be gained from that communication and understanding.
Where companies struggle is where they have used assumptions that are not always necessarily monitored. Have they remained relevant? Are they still in scope? Going back to the experts, and keeping that engagement going, is where we think that the process helps. This is all about evidencing good decision-making. We are not trying to say that that is not happening now in terms of decisions being made. But certainly we do not see, as a whole, the robustness of the ability to provide the evidence that would stand up to challenge, that ability to have granularity and control. Certain companies have expert judgement registers. This, in many ways, is a management process for dealing with issues in the business and being able to ensure consistency, and to ensure standards are maintained. This is trying to put everything on a level playing field, effectively, in terms of the decisions you make when modelling assumptions and so on. It is trying to give that overall comfort that the judgements you have made are sensible and controlled.
I think boards are much more educated now than they would have been 5 to 10 years ago. They are much more aware of what they have to do. Whether or not they fully understand it is another matter. Certainly the amount of challenge I have seen from non-executive directors in past years has been much heavier. I think the expectation among board members is much higher. The ones that I have spoken to are very aware of what they are expected to do. I think you need to give some merit to individuals for probing, and certainly we see more pushing back than we have done previously from board members who want to understand more and see the evidence.
Mr Barnes: The framework is designed to work on repeatable actuarial processes like valuations and capital setting or pricing. It is not intended to capture, for example, one-off M&A decisions.
Solvency II has been borne by the lessons the regulator has learnt from failures, near-failures and partial failures.
Ms S. P. S. Parikh, F.I.A.: It is a question rather than a comment. I wanted to try to understand more about whether it is critical that the person setting out the expert judgement is impartial to the outcome or is not going to be influenced by the outcome because otherwise they could find evidence that suits the outcome that they want.
Mr Ashcroft: I think the reality is this. We are talking about experts. You do not suddenly go out and hire all the specialists concerned. Much of the time the experts you use are experts internal to the company. So there is always a risk of bias within any decision-making process. Part of it is recognising that that is a risk. But this again is where some validation tools come in useful. There are other ways of trying to gain comfort that your internal bias does not dominate in the paper.
Part of this is to be self-aware. There is a natural tendency to always want the right number. You cannot help that. I think that it is making sure that that challenge exists – certainly at board level. I think those validation tests are critical things to make sure that you have that suitable challenge. Is that fair?
Mr Barnes: Yes. I would add to that that any process gives you a safe environment, or at least a safer environment. You can say, “Well, I would like to agree with you, but my process says I have to do that”. I think it helps in that regard.
I would say you have to involve those with a stake in the outcome. Otherwise, the expert judgements you make are never going to stick.
The Chairman (closing the discussion): I will try and sum up the points from tonight’s presentation and discussion.
It is important that we understand what an expert judgement is, and an emphasis, I think, is being put on the word “expert” and not just “judgement”. It is important to determine who truly is an expert. But then also to have an expert judgement policy is probably a sensible idea. I certainly take the point that such a policy could be embedded elsewhere, perhaps as part of another policy.
We also have a reinterpretation of the actuarial control cycle.
Solvency II hinges on proportionality and this area is no different. So making sure that all of this effort is proportionate is important. We have just heard again about the need to identify in the process any evidence of bias or at least a conscious use of groupthink.
I like to think of this as “marking to model”. If you are taking this to a board, the board should not accept this expert judgement blindly: it should be able to challenge it. It cannot challenge it if it does not understand how the judgement has been taken. Indeed, the governance around that judgement is important from a validation perspective.
Validation was mentioned several times. This clearly is an important point. We do not want to get into an expert war of a first line having its experts, a second line having its experts too and so on. But I think that it is important that you can have some sort of challenge, which could be from having different experts giving their views. But there are other ways of doing this, which were also mentioned.
With all of this, anything around Solvency II hinges on the ability to communicate. That is communicating between the elicitation manager and the expert, as well as between the expert and the decision-makers, the board. I think being able to keep the process simple enough to cover all the key points, but to communicate this to the users of information, and for them to understand how they can take decisions, is clearly important.
I take the point about regulatory change. I would view that as another set of experts giving input. It is important to understand where the data is coming from if it could lead to the two sets of experts giving different conclusions. I take the point that you do stand, as Michael [Ashcroft] said, a much better chance of having a decent argument or discussion, if you have a well set out rationale for why you came up with that information. Certainly, that enables a much better level of discussion to be had.
The final point about the board is important. The board has to take ownership of all of this and it has to be able to understand not only why the decision was taken, but the range of plausible answers and the limitations.
I should like to take this opportunity to express my thanks to Peter (Heffernan) and to the authors of the paper. I thought it was a really useful document and I think that we have had a really good discussion tonight.
