The Chair (Mr M. A. Pomery, F.I.A., Past-President): The main purpose of this evening is the presentation of a paper “Reserving and Uncertainty: a Meta-Study of the General Insurance Reserving Issues Taskforce and Reserving Oversight Committee Research in this area between 2004 and 2009.”

I should like to invite Elisabeth Gibson to make the opening presentation.

Dr E. R. Gibson, F.I.A. (introducing the paper): Over a hundred actuaries from a dozen countries have contributed to the source material of which this paper is a meta-study, and many of those actuaries are in the room today, not only sitting at the high table but also in the audience. I would like to give a word of thanks to everybody who has contributed to this work.

Most of the authors of the paper have had two roles. Firstly, they have been involved in supervising the work of the working parties, taskforces and committees which have carried out work in this area in the last five years, and also they have been responsible for pulling together this paper into what is, hopefully, a coherent story of that work. I am particularly grateful to them for that, especially to Dr Bruce, who did the lion's share of the work of pulling together the paper. Also, thanks to the Profession's scrutineers for their help in making this a better paper than it would otherwise have been.

I do not plan to present all of the paper. We were very careful to make it manageably short, so I hope you will be able to read it yourselves. What I will do is dip into a few parts of it, give you a few highlights, and tee-up Mr Fisher to open the debate.

May I start, first of all, with what I call a ‘range of ranges’. One of the findings of GRIT (the General Insurance Reserving Issues Taskforce) was that, as a profession in the general insurance area, we use the word ‘range’ to refer to different things, and we are not consistent in the way in which we use the word ‘range’. In particular, we found that there are two quite distinct senses in which we are using the word ‘range’, sometimes interchangeably, and sometimes apparently unaware of the conflict between the two.

One of those is the sense of a range of reasonable estimates or a range of best estimates. This is particularly prevalent in the US, but also quite common in the UK – certainly up until mid-2000.

The other sense of the word ‘range’ is the range of possible outcomes: what the eventual total claims come to compared with the reserve estimates that you are now making. These two ‘ranges’ relate to different sorts of uncertainty. The range of reasonable estimates, or range of best estimates, is about uncertainty in relation to the parameters in your model. If you think about it from a purely statistical point of view, a range of reasonable estimates might be those best estimates that are equally valid as a mean given the data you have, and given the model and parameter choices you are making. You cannot really say one is better than the other; there might be a relatively tight cluster of estimates that have equal validity.

Part of the problem with this ‘range of reasonable estimates’ is that it is not actually an unambiguous term even when you do differentiate it correctly from the range of outcomes. It is still ambiguous because we also sometimes use a ‘range of reasonable estimates’ to mean something different. We use it to mean a range of numbers that are appropriate to be carried in your accounts for a particular purpose. Of course, that has a different meaning again.

The range of reserves that is appropriate, or suitable, for accounting purposes is subjective and depends on both the purpose in question and the accounting regime. It is not the same as a range of numbers that are equally valid.

So, one of the original observations from GRIT was: “We really need to take care when using the word ‘range’. We, ourselves, must understand what we mean, so that our readers can understand what we mean.”

If we are talking about a range of outcomes, and somebody thinks we are meaning a range of numbers that are equally valid to carry on the books, then we really have a problem.

We also got to thinking about these different ranges and asked, “What should we be communicating to our clients about ranges? What type of range should we normally be communicating?”

One of the most controversial recommendations of the GRIT paper was that actuaries should, in fact, give a quantitative indication of the uncertainty in outcome (the second type of range). We should actually put a number on that uncertainty. In fact, following the revision of GN12 in 2006, that became something which actuaries should normally do. Since then, as a profession, we have developed ways of doing that.

It was quite controversial at the time. We were talking about it over the last couple of weeks when preparing for tonight. We think, perhaps, it is less controversial now than it was then. But maybe that is something that Mr Fisher will touch on when he introduces the debate.

That is a little bit of context about why GRIT then recommended further work. We have all these things buzzing around: ranges, uncertainty, and best estimates. So we set off some workstreams to try to explore these areas further and raise the game of the Profession in this area.

There were three main areas on which we focused, which the paper goes through. The first is best estimates themselves; what do they mean and what can go wrong in formulating best estimates? What methods do we normally use?

Secondly, is uncertainty. How does one quantify the uncertainty? Having made this controversial rod for our backs that we should do so, how are we going to do that, and what methods should we use?

Then, thirdly, also in relation to reserving uncertainty, is the problem of the range of ranges: how do we effectively communicate this uncertainty so that our customers, who are often not actuaries, understand what we are talking about and get some value out of what we are saying to them?

I am going to touch briefly on those three areas. There is a lot more in the paper.

In relation to best estimates, we thought about a number of elements. First was how actuaries respond to the influence of others, particularly the influence of underwriters, in the way they set their estimates. Papers have been written on this before and, as well as looking at those, we thought about it in the context of the reserving cycle and the underwriting cycle.

We also thought about how we use the normal estimation methods (the chain ladder and the Bornhuetter-Ferguson methods), including the diagnostics that we do and the backtesting that we do, or do not do, when we use those methods.

Then, finally, looking at the reserving cycle more generally: what is causing it and how do we allow for it? The working party was set up to focus on those areas.

I will touch on some of the key findings on those areas now. There is more in the paper and a lot more besides in the original underlying papers. On reserving cycles, we looked at the earlier paper by Archer-Lock et al. from 2003. That paper found that the tail length – the development pattern length – could vary with the underwriting cycle. In soft markets business could develop more slowly, and so, if you are using a pattern that does not allow for that, you might underestimate the reserves. It also found that rate indices tend to understate the amplitude of price changes, both favourable and unfavourable, so that again the Bornhuetter-Ferguson method would tend to lag true rate decreases and increases.

The Reserving Oversight Committee (ROC) paper that followed on reserving cycles reported similar findings. It also set out some practical structural steps to mitigate these problems.

We found that we do not really believe that terms and conditions are a major contributor to the reserving cycle, but rather that changes in terms and conditions caused step changes in reserving development patterns, but not really cycles as such. It is possible to allow for these; some people do, but some do not, which is a problem. However we felt this had less of an impact than some people might think, and the reserving cycle was caused more by development pattern changes of rating indices than terms and conditions themselves.

Steven Fisher led a working party on effectiveness of reserving methods which had vast numbers of actuaries engaged in trying out these reserving methods on some made up data and some real data. What we found in this rather artificial environment was that actuaries produced quite a wide range of different reserving answers – perhaps a surprisingly wide range. There did seem to be some systematic conservatism going on, although it is possible that this was really a result of the artificial environment and that people were slightly unnerved by not having access to real-life data, and that was causing them to include margins in their estimates.

What that study did establish was the, perhaps, unsurprising result that premium is a better indicator for ultimate claims in early stages of development, and claims reported is a better indicator in later stages of development. That, at least, reassures us that our habitual use of the Bornhuetter-Ferguson method in early stages of development and then switching to the chain ladder is well-founded.

That is a summary of best estimates.

There was a little bit of controversy surrounding the quantification of reserving uncertainty. Partly the controversy was over the interpretation of the “should normally” requirement. Then, when it came to look at the methods themselves, we encountered some unexpected challenges. Papers were presented at GIRO in both 2007 and 2008 entitled, ‘Best Estimates and Reserving Uncertainty’, in which we explored the methods that are normally used to quantify uncertainty – the more stochastic methods such as bootstrap, Mack, and others. We found a number of difficulties in using these methods in practice. There is rarely enough data to calibrate the method to produce credible answers of high percentiles. Since these methods are often used for capital and solvency purposes, it is often high percentiles in which you are most interested, where you have, arguably, the least data with which to calibrate.

We also felt that these methods do not normally apply to real-life situations, because the underlying assumptions of the methods tend not to match reality.

Thirdly, we found, rather by accident, a strange problem with the methods. We were trying to test what would happen if you started with methods the underlying assumptions of which were perfectly satisfied by artificial data and then looked at how much worse they became if you perturbed the way in which the model fitted the data by making it deviate from the perfect fitting in one way or another. Our control experiment was to say: “Let us imagine a universe where these models are perfectly satisfied by the data and let us see how well they work there, and then we will switch off some of those compliances and see how much worse it becomes.”

In fact, we found that they did not seem to work well when the models’ assumptions were perfectly satisfied. That threw us a little.

Further work has been done by Peter England, and others, at GIRO this year on that subject. Maybe we will cover that in the debate. There are still questions to be answered in that regard. I think it is an area for further thought, although I would agree with the points Peter England made at GIRO about there being insufficient data to properly calibrate the models, and also the way in which he used the method in terms of segmentation.

That is quantification. There are still some challenges. Perhaps a fundamental problem on quantification is a very practical one. That is, if you use a chain ladder or a Bornhuetter-Ferguson method, or some combination thereof, to come up with your best estimate, and if you then try to quantify your uncertainty using one of the stochastic methods the two are invariably not consistent. You end up with a mean from the stochastic method that is not the same as the best estimate that you have selected for the deterministic method.

There seems to be, inbuilt, a fundamental inconsistency in the way we think, and I will come back to that point at the end.

Now to the topic of communicating reserving uncertainty. There was a paper in 2007 by Barlow et al. that recommended a clear vocabulary in communication of uncertainty so that the users of our work could both understand the causes of the uncertainty, and also the scale of the uncertainty. The paper focussed on those two areas.

In relation to causes, we felt we should be disclosing the key drivers of the uncertainty and giving some descriptive language, maybe illustrative examples, supporting any quantitative information that is given.

That paper sets out a framework involving a combination of percentiles and language which will help us achieve a more consistent approach when talking about uncertainty of outcome.

Finally, there is an interesting backdrop at the moment in relation to these subjects, both internationally and in the UK.

The IAA has a Risk Margin Working Group which has produced some material on characteristics of acceptable methods for calculating risk margins.

Solvency II introduces two particular issues: the challenge around the way that we calculate the mean with the probability weighted distribution and also the issues surrounding risk margins.

One of the interesting challenges of Solvency II is that it arguably calls for a convergence between these two usually inconsistent things: the best estimate and the uncertainty. It is perhaps not good enough that we just bodge the two together and reverse-engineer one to the other. Maybe Solvency II will eventually require us to do better than that, and have some bringing together of those methodologies.

There are similar issues with the International Financial Reporting Standards (IFRS) where, although the risk margin is probably going to be a little bit different, the same principles apply. It is worth noting that, in the US, an actuary must state if he or she believes that there is a significant risk of material adverse deviation. They also need to say what they mean by “material”, giving an actual quantum. They must also compare their range of reasonable estimates with the carried reserves. Those are recent requirements, moving in a similar direction.

In Australia they have taken a slightly different approach. They have not really separated the best estimate and risk margin in the same way. Instead, they have adopted the 75th percentile approach to reserving. This is not just about statistical techniques but also judgement.

In Switzerland, with the Swiss Solvency Test, that is a risk-based approach and a 1-in-200 test. So there is an interesting move internationally to require actuaries to start grappling with these issues a little more.

For us in the UK, we had a debate at GIRO on stochastic versus deterministic. There is a similar debate now going on in the world of climate change about how climatologists are going to communicate the uncertainty in their forecasts for future climate. Some think that they should be giving a probability weighted universe for future climates; others think that that is too hard to understand and what is needed is to give specific stories and bring it to life.

I was listening to Professor Nigel Arnell recently, who is the Director of the Walker Institute at the University of Reading and one of the UK's leading climate scientists, and his view is that they need both. Climatologists need to bring together both their deterministic communication methods and their stochastic communication methods if they are going to communicate the full texture of a scientific understanding to non-scientists in a way that their users can understand. That is probably true of us as well.

So, let me conclude by saying that the final word of our paper really is a call to arms. Like the world of physics, where they are searching for their unifying theory by bringing together general relativity and quantum mechanics, we actuaries have to find a way to bring together the chain ladder and the bootstrap. We have to find a way to find the ‘theory of everything’ that enables us not to have to bodge these two methods together, but hold our heads up and have a reserving method which does both best estimates and margins. That would surely be the Profession's magic bullet for Solvency II and IFRS.

Mr S. Fisher, F.I.A. (opening the discussion): I would like to begin by thanking Dr Gibson for her very interesting and engaging presentation this afternoon.

Today's paper highlights a large volume of practical research that has been conducted over the last 5 years by GRIT and ROC, together with various working parties sponsored by these two bodies. This research has helped to move forward the debate in respect of reserving and uncertainty, yet there remains much work to be done. I would like to take a few moments now to highlight a selection of topics that may benefit from further discussion today.

First, one of the notable changes of recent years is the increased focus by general insurance actuaries on the reserving cycle. As today's paper points out, this is an issue long acknowledged within the Profession, but it was the work of the Cycle Survival Kit Working Party in 2003 which drew attention to the likelihood that the reserving cycle was being driven by actuarial projections, and not only by the impact of management adjustments to actuarial estimates within booked reserves. It is certainly hoped that, given the higher profile of this issue within the Profession, as well as efforts to improve the quality of rate indices, the impact of this issue will be somewhat reduced as we emerge from the latest soft market. I am interested to hear views from the audience as to whether such hopes are well-founded.

Second, as illustrated by the successful debate that took place at this year's GIRO conference, the question as to the relative merits of deterministic and stochastic methods remains at the forefront of our professional discourse. That debate appeared to point towards a consensus that the traditional deterministic methods still bring great value to the reserving process, but that further research into stochastic techniques – and especially the practical application of such techniques – would be most welcome. I would like to invite contributors to this discussion to consider how – or indeed whether – it will prove possible to address some of the practical challenges associated with implementing stochastic approaches to reserving, while continuing to address other, potentially conflicting, objectives, such as the need to communicate our assumptions clearly and the requirement (soon to be reinforced under Solvency II) for backtesting of assumptions.

Third, the debate between proponents of deterministic and stochastic methods itself forms a part of a wider dialogue regarding the effectiveness of reserving methods more generally.

If I might be permitted to dwell for a moment on the conclusions of the Effectiveness of Reserving Methods Working Party (and perhaps at this point I should declare a personal interest as the chair of that group), then I should highlight that one of the key conclusions of the working party's most recent investigations, as reported to GIRO this year, is that there is no single ‘perfect’ or ‘ideal’ method. Indeed, if it were to be possible to identify a ‘route-map’, or recipe, for sound reserving, it would consist of some probing diagnostic analysis that uncovers the underlying features of the data, taken together with a clear understanding of the underlying drivers of the business, so that sound judgements can be made about the right method and assumptions for the circumstances under investigation.

Fourth, one of the great controversies of recent years has been GRIT's recommendation that the communication and quantification of uncertainty should be mandated within professional guidance. This recommendation was, of course, adopted, and has now been in place for a number of years. It would be interesting to hear from contributors to today's discussion whether this requirement remains as controversial now as it was when first introduced.

Fifth, while there has undoubtedly been a substantial move within the Profession to quantify the uncertainty that surrounds best estimates, there also remains doubt surrounding the current toolkit available to actuaries for this purpose. The Best Estimates and Reserving Uncertainty Working Party in 2007 and 2008 highlighted that the Mack and Over-Dispersed Poisson methodologies appeared to understate the actual variability at the most extreme percentiles, although I note that elements of these conclusions have been challenged in this year's GIRO paper by Peter England and Martin Cairns. Nevertheless, the question as to whether we have the right toolkit to quantify uncertainty is certainly one for further debate.

In conclusion, while both GRIT and ROC have contributed greatly to the Profession's understanding of reserving issues, there remains a great deal still to debate. I look forward to what should be a most interesting discussion.

Mr P. J. Copeman, F.I.A.: The authors of this paper have provided a very useful summary of the significant amount of work that has been undertaken in recent years under the auspices of the General Insurance Reserving Oversight Committee following on from GRIT. I would suggest that all General Insurance (GI) actuaries working in reserving should read the paper as, in my view, it contains much useful material for their work.

I am pleased that the authors have seen fit to include, in Section 1.2, a useful summary of the difference between the two main types of ranges (reasonable estimates or reasonable best estimates, and possible outcomes). This simple but fundamental distinction is, in my experience, still not always understood by some of our customers and, as actuaries, it is important that we remind ourselves of the need to explain it clearly, as in this paper.

Paragraph 1.3.6 refers to the controversial recommendation of GRIT that actuaries should seek to quantify the uncertainty in outcomes. I recall that those of us on the GRIT committee who had misgivings about this recommendation were eventually swayed, amongst other reasons, by the argument that this recommendation (which was subsequently enshrined in GN12) would force the Profession to ‘get its act together’ in this important area. It does seem to me, however, that despite the valuable work conducted by ROC and others, we have not made progress as quickly as had been hoped. I also note in passing that GN12 will be removed soon, and that TAS R, whilst intending to have similar effect on this issue, is arguably not as explicit in the area of quantifying uncertainty. I stand to be corrected on that point!

Paragraph 1.4.2 reminds us that under Solvency II, extreme outcomes are not to be excluded from best estimates (being a mean of the whole distribution). This is perhaps only of academic interest for some short-tailed classes of business, but could be significant for longer-tailed classes such as employer's liability.

Subsection 2.4 discusses the work on “Effectiveness of Reserving Methods”. This was always going to be one of the most challenging workstreams on which to make progress, requiring a great deal of effort in testing methods, and in assessing, validating and coordinating the results. It is therefore not too surprising that the thoughts in this area contained in this paper are rather inconclusive. I do note, however, that the latest presentation by the working group at GIRO contained some interesting observations which should be taken further. It is of no surprise to me that the first conclusion presented was that no one method (from the wide range examined) works best in all situations.

Section 3 contains a good discussion of the difficulties and pitfalls when attempting to quantify uncertainty. There are no clear conclusions from the section, but I would suggest that caution is required when using the analytical methods currently in vogue, and that whilst undoubtedly helpful they should be overlaid with a large dose of common sense.

The paper quite appropriately includes, in Section 5, mention of the changing insurance industry background to the issue of reserving, in particular Solvency II and IFRS. I do have some concern that the understandable desire on the part of regulators and others for consistency of standards in reporting results and assessing capital needs will not be readily achieved, bearing in mind all the difficulties referred to previously.

My final point relates to the very last paragraph of the paper (paragraph 6.3). The sentiment in this section is understandable, in particular the desire for a unification of approach for assessing uncertainty and best estimates. There is a danger, however, that the phrase “a unifying method” could be interpreted as implying one method for all types of business in all situations. That is, in my view, an unrealistic goal bearing in mind the vast variety of different reserving situations, and considering the ‘no one perfect method’ conclusion mentioned earlier. In addition, experienced General Insurance actuaries should not be afraid of exercising judgement in looking at the results of different methods, understanding the features in the data or the business which may cause the results to differ, and then using their experience to assess both the best estimate and the level of uncertainty. That is what we are paid for. It may not satisfy the desire of some for a neat mathematical solution but it represents reality in the difficult and uncertain world in which we work. The challenges of working in this difficult and uncertain area are why we, as General Insurance actuaries, enjoy our work, I believe.

Dr L. M. Pryor, F.I.A.: I welcome this paper, and its emphasis on uncertainty is particularly timely.

However, I was disappointed to see no mention of the BAS and its standards in subsection 4.2. In September the BAS published its Standard on Reporting Actuarial Information (TAS R), which had been preceded by an exposure draft earlier in the year. TAS R places a great deal of stress on the importance of communicating uncertainty and on the need for actuarial information to be understandable by those who are expected to base decisions on it. For example, TAS R contains principles that both the nature and extent of any material uncertainty is communicated to the user of actuarial information. It also covers the need to be clear about the meaning of probabilities and to provide clarification and correct misunderstandings as the need to do so arises. The principles in TAS R thus go quite a long way beyond the requirements in GN12 which will, in any case, be withdrawn within the next couple of years. Although TAS R will apply to only a small proportion of GI reserving work from April 2010, we encourage wider adoption.

The point that Peter Copeman made about TAS R possibly not going as far as GN12 is not correct because TAS R places emphasis on the nature of the uncertainty, which I think covers the point about the difference between a range of reasonable estimates and a range of possible outcomes. Another reason, though, why we use the word “extent” rather than requiring quantification is that TAS R, like all our standards, is a principles-based standard, and TAS R, especially as a generic standard, is meant to be very widely applicable. It is not clear to me that requiring quantification in all circumstances is realistic, even for reserving in general insurance, and it is certainly not realistic for all uncertainty across a broad range of actuarial work.

That brings me to my second point. I very much welcome the proposal in paragraph 4.6 for the development of a common vocabulary for the clear communication of uncertainty. However, I note that nearly all the issues identified in section 4, and indeed in the paper as a whole, are applicable to, or have close parallels in, other actuarial fields. Uncertainty is an issue that arises over and over again: in pensions, in life insurance and in general insurance. The basic concepts involved are essentially the same.

It would surely be a mistake to develop a vocabulary for use only in the GI field when such a vocabulary is needed across the whole actuarial community. I would encourage GI actuaries to work with other groups within the Profession to ensure that there is consistency across the various fields, and that there is no unnecessary duplication of work. It is time to start breaking down the silos of individual practice areas rather than reinforcing them.

In summary, this is a very useful paper that provides much food for thought, and contains a clear identification of some very important issues.

Mr J. B. Orr, F.F.A.: I think this is a very important paper; GRIT is an asset of the Profession whose value is growing. We should be looking to make the most of the asset that we have been given by Tony Jones and the others.

The specific point I want to make is in relation to cash-flows. Underlying all reserving work is consideration of how cash-flows are emerging, how they point to the emergence of the ultimate outcome, and also what we might draw from claims notifications.

A transaction level approach is highly relevant to a better understanding of those cash-flows. Of course, a wholesale move to such approaches would not be economic, as the people we advise would resist that strongly, and probably rightly so. However, I think that there is an argument for looking at particular cases, to do a ‘drill–down’ and look at the cash-flow basis for reserving and then use this to clarify or confirm that the portfolio level methods – and remember the chain ladder method and other approaches are working at the portfolio level – are valid and give us the result that we are seeking, whether it is in relation to our best estimate expected value or otherwise.

Next, an observation about risk and change.

I was reflecting on a presentation by Stephen Richards about breaking down barriers between actuaries and other professionals looking at mortality. I began as a pensions actuary in the late 1980s and worked for about six years in pensions consulting. We did not give much serious consideration to mortality trends then. However, mortality improvements, which we all, it is to be hoped, will benefit from, have been a major driver of the change in the risks for life offices and pension funds.

What Stephen Richards identified as the key change that made mortality improvements so much more important, was the drop in interest rates. In the time that I was a practising pensions actuary, we experienced interest rates in double figures. Now we are looking at interest rates of 2% to 4%, and closer to zero at shorter durations. This change has exposed some of our key assumptions.

The lesson I draw from this is that, if we continue to operate with non-explicit margins in our work, say as implied by an undiscounted approach to reserving, then we run the risk of not really understanding how risk might present itself in the future.

The way in which foreign exchange risk has come around, with the substantial fall in Sterling against the Dollar at the end of 2008, has been a wake-up call. The FSA, as the regulator, is challenging firms more frequently on understanding and mitigating their currency exposures.

The last point I want to make is that as an economy we face two possible worlds, and I do not know which one we are going to end up with: a world in which quantitative easing might lead to a burst in inflation or a world in which we follow a relatively flat ‘L-shaped’ future for economic recovery, in which we have a fairly stagnant economy and limited inflationary pressure.

If we do experience a burst of inflation, I think a lot of the methods we use at the moment are going to be caught out. We are exposed if we do not go back to a more explicit approach, to thinking about what is driving the claims that we are estimating.

Mr J. P. Ryan, F.I.A.: This valuable and interesting paper has given us a lot to think about. It probably holds the record, certainly for a meeting I have attended, of having the fewest words per author of any paper!

However, this is not a criticism. It is the old joke, “If you want the 100 page report, you can have it tomorrow; but if you want the one page report, you will have it in three months”. The authors are to be congratulated on that conciseness because they bring out a lot of very interesting ideas very clearly, for which I am very grateful, and I am sure many other people will be as well.

I agree the need for a vocabulary and, perhaps, better language when discussing reserve uncertainty. We should distinguish between the language for communicating between actuaries and also for communicating to other people. They are essentially different.

You could regard communicating to the user of an actuarial report as speaking French and communicating to a French person, whereas the more sophisticated English version will have a different language and different concepts, and you will want go into a different level of detail.

I have what I think is a very useful analogy from physics: quantum mechanics and relativity theory.

I first started looking at non-life claim reserves almost 40 years ago. It was really, at that time, akin to the stage of Newtonian mechanics. We were struggling and learning how to do chain ladders and the likes. We had to put them into practice. We have clearly moved on a lot since then and there are many more people now involved.

The area of reserve uncertainty is still very much in its infancy. Much of trouble that we have communicating it has its origins in its underdeveloped state – that we still have a lot to do. It is a little bit like comparing and trying to reconcile Newtonian mechanics with quantum mechanics and relativity theory. We are at that sort of level and I think over the next 10–20 years we are going to do a lot more work on uncertainty.

The paper refers to Gibson et al. (2007) where they separate out the variations of uncertainty into process parameter and model error. It is important to realise that these actually affect the calculation of best estimates differently to calculating uncertainty. Model error probably does not play a major part in the overall best estimate, but it has a major impact on the tail of the extent of the uncertainty. We, therefore, need to develop the language and the methodologies to deal with that.

We have made a good start at it but we are still only at the beginning.

This is where the parallel with the development of physics is helpful. Newtonian mechanics served the world very well for a long time. Nobody criticised Newton for not having sorted out relativity or quantum mechanics. Quantum mechanics and relativity have taken physics a long way over the last 100 years. But there is still no unified theory reconciling the two. We need to think about that when we are trying to develop our own.

My experience over the years is that, if you look over a development triangle, for a large range of that triangle the variation is directly proportional to the outstanding reserves. A simple percentage loading is fairly accurate – it is fairly stable between lines of business, subject to some variations between companies. But that approximation breaks down at short durations and also at long durations. These, unfortunately, are the areas where it is most important.

Again, the parallel with physics is useful: the fact that the issues that affect short-term durations are different from those of the tail. The techniques that are required are different, and so on.

Trying to work those out, I think, is going to be a challenge for us. We have started to make an effort in those areas, but there is scope for further research.

Talking about the cycle, again that has a parallel with quantum mechanics. The uncertainty principle to some extent applies to that. We need to recognise that as we analyse uncertainty, some elements will affect future uncertainty. It is worth considering that impact when doing our thinking in this area.

I have always expounded the theory that actuaries tend to exaggerate the cycle. This is not something that we should necessarily take as a criticism. It is in fact a compliment because the more accurately you can estimate the reserves, the greater the inherent cycle, because, basically, insurance is a cyclical business because of the way the laws of economics work. It is a very price-sensitive article. When you have excess capacity you are going to get a soft cycle, and vice versa. The more accurately you can price your product, the more you are going to exaggerate that cycle.

Mr D. I. W. Reynolds, F.I.A.: I want to make one simple point about accounts that covers the accounts of pension funds, life insurance companies or general insurance companies.

I start by welcoming one particular aspect of this paper: nowhere in it did I see the concept of mark-to-market. I trust we all accept that market value and fair value are not equivalent concepts. They may often be the same, but they are not equivalent.

I should like to suggest that we get closer to the accountants and we ignore the term “best estimate” and use “fair value”.

My view of fair value of a reserve is figure that is just as likely to be too much as too little – I presume a 50 percentile. I acknowledge the difficulties of actually producing that number, but I will leave that aside.

I think we can also cope with the accountants’ view of prudence by incorporating a risk margin. The point I want to make is we must always separate out the fair value and the risk margin. We should not bring them together. I think the Australians, in looking for a 75% percentile reserve, are in error – it is very difficult to see what margin has been incorporated.

The International Actuarial Association Risk Margin Working Group has set out, as outlined in paragraph 5.2, some very crucial points as to how the risk margin will vary depending upon the nature of the risk. Unless you separate it out from the fair value, you do not know whether or not you are including a proper risk margin.

Solvency II seems to be taking us that way. I think they use the term ‘best estimate’, which is a pity. They then show a separate risk margin. One of the arguments against this – and maybe this is what influenced the Australians – is that, if you show a fair value reserve and then a margin, the taxman may take out the margin, and you will pay more tax. That is rather specious. What you are actually saying is we are going to confuse the world in order that we may confuse the taxman – and you probably will not.

I think when we are talking about reserve and margin we do have to keep them quite separate.

Mr Ryan brought in the concept of the uncertainty principle. My understanding of Heisenberg's principle in physics is you cannot know where something is and its momentum. I think we also have an uncertainty principle. By the time we have done the calculations, things have moved on. One of the things that we should be thinking about in all of this is not to make things more complex if, as a result, they take longer to produce. So we should be looking to see how we speed up our calculation processes.

Maybe, in congratulating the authors, I could say this is one case where GRIT has produced a pearl!

Mr P. H. Hinton, F.I.A.: I do not believe in magic bullets. When we are talking about reserving, it is very easy to think about data as the data that underlines the chain ladder but that is only part of the data that an actuary will be using. As well as data relating to the years which are sufficiently like the business you have on your books to be able to use directly, there are also data relating to business written 40 or 50 years ago which tells you something about the business of insurance, and there is also information relating to other companies.

When you are setting a best estimate which takes into account, as Solvency II does, the notional mean and therefore takes into account extreme possibilities, which by their nature are unlikely to be represented in the reserving data as narrowly defined, you have to bring into account the possibility of inflation taking off again. You have to take into account something like asbestos happening again, and so on.

Then, when you come to look at uncertainty and describe it, you also have to mention these as explicit possibilities and remind people that they are not as unlikely as they might seem – they have happened within living memory. The probability is almost certainly rather less than 1 in 200. We are concerned about the 1 in 200 event.

Then when we are talking about uncertainty, since our memories do not go back 200 years, we probably need to add on something over and above that.

Mr A. D. Smith (a student): I should like to thank the authors for this paper, and for compiling a useful synthesis of the voluminous input from GRIT and other working parties. I should like to focus some remarks on Section 6 which looks at the disconnect between the methods we use for best estimates and the methods we use to quantify uncertainty. As the paper notes, a unified approach currently eludes us.

I do not think that this is a mathematical or statistical problem. I think it is a problem of governance. The awkward part is the effect of management on filtering results. Where there is a range of best estimates, all equally statistically plausible, it does not follow that management is indifferent to which of those is disclosed.

Preference might be reflected in several ways: by direct request; by actuarial self-censorship; by management selection advisers; or an element of all of three.

Of course, this issue is not unique to actuarial work. This morning's papers are full of the story of a Government chief drugs adviser and his dissatisfaction with how the Government has filtered his advice.

If the existence of filtering is acknowledged, we can ask the empirical question, which is partly considered in subsection 2.4, of whether or not management are filtering results in closer estimates to the actual outcome.

I think, in principle, we can test this hypothesis by a three-way comparison based on historic FSA data: firstly, of five year old management estimates; secondly, of mechanically derived estimates from triangulation data (so I am disallowing the extra data that Mr Hinton has just mentioned); and thirdly, what we now know five years later.

That would give a way of testing the hypothesis of whether all the extra data and the extra management had actually improved the reliability of the estimate. As far as I know, that analysis has not been published comprehensively.

If the answer is, ‘Yes, management filtering has improved the liability estimation’, then I think that the pure mechanical methods for reserve variability would overstate the actual reserving error. The reason is that the management intervention reduces the amount of variability in the estimate. It makes it closer to the ultimate outcome.

In that case, your uncertainty estimates from your mechanical method, be it Mack, bootstrap, or whatever, should be narrowed to reflect the additional information in management views.

Once you have done that I think you bring back together the best estimate and the variability models: they come back into line.

There is another possible answer. The answer could be, ‘No’. You find that management filtering has not improved the liability estimation at all. Then you would have to ask, ‘Why is it needed? Why are we doing this?’ Maybe it fulfils some other purpose like smoothing reported profits over the underwriting cycle.

So, if management filtering is worthless from a statistical point of view, then you might argue for just removing it, which would also close the gap between the best estimate and the variability estimation methods.

I do not think that is a conclusion that many of us would like to reach: that management filtering has no value. It is possible that the fear of such a conclusion stands in the way of publishing analysis of FSA returns. But even if that is the conclusion, I would rather know than remain in ignorance.

Professor R. Macve, F.C.A., Hon F.I.A.: As you know, I speak as a chartered accountant, albeit also an honorary actuary. I just want to note the frequent stress on the importance of good communication with users and clear vocabulary.

Mr Reynolds has already referred to some accounting terminology, and in this regard I would remind you, yet again, that what is ‘reserving’ to an actuary (primarily to make the necessary technical provisions for claims liabilities) is ‘provisioning’ to an accountant, to whom ‘reserves’ are ‘spare’ owners’ funds to be used as you please, or possibly held to satisfy regulatory requirements. They are not amounts required to cover liabilities.

The accountants – and probably most users of accounts – would include the additional amounts held to meet capital adequacy requirements under solvency regulations, i.e. the ‘cushions’ required over and above the provisions for liabilities, as a part of the owners’ funds (i.e. part of ‘capital and reserves’).

So I think the terminology does not help. It is not the most serious issue, of course, but it does not help communication. It is quite proper in a meeting of actuaries for you to talk to each other as actuaries. But when you are talking to the rest of the world, as has already been said, you want at least to show that you understand how these things translate.

Knowing whether you are going to call it a provision or reserve does not make it any easier to know how much is needed and how amounts should be measured. The FASB, in the US, and the IASB have been wrestling since 2002 on how to amend the current rules in IAS37, Provisions, Contingent Liabilities and Contingent Assets. In June 2005, the IASB issued an Exposure Draft of Amendments to IAS37 but it met such widespread opposition that the Board is still working on revising the standard. Any resolution will of course interact with the Insurance Contracts (Phase II) project on which, as the paper notes in subsection 5.7, the Board is also still working.

However, the terminology does now seem common to IFRS, the IAA, and Solvency II (all refer to terms like ‘best/current estimate plus risk margin’), although Solvency II, as described in subsection 5.6, appears to confuse the issue again by including in the risk margin the cost of the ‘solvency capital requirement necessary to support the insurance and reinsurance obligations over the lifetime thereof’. Those should be in ‘reserves’ (in the accountants’ sense) not in ‘provisions’!

The issues at stake are also reflected in the similar recent debates over how banks’ loan-loss provisioning could be improved – in particular in respect of allowances to reflect the stage of the economic cycle – although, of course, the time periods involved there may often be shorter than for the more difficult long-tail insurance or pension liabilities being discussed here.

As has already been mentioned, there are inter-relationships between the concepts being discussed here and the way in which the concept is discussed in the accounting standards. So this is just a plea for more clarification on vocabulary and emphasis on communication.

The Chair: Thank you, Richard. We are always delighted to have a contribution from one of our Honorary Fellows.

Mr C. D. Daykin, F.I.A.: I would like to add my congratulations to the authors not only of the meta-study but also of the other studies that go to make up the overall group of GRIT and ROC papers. Subsection 5.6 of the paper refers to Article 76 in relation to Solvency II. I want to draw attention, in particular, to Article 47 of the Solvency Directive, which refers to the actuarial function.

Article 76 sets out the pattern of best estimate and risk margin. I am not going to enter into that debate further tonight, although I would just add that when the European Union started on this road they probably anticipated that they could have relied on the IASB to get an international accounting standard in place, which they would then have used as a basis for their understanding of best estimate and risk margin. But as that is not happening in the timescale necessary, Solvency II is going out alone on this path.

Article 47 refers to the actuarial function, which is for both life and non-life companies. So this does in effect create a statutory role for actuaries in the non-life field. Although it does not specify that the actuarial function must be carried out by actuaries, the assumption must be that actuaries will be the most natural people to carry out the actuarial function.

Article 47 focuses on some of the roles of the actuarial function, which include estimating technical provisions and the need to test the outcome against the estimated cash-flows. It also talks about explaining to management the uncertainty in the technical provisions and some other things, such as giving an opinion on the underwriting policy and contributing to the risk management function.

One then has also to look at what the risk management function and the Own Risk Solvency Assessment (ORSA) involve. So there are a number of elements which go beyond the technical provisions set at best estimate plus risk margin, and also introduce an assessment of other aspects of uncertainty beyond that within the estimates.

Article 47.2 refers to professional and other standards and hence challenges the Actuarial Profession to develop appropriate standards that will contribute towards the convergence of practice, which is envisaged by CEIOPS and the European Commission in connection with Solvency II and its roll-out across the European Union.

About a month ago the Groupe Consultatif, the umbrella organisation for the actuarial bodies within the European Union, submitted a substantial report to CEIOPS, the over-arching body for the regulators, on the question of professional and other standards. It put forward some proposals for various standards which would be needed, including ethical and communication standards. The communication standards would overlap with some of the BAS standards which were referred to earlier. Also it was suggested that there would be a set of technical standards. There will, no doubt, be ongoing debate between the various stakeholders as to the level at which these should be set. There are some within the European insurance industry who would like them to be quite detailed and specific. Others would like them to be very high-level principles. The Groupe Consultatif has settled somewhere in the middle: principles-based but with more detail than is currently proposed, for example, in the BAS insurance draft standard.

So, this is an area which will be developing rapidly over the next year or so. We do not yet know how the Commission and CEIOPS are going to play this in terms of who will actually set the standards. The Groupe Consultatif has suggested that it could play a major role, but clearly there will have to be a consultative process, and others will have to be involved. I think it is clear that the Profession, particularly in the UK, in relation to the general insurance expertise, will need to play an active role in helping in the process of defining just what those standards should be, in order to help convergence of understanding of how technical provisions should be set in accordance with the principle of best estimate plus a risk margin.

Mr C. D. O'Brien, F.I.A.: I should like to talk a bit about words and a bit about numbers.

Words are important; the way that actuaries describe the general insurance reserves – or maybe I should say provisions – does have an effect.

Some of you may remember the Independent Insurance Company, which used consulting actuaries to comment on the provisions that management were making in its accounts. The wording used by the consulting actuaries in later years before the Independent folded changed in a way which clearly illustrates that the consulting actuaries became increasingly uncomfortable with provisions put forward by management.

The words need to be read carefully; they convey something. I was therefore interested in the words that were suggested in paragraph 4.5.5 of the paper as to how we might illustrate the level of uncertainty in provisioning in the future. Those words are associated in that table with some specimen probabilities.

So I move on to numbers. If I see a comment such as it is “likely that the outcome will lie below this estimate”, which is associated in the table with a 90% probability, and if I see accounts 100 times, I will ask, ‘To what extent was the outcome within that 90% figure?’ or ‘How many outliers were there in comparison with what I would reasonably have expected, having read those words?’ I have some issues about how, in retrospect, you check the accuracy of the initial reserve, but I leave that to one side.

The issue is that the way these words are used, if this table is adopted, will lead people to have a numerical understanding of the uncertainty in the reserves. Bear that in mind and think about the uncertainty in reserving at the moment. There is some understanding of the degree of uncertainty about provisioning, but it would be useful if there could be some further work on that to complement this paper so that we can understand the extent to which there is uncertainty and whether this is a major issue or an issue which is already under control. We have data in FSA returns, and data in firms’ reports and accounts about how the outcome is different from the provisions that were set. I suggest that merits some further study.

Mrs K. A. Morgan, F.I.A.: Unsurprisingly, I am going to talk about Solvency II. What I have noticed in my time at the FSA is that many UK insurance firms have focused on the internal model aspects of Solvency II and, whilst it is important, we must remember that there are lots of other parts of Solvency II. There are all the Pillar II requirements about the Own Risk and Solvency Assessment (ORSA), governance requirements, all the reporting requirements, and all the other elements of Pillar I, including technical provisions.

Internal models are voluntary. Not every insurer is going to use an internal model, but every single insurer is going to have to calculate their technical provisions using the Solvency II rules.

The other thing to remember is that 1st November 2012 is now less than three years away, as yesterday was 1st November 2009. In 2012 we will not be working things out on a best estimate basis, as we have done for QIS 1, 2, 3 and 4. This should be the time when firms do the QIS exercise properly. QIS 5 is happening next year; that will be a good time to try out some of these techniques on a real basis.

One of the main things underpinning Solvency II is understanding risk, and hence understanding uncertainty. This is not just important for regulators. In fact, I think it is much more important for firms’ management. If I were running an insurance company, I would want to know where the big sources of uncertainty were. Technical provisions are the biggest item in the balance sheet. If that number changes by a small amount then that can wipe out the profit in one year or more than double it. So, basically, I think, as a profession, we need to stop navel-gazing and get on with trying to solve this problem. We need to think about the state of technology now. This was something emphasised in the GIRO debate in Edinburgh. We need to think about what the problem really is. We are trying to model the real world and come up with a mean and an estimate of the uncertainty in that mean, and think about what period we are trying to model that uncertainty over.

We also need to think about what we can learn from other areas. For example, we can learn from our life actuarial colleagues, and also the accounting profession. Perhaps we need to work together, as Dr Pryor mentioned.

There is so much to do, I should like to finish by saying we should ‘just do it!’.

Mr A. S. Collins, F.I.A.: With reference to the comments in Section 6, given that stochastic statistical models are able to show a distribution, together with a mean, should we not use such methods, where appropriate, to produce best estimates? It might be controversial, but if actuaries put the same effort into preparing results based on stochastic methods as they did into their previously assessed deterministic methods, maybe they will find out that they will not need to align the results to a certain figure in the same way they did previously.

If actuaries did this and believed in the output from their models sufficiently for them to trust the models that they use to assess the best estimate, then the recipients of actuarial models would be more likely to have faith in the conclusions being discussed around uncertainty.

Mr P. R. Archer-Lock, F.I.A.: I just want to respond to some of Mr Smith's comments, which I thought were very useful comments, and I also want to pick up on a couple of his points.

Firstly, in terms of the analysis of FSA returns, and looking at cycles, that is what the reserving cycle working party tried to do. The fundamental problem we faced was not knowing what the actuarial best estimates were corresponding to the numbers in the FSA returns. You can look at analysis of cycles in the past based on what was actually booked, but separating out what an actuary would have done at the time is nigh on impossible unless you have your own company's data. That is where we hit the brick wall. If anyone has any suggestions as to how we could overcome that, I would be very interested to hear them.

Secondly, in my experience, a common reason that your best estimate is not close to equalling the mean coming out of your uncertainty modelling is because you are deliberately making adjustments to the chain ladder or other methods because of a particular large claim or other information that is not necessarily coming from an actuarial background. The information could, for example, relate to a particularly large claim for which you have a lawyer's report. You are factoring into the best estimate to be carried in the reserves the information coming out of that lawyer's report. You may not have the information to convert that individual estimate into an overall distribution. Even if you derive an overall distribution by making some assumptions, you then have to decide, if you have done a bootstrap method, how much of that uncertainty on the large loss is already implicit in the bootstrap and how much is not.

In setting a best estimate reserve, we often try to do a much more detailed exercise than we do in our reserve uncertainty. The challenge for the Profession is not converting a triangulation method into an uncertainty method, on which a lot of work has been done, but how we look at areas where we are deliberately building other people's judgements into our best estimates, and how do we translate that into a range. That is going to require quite a lot of input from people other than actuaries.

Dr S. M. Coutts, F.I.A.: There are a number of people at this meeting tonight who were at the very first GIRO conference in Norwich. There are five or six I notice, from not so many there. I think that they would agree that the actuaries who were at that conference would be proud to see how the next generation of actuaries have generated new concepts in general insurance for the next generation to move forward.

For those people who did not know how ARN came about, it is not what you think it is. It is called, in the spirit of Brave New World, After Norwich!

I should like to go back to a bit of history. My point is we have talked about uncertainty. No one seems to want to talk about distributions or where it all comes from. We seem to have taken as a basis that the chain ladder method is it, and we are going try to work around that.

I should like the working party to say, ‘Hold on! Maybe we should be looking at what is driving this chain ladder, whatever the distributions are.’ For example, why do we not look at the distributions within each one of these cells to see what actually is the uncertainty there? The Mack, basically, does an aggregate of those cells, as somebody mentioned earlier.

It is funny that the grandfather of actuarial distributions was a very lovely actuary called David Craighead. He was before chain ladder in the Lloyd's market. He used cumulative distributions, and he did that in the 1960s. Not only did he do that, he also did it by underwriting year, so he was able to get individual distributions for underwriting years and variances. We then moved on to the chain ladder method.

Then some statistical models were developed. One of them is ICRFS, which some of you know. It is basically a general linear model with three parameters.

It started off with a distribution concept of log-normal, but then moved to a non-parametric spline type of analysis. Distributions were therefore lost and back came the non-parametric chain ladder method.

My plea to the working party, and future working parties, is that they start looking at the real underlying distribution, like a statistician or physicist. I must admit I find it difficult to understand the parallel between physicists and what we are hearing tonight: those scientists would go back to the original data, and have a look at what that was and that would tell us a little bit more about the original distributions and uncertainty.

Mr C. A. Brooke-Taylor, F.I.A. (closing the discussion): In closing, I will try to do the one page version rather than the 100 page version!

I am privileged to be here at what is possibly a fairly pivotal moment for the general insurance discipline of the Profession. The paper, as has already been said, is a summary of a lot of work that has been done, particularly over the last five years. I have to admit that I read this paper but not very much of the underlying material. There has already been a recommendation to every GI actuary to look at this paper. I would go further than that and say: pick out the pieces that particularly appeal to you and drill down into the research which has already been done.

I should like to bring out a number of themes, which quite helpfully came out during the debate. They are the same themes I wanted to bring out as I was preparing.

The killer point for me is we have to be able to communicate uncertainty to our users in a way that they understand. Fundamentally, the information is only useful if it is used, and it will not be used if the user does not understand it properly. There is quite a lot of discussion around the way in which we communicate the results of our analysis, the discussion of the difference between estimating the uncertainty in the estimates we are making versus estimating the uncertainty in outcomes. That is a very important distinction.

There is also discussion about the different forms of professional guidance we have. Most of it is very helpful in setting out the ways in which we can communicate more clearly. But, underlying that, there is a problem with vocabulary. We have the vocabulary challenge, as Dr Pryor pointed out, across our fields of practice. This is not peculiar to the GI industry.

As well as communication of the outcomes, there is also concern about communication of the possible implications of those outcomes.

One of the ways in which we can help to understand uncertainty, and communicate that uncertainty, is to consider the underlying transactions behind what we are measuring. This leads also to a topic which was not particularly well brought out in the discussion, and that is scenario testing.

There was reference to trying to illustrate uncertainty with deterministic methods, but most of the focus on uncertainty was around stochastic methods. It is important that we can explain, and our users can understand, the real-life situations that might originate or might come out of unexpected outcomes.

Why is that? It is because they need to be prepared to take action in the eventuality that those outcomes do transpire. It is not sufficient just to acknowledge that they might happen, we have to make sure that we can react to them to avoid wasted opportunity cost, to avoid wasted management time, to avoid ultimately, in extreme circumstances, the destruction of the firm.

Also on the topic of communication, the paper itself exemplifies the succinctness and clarity with which we should seek to communicate uncertainty when setting out our reserve estimates. I am sorry for using the word “reserve”. I think there is a very interesting point there that the very word we use around this debate is a word which does not have the same meaning to other professionals who are also advising the same clients.

Finally, on the topic of communication, a plea that words are particularly important. I exhort us to use some of the suggested wordings in the paper to help to convey a more quantitative understanding of the uncertainty in the outcomes.

The flipside of the debate, and there are really two sides to this, is do we work on communication or do we work on refining our methods? I think there was a debate on this topic at the last GRIT presentation a couple of years ago.

A method is just a model, and a model is never going to emulate the real process properly. It is a tool to help understand the process and a tool to help make decisions based on one's understanding of that process.

While it is laudable to seek an improvement in the methods that we have available to us, we should not get hung up about the perfect method or the one single method being applicable to all situations, which of course none of us believes – an important point which was brought out in the discussion.

Also, within the establishment of the method with which we estimate uncertainty there is a lot of discussion about the method by which we communicate that uncertainty, and how we represent it. In the discussion in Section 5 about the international perspective there are examples of different ways in which that is done. I will pick up on that later.

One of the key points that I think was brought out is the fact that it is quite possible to mask big holes in our understanding by burying them under implicit margins. Heisenberg's uncertainty principle aside, I think there is a great deal of value in getting a better understanding of what is really happening rather than just hiding behind those margins.

Having said that, there is also a huge amount of value in knowing when to approximate. One of the key principles that I was brought up on was the principle of parsimony. As we increase the number of parameters that we use in the modelling, there is a huge risk that (a) we get too hung up on getting those parameters right and (b) we end up thoroughly confusing ourselves and losing the meaning of what we are trying to convey.

Mrs Morgan referred to Solvency II and brought out the point that Internal Model Approval is not the be-all and end-all of Solvency II. This is a key point of debate that we are having within the FSA at the moment: the fact that there is a risk that we spend too much time focusing on the models and not enough time discussing those models. This is something in the context of everything that we do as actuaries that we have to take into account.

On that subject, Mr Hinton brought out the point that the data we use are not just the data that feed into our chain ladder models. There is lots of collateral information and, more to the point, there is lots of historical information that we might not be able to build into our statistical models but which is very useful in the discussions which we are having about what those models are telling us. It can help us to formulate a much more rounded view of the uncertainties facing us than can the output from the models on their own.

Touching on this, there is really a debate about the extent to which one should bring judgement, be it into the deterministic estimate of best estimate or the stochastic models themselves. A number of commentators spoke about the extent to which management judgement can increase, reduce or indeed introduce bias into the results that are finally presented. This is something that needs to be dealt with in a much more explicit way; but, again, the intellectual power that we as a profession have should be devoted at least as much to the discussion of those results and helping our clients and colleagues to understand those results, as it is to coming up with some numbers in the first place.

One of the themes which I thought was important was what I call the cost-benefit analysis of what we are doing. Users only use the results if they understand them. The user also uses the results if they think they are relevant to the question they want answered. I think we have to be very careful again, as a profession, that we only deliver information that is useful to our clients. We may have to help them to understand what is useful. That is another field of communication. We have to be careful that we do not get too hung up on protecting ourselves professionally and forget to give something useful to our clients.

Mr Copeman said he was disappointed that we have not really got our act together in quantifying the uncertainty in our results. That takes me on to a point which came out a couple of times about, in my view, whether or not we have set ourselves an impossible challenge. Setting ourselves the challenge of quantifying uncertainty, via GN12 or otherwise, perhaps was a good thing to do. Given the development that we have seen in expectations over the communication of uncertainty, particularly in the international context with Solvency II, etc., I think we should be asking ourselves whether or not those expectations are now too high. I think the peripheral question might be the extent to which we, as a profession, have contributed to raising those expectations. We also need to think, as we do with some of the thinking that Mr Daykin has asked us to do on the development of Solvency II standards, about what we should be doing to manage those expectations and making sure we do not set ourselves up for a fall which may ultimately come.

Regarding approximations and useful information, Mr Reynolds made the point that things tend to move on. I really do believe that timely information is at least as important as accurate information. It is quite evident from the research that has been done in the paper, and the work underlying this paper, that we still have a lot of work to do in improving the accuracy of the work that we deliver. But that accuracy should not come at the expense of timeliness, and we have to bear that in mind.

To draw an analogy with the work that our life assurance colleagues tend to do, I do question the true value of what the farms of computers that are busy crunching away at stochastic models that take ages to close a year-end can bring. So we need to be careful as we become excited about how ‘whizzy’ are our computers these days.

Culture is one of the important considerations that the FSA makes when it is assessing the risk inherent in a firm; its management, governance and culture. Mr Smith suggested that the barrier to finding a unifying method between stochastic and deterministic measures of best estimates might be a cultural or governance issue at least as much as a statistical issue. I have much sympathy with that view.

If the management filter brings us closer to the real results, I would be very surprised. If the management filter takes us further away from the real result over the cycle, I would not be surprised at all. But I am pretty sure that the management filter introduces a degree of bias into the results at any given stage.

Reference was made to the potential for analysing FSA returns to do some backtesting. We have done quite a bit of work on that and some has been put into the public domain. I think that Mr Archer-Lock made a very important point: we cannot second-guess the work which actuaries did five years ago if we do not know what work the actuaries actually did. Being able to bring out the difference between the actuarial estimate and the management decision would be an interesting avenue to explore which, clearly, they have done in the US.

Finally, I promised I would talk a little bit about the international perspective. Section 5 brings out a number of different ways in which we might present the outcome of our uncertainty estimates. None of the bodies involved has given us particularly helpful suggestions on how we might estimate that uncertainty in the first place. We have to be careful that we do not get too hung up on whether or not the cost of capital approach is correct at the expense of working out what the uncertainty is in the first place.

Mr Reynolds also made a plea that we do not blur the best estimate, however we define it, with the risk margins, and we really must separate the margins from the best estimates.

One of the thoughts that came through in the latter part of the discussion also alludes to these different international perspectives. That is, perhaps, the difficulties we may have in working out how we express ourselves could arise from the confusion as to the purpose of some of our estimates. Reference has been made to the taxman; reference has been made to regulators. I am not sure whether any reference was made to shareholders, but they are an equally important stakeholder in reviewing the results of what we are doing. Very simplistically, it is a case of horses for courses, and however we estimate the uncertainty in our results, we have to communicate those in a way that is meaningful to the user and for a purpose that suits them.

I should like to conclude by thanking again the authors of the paper. This is a very readable overview – it must be, because I read it! It is an overview of a huge amount of work performed within and outside the Profession on this topic.

In my view, they brought out most important issues that continue to require consideration. They have also illustrated the progress that has been made to date but, unfortunately, it has been made more clear to me that the end goal of making advice on reserving uncertainty truly valuable seems further away now than, perhaps, it did in 2004.

Dr Gibson (responding): First of all, let me say a word of thanks for all of the kind words from the contributors, and also thank you for the challenging words as well.

A lot has been said about communication and language. I will respond on a few points there. First of all: ‘provisions’. For those of you who do not know, and who are thinking that means the food that you take with you on a camping trip, it means ‘reserves’. Best estimates and fair value – we heard about that from Mr Reynolds. I am reminded when Tony Jones was doing the original GRIT and he interviewed some of the chief executives of companies and said, “What does ‘best estimate’ mean to you?” One of them replied, allegedly, “The best estimate is the estimate that I like best – and that is probably the lowest one.”

That reminds us of the dangers of the phrase ‘best estimate’. Maybe ‘fair value’ is better.

I think the last word on language has to go to ‘meta’. My Merseyside comprehensive school did not offer Latin, although I did benefit from studying physics. That is not a disadvantage to me here. I am reliably informed by Caroline Barlow that ‘meta’ is Greek and not Latin. It means the bringing together of different studies. A meta-study, as Mr Reynolds has pointed out, is a study of studies.

The point was that we were not trying to produce new material in this study. Perhaps that is one of the reasons why we failed to pick up on some of the recent TASs, for which my apologies, although I would pedantically point out that subsection 4.2 is entitled “Professional Guidance” and the BAS is dealing with professional standards, which is rather different. Perhaps it was out of scope anyway. But you can be reassured that we will be very interested indeed in TAS R and the other TASs. I was interested to hear your response to the concern about quantification of uncertainty and why it is worded the way it is. It is something to which we will pay more attention going forward.

A number of people spoke about backtesting. As Mr Archer-Lock mentioned, the 2003 reserves cycle working party did look at this. In fact, this is how they got to grips, if I am correct, with the reserving cycle because it seemed to be that management influence was cyclical according to the reserving cycle to some extent, or at least that it was distorting the picture.

A word of caution to us all is that the big handicap for backtesting in the past was that you could not clearly see the actuary's best estimate. Of course, going forward under Solvency II, you will be able to see it clearly, with the reserve margins separate. So perhaps there will be further opportunity for backtesting, although I do see a head being shaken in the audience which suggests that not everybody is in agreement.

Finally, to Article 47. We are looking at the Article 47 material in the Reserving Oversight Committee. Kendra Felisky is chairing the working party, looking at reserving for Solvency II, so that is something of which you will see more later.

To wrap up; we are not done yet. We have been going for five years and, as a number people have said, perhaps we have succeeded in understanding how big the problem is and are beginning to articulate it a little bit more and realise that we still have quite a long way to go. It is to be hoped you will expect more of the same in the future, and maybe we will get ever closer, maybe not to a magic bullet, which Mr Hinton does not believe in, but at least to a more coherent and more robust approach to reserving, communicating and quantifying uncertainty.

The Chair: It has been a great privilege for me to chair this meeting.

When I was President, I made a number of speeches and wrote articles on the topic of communicating uncertainty. It was something that had concerned me during my career as a pensions consultant. When I became President, I discovered it was something that the GI practitioners were exercising their minds over. At almost exactly the same time I sat ex-officio on the executive committee of the CMI and discovered that the actuaries producing mortality tables were exercising their minds enormously about this issue of uncertainty and communicating it.

In essence, what I was saying in the speeches and articles was that this was probably the biggest challenge facing actuaries: how we communicate uncertainty to our clients and our customers. The only answer I have, and it is not a very helpful answer, is we are going to have to work very, very hard at it.

This has been a tremendous contribution tonight to enhancing that debate. I simply add my support to what Dr Pryor said about not sitting in silos. This issue affects every single branch of the Profession and every single actuary in the country. It is also very topical because of the advent of Solvency II. It seems to me that Solvency II is a major issue for the insurance industry and the actuarial profession right across Europe. I would not be surprised in a few years’ time if it, or something very like it, does not spread beyond Europe.

When I became President, the first UK actuarial convention I attended was the GIRO Conference in 2004. It is interesting that the title of this paper spans the period 2004 until today. I was struck at that GIRO convention by the enthusiasm, the vitality and the vibrancy of the general insurance branch of the Profession. Everything I have heard tonight, and read since, has convinced me that that was the right analysis. So I wish you all the very best for the future. Keep up the good work. I shall continue to spread the gospel that the other branches of the Profession should take note of what you are doing in this area.

I should like to express my own thanks, and I am sure the thanks of all of us, to the authors, the closer and to everybody who has participated. I think we have had an excellent discussion.