Health policy making in Australia, as in other Western democratic contexts, has increasingly been characterized as a process of evidence appraisal (Reference Lomas, Culyer, McCutcheon, McAuley and Law12). Mirroring the developing prominence of evidence-based medicine, “evidence-informed health policy” has emerged as a dominant evaluative framework in which decisions routinely foreground the outcomes of primary research, systematic reviews (SR), and health technology assessments (HTA).
While some practitioners critique the notion that “evidence-based healthcare. . .can make an unproblematic transformation into evidence-based policy” (Reference Lin, Gibson, Lin and Gibson11), a growing acknowledgement that decision-making processes should be both explicit and public has fuelled the evidence-based policy movement. Such concerns are particularly salient with regard to healthcare “disinvestment” – policy reform oriented to supporting safe and effective care within sustainable health systems (Reference Elshaug, Hiller, Tunis and Moss7).
In Australia, SR evidence forms the cornerstone of HTA and related policy processes. Within a disinvestment framework, understanding how SR findings fit within the rubric of “evidence,” how they are used to inform decisions, and how this evidence is negotiated by expert clinical stakeholders, is integral to the establishment, evaluation, and success of reform. Using a case study, this investigation describes an example of evidence-informed debate around potential disinvestment policy, situated within a case study of pathology testing for vitamin B12 and folate. More specifically, we address the role of SR in informing such debates, and the negotiation of SR evidence by an expert group of clinical stakeholders.
Systematic Review in a Health Policy Context
Systematic reviews have become central starting points to framing evidence-informed health investment decisions made at local, provincial/territorial, and federal levels in several international jurisdictions (for examples, see 1;Reference Menon and Stafinski16;Reference Pearson and Rawlins19;Reference Sampietro-Colom, Ausa, Briones and Gol21).
However, determining the role of SR in the context of disinvestment decisions has been less straightforward. Early work in the Australian context has suggested broad stakeholder support for disinvestment processes to be aligned with existing evidence-based policy processes for investment decisions (Reference Elshaug, Hiller and Moss6), while Spanish work in this area has also emphasized the use of SR in disinvestment decisions (Reference Gerdvilaite and Nachtnebel9). The United Kingdom has focused more broadly on using guidelines as the basis for disinvestment recommendations (including those based on Cochrane Collaboration SRs), while Program Budgeting and Marginal Analysis (PBMA) approaches, among others, have been applied in Canadian provinces and regions (Reference Gerdvilaite and Nachtnebel9).
For new and emerging technologies and services, comprehensive SRs are an integral part of HTA but they alone do not constitute an HTA. For disinvestment health policy decision making the precise role of SR requires investigation, given certain conditions unique to disinvestment policy, including: contention around the identification of candidate technologies; the apparent greater “strength” of evidence required for a disinvestment decision; the contestability of evidence; and the role of stakeholders in these decisions.
To inform further the pragmatic application of SR methods to disinvestment decisions, this study addresses broad questions around the role of SR within disinvestment policy processes. The discussion focuses on how clinical stakeholders perceived and negotiated SR evidence within a specific case study with potential for disinvestment: pathology testing for vitamin B12 and folate.
METHODS
Selection of Case Study
Lead members of this research group have proposed criteria by which to “flag” potential technologies that might warrant further investigation under disinvestment programs (Reference Elshaug, Moss and Littlejohns8). These criteria include among others: geographic variations in use; temporal variations in volume; public interest or controversy; or nomination from clinical groups. This process was adapted for topic selection in the current study, resulting in a wide range of candidates being flagged. Tests for investigating vitamin B12 and folate levels—most frequently ordered conjointly in Australia—were nominated by clinical experts to this research group. Further scoping revealed that they met several additional “flag” criteria from this group's identification and prioritization frameworks (Reference Elshaug, Moss and Littlejohns8); a full account of this is recorded in a separate manuscript (Willis et al., unpublished data 2012). Briefly, since 2001, usage of a combined vitamin B12 and folate test through Australia's universal health insurance scheme, Medicare, has experienced an annual average growth of 21.78 percent (15) in contrast to 6.3 percent for all pathology services (14). International differences in recommended test usage are also apparent (3;10;17;20), while usage in some clinical scenarios appears to conflict with available clinical guidelines (Reference Bayram, Britt, Miller and Valenti2). Finally, given this was a primary research study, it was decided by the lead investigators that these tests offered characteristics (e.g., low level of controversy, high volume, low cost per-item) which would balance the other case study chosen for investigation, assisted reproductive technologies.
Systematic Review
Diagnostic accuracy (a central marker of the effectiveness of diagnostic testing) formed the foundations of a SR on the effectiveness of vitamin B12 and folate tests in determining the presence or absence of deficiencies (Reference Willis, Elshaug and Milverton22). A protocol for guiding this SR was developed in consultation with a clinical pathologist, health services researchers, epidemiologists, health economists and experts in HTA. The SR aimed to synthesize published evidence relating to the diagnostic accuracy of the three most commonly used tests for investigating vitamin B12 and folate deficiency: serum B12, serum folate, and red cell folate assays.
Results from the SR and meta-analysis suggested that the serum B12 assay has limited ability to identify patients with symptoms amenable to vitamin supplementation, noting poor positive (PLR) and negative (NLR) likelihood ratios across patient sub-groups. Similar results were found for serum folate assays, while there was insufficient evidence available to complete a similar analysis for red cell folate assays. As noted in the review, the existing evidence base is hampered by lack of an internationally recognized “gold standard” for diagnosing either vitamin B12 or folate deficiencies; imperfect reference standard tests; and inconsistent “cut-off” values used to distinguish deficient from nondeficient patients (Reference Willis, Elshaug and Milverton22).
Stakeholder Engagement
Following completion of the SR, a series of iterative stakeholder engagements was undertaken, involving test requesters (primary care physicians and specialists), test providers (pathologists) and key decision makers (federal government policy advisors). Details of this engagement process are provided elsewhere (Watt et al., unpublished data, 2012). Briefly, each group was consulted separately in professionally facilitated sessions that sought to explore reasons for the changing usage of B12/folate tests, to discuss the evidence arising from the SR, and to determine potential avenues for reform (see Figure 1).
Figure 1. Evidence generation and stakeholder engagement process for vitamin B12/folate pathology testing case study.
Nominations for participants within the test provider group (the focus of this manuscript) were sought from key stakeholder bodies in the field of pathology (see Table 1). Nine nominees and key opinion leaders (pathologists) took part in the engagement held in Sydney, New South Wales in May 2011; three females and six males. During this 4-hour session, participants were presented with the outcomes of the SR, supplemented with information modules pertaining to relevant health funding policy and economic considerations to support decision making (see Figure 1). Following deliberation around this evidence, participants were invited to discuss their experiential knowledge and perspectives on the issue, before formulating a response to the question “What should be considered when making decisions about how much we should publicly subsidies B12/folate pathology tests?”. After obtaining consent from participants, deliberations were audio-recorded and transcribed verbatim by an experienced stenographer.
Table 1. Composition and Characteristics of Participants (Pathologists)
a Including National Coalition of Public Pathology, Australian Association of Pathology Practices; Australasian Association of Clinical Biochemists; Healthscope; Sonic; and Primary Health.
Data Analysis
The theoretical perspective of this study is broadly social constructionist, our analysis grounded in the assumption that talk reflects both the interactional context in which it is generated, as well as broader patterns of sense making that can have both ideological and material consequences (Reference Edley, Wetherell, Taylor and Yates5).
Our focus is on how expert stakeholders (pathologists) made sense of “evidence” within a data-informed discussion on a disinvestment topic. In particular, we attend to participants’ framings of “what counts” as appropriate evidence for disinvestment, the language used to justify their claims, and the implications of these arguments for engaged policy processes around disinvestment more broadly.
Transcripts of the stakeholder engagement followed the model for thematic analysis as outlined by Braun and Clarke (Reference Braun and Clarke4). Through a process of open coding, the whole data corpus was worked through to build up a file of instances referencing “evidence” with regard to the case study of B12/folate testing. The main repetitive patterns evident across these accounts were identified and coded (axial coding). The whole corpus was then re-read in light of these codes, and over-arching themes (constructed so as to emphasize internal convergence and external divergence) were subsequently identified (Reference Braun and Clarke4). These themes, and representative data extracts, are outlined (in no particular order) in Table 2, and we analyze each in turn to elaborate their implications with regard to questions of evidence in disinvestment policy.
Table 2. Quotes Illustrating Pathologists' Position That B12 Testing Is Appropriate
In representing extracts of policy advisors’ talk, the transcription protocol is as follows: [] indicates clarification of meaning; . . . indicates words omitted for ease of reading; (. . .) indicates inaudible speech.
RESULTS
Pathologists questioned the usefulness of the outcomes of the SR presented to them at the outset of the engagement, and the appropriateness of using systematic reviewing per se as a means of informing funding policy around vitamin B12/folate testing. More specifically, pathologists: (1) Contested the terms of the problem driving the review, arguing that the B12 element of the combined B12/folate test (but not the folate) does not represent a legitimate candidate for disinvestment discussion. Note: that pathologists did nominate this as a case study points to a lack of agreement within the group (see point 2). (Reference Bayram, Britt, Miller and Valenti2) Reframed the terms of the problem that ought to have driven the review, suggesting that only folate testing is a suitable disinvestment focus (given that mandatory fortification has made the identification of folate deficiency rare), and that a clear definition of deficiency must be established before the effectiveness of vitamin B12 testing may be usefully investigated using SR methods.
We address each of these arguments, and their consequences, in turn.
Contesting the Terms of the Problem
A central theme in the pathologists' deliberations was that the usage of testing for levels of vitamin B12 (as opposed to folate) is currently appropriate, a claim that drew into question the fundamental value of undertaking a SR in this area.
Table 2 presents a summary of participants’ arguments in support of this position. These arguments have significant implications, not only for this case study, but for the more general role of SR within disinvestment-oriented policy processes. For example, arguments 1–3 undermined the driving rationale of a “review for disinvestment” in this area. Holding that B12 testing remains imperative for a range of presentations, and that utilisation reflects genuine “medical issues” rather than the vested “interest” of laboratories, pathologists rejected the notion that this item should be subject to disinvestment discussion per se. Although a consensually held position across the engagement, it must be noted that this argument is at odds with the original nomination of these B12/folate tests by other members of the pathology community as warranting disinvestment consideration.
Within a disinvestment process, participants’ rejection of a technology as a legitimate disinvestment candidate represents a significant barrier to productive stakeholder engagement. In this case, a rejection of what was perceived to be the guiding problem (“B12 test utilisation is inappropriate, thus we should consider disinvestment”) forestalled potential conversations with pathologists around the effectiveness of this test as indicated by the SR. The notion that disinvestment might represent a more nuanced outcome than blunt rationing (to fund or not to fund) was not entertained within the arguments outlined. For example, pathologists' claim that B12 testing remains imperative for particular populations—a notion that could have supported calls for “restrictions in eligibility” for this test to those specific groups—served in the examples cited in Table 2 to argue in favor of simply leaving the test alone. Stakeholder interpretations of review outcomes may be colored by “what stands to be lost”, and a defensive stance around accountability for perceived overuse (see argument 3) may preclude discussion around more fine-grained disinvestment options.
Arguments 4–5 presented in Table 2 also have implications for the use of SRs in disinvestment contexts. With regard to the case study of B12, these claims suggested that as long as deficiencies continue to be found, and there exists no better available assay, we have no choice but to “use the test we have.” These arguments served to position the outcomes of the SR as “purely academic,” and that indications of a test's accuracy are useful only when alternative tests are available. In turn, these claims precluded engagement with disinvestment concerns around quality as indicated by the review (e.g., at what point might it be wise to restrict or remove a test even in the absence of alternatives?). At the same time, the superiority of an alternative assay (holotranscobalamin) was asserted within these arguments without empirical evidence of its superior accuracy. Each of these claims suggests the importance of explicitly discussing issues of evidence (what counts and how to count it) in both the framing and interpretation of SRs for disinvestment.
Argument 6, holding that any partial disinvestment/restriction to pathology testing will represent additional costs, suggests an inevitable resistance to SR outcomes in this context. Without unequivocal evidence of harm, inaccuracy or superior alternatives it seems that review outcomes indicating support for “refinements to indications” may always be susceptible to arguments foregrounding the “burden of regulation.” This finding indicates the importance of enhancing SR outcomes with economic analyses at an early point in an engaged policy process in order that discussion of the costs and benefits of disinvestment options may remain appropriately evidence-based.
Evidence and Judgment
As indicated above, there was general agreement among pathologists that the holotranscobalamin assay (as yet not publicly funded in Australia, nor widely available) represents a superior B12 test than that currently used in most Australian laboratories. Interestingly, the supposed superior accuracy of holotranscobalamin was not taken as evidence that disinvestment from the current B12 item may be justified. On the contrary; participants argued that inaccuracy is inherent in all testing, an issue suggesting that decisions about what constitutes “appropriate test utilization” rely, not on evidence such as that generated from SR, but on “judgments” around sensitivity versus specificity, the value of screening, and the definition of reference standards and ranges for deficiency; all questions of values.
Sensitivity Versus Specificity
Participant (P) 6: Even that test [holotranscobalamin], while it's a good test, the predictive value will be limited. You'll still have false positives and false negatives no matter what you do.
The Value of Screening
P9: The difference between therapeutics and diagnostics is absolutely enormous . . . [In therapeutics] you can measure side effects, efficacy and costs much more easily than in diagnostics, where essentially we're offering screening for disease and if you're doing your job properly most of the results are going to be normal. It's a philosophical question, how much investment the community wants to make in screening.
The Definition of “Reference Standards” and “Reference Ranges”
P6: The whole study you [the research team] have done is predicated on reference ranges and I know some people who have used MCV [mean corpuscular volume] as the standard. There's no problem with B12 until the MCV is elevated. Well, you miss 80 percent of all cases if that's the reference standard you're using.
P2: In the case of B12, the problem is defining reference ranges. It's really very different depending on how you measure it, and the link [from] the appropriate reference range to clinical impact on the patient is really, the evidence around that is very poor.
As these examples illustrate, the argument that decisions around what constitutes appropriate testing are ultimately “judgment calls” worked to contest the notion that disinvestment policy should be guided by SR evidence in cases where evidence of effectiveness is deemed to be “complex”. In these extracts, the criteria by which “appropriate testing” may be measured in evidence are complicated by the claims that “all tests are fallible”, and that investment in tests as “diagnostics” or as “screening” represents a “philosophical” decision.
The final argument outlined above—that inconsistent reference ranges for deficiency cloud analysis of B12 test efficacy—served as the most central claim against using a SR to guide disinvestment in this area. The argument that you “cannot systematically review studies of B12 testing if reference ranges are inconsistent” functioned, not only to undermine the outcomes of the review presented, but to reframe the central problem and corresponding disinvestment solutions.
Reframing the Terms of the Problem
The problem is inconsistent reference ranges, so the answer is more research
By arguing that B12 test usage is not perfect, but “appropriate,” and that any problems identified rest with inconsistent definitions of deficiency, participants contested the terms of the problem driving the SR. In turn, they contested the applicability of its outcomes in framing future policy. While participants attested to the value of systematic reviewing for disinvestment in the case of pathology items whose reference ranges and clinical expression are clear, they rejected the method in this more “complicated” case.
P2: Where you have a test of a very reliable reference range that everybody agrees on and is appropriately linked to the clinical expression of the problem, yes, systematic review in the way you've talked about is very reliable. In this situation the thing that is complicating it is the confusion about what is the reference range of B12, so applying that tool to B12 measurement. . . does not work because of the complexity around and the uncertainty.
Reframing the problem as one of “reference range uncertainty” (rather than “inappropriate usage” or an “ineffective test”) served to position further primary research as an appropriate next step within a disinvestment program. Significantly, the argument that such research should be undertaken within laboratories (rather than universities or government departments) served both to position pathologists as the appropriate arbiters of a disinvestment agenda and, potentially, to justify increased expenditure in this area.
P2: One of the real concerns we have about where funding is going for pathology . . . is it's trying to strip away the professional and scientific basis of this on the grounds that, saying that this is a lousy test, just stop doing it, we're not going to fund these any more . . . and what that will do is take people like the ones sitting around this table out of the process. . . .I would say that it is not NHMRC research, it is appropriate continued research in any professionally high standard, high quality practice that's public or private . . . but it has to be recognized, the importance of that to the Australian health economy [so] that the pathology lab remuneration processes include a provision for that activity to continue.
This argument is significant in that it supports a disinvestment path that, unlike systematic reviewing, rests upon stakeholder-defined criteria of “quality” and “standards”.
The problem is folate, so we should “split the item” and “educate the doctors”
As outlined earlier, a second arm of participants’ arguments against disinvestment from the combined B12/folate item number rested upon the notion that only folate testing is “really a problem” because mandatory fortification has significantly reduced the identification of deficiency. This argument worked, at one level, to offer an opportunity for disinvestment that pathologists described as being legitimate and evidence-based.
P3: I think discrimination post fortification is probably the issue, in terms of folate at least that it's not really required any more. . .
P1: I think it could boil down to that . . . we recognize that with folate fortification, the scene has completely changed and I guess it's really as simple as getting the IT systems in surgeries to reflect that . . . if there's a button they press which gives a combined test, if that was de-linked it would be much more sensible.
While offering a potentially useful disinvestment option, this suggestion explicitly attributes responsibility for over-expenditure on folate tests to the clinicians who request them (not pathologists). This suggestion does not attend to the claim (presented by one participant but roundly rejected by others) that private laboratories may use the ambiguity inherent in the combined B12/folate item number to test red cell folate (for which the laboratory receives a rebate) rather than serum folate (for which they are not paid).
P8: The only reason private labs do red cell folate is because there's a rebate for it. According to the wording of the item number you do not get paid for serum folate. . .
Moreover, the depiction of folate testing as an appropriate candidate for restriction on the grounds of fortification appears to set a very high burden of proof for disinvestment in pathology.
P2: Maybe we are saying folate should not be done and B12 should be, unless you have [B12] fortified food.
Ultimately, pathologists' reframing of the “B12/folate problem” as a problem of “folate alone” deflects attention from the complexity of B12 testing. Instead, it makes way for a solution that accords with pathologists' own interpretation of appropriate evidence for disinvestment.
DISCUSSION
Tests for B12 and folate are legacy items; if subjected to contemporary HTA processes as new or emerging “investment” items, the evidence base is such that it is questionable as to whether or not they would receive a favorable funding decision. This research points to the added complexity inherent in “disinvestment” decision making. Due to the distinct parallels between investment and disinvestment decision making—focusing as they both do on safety, effectiveness and cost-effectiveness—it seems logical that disinvestment be situated within the HTA/SR paradigm. The process outlined here makes explicit the central role and contribution of SR to disinvestment discussions with clinical stakeholders, in line with evidence-informed notions of robust, defensible and sustainable health policy. However, findings from this study suggest there are some limitations to this approach, grounded primarily in stakeholder's contest of the evidence and its framing.
While an evidence-informed deliberative stakeholder engagement may contribute unique solutions to a disinvestment discussion that may not have been previously considered, there are likely instances when a full SR may be of uncertain value to disinvestment initiatives, including: when there is lack of a contemporary evidence base reflecting current practice; research questions that are more appropriately addressed using existing population-based data sources (e.g., clinical registries); genuine orphan procedures with no comparator technology; co-dependent technologies; or when the salient questions relate to how a technology is applied rather than the technology itself. The manner in which these stakeholders re-framed this problem as one of “reference ranges” or “folate alone post-fortification” suggests the importance of involving stakeholders (particularly clinical) in framing the review that seeks to inform a disinvestment question. Also, their input suggests that disinvestment might look different in relation to different questions (in this case primary research may be a necessary clarifier in a disinvestment program).
Additionally, the selection of the most appropriate synthesis products may best be done in consultation with a range of stakeholders, including clinicians, to ensure methods fit for purpose. Multiple strategies for synthesizing an evidence-base exist, including (but not limited to) SRs, meta-analyses, realist synthesis (Reference Pawson18) or rapid reviews. The appropriateness of each “synthesis product” for informing policy varies based largely upon the types of questions being asked of the evidence base: how has this problem arisen; how does the magnitude of this problem compare with other problems; what are the impacts of an intervention across specific population sub-groups; what is it about this intervention that makes it ineffective in this situation; or, how cost-effective are alternative policy or clinical approaches (Reference Mays, Pope and Popay13)?
Common to each synthesis product ought to be the opportunity to articulate where gaps in knowledge exist and how they might be addressed. The present analysis suggests that early, broad-based engagement with clinical stakeholders may have assisted in guiding the SR toward alternative gaps (e.g., reference range inconsistency), and identifying requisite primary research processes for addressing them. However, HTA structures in various countries may not have capacity to fill these identified research gaps. For example, funding may not be available to carry out additional primary research to clarify reference range uncertainty.
Within programs of investment/disinvestment, some jurisdictions are beginning to move toward a more consultative approach of review co-development, working alongside sponsors and stakeholders to create policy-pragmatic research. Australia is trialing SR production based on a co-developed Decision Analytic Protocol (DAP) between the review sponsor and the Medical Services Advisory Committee (MSAC), which guides the direction of a subsequent HTA incorporating a SR and economic/financial modeling. While this might flag a research gap, it does not fill it (though the review might shift in direction, if feasible). Other approaches such as “coverage with evidence development” or the “future research needs” program of the Agency for Healthcare Research and Quality could contribute important knowledge here.
While these co-produced approaches hold genuine appeal for disinvestment programs, they need to be managed carefully. Three key characteristics of disinvestment-focused initiatives pose challenges to this process. First, disinvestment risks continued perception that it is a negative, rationing approach, rather than one espousing quality and safety. Second, co-developed disinvestment reviews may be at risk of being redirected toward ancillary issues, while neglecting critically important questions related to technology effectiveness. Finally, information asymmetry between clinically engaged experts and policy or decision makers may preclude the truly collaborative, informed and technical discussions required to generate genuine change. Overcoming these challenges will require innovative approaches to co-produced evidence syntheses, broad-based stakeholder engagements and a sustained commitment from clinicians and policy makers alike.
A stakeholder-engaged co-produced disinvestment process for building evidence synthesis products may need to consider from the outset the terms of the review; what constitutes evidence in this context; the advantages and disadvantages of different forms of evidence; and how a synthesis should be structured. Expectations on clinicians are high, as they are charged with the responsibility of representing their patients, their profession and the population as a whole. It is, therefore, critical to identify ways of supporting clinician partners in negotiating these tensions.
CONCLUSION
While engaging vested stakeholders in processes such as this poses significant challenges, if disinvestment is to remain focused on improving the quality and safety of health systems it will continue to require clinical stakeholders to inform, guide, and invest in change.
Systematic reviews and evidence syntheses are important components of quality and safety, and for informing health policy decisions, including those related to disinvestment. Trends toward co-development of SRs for informing investment decisions suggest that there may be value in similar approaches for informing disinvestment decisions, although avenues to fund identified evidence-gaps will be required.
CONTACT INFORMATION
Amber M. Watt, MPH, Senior Research Officer, Cameron D. Willis, PhD, Postdoctoral Research Fellow, Katherine Hodgetts, PhD, Research Fellow, School of Population Health and Clinical Practice, The University of Adelaide, Adelaide, South Australia, Australia
Adam G. Elshaug, MPH, PhD, Sidney Sax Fellow, School of Population Health and Clinical Practice, The University of Adelaide, Adelaide, South Australia, Australia; Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts
Janet E. Hiller, MPH, PhD, Associate Dean (Research), Professor of Public Health, Faculty of Health Sciences, Australian Catholic University, Fitzroy, Victoria, Australia; Adjunct Professor, School of Population Health and Clinical Practice, The University of Adelaide, Adelaide, South Australia, Australia
CONFLICTS OF INTEREST
Adam Elshaug has acted as consultant to the Australian Government, Department of Health and Ageing (as has Prof Hiller) and the Canadian Agency for Drugs and Technologies in Health (CADTH). The other authors report they have no potential conflicts of interest.
