A PERSONAL EXPERIENCE: DISJOINTED HEALTH TECHNOLOGY ASSESSMENT

A couple of years ago, one of us (GJvdW) attended a presentation on a national study on extra-corporeal membrane oxygenation (ECMO) in newborn children that had been conducted in the Netherlands. ECMO, also known as the heart-lung machine, is used in newborn children with potentially life-threatening conditions such as diaphragmatic hernia and meconium aspiration. The principal investigator presented results on adverse outcomes, survival, and cost-effectiveness. After the presentation, someone from the audience raised his hand. “I served on the committee,” he said, “that decided on the funding of this project. One of the reasons why we believed the study was important, was that ECMO, when used in newborn children, also raises several complex ethical issues. We wanted those issues to be addressed, too. Could you enlighten us, and share with us the results of that part of the project?” The principal investigator was taken somewhat by surprise. “Ah, yes, the ethical issues,” he said. “Now, these were addressed by a researcher from another department. I'm afraid I am not fully cognizant with the results of that part of the study. I think you should contact them.”

The answer was somewhat unsatisfactory. Indeed, in the final report of the project there appeared to be a separate part on ethical issues associated with the use of ECMO in newborn children. It dealt with the dilemmas that ECMO teams are facing when a newborn child, despite ECMO support, fails to thrive: should ECMO support be discontinued at some time, knowing that it will result in the death of the child? Furthermore, it addressed the cases where anti-coagulation, a necessary component of ECMO, had resulted in extensive brain hemorrhage, severely compromising the health prospects of the child. It raised the question how parents should be informed about this and be asked for their consent to proceed with ECMO. It also dealt with the question what should be done in cases where parents choose to waive treatment, while the ECMO team is convinced that the child should be given a chance.

TO INTEGRATE OR NOT TO INTEGRATE?

So there we are, then. A team of researchers has, to the best of their ability, examined different aspects of ECMO. The question is: what next? Some might say: not much. That is, not much in terms of health technology assessment (HTA). The study has been completed, it is now for policy makers to make up their mind and decide whether, and if so, how, to proceed with ECMO, taking, hopefully, into account the outcomes of the study. Those who take this position maintain a clear-cut distinction between assessment and appraisal: collecting facts (assessment) on the one hand, and deciding on what follows from those facts on the other (appraisal).

Others hold that such a process lacks transparency and consistency. They argue that some procedure is needed to integrate evidence, decision criteria, and the relative weight that is given to each of those criteria. They suggest that some form of Multi Criteria Decision Analysis (MCDA) would do the job. In this study, we will hold that both of these approaches are unsatisfactory. We will argue that some sort of integration is required, but that integration is best achieved by working out the ends that are sought by the use of a specific health technology and by working out the mechanisms that are thought to be involved. In that way, we would be in a better position to understand and discuss why and how certain facts could be relevant to the assessment task at hand. We will elaborate our argument by first exploring the meaning of integration.

INTEGRATION: THREE ASPECTS TO TAKE INTO ACCOUNT

To explore the meaning of integration, imagine a medical doctor who is held in high regard by her patients. When patients are asked what it is that they particularly appreciate in the doctor's manner of conduct, two elements stand out: the doctor's extensive medical knowledge and skills, and her humane demeanor. In addition, patients point out that the doctor succeeds in integrating these two aspects in her clinical practice in a natural way, the two features are manifest in her general behavior toward patients.

From this example, we would suggest that the following aspects of integration may be distinguished: (a) two or more elements that need to be integrated (the “integranda”). In the example: medical knowledge and skills on the one hand, and a humane demeanor on the other; (b) something in which the integration is achieved and becomes manifest (the “integrator”). In the example: the medical doctor's conduct; and (c) something of value, that may be considered the whole point of the integration. In the example: a treatment of patients that is both professional and respectful.

We suggest that these aspects may be common to integration more generally, and that, to get a better grasp of a specific instance of integration, these three aspects need to be specified: what is being integrated, what is the result, and for what purpose? This also creates the possibility for assessing how successful a specific attempt at integration has been: to what extent did it help to realize the desired outcome? To better understand the meaning of integration in the context of HTA, then, we need to address the three questions above, too. What needs to be integrated (the “integranda”) is, we believe, relatively straightforward: these are the various aspects that are usually addressed in HTA, such as clinical effectiveness and cost-effectiveness, safety, and the wider ethical, legal, and social issues.

What is less obvious in the case of HTA, is what these various aspects are being integrated into (the “integrator”). In the following, we will suggest that it is a particular conception of the ends that are being sought by the use of a health technology that can fulfill this role. We will introduce a researcher who is conducting an HTA of cochlear implants for deaf children. We ask this researcher one, simple question: why did you focus on these particular aspects in your HTA? In other words, we are asking for an explanation of the selection of data that the researcher has made. We will then explain that offering such an explanation requires the articulation of some evaluative framework, specifying the ends that are being pursued. We will then argue that such frameworks have integrative capacity: they provide the coherence between empirical observations, underlying values, and assumptions about how health technologies lead to outcomes and how context matters.

FRAMEWORKS FOR INTEGRATION

So, imagine that you meet a colleague at a conference. She tells you that she works as a researcher at an HTA Agency, working on a project on cochlear implants for deaf children. She explains the characteristics of this technology and shares with you what she has found in the literature so far. She was particularly impressed by a research group from Sweden who had used a variety of methods to assess the outcomes of the cochlear implant in deaf children (Reference Preisler, Tvingstedt and Ahlström1). This group of researchers had been making video recordings of a group of deaf children before and various periods after they got the implant, at home and at school or in the Kindergarten.

They had followed these children for prolonged periods of time, assessing how these children behaved and performed in their own environments and developed, among others, reading skills. They had paid specific attention to how the (hearing) parents communicated with their deaf child. They had also made inquiries about the accessibility of specific services for the children and their parents. Upon hearing this, there is probably one question that stands out and that you should ask: Why was she particularly interested in these aspects? Not being familiar with the subject, it is not immediately obvious to you how the various aspects that she had been reading about hang together and how they bear on the value of this technology.

In other words, you are asking for an explanation of the selection that this researcher made when conducting her HTA. In the absence of some explanation, the selection is not fully intelligible to you. Moreover, because the assessment was conducted at an HTA agency, aimed at supporting public policy making, we may actually demand that the researcher is capable of providing a satisfactory explanation. Only then, we would be in a better position to understand why she collected these particular facts from the literature, rather than others. Also, we would be in a better position to judge whether we find the selection that was made reasonable, or acceptable.

EVALUATIVE FRAMEWORKS: MAKING SENSE OF EVIDENCE

Now, let us assume that the answer of our researcher is that she collected the evidence and information because she wanted to find out how (a) deafness interferes with a child's capability development and (b) how a cochlear implant impacts on this. She explains that her starting point is the concept of capability, which was developed by the Noble prize laureate Amartya Sen. Capability stands for the real opportunities that people have to become who they want and do the things that they have reason to value (Reference Venkatapuram2).

In Sen's view, such capability is determined by the combination of people's internal features, their external environment including their resources, and social and physical context. Sen developed this concept as an alternative to utility. Sen's account of capability is currently not widely used as an evaluative framework in the context of HTA, although some steps have been taken in this direction (Reference Anand3–Reference Canoy, Faber and Munneke6). The key question that derives from the framework is what real opportunities people have in a society to be and do things they have reason to value.

When we see instances where such capability is seriously constrained, we need to ask ourselves what likely causes of this might be, and whether we see opportunities for rectification, bearing in mind that these may be sought by impacting on capacities, resources, contextual factors, or combinations thereof. Opportunity costs are still a valid consideration in this context, but they take on a specific meaning: if we spend finite public resources on one program, what opportunities for rectifying gross inequalities in capability are thereby foregone? Clearly, adopting the capability approach as an evaluative framework entails specific informational requirements, different from those evaluative frameworks currently used.

The point of the story is this: we can only understand the selection that the researcher has made in her inquiry, once we know that she made this selection on the basis of Sen's capability concept as a descriptive and evaluative framework.

THE POINT OF INTEGRATION IN THE CONTEXT OF HTA

Committing ourselves to the capability approach as an evaluative framework would, in the case of the cochlear implant, comes down to the following: (a) Recognizing that one of the major reasons why we are concerned about severe deafness in children is that there is abundant reason to assume that it can, if not properly attended to, very seriously affect their capability throughout life; (b) Acknowledging that a judgment of the value of the cochlear implant and associated rehabilitation will to a large extent be based on evidence to what extent it can prevent deafness from having this type of impact (capability deprivation); (c) Accepting that other options for supporting deaf children and their families will be identified and valued with regard to their potential to protect and expand the children's capabilities, too. This would include the teaching of Sign language and acculturation to Deaf communities; (d) Acknowledging that we need to identify the causes of the wide differences across deaf children in terms of the benefit (in terms of capability development) that they gain from treatment and support, and that, for this, we need to delve into the resources that are available to them and the factors that enable them to convert those resources into valued achievements.

Basically, it explains why we consider certain aspects relevant by relating them to a specific evaluative framework (such as Sen's capability approach). In the case of ECMO, it would be helpful to state explicitly, at the start and throughout the assessment phase, the ends (i.e. outcomes) that we hope to achieve by using the technology. It would be very helpful to acknowledge that there is not just one end, but that there are multiple ends. It would be helpful to acknowledge that these ends can, and usually do, conflict with each other. It would also be very helpful to acknowledge that a commitment to a generic end such as capability development (fortunately!) does not uniquely prescribe what sort of treatment and support should be provided to severely deaf children.

Instead, this requires practical reasoning, and methods have been developed to do this in a more structured way (Reference Richardson7;Reference Richardson8). This, then, brings us to the third and final aspect of integration: why is it worthwhile to strive for, what is the point of it? Our answer to this question would be: it helps members of a community, in formal and informal (policy) settings, to collaboratively explore how technological options can help them realize specific ends that they consider important. HTA, then, is a means to systematically produce and interpret empirical findings, which derive their relevance from a concomitant deliberation on ends that are considered worthwhile to pursue. It is also for this reason, that we think that a full algorithmic solution such as MCDA falls short as an integrative device. Rather than algorithms, we think that argumentative (deliberative) approaches are needed to clarify how final outcomes and technological choices can be brought to bear on each other. In the following section, we will briefly summarize which steps have been taken in the INTEGRATE-HTA project to develop guidance that may help HTA researchers to conduct their analyses in a more integrative way.

INTEGRATION AND INTEGRATE-HTA

Integration has been addressed in the INTEGRATE-HTA project in a step-wise process (for details, see Wahlster et al.)(9).

First, by emphasizing the importance of scoping in the context of HTA (Reference Brereton, Ingleton and Gardiner10). The relation is as follows: people can, and often do, differ in terms of how they operationalize or specify the evaluative frameworks that they (usually tacitly) use when evaluating certain situations, events, or acts. If they do, they will probably differ in what they consider relevant. If this is not acknowledged at the outset, a formal assessment risks producing outcomes that are relevant to only some of the stakeholders involved (Reference Guba and Lincoln11). To avoid this, we recommend that a scoping exercise is conducted before the actual data collection. For an elaboration on the methodology of scoping, see the paper by Brereton et al. in this issue (Reference Brereton12).

Second, integration was addressed by emphasizing the importance of logic models along the conduct of the assessment. A logic model can be conceived as a graphical representation of an evaluative framework, when specified for a particular problem or issue. It graphically represents what sort of outcomes are considered important, what mechanisms are thought to be involved in the onset of those outcomes, and what contextual issues may be at play. The methodology of constructing logic models in the context of HTA is described in more detail by Rohwer et al. (Reference Rohwer, Booth and Pfadenhauer13).

Third, recognizing that a variety of outcomes may be considered relevant when assessing the value of a specific health technology, the INTEGRATE-HTA project has tried to systematically identify research methods that may be appropriate to explore specific types of outcomes, including socio-cultural, ethical, and legal aspects. For a further elaboration of these methods, see the study by Lysdahl et al. in this issue (Reference Lysdahl14).

Finally, the INTEGRATE-HTA project has tried to systematically identify approaches that are currently used to achieve integration of outcomes in one way or another. The results of this systematic review of the literature can be found in the guidance at the project's website (www.integrate-hta.eu). When looking at this body of literature, we can see that integration is frequently conceived as commensuration: an attempt to express different value dimensions in a single common metric. MCDA is probably the best known example of this approach. It consists of the following steps: (a) Identifying a range of alternative options that may be considered to achieve a specific objective (e.g., reducing the burden of disease associated with HIV/AIDS in a specific community); (b) identifying a set of relevant evaluation criteria; (c) assessing, through empirical research, literature research and / or expert consultation the relative performance of the different options on the various evaluation criteria; and (d) weighing of the various criteria through some formal or informal procedure.

The exercise, then, results in a ranking of the various options, determined by their relative performance on the various criteria and the relative importance of these criteria. Clearly, in this approach no attempt is made at articulating an evaluative framework (such as Sen's capability approach) which can account for the relevance of particular observations or considerations when evaluating a specific healthcare technology. As such, these methods fail to clarify why and how specific aspects are relevant, whereas others are not. Commensuration may look as an attractive procedure, collapsing a wide range of value dimensions into a single metric. It produces a single winner, and has an attractive semi-quantitative ringing to it. However, to what extent values are, in fact, commensurable, or should be considered as such, is a complex and contentious issue (Reference Richardson8). It is beyond the scope of this study to explore this issue further. Suffice to observe that MCDA is a largely technical approach which seems to be at variance with more deliberative approaches to value assessment with a focus on practical reasoning, as proposed in the INTEGRATE-HTA project.

CONFLICTS OF INTEREST

The authors have nothing to disclose.