It has been argued that a central reason for health technology assessment's (HTA's) relatively limited success in regulating use of undesirable technologies is the narrowness of the evidentiary basis on which it relies (Reference Johri and Lehoux8). Many decision-makers have realized that information about clinical efficacy and cost-effectiveness is insufficient; understanding the social and ethical implications of introducing new health technologies is also vital. Over the past 10 years, it has been increasingly acknowledged within the field of HTA that qualitative research can be used for examining the perspectives, meanings, values and interests of different stakeholders, including patients and informal carers. It may thereby enable an in-depth understanding of the context in which the health technology will be embedded (Reference Leys13). Consequently, producers of HTAs now often seek to incorporate data from the local context in which the technology will be implemented, requiring integration of very diverse types of data. Although this is a well-known reality of the daily work of HTA-producers, no systematic approach for integrating and contextualizing qualitative and quantitative research findings has yet been agreed upon.
Methods for synthesizing data from primary qualitative studies, such as meta-ethnography and comparative thematic analysis, have already been described (Reference Marston and King14;Reference Noblit and Hare17;Reference Sandelowski, Trimble, Woodard and Barroso22). Recently, methods for comparing and combining results from literature reviews of qualitative and quantitative studies have also been proposed (Reference Thomas and Harden23). However, the integration of both primary and secondary research in one single review remains a key challenge.
Drawing on the conduct of such an HTA report on the psychosocial needs of people suffering from chronic fatigue syndrome in the Canadian province of Québec, this paper aims to address this methodological gap. It will describe a method for combining findings from systematic literature reviews of both qualitative and quantitative studies, with data from a primary qualitative study.
BACKGROUND
In 2007, the Québec Ministry of Health commissioned AETMIS to conduct an HTA which examined the effectiveness as well as organizational, professional and economic aspects of techniques and interventions to diagnose and manage chronic fatigue syndrome (CFS; also known as myalgic encephalomyelitis or ME).
CFS is a condition characterized by pathological exhaustion (particularly after exercise), memory problems, flu-like symptoms, and headaches (Reference Fukuda, Straus and Hickie5). Prolonged, extensive loss of physical and cognitive function is common. Furthermore, CFS-sufferers are often considered mentally ill or just plain lazy by both health professionals, friends and relatives, and this adds considerably to their suffering (Reference Gilje, Soderlund and Malterud6;Reference Komaroff, Fagioli and Doolittle9;Reference Larun and Malterud11;Reference Ross, Estok and Frame21).
The cause(s) of CFS remains unknown, although it has been linked to certain viral infections, and there is currently no effective diagnostic test or long-term treatment. The Québec Ministry of Health, therefore, specifically requested that the HTA conducted by AETMIS should identify care and services interventions relevant and useful in addressing the psychosocial needs of CFS-sufferers. This paper describes the innovative method developed and used by the authors to review evidence and integrate it with contextual data for this specific part of the full HTA. The following sections describe this method step-by-step.
MULTI-SOURCE SYNTHESIS
Framing the research
The executives of AETMIS decided that a social science approach would be useful for an in-depth examination of the psycho-social needs of CFS-sufferers. As the focus of the investigation was clear from the outset, a deductive approach was applied. This entails the application of a preliminary theoretical framework, which helps identify the core issues at play (equivalent to hypothesis testing within the natural sciences). A preliminary literature search revealed that loss of function and ability to fulfill daily roles was problematic for CFS-sufferers (Reference Ross, Estok and Frame21), and difficulties coping with stress could trigger or prolong the illness (Reference Ax, Gregg and Jones1). Consequently, theories about the impact and management of loss of function, and theories on coping strategies were integrated in the theoretical framework.
Then, VHP and PD (in consultation with PL) formulated the following research questions: (i) What are the physical, emotional, and practical problems experienced by CFS-sufferers? (ii) How do CFS-sufferers cope with these problems? (iii) What are CFS-sufferers' experiences with treatment and support interventions in Québec; which types of interventions are (in their opinion) beneficial, and which are needed to provide better support?
To address the first and second questions, a systematic literature search was conducted, while conducting interviews with CFS-sufferers was seen as a relevant strategy to answer the third question, given the need for contextualized data.
Literature Review
A systematic literature search was carried out across carefully selected databases. The databases were chosen because they included publications from disciplines likely to examine the psycho-social aspects of CFS (see Table 1). An effort was made to locate nonpublished studies such as master theses, conference abstracts, etc. Nonscientific literature such as newspaper articles, newsletters from ME/CFS support organizations in Québec, and patient testimonials was also gathered, and later used for triangulation of the findings (see page 242). The databases were searched using the search terms provided in Supplementary Document 1, which can be viewed online at www.journals.cambridge.org/thc2011015.
Table 1. Databases and Journals Searched for Literature (Reference Pedersen18)
All titles and abstracts were screened by VHP as recommended in the PRISMA statement (Reference Moher, Liberati, Tetzlaff and Altman16). Figure 1 shows a flow diagram of the retrieved and selected literature. A total of 72 papers were read in full by VHP. Study quality was assessed using a well-known checklist for qualitative studies (Reference Mays and Pope15), and tools from the Critical Appraisal Skills Programme for systematic reviews, case-control studies, and prospective cohort studies, respectively (20). Cross-sectional studies were assessed using an assessment tool developed by the Institute for Public Health Sciences at Yeshiva University (7). Twelve studies of low quality and twenty-nine studies which did not meet the inclusion criteria were excluded (18, pp. 84–88). In total, thirty-one papers were included; sixteen quantitative studies (including two systematic reviews) and fifteen studies using qualitative methods (including one meta-analysis).
Figure 1. Flow diagram of studies retrieval and selection (Reference Pedersen18).
Synthesis of Findings from Literature
Findings from the included quantitative and qualitative studies were synthesized using analysis techniques from primary qualitative studies, as suggested in various papers (Reference Baxter, Killoran, Kelly and Goyder2;Reference Noblit and Hare17;Reference Sandelowski, Trimble, Woodard and Barroso22;Reference Thomas and Harden23). Key concepts from the included studies were identified by VHP and “translated” into analytical themes using constant comparison (Reference Thomas and Harden23). For instance, findings describing physical sensations and problems associated with CFS were described under various themes in the original studies. Using constant comparison, these themes were merged under the analytical theme “Functional limitations.” Findings from the quantitative literature were subsequently grouped under descriptive headings, for instance “Correlations between depression and CFS.”
The two lists of qualitative analytical themes and quantitative descriptive headings were then compared and merged under the analytical themes if they were reported findings on the same subject. For instance, findings from quantitative studies concerning unemployment rates, workdays and financial losses associated with CFS were presented under the analytical theme “Loss of work,” which reported findings from qualitative studies about the impact of CFS on sufferers' experiences of losing their work ability. The synthesis of qualitative and quantitative literature resulted in a total of twelve themes.
Design of Interview Guide
To determine the relevance of the literature findings within the Quebec context, the interview guide included questions about each of the twelve themes found in the literature. Furthermore, the interview guide included background questions (time of diagnosis, received diagnostic procedures and treatments etc.) and questions on suggested care improvements for CFS-sufferers (Supplementary Document 2, which can be viewed online at www.journals.cambridge.org/thc2011015). As people with CFS often have cognitive problems, two CFS-sufferers from a local patient organization reviewed the interview guide to ensure the questions were relevant and the language was clear and comprehensible.
Collection of Primary Data
Individual interviews were conducted to get sufferers' personal experiences on their illness and their encounters with the social services and healthcare system.
Participants were recruited from a large, specialized practice in Montréal, where physicians distributed an information sheet to patients diagnosed with CFS. From a pool of thirty-eight volunteers, twenty interview participants were specifically selected to ensure variety across demographics and illness duration. In total, nineteen individual interviews were carried out (one person withdrew participation before the interview). Two interviews were subsequently excluded from the study as one participant had not been diagnosed with CFS by a health professional, and one sound file was technically unusable. Seventeen interviews were transcribed verbatim. Transcripts were subsequently checked against the recordings by a research assistant.
Coding of Primary Data
Findings from an analysis of interview data—usually described as “themes” in qualitative research—are topics which are considered of central importance to several interview participants. Themes are identified by “coding” the interview transcripts; they are often closely related to the theoretical framework (on which the interview questionnaire was based), but are also inductively identified through the data. For each of the twelve themes initially identified in the literature review, the corresponding interview content was copied into a separate document. Then during the coding process, new codes were created to address topics uncovered by predefined themes. For instance, during the interviews, it became clear that a theme from the literature, “Practical problems,” consisted of several distinct themes. This theme was, therefore, split into the sub-themes; “Work,” “Home life,” and “Finances.”
After the initial coding, the first ten interviews were re-coded to ensure that codes developed over the process were robust and consistent. The final list contained a total of thirty-seven codes, comprising the twelve themes from the literature and twenty-five themes that had emerged from the interview analysis.
Synthesis of Primary and Secondary Data
The analysis was conducted using a thematic approach. This involves grouping data in themes, then examining the whole data set to ensure that all the manifestations for each theme have been accounted for and compared (Reference Noblit and Hare17).
The grouping of themes was performed by reading the selected interview sections in each theme document and comparing these documents to identify connections between themes. The identified connections were used to draw up an Analytical Map (Table 2) where all the identified themes were organized in relation to each other. Themes were grouped and regrouped on the analytical map until all theme documents had been analyzed. This resulted in four overall categories each representing a “cluster” of interrelated problems, coping strategies and associated needs as they emerged in the interviews.
Table 2. Analytical Map (Reference Pedersen18)
Triangulation with Nonscientific Literature
Triangulation means that different research methods are used to collect data about the same phenomenon. This increases the thoroughness of the analysis and robustness of the study findings (Reference Pope and Mays19). Synthesizing data from a systematic literature review with findings from interviews is in itself a form of triangulation, as several data collection methods are used to gather data. However, to ensure that the study findings would have local relevance, nonscientific literature retrieved in the literature search served as a context-specific triangulation. The retrieved nonscientific documents (see reference 18 for full list), such as newsletters from CFS patient organizations and newspaper articles about CFS were read by VHP, and any themes reported in these documents were noted. The nonscientific documents helped contextualize the interviewees' perspectives by comparing and contrasting them with those made public by the broader Quebec community of CFS-sufferers.
Conclusion of Combined Data Analysis
For each of the four identified problem categories, findings from the literature, interviews and nonscientific literature were compared, and similarities and differences highlighted. In this way, the core issues which were of particular relevance to CFS-sufferers in Québec became clear. For instance, the final analysis confirmed literature findings: physical and cognitive incapacities associated with CFS are distressing, potentially leading to unemployment and serious financial problems. However, the interviews showed that for CFS-sufferers from Québec, the loss of autonomy (physically and economically) which followed from these incapacities was perceived as deeply shameful and frustrating. Another important finding, owing to the synthesis of data from different sources, was that CFS-sufferers from Québec appear to make use of certain coping strategies which they consider beneficial and which are not described in the literature (Reference Pedersen18). The triangulation confirmed that these coping strategies were not only used by the interviewed participants, but were also mentioned in patients' organization newsletters and other nonscientific literature.
Policy Recommendations
Based on the analytical conclusions made for each category of problem, recommendations were made for changes in existing care and support services, and for possible new methods of support, which could address the problems identified. To this end, existing health care, social service policies, and laws were consulted to verify anecdotal interview statements and suggest changes. For instance, offering CFS-sufferers more practical help with domestic tasks was proposed. This would allow them to conserve energy for work, child care or socializing, thereby increasing physical autonomy, social involvement and financial independence. The recommendations thus focused on interventions providing care and services suggested in the interviews (and to some extent in the literature review).
DISCUSSION
Scholars have increasingly called for, and proposed methods for retrieving and synthesizing studies that address patients' perspectives and ethical issues (Reference Dixon-Woods, Agarwal, Jones, Young and Sutton3;Reference Droste, Dintsios and Gerber4;Reference Leys13).
When a given HTA report examines an intervention that is complex and may involve “multi-factorial outcomes rather than a causal chain” (Reference Baxter, Killoran, Kelly and Goyder2), a multi-source synthesis approach has the potential to contribute more significantly to policy-making, because it addresses specific knowledge gaps in the literature while simultaneously summarizing available evidence.
For the study of psychosocial needs of CFS-sufferers in Québec, the Multi-Source Synthesis method outlined in this paper proved very useful for contextualizing literature findings with local context data. Not only did the findings confirm several issues identified in the literature; the inclusion of interviews and nonscientific literature provided additional information about coping strategies used by CFS-sufferers in Québec and organizational issues specific to the Québec society (Reference Pedersen18). For instance, CFS-sufferers' access to financial and social support was restricted because CFS was not categorized as a disabling health condition within the Québec social service system at the time of the study. Because these specific issues were identified, policy recommendations resulting from the analysis were “actionable” and more pertinent to the local context than recommendations based on a systematic literature review alone. It has been argued that access to a broader spectrum of evidence could help policy makers make difficult decisions in a more transparent and legitimate way, thus enhancing the probability of regulating use of health technologies (Reference Johri and Lehoux8). The relevance of using Multi-Source Synthesis in HTA is precisely that it combines such transparency with more dynamic knowledge acquisition, thereby ensuring that decisions about medical technologies are made on a broader and more robust knowledge base.
Although it has long been acknowledged among HTA researchers that users' perspectives on healthcare services and technologies should be examined and incorporated into HTAs, such extensive evaluations are still not standard (Reference Lehoux and Williams-Jones12). One explanation for this could be the current paucity of well-described methods and practical guidelines for how to include such data. The Danish agency for HTA (DACEHTA) has recently published additional chapters for their methodological textbook, regarding the use of qualitative data for assessing patients' perspectives on health technologies (Reference Kristensen and Sigmund10). However, these chapters appear not to contain information about synthesis of literature findings with data from primary studies. Although many HTA producers are undoubtedly familiar with some of the processes described in this study, Multi-Source Synthesis offers a step-by-step approach to synthesis of data from multiple sources (Fig. 2). We, therefore, hope that this method can help HTA producers by saving them time and resources otherwise spent on developing original methods each time they conduct needs assessment studies, while simultaneously enabling increased use and utility of qualitative data in HTA. The use of more clearly described analytical techniques, such as Multi-Source Synthesis, would enhance transparency when synthesizing qualitative and/or contextual data, thus enabling comparison with between studies. Furthermore, the method incorporates an internal validation of the analytical results through triangulation with nonscientific sources. Thus Multi-Source Synthesis can enhance the robustness of qualitative and contextual data synthesis used within HTA, while at the same time increasing the meaningfulness of findings to decision-makers.
Figure 2. Overview of Multi-Source Synthesis.
As with all methods, using Multi-Source Synthesis also has some limitations. It has been argued that combining findings from studies using qualitative and quantitative methodologies can make it difficult to develop theories about the findings, because of the different epistemological foundations of the two types of studies (Reference Dixon-Woods, Agarwal, Jones, Young and Sutton3). Thus Multi Source Synthesis is probably not suited for theory development, but instead seeks to provide nuanced and contextualized descriptions of empirical phenomena. Furthermore, Multi Source Synthesis is mostly relevant for HTA's for which analysis of user needs or patients/providers perceptions can influence the effectiveness and appropriateness of management and treatment options. Whereas HTA's on medical techniques or drugs would rely more on data from clinical trials describing their clinical effectiveness, in which case a traditional meta-synthesis would be a more appropriate method. More practical limitations are that the method requires skill in qualitative data analysis and by nature is time-consuming, making it somewhat more costly than systematic reviews alone. However, investing in research which, while taking longer, ultimately provides more relevant answers to decision-makers' questions, is often worthwhile. This is particularly true when there is no data from the local context and patients' needs are left unaddressed.
The subjective nature of data selection can also be considered a methodological limitation. It is crucial to remember that findings from quantitative and qualitative studies often examine fundamentally different things. Consequently, it is not always possible to integrate data under one theme or headline. Outlying or contradictory findings should be presented openly so their relevance to the overall aim of the research can be discussed. Finally, it is important to keep in mind that using Multi-Source Synthesis is to some extent a pragmatic activity, which takes time and skill to master. As it is also a new method, it is, therefore, both likely, and desirable that it will be adjusted to different settings.
CONCLUSION
Multi-Source Synthesis offers a new method for systematically integrating quantitative and qualitative data from literature and primary studies. This method has some limitations in that it is time-consuming and requires skill and experience in collecting and analyzing qualitative data. However, the advantages of the method are substantial. It makes it possible to inform decision-makers directly on issues important to patients and other users/stakeholders in their local area, while simultaneously allowing for comparison with findings from other countries/areas. In this way, scientific knowledge can be made immediately relevant to the local area where new services or health technologies are to be implemented. Using Multi-Source Synthesis thus enables the provision of a solid knowledge base for decision-makers, while at the same time including the perspectives of local users and other stakeholders.
SUPPLEMENTARY MATERIAL
Supplementary Document 1
Supplementary Document 2
CONTACT INFORMATION
Vibe Hjelholt Pedersen, BSc, MSc (Anth), MSc (HTA) (vibe.pedersen@kcl.ac.uk, vibehp@gmail.com), Research Associate, Department of Specialist Care, King's College London, Florence Nightingale School of Nursing and Midwifery, 57 Waterloo Road, London SE1 8WA, UK
Pierre Dagenais, MD, PhD (pierre.dagenais@inesss.qc.ca), Associate Professor, Department of Health Administration, University of Montreal, 1420 boul. du Mont-Royal, Montréal, Québec, Canada, H2V 4P3; Director Methodological Support and Development, Institut national d'excellence en santé et en services sociaux (INESSS), 2021 av. Union, bureau 10.083, Montreal, Quebec, Canada H3A 2S9
Pascale Lehoux, PhD (pascale.lehoux@umontreal.ca), Full Professor, Department of Health Administration, University of Montreal, P.O. Box, 6128, Branch Centre-ville, Montreal, Quebec, Canada H3C 3J7
CONFLICT OF INTEREST
All authors report they have no potential conflicts of interest.
