Health technology assessment (HTA) practices that separate facts and values by first collecting all facts on a health technology, as a basis for subsequent assessment in terms of values, obscure the value-ladenness of the particular features built into the technology in the first place. Also, the professional logics informing what facts to collect, and which outcome measures are relevant, are “self-evidently” adopted. Recognizing this, the VALIDATE (VALues In Doing Assessments of healthcare Technologies) approach holds that empirical analysis in HTA should always be viewed in conjunction with the interpretive frames (IFs) from which such analysis makes sense (Reference Van der Wilt, Bloemen and Grin1). An IF is defined as a perspective on a particular issue in a particular context comprising the problem definition and solution assessment of that issue; and generic beliefs that form the lens through which the issue-in-context is perceived: background theories and normative preferences (Reference Grin and van de Graaf2).
By including stakeholders and taking into account their IFs in HTAs, when considering safety, clinical, economic, and wider social and ethical implications of a health technology, VALIDATE seeks to complement current approaches to HTA in order to realize this. Additionally, VALIDATE acknowledges that, in order to make an effective contribution to decision making, HTA must promote that different stakeholders collaborate and accept the outcome, through explicitly aligning their perspectives already in defining the policy problem (3). Against this background, in the VALIDATE approach HTA researchers need the following knowledge and competences:
-
1. The ability to recognize that all types of evidence (on e.g., safety, effectiveness, social, and ethical implications of health technology) also reflect value statements and evaluate them through ethical methods.
-
2. Knowledge and skills that help them in reconstructing, exploring, and critically evaluating stakeholders’ IFs and their relationship with the questions and evidence that are to be taken into account in an assessment.
-
3. An understanding of how to integrate the above insights, and choose proper HTA methods, into an effective policy advice.
These items have been considered the desired learning outcomes of the training program (comprising an e-learning course, and internship projects), as well in an associated handbook (Reference Oortwijn and Sampietro-Colom4) and a consensus statement, developed in the VALIDATE project. These products are all freely available from the project’s Web site (www.validatehta.eu).
E-Learning Course
The e-learning course (see Figure 1) introduces participants to the VALIDATE approach. The course material consists of texts, videos, and assignments that introduce participants to concepts and methods from HTA, policy sciences, philosophy of technology and ethics, in order to improve their ability to explain and explore the normative nature of HTA, the relation between empirical and normative analysis, identify policy relevant questions, and explore and evaluate the perspectives of stakeholders. In the first three modules participants develop and check their understanding by intermediate assignments on a central case study: applied behavioral analysis (ABA) for children with autism. In the final module they write a report and give a presentation, applying the acquired knowledge and skills to the case of population screening for lung cancer. Modules were designed to be completed in one week, but participants could, and (due to work or study pressure, and personal circumstances including COVID) often did take more time to complete. Participants could enroll at arbitrary moments. Information on enrolment and participant characteristics at (www.validatehta.eu/characteristics-of-validate-e-learning-participants/).
Figure 1. Structure of the VALIDATE e-learning course, comprising four modules of a workweek study-load each.
Internships
In addition, a VALIDATE internship, was offered as an opportunity to apply the knowledge acquired to a real-life case in a national HTA organization, a policy unit or an academic or hospital-based HTA setting. Students were eligible for taking an internship after successful completion of the e-learning course, and by expressing their interest in an internship project. Actual enrolment into an internship project was based on availability of projects. Objectives are, in addition to the usual, generic objectives of being an intern, to gain practical and professional experience in performing a comprehensive HTA in a real work environment wherein the study of safety, clinical, and cost-effectiveness of healthcare technologies and their wider ethical, legal, and social implications, and views of stakeholders, are closely integrated.
This article provides first, a formative evaluation of the VALIDATE e-learning course, drawing lessons on benefits, enablers and barriers identified by the first tranche students; and, second, an exploration of how, for what purposes and with what outcomes the VALIDATE approach was used in internship work, and how that was shaped by institutional contexts.
Methods
The evaluation of the e-course draws on the feedback received from the participants who completed the course. At the time of writing, out of fifty-one enrollees, twenty-two by then completed the course; the other twenty-nine were at different stages in their training. Nine participants filled out the questionnaire, while all provided feedback orally, and sometimes by mail, some of which, though unstructured, could usefully inform the analysis. Participants were asked to indicate (on a five point scale) to what extent they had acquired nine sets of knowledge and competences; to give their qualitative views on what the course had brought them overall; and to “grade” the course on a regular 1–10 scale (see Table 1). In addition, they were asked to answer three open questions via the VALIDATE Web site (www.validatehta.eu): in what way(s) did you benefit from the course? What was the best part of the course? and finally, how could the course be improved? Students provided informed consent for using their anonymized responses for scientific reporting. Forms and full data can be obtained upon request from the corresponding author.
Table 1. Evaluation Outcomes (n = 9) of the VALIDATE E-Learning Course in Terms of Having Acquired the Competences and Skills Specified in Nine Specific Learning Objectives Plus Overall Satisfaction on Learning Outcomes
Notes. The final two rows show the grades given by participants for the course as a whole. Frequencies lower than three have been marked white; higher frequencies in increasing sheds of gray.
HTA, health technology assessment; IFs, interpretive frames.
While the limited number of participants obviously does not allow for a summative evaluation, we relied on grades given to focus the analysis on outliers and to points of strong agreement between participants. To interpret these outcomes as a basis for our formative evaluation, we used the qualitative explanations given by course participants and the answers to the three open questions as well as orally obtained feedback.
To explore the what, whereto and how of the use of the VALIDATE approach in internships and how that was shaped by institutional contexts, three internship projects will be discussed, selected to represent a variety of contexts: one performed for the Dutch National Health Council, although not as part of its regular routines for advising government; one by an HTA professional within and for her own hospital in Spain; and one within in an Italian school of medicine HTA group as an academic exercise, exploring what the approach could contribute to the intern’s PhD work. The case studies will discuss institutional context; the objective/central question of the project; the analytical argument, with due attention to the ways it dealt with IFs and values in the analysis and how it formulated the policy problem; and conclusions.
Data were obtained by, first, reading the final report of the internship and mapping how the VALIDATE approach was followed in an open, structured format. It mapped, first, the context in which the HTA was undertaken; second, the process in which the intern portrayed the “problematic conditions,” elucidated the variety of “problem definitions,” and formulated the research question and/or objectives of the HTA; third, whether and how the intern used VALIDATE to reconstruct IFs and do value analysis; and, finally, findings (including the policy problems and the solutions arrived at). Next, the intern was interviewed by the first author of this article (J.G.) to supplement these data as well as to gain information and insight in the HTA process, including key choices made. In one case, the video recording of an intern presenting her work at an international HTA conference was used as an additional source. In two cases also the supervisor was interviewed and provided input, respectively. Due to COVID-19 restrictions all (open, structured by the format) interviews were done online, through Zoom or WebEx videocalls.
Results
E-Learning Course
The results of the questionnaire amongst e-learning participants are presented in Table 1.
While most participants were largely positive on their learning achievements, one participant had mixed feelings at best. While this participant graded the course with a 5 (on a 10-point scale), the participant also noted “agreed” when asked whether “overall, has the course improved your knowledge and skills?” and indicated that the course would be more enthusing when web lectures would complement texts and assignments. Personal contact is also appreciated by other participants: several of them declare, unasked that they have appreciated the help, feedback and engagement of the course leader. The participant grading the course with a 5 responded to an open question: “I have never had a course in HTA, so I learned a lot of basic things and especially the final assessment and the presentation were useful for me since I learned how to make such a HTA report and think about it.” Apparently, without some background in HTA (and underlying fields like social science and ethics) taking the e-learning course is just asking too much: this participant felt to have learned too little on two topics on which most other participants thought they had learned a lot (“identify policy relevant questions… ‘’ and ‘explain how stakeholders’ commitments to underlying values and assumptions affect their initial definition of a health problem… by … reconstructing interpretive frames”). Indeed, others frequently mention that the course in interesting ways complemented or enriched their existing knowledge and insights on HTA.
Starting with learning achievements pertaining to objectives I and II above, an important first point of agreement between most participants is that virtually all felt to have acquired really good knowledge and skills to “explain the normative nature of HTA and the relation between empirical analysis and normative inquiry.” One typical open question response is “I am a biomedical student and … was really in favor of the classic HTA approach (I regarded cost-effectiveness and safety as the most important). However, this course offered me the ability to think on a whole new level, by making me take other aspects like ethical and social aspects into account from the start of the inquiry… it is really important for scientists like me to also think of these other aspects instead of only using their own conservative view.” Similarly, another participant wrote: “The ‘classic’ models of HTA [hold] that it is possible to collect and synthesize facts in a value-neutral way…, and that value judgments enter the process only later… The VALIDATE course really broadened, or even changed, my view on HTA by showing that value commitments are already operative during the assessment stage (by defining what is and what is not considered relevant).”
This may for some partly be related to knowledge pertaining to IFs: the best part was the “part concerning the reconstructing of IFs… in which it is made clear that the value commitments (and its relation to the background theories) of the stakeholders play a very important role in their positions and thus should be taken into account from the very start.” Another student noted: “For me, learning how to reconstruct an interpretive frame was the most instructive and interesting part of the course. Being able to take into account the beliefs and values of different stakeholders… is definitely a great skill to have as biomedical researcher.” On learning about “ethical argumentation models to evaluate underlying normative statements and values” answers were still positive, but most participants assessed their achievements here somewhat lower. Responses to open questions do not explain this, but interviews and provided feedback revealed that many found this a difficult course element. The assignment devoted to this was seen as a tough one, as applying the relevant model (of specifying norms, as developed by Richardson (Reference Richardson5)), requires that one can identify the general norms invoked in a particular discussion, for example, to identify the potential value of ABA for autism. In addition, to be able to formulate different specifications of these norms also appeared challenging: it involves “a lot of reading” and discerning a variety of interpretations of the same norm, to develop the competence in interpretation. “It would be nice if one of the assignments would be together with another student.”
Regarding “reconstructing, exploring, and critically evaluating stakeholders’ interpretive frames and their relation with HTA questions and evidence” per se, most (5/9) participants indicate that they have acquired really well the associate knowledge and competences to reconstruct IFs. “The main thing I’ve learned during the course was how different backgrounds can affect definitions of health problems. I realized this particular part of research is very interesting as you get to be involved with ethical dilemmas and different stakeholders.” About the same number feel they have achieved really well (4/9) or reasonably well (5/9) a capacity to “critically appraise stakeholders’ background theories.”
Finally, in terms of overall learning outcome III (understanding how to integrate the above insights, and choose proper HTA methods, into an effective policy advice—cf. introduction), all indicate to have learned to “use concepts and methods from policy science” (4/9x agree; 5/9x strongly agree), to “choose an appropriate approach to policy analysis” (4/9 agree and 5/9 strongly agree, including the outlier), and (except the outlier) to “decide on the appropriate scope of an HTA” (4/9 and 4/9, respectively). One typical quote mentions the “how” and “whereto” of integration: “it is important to identify the coherence, differences and commonalities between these frames (particularly in the second order beliefs) in order to identify the aspects that have most potential for getting closer to a solution for the problem.” Another illustrates part of the struggle occasionally experienced: “when it comes to medical sciences students, I think that the level of English used in the course is sometimes slightly too high for them… would have liked … more background information… on social topics like these. The same goes for the philosophical concepts … level of the course was above average when comparing it to other biomedical courses … However, this is not necessarily a bad thing (…).” Several participants mention the use of one example (autism) throughout the course and the intertwinement of reading and writing assignments as key assets to understand (especially the toughest) notions; two others mention the use of another case (lung cancer screening) in the final assignment as promoting proper understanding.
All in all, most participants strongly agreed they had achieved a capacity to “identify policy relevant questions to be addressed in a health technology assessment” (7/9). This appears the most practical and thus tangible implication of understanding differences between stakeholders. Regarding the acquired insight and competence to choose proper HTA methods assessments were again evenly (3;Reference Oortwijn and Sampietro-Colom4) distributed over “agree” and “strongly agree”; feedback conversations indicate that some found it difficult to make the connection between identifying the type of policy problem and deciding upon which standard HTA methodology may be appropriate to address such policy problem. This topic admittedly, gets relatively little attention in the course and ways for further elaboration of the subject will be explored by the VALIDATE faculty.
These outcomes are confirmed by the answers given by six interns (others than the nine above) to open questions in the online evaluation form of their internships (for details, see www.validatehta.eu). [The internship] “allowed me to apply the notions studied during the VALIDATE course… I found very interesting” (#Reference Van der Wilt, Bloemen and Grin1); “allowed me to grasp in a more comprehensive way the novelty of the VALIDATE approach: formulating what the problem is and also finding myself dealing with a different problem from the one I was expecting” (#4); For #2 the best part was the “opportunity to see multiple points of view regarding a given health issue, trying to understand what are the reasons of the various actors involved and giving weight and value to their ideas”; for #6, “the greatest discovery was that there are other ways of doing research …, which touches upon personal views and stories. This was an aspect I really missed during my studies.”
Lessons from the Internships
Below, we draw on three internship projects to explore the variety of purposes for which the VALIDATE approach was used, how it was used and how the wider (national health system) and narrower (place of internship) institutional setting mattered.
Internship #1: Big Data for Health Screening
The first intern (I-1) was a Master Student in Biomedical Sciences at Radboud University, Netherlands. He was advised to do an internship with the Health Council of the Netherlands, an advisory body providing solicited and unsolicited advice to the government and Parliament, on the emerging field of personalized population/artificial intelligence (AI) based screening, an issue that was on the Health Council’s 2021 annual work program (6). A Senior Policy Adviser (SPA) of the Health Council was interested in exploring VALIDATE’s potential in the Health Council context, yet also realized that using VALIDATE for an official Health Council project would impose a lot of extra demands and constraints on I-1’s work. The SPA therefore gave I-1 room to do his own project, shaped as an advice to the Health Council, while providing him ample support (interview with SPA).
I-1 started the analysis by outlining the key conditions constituting the issue in his internship report. Referring to an earlier report by the Health Council, the first two were developments in the combined use of big data, particularly genomics, and risk stratification, in particular screening for early stages of disease conditions and the expectations of risk prediction and stratification based on big data. Third, gene expression regulation turns out unexpectedly complex. On diverse perspectives, I-1 summarized the view of Radboud University Professor of Philosophy, Digitalization and Society, Tamar Sharon (Reference Sharon7): “The use of AI in health care in general has however been related to multiple ethical, legal and social issues, and thus to human values, which probably results in different perspectives on this topic and thus in a serious debate.” For instance, regarding the “Googlization of health research” (GHR), various stakeholders mention the “common good” as an important value in assessing the usefulness of GHR but they have “different conceptions of the common good and thus different perspectives on GHR.” Thus, the central question became: “what are the different perspectives and values related to the use of AI for population-based screening in the Netherlands and what implications do they have?”
After selecting stakeholders on the basis of document analysis, I-1 designed the HTA to include, successively, reconstruction of typical interpretive frameworks, a literature study to support, deepen and critically scrutinize the background reconstructed theories; and validate the results through an expert interview with professor Sharon.
The intern noted that actually executing the HTA proved quite challenging. I-1 pointed to the difficulty of “navigating between the languages of science and of practice.” While VALIDATE draws attention to value debates on health technologies, I-1 was not too acquainted with such debates that are often not well reflected in scientific literature; here newspaper clippings and other documents presented by the supervisors were gratefully received and turned into literature searches. Second, the common (and prescribed) format for the report (“Problem–analytical framework–method–results–discussion–conclusions”) “felt sometime like a constraint on ‘natural flow of my thoughts’ when following VALIDATE: in this approach the problem was not given at the outset, but to be found and evolving over time” (int I-1). Finally, scoping, that is, sensibly integrating various problem definitions from different IFs into an overall research problem and questions, appeared difficult.
In spite of these struggles and of being initially overwhelmed by many IFs, I-1 identified two crucial factors: “at some stage, it struck me [that] in diverse scholarly articles I had been reading as well as in newspaper articles and interviews, ‘explainability’ [i.e. the extent to which the results of an AI application can be explained] and trust appeared really important.” These two notions together proved a useful focus for reconstructing a manageable set of IFs. So, his understanding of underlying factors that shaped views and societal debate was one key help in demarcating the problem; the other was the VALIDATE recommendation to be policy relevant, which helped to determine a proper focus.
I-1 thus arrived at four ideal typical IFs. Two consider AI as basis for screening but with still insufficient trust from end-users and stakeholders; one attributes this to the lack of scientific evidence (IF #1), or the too limited “explainability” of AI (IF #2). The other two expect personalization of population-based screening by means of AI, but either think this will imply more responsibility for end-users than they want to or can take (IF #3), or consider it not feasible in population-based screening as currently organized (IF #4).
Carefully reflecting on his results, I-1 concludes that “trust is a main requirement related to the use of AI for population-based screening, which is generated by valuing empirical evidence on the health gain and benefit-harm ratio and the concept and value of explainability.” Pointing to the existence of different perspectives and uncertainties on explainability, the report recommends making policy makers aware of these IFs. In order to address challenges related to responsibility, purposeful collaboration is recommended between policy makers, (potential) end-users, AI-developers (and other scientists) in policy development and research. Finally, to deal with the challenges on data collection, I-1 recommends to focus future research on explainability, side-effects and sensitivity of data (sharing).
SPA shared the observation that the organizational routines and culture may hamper adoption of a method such as VALIDATE, which relies on acknowledging the existence of different expert perspectives, rather than claiming “the” view of “science.” “This is a dilemma. It is rooted in the Council’s inclination to base authority on expertise, which is understood as objective, which is shared by the field of healthcare/medicine’s and reinforced by the societal skepticism on expert advice. The dilemma is, of course, how to solve problems of expertise-based decisions while maintaining expertise’s central role.” (int SPA) Having noted this, the SPA continues to ponder on using the VALIDATE approach “as a framework to evaluate earlier work by the HC on issues, to show how value judgments are implicit in these exercises, and how they got there.”
Internship #2: Assessing OntoPharma, a Clinical Decision Support System
A second intern (I-2), a biologist/biomedical scientist by training, is a junior HTA scientist at the hospital-based unit of Hospital Clínic Barcelona, Spain. She explored VALIDATE by applying it to a specific clinical decision support system (CDSS), OntoPharma, being studied by the hospital. The intern mentioned that the pharmacist, who was in charge of the study, was very interested in the VALIDATE approach, having “never seen an approach like VALIDATE to her projects.” (int I-2) The pharmacist was very stimulating and supportive, establishing contacts with stakeholders and collaborating in data analysis and abstract submission of the results.
A central problem on the level of (international) health care is the frequency of adverse events (AEs) linked to medicines in hospitalized patients, causing 1 death/100,000 people/year in the EU according to the WHO (8). In hospitals, key conditions are that “proper prescription behavior depends on the ability of the clinician to focus all the attention on memorizing and analyzing large amounts of data… [while] clinicians are aware of drug contraindications, drug interactions, dosage adjustments, etc. … this information is not always taken into account … However, CDSS also have limitations…: reporting of inappropriate alerts, excess number of alerts (leading to alert fatigue), or lack of interoperability with different electronic prescription systems,” as described in the internship report. I-2 is intrigued: “Everybody knows it, nobody mentions it.” (int. I-2).
Against this background, I-2 undertook an HTA to evaluate CDSS, in particular OntoPharma. More specific objectives were to provide an “integrative framework … using VALIDATE methodology, to gather information on the consensus level on the existing empirical evidence… and on the value and acceptability of CDSS within the healthcare professional’s community; to gather a deeper understanding of the different stakeholders perspective (nurses, physicians, patients, pharmacists, and IT) on CDSS.”
I-2 proceeded with a review of 12 studies, selected from 202, on the cost-effectiveness, outcome measures and values articulated and barriers and facilitators for CDSS optimal implementation, to prepare the questionnaire for a round of interviews. Noting that literature (implicitly, by default) only comprises the views of physicians, the intern decided to interview, in addition to two physicians, three medical informatics experts, two nurses, two clinical pharmacists, a CDSS company representative, an Electronic Health Record developer and a consultant. All interviews were recorded, transcribed, and coded to identify and illustrate the key themes of concern.
The intern concluded that people are generally very open to give their opinion on a topic of their concern. Also, I-2 signals major differences between perspectives in scientific publications and different stakeholders’ problem definitions and evaluation frameworks. Reflecting on the project, I2 sees the added value of VALIDATE as providing “a different mindset on HTA, which especially helps to understand much better diverse stakeholders’ perspectives and thus to understand better the problem you are working on.” (int. I-2). The hospital supervisor’s evaluation stipulated that these outcomes precisely matched her expectations on the internship, which “has been able to ‘open the mind’ of other researchers/clinicians to have a broader view on HTA.”
Internship 3: Umbilical Cord Blood Collection
The final intern, I-3, is a PhD candidate in Biomedical Sciences and Public Health at the Catholic University of the Sacred Heart, Italy, whose research focuses on biobanks management and the ethical, legal, and social issues involved. Together with her supervisor (S) the intern decided to focus the internship on umbilical cord biobanking in Italy. The research was prompted by her personal observation that there seemed to be little participation from Italian women in collecting umbilical cord blood (UCB) upon delivery (int I-3). This original thought was confirmed by literature search that showed that only 1 percent of Italian parents opt to donate to the public UCB biobank (Reference Grieco, Lacetera and Macis9).
In Italy, only public donation of blood (i.e., collection for storage and use intended for others) is supported by the National Health Care System and legally allowed. There has been extensive staff training and public information campaigns. The biobanking is public, collection is on voluntarily basis and not remunerated. Only subjects who, at the moment of birth, are already (likely to become) in need of their own stem cells are exempted from these rules. Those who wish to store their child’s UCB privately must arrange and pay for storage abroad. The main problem condition is that low participation yields low cost-effectiveness.
I-3 started with a literature search which to her surprise did not confirm her initial expectation that poor participation would derive from some major controversy around the practice itself. Parents and health care providers (both those in responsible for mother and baby before and upon delivery, as well as those who would benefit from the donation for the treatment of their patients) do not have objections to the procedure, besides the fact that a protocol must be followed in order to perform it safely for mother and baby. The question thus became: how it is that a safe, efficient, ethically, and socially acceptable and accepted procedure that could bring substantial benefit nevertheless sees such limited participation? Does it even make sense for the healthcare system to continue such practice and policy?
To frame the issue, I-3 formulated questions like: who exactly can be identified as a stakeholder in this scenario? Do these stakeholders deem the practice of public UCB biobanking to be safe and desirable? Do the stakeholders have the means to offer/access this service? What is the context in which these stakeholders act? A second round of literature search was performed in order to determine what are the barriers and facilitators for a better implementation of the collection system was found to be necessary.
The search words used were terms related to UCB banking combined with terms on women’s/expectant parent’s knowledge. No publication date limits were set. A total of thirty-two articles were found. While all of them appeared relevant, most of them were very specific about a single area (at times, a city or a hospital). Hence, more weight was given to those pertaining to the Italian situation. Papers referring to the issue in other countries were also taken into account when relevant.
One thing that became evident through the internship report was the fact that no literature is found that discusses in depth expectant parents/mothers’ contexts when they are learning about this practice, formulating their choice at home, giving their consent, and then finally confirming their position directly after birth, in the delivery room. The diversity between practices in different hospitals, reported in some articles, is also not really investigated. In order to better evaluate the issue contacts with the stakeholders would have been beneficial to the report. While it was not possible to do so within the scope of the internship, the intern recommended that “future research should promote interaction among the various stakeholders.”
Discussion and conclusions
Based on the findings, we conclude that the VALIDATE training program has significant added value according to e-learning participants, interns and supervisors. Especially appreciated were the capacity to uncover and assess values implicit in evidence, and understand an issue better by knowing the variety of problem definitions prevailing amongst stakeholders and understanding their relations by reconstructing IFs. Course participants indicated they feel they have learned a lot on these issues. Internship experiences and results indicates that, apparently, the course sufficiently prepares for using VALIDATE in differing real-life HTA contexts, integrating the VALIDATE approach with standard HTA approaches. We conclude that the VALIDATE e-learning course (i) shed a different light not only on a health technology and its use, but also on the policy problem at stake; (ii) is most productive as a course that deepens and broadens pre-existing HTA knowledge and competences; and (iii) requires some prior training in standard HTA.
All internship projects involved, indeed, redefining “the” problem in unexpected but productive ways, by being more attentive to different perspectives (of e.g., nurses, pharmacists, and I-2), to criteria that are often neglected in conventional HTA (like “explainability,” I-1) and to diversity in user contexts (I-3). Having said so, especially two learning objectives appear relatively demanding. First, course participants find it more difficult to analyze how IFs shape definitions of the problem and understandings of evidence. Maybe VALIDATE should comprise more than one exercise on this, also in view of the fact that the use of argumentative methods in ethics, including the method of specifying norms, is considered challenging in literature (Reference Demarco and Ford10). Second, sometimes a deeper understanding of the differences between IFs could have been provided by surfacing differences of view regarding (i) the preferred relations between health professionals and patients, and (ii) the determinants of health and diseases that are emphasized in background theories—compare Van der Wilt (Reference Van der Wilt11). While Grin (Reference Grin12) discussed them in the VALIDATE handbook, these elements are also somewhat under-emphasized and under-exercised in the e-learning course. Maybe more fundamentally (as it underlies the difficulty in considering these points, which simultaneously is the main reason for putting less emphasis on them in the course), analyzing these issues also goes beyond standard biomedical curricula and has limited basis in literature—in that sense, this observation also contains a message to the field.
The evaluation also shows some challenges in “doing” VALIDATE. First, interns point to a tension between applying VALIDATE, discovering and scrutinizing “what the problem really is,” and classical styles of research reporting based on a stable, unambiguous problem. Second and relatedly, they struggle with navigating between the worlds of science and practice. Third, existing studies have important limits that hamper their use as secondary sources for reconstructing interpretive frames and ethical assessment: they rarely pay attention to diversity of perspectives within the discipline, the value dimensions are under-articulated at best, and problem definition is often routinely done, and not problematized at all. While we may take this as an indicator for precisely the relevance of the VALIDATE approach, it leads to a need for (potentially time consuming) additional data gathering. Fourth, and complicating this problem, precisely data gathering on and evaluation of value dimensions is experienced as really challenging by most participants. It may help to point users of the e-learning course more explicitly to the VALIDATE handbook chapters by Sacchini and Refolo (Reference Refolo and Sacchini13), and Hofmann and Sandman (Reference Hofmann and Sandman14) that deepen the material on this issue; but this experience also suggests a need to pay more attention to these issues in standard HTA curricula (see also Gagnon et al. (15)). Similarly, it may help to point more urgently in the course material to the Oortwijn’s (Reference Oortwijn16) chapter on scoping in the VALIDATE handbook.
Taking these things together, we think it is worthwhile to monitor and publish future uses of VALIDATE, their added value and the limits encountered. With due modesty, we encourage the field to scrutinize such experiences and lessons, and consider whether VALIDATE should become a more common element of the field, as a contribution to address normativity of HTA as reflected in the new definition of HTA (Reference O’Rourke, Oortwijn and Schuller17) and the views expressed in by Gagnon et al.’ s (Reference Gagnon, Legault and Bellemare15) survey. Essential for that would be to integrate it in curricula and to make attention to ethical assessment and diversity in background theories much more common in HTA research.
Funding Statement
The VALIDATE project (www.VALIDATEhta.eu) was financially supported by the Erasmus+ Programme of the European Union, Contract Number contract 2018-1-NL01-KA203-038960.
Conflicts of Interest
The authors declare that they have no conflict of interest.
