Many scientific uncertainties about the effects of treatments in health care have been identified (1). The uncertainties may for example result from a lack of research or that the research conducted is scientifically flawed (Reference Altman2;Reference Chalmers and Glasziou3). An important aspect is that the researchers address questions that are relevant to the patients and clinicians. This is not always the case, as illustrated by a study showing that patients with osteoarthritis of the knee want more trials on surgery, education and advice, physical therapy, and complementary therapy, instead of drug trials (Reference Tallon, Chard and Dieppe4). However, most of the randomized controlled trials in this patient group comprise evaluations of pharmaceutical interventions.
Moreover, research priorities are generally not set by a democratic process. Resources are usually awarded to research projects submitted to funding bodies by scientists (researcher-initiated studies). Such a “responsive” mode of funding may overlook the views of the people affected by the condition in question and is more appropriate for funding of basic science projects (Reference Lloyd and White5).
How can patients and clinicians influence the research agenda for a specific condition? The nonprofit British organization James Lind Alliance (JLA) was established in 2004 and is coordinated by NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC) (6). JLA supports the process of convening patients and clinicians in joint research priority setting exercises (Reference Cowan7). Patients and clinicians form so called priority setting partnerships (PSPs), which identify, confirm, and prioritize the ten most important unanswered questions about a particular condition. The process developed by JLA is described in detail in the JLA guidebook (8). To date, forty topics have been subjected to prioritization with the support of JLA.
A health technology assessment (HTA) report published in 2013 by the Swedish Council on Health Technology Assessment (SBU) disclosed uncertainties about the diagnostic accuracy (in terms of sensitivity and specificity) of all tools used in Sweden to establish a diagnosis of ADHD (9). The same applies to all thirty nonpharmaceutical treatment methods used for ADHD in Sweden (in terms of benefits, risks, and costs). Cognitive behavioral therapy could be effective when provided as adjuvant therapy to adults with ADHD who have persistent symptoms despite drug treatment (9). However, this has been shown in two studies performed by the same group and must be confirmed in independent studies.
With respect to pharmaceutical interventions, there is some evidence to support medication with methylphenidate (psychostimulant) or atomoxetine. In children, adolescents, and adults with ADHD, both drugs relieve symptoms during short-term treatment (3 weeks to 6 months). However, the effects of long-term treatment have not been assessed.
The aim of this project was to gain an insight into the perspectives of consumers (i.e., those with ADHD and their close relatives), as well as school, health, and correctional services personnel (here defined as “professionals”), as to which ADHD-related questions warrant most urgent investigation. The method for achieving a top ten list of the most important unanswered questions was based primarily on the JLA guidebook, with certain modifications as described below. A previous prioritization of scientific uncertainties relating to ADHD has been published (Reference Gaynes, Christian and Saavedra10). The major and most important difference in this prioritization project is that only consumers and professionals were involved whereas the previous prioritization also involved funders and researchers. The exclusive involvement of patients and clinicians is a key characteristic of the JLA method.
METHOD
Fourteen participants were recruited by SBU to form a working group, for the purpose of prioritizing scientific uncertainties related to treatment for ADHD. The working group consisted of seven individuals with ADHD themselves and/or closely related to someone with ADHD (i.e., parents with or without ADHD diagnosis), along with seven professionals (two psychologists, a psychiatrist, a primary care physician, a corrective services officer, a school counselor, and a specialist educator). The gender, age, and geographical distribution of the participants was balanced and different professional and personal perspectives were represented. Three of the participants represented the Swedish interest organization Attention, and the others were either members of Attention (but did not participate as representatives of the organization) or nonorganized. All members were asked to fill in a conflicts of interest form.
The JLA guidebook was followed for the prioritization process (8). However, the JLA approach was not adopted for identifying unanswered question of concern: instead the working group prioritized the uncertainties related to treatment methods listed in SBU's report on ADHD (9). All of the covered methods are used in Sweden to some extent and range from pharmaceuticals to psychological and psychosocial therapies, including multimodal treatments and various support and educational programs. The thirty-nine uncertainties are presented in Table 1, and are also listed in a database on SBU's Web site (11).
Table 1. Long List of Uncertainties Including Ranking, Scores, and Number of Votes by the Consumer Group and Professional Group, Respectively, as well and the Combined Ranking of Both Groups (i.e., Total Ranking) after the Interim Priority Setting Exercise
Note. The table contains the long list of uncertainties included in the priority setting (39 in total). During the first step of the priority setting, each member of the working group independently selected his/her ten most important uncertainties from the total of 39. Rank 1 was the highest and received a score of 10, rank 2 a score of 9, and so on. The uncertainty which received the highest score in each group (consumers and professionals) was ranked as 1, the one with the second highest score ranked 2 and so on. The scores from both groups were then combined into a total ranking, of which the top 20 items were subjected for further prioritisation during the workshop. The table is divided into rankings 1–20 after the interim priority setting exercise (i.e. short list), ranking 21–34, and the 5 uncertainties that were not given a ranking by either group.
Effects refer to both positive and negative effects. In this context, a positive treatment effect refers to improvement in the core symptoms of ADHD (inattention, hyperactivity, and impulsiveness).
a There is evidence that both drugs are more effective than a placebo; however, it is unclear whether one drug is better than the other.
b Includes Barkley's parent training program, KOMET, COPE, and ”The incredible years.”
c “Teacher support,” given for example by counsellors, social workers or psychologists.
d Including neurofeedback.
C, children; Ad, adolescents; A, adults; no., number.
The working group was not required to consider the feasibility of conducting clinical studies to bridge the uncertainties. For example, issues such as resources, research ethics, or research methodology were not taken into account.
As some participants might have reading difficulties, it was important to adapt the written material distributed to the working group. One of the members of the working group provided feedback on the material and recommended modifications that would make the written text more accessible. For example, the text was shortened and revised to be less formal.
SBU's role in the process was merely to administrate and facilitate. SBU did not have a vote in the prioritization. It took approximately 4 months to complete the prioritization process.
Interim Priority Setting
To reduce the thirty-nine uncertainties to a short list of twenty items, an interim priority setting exercise was undertaken in May 2014. Each member of the working group independently (or together with colleagues or relatives sharing the same perspective) selected his/her ten most important uncertainties from the total of thirty-nine (Table 1), and returned it to SBU by e-mail within 3 weeks. Rank 1 was the highest and received a score of 10, rank 2 a score of 9, and so on. The scores submitted by each member were calculated by SBU on a spreadsheet (consumers and professionals separately). After the scores from both groups were added together, the twenty uncertainties with the most points (based only on scores, number of votes were not taken into account) were compiled into a short list for the workshop (Table 1).
Before summer, the members of the working group received the shortened list of uncertainties, in no particular order, by e-mail. They were again asked to select their top ten most important uncertainties, which they should bring to the final priority setting workshop held on 25 August 2014. In preparation for the workshop, the working group received a modified version of SBU's ethical guidelines to consider ethical aspects (but not research ethics) that might have a positive or negative influence on prioritizing of certain research questions (Reference Heintz, Lintamo and Hultcrantz12).
Final Priority Setting
This stage of the process involved a 1-day workshop, conducted in two sessions. The initial session consisted of group discussions in three small subgroups comprising half consumers and half professionals to achieve balance: two subgroups with four individuals (two consumers and two professionals), and one subgroup with five individuals (two consumers and three professionals). Six facilitators (SBU officials) moderated the discussions (two in each subgroup). Facilitators ensured that everyone had the opportunity to be heard, took notes from the discussions and documented the scores of the rankings. For the discussions, each of the twenty uncertainties was written on separate A4-cards. Ranking information from the interim process was printed on the back of the cards, including the total rank as well as the rankings assigned by the consumers and the professionals presented separately.
The subgroups were charged with compiling a ranked list of ten uncertainties they believed should be prioritized for further research. To begin the process, each individual presented their choices, including a short justification, to their subgroup. Subsequently, the members of the subgroup worked together to build a consensus that would best represent the views of the group. At the end of the exercise, two of the facilitators summed up the ranking scores of all three groups. Uncertainties that had not been given a ranking were removed.
The second session consisted of a plenary discussion involving the entire working group. The goal of the plenary discussion was to arrive at consensus on a final top ten list. The members of the working group gathered around a large table where the A4-cards had been distributed by the facilitators according to a “diamond-shape,” with higher ranked questions at the top, medium ranked questions in the middle and lower ranked questions at the bottom. The members then moved the cards around during the discussion and finally constructed a linear shape with a ranking from one to ten. Justifications for choosing the top ten research questions were noted by the facilitators during the discussions.
RESULTS
Interim Priority Setting
Of the thirty-nine uncertainties, thirty-four were given a rank by at least one person. Eleven items were chosen by only one or two individuals and thirteen by three or four individuals. The remaining ten were selected by between five and ten different individuals. The twenty uncertainties with the highest scores are presented in Table 1. The total scores among the top twenty ranged from 17 to 67, and each was selected by at least three individuals, with the exception of one which was chosen by only two. The highest consistency among the working group was seen among the top nine items which were all selected by at least seven individuals.
When the prioritization scores were broken down into the two subgroups, twenty-nine of the thirty-nine items were given a rank by the consumer group and twenty-eight by the professional group. The professional group selected five items that the consumer group did not (life style advice, treatment with atomoxetine in combination with motivational interviewing and comorbid substance use disorder, multimodal therapy with comorbid substance use disorder, psychodynamic therapy, Marte Meo), and the consumer group chose six items that were not ranked by the professional group (family therapy, dialectical behavioral therapy, behavioral therapy, milieu therapy, time out, magnesium). There was more consistency among the top five items in the merged list; they were all ranked within the top ten in each group.
Final Priority Setting
Thirteen people participated in the 1-day workshop (one of the members representing the consumer group was unable to attend). After the subgroup discussions, fifteen uncertainties remained, namely teacher support, atomoxetine versus methylphenidate, multimodal therapy, risk of developing drug dependence later in life when treated with methylphenidate during childhood, parental support programs, network therapy, life style advice, computer-aided working memory training, melatonin, cognitive behavioral therapy, methylphenidate medication in substance abusers, supported conversation, psychoeducative treatment, and omega 3 and 6 fatty acids.
During the subsequent plenary discussion, the working group decided that the four items with the highest ranking from the group exercise should be included automatically. The working group considered that the question warranting highest priority was whether treatment of ADHD with methylphenidate during childhood increased the risk of drug dependence later in life. Parents and healthcare personnel alike believed it was important to know for certain if this treatment is linked to drug addiction. In part, to avoid unintentionally harming these children, and in part to address the heated public debate this issue has generated.
Another urgent topic was the influence of teacher support, because the school has such a pronounced influence on a child's future. The group argued that the right support at school could even make other interventions redundant. Multimodal therapy is widely used and comprises many different forms of psychological treatments (e.g., cognitive behavioral therapy, stress management groups, family therapy, and aggression replacement training), often in combination with pharmaceutical treatment. It was, therefore, deemed important to determine whether these forms of treatment are effective. The question of which pharmaceutical, atomoxetine or methylphenidate, is most effective and causes fewest side effects was also ranked among the top four after the subgroup discussions.
Discussion then turned to which five items should be removed. There was some debate as to the most important forms of therapy to be included in the final list. Finally, family therapy, network therapy, and cognitive behavioral therapy were removed, as the group reasoned that they were included in the research question regarding multimodal therapy. One participant argued for life style advice; however, this was eventually excluded. The final item to be removed was supplementation with omega 3 and 6 fatty acids although there was a debate about the importance of research on this topic. One view was that the risk of side effects is probably low and study of the topic is, therefore, less urgent, whereas others argued that there may be risks associated with doses exceeding the recommended daily intake. Some considered it to be an important question from an equality perspective, as not everyone can afford the treatment.
An important topic for priority was the effect of methylphenidate in cases of substance abuse, due to the lack of consensus as to whether those who suffer from substance abuse should be treated with pharmaceuticals. Parents of children with ADHD carry a high burden in meeting the child's needs. Many parents have a diagnosis themselves, which can further complicate the situation. Because there are several parental programs available, the working group considered establishing the positive and negative effects of such programs a prioritized research question. Supported conversation is widely used in Sweden; thus, it was deemed important to understand which elements were key to making this intervention helpful to individuals with ADHD. The group further motivated the inclusion of supported conversation to the final top ten list because it is relatively inexpensive and uncomplicated to deliver.
Computer-aided working memory training is a treatment that could be widely distributed if proven effective and could be an appealing method for many. Psychoeducative treatment is widely used and, therefore, considered important to evaluate. Finally, melatonin is inexpensive and has been used for a long time to treat sleeping disorders. However, it is not an approved pharmaceutical in Sweden; thus, each prescription requires approval by the Swedish Medical Products Agency. Research was considered important because pharmaceutical companies are not likely to perform research on an inexpensive off-patent compound such as melatonin.
Once agreement had been reached on the final ten items, ranking them from one to ten was the least time-consuming. The final top ten list was supported by all members of the working group (Table 2).
Table 2. Ten Most Important Research Questions Including Changes of Ranking Order during the Prioritization Process
Note. Numbers represent total ranking orders after the interim step (i.e., consumer and professional groups combined ranking), small group discussions during the workshop (combined ranking of the three groups), and the final ranking after the plenary discussion. 1 = highest priority. Effects refer to both positive and negative effects. In this context, a positive treatment effect refers to improvement in the core symptoms of ADHD (inattention, hyperactivity, and impulsiveness).
Among ethical topics discussed were equality of access to treatment (for example, who is offered a parental support program, teacher support, and computer-aided working memory training). Another topic of discussion was the influence of special interest groups who are opponents or proponents of certain treatment methods, for example, methylphenidate medication for children or those with substance abuse problems.
When comparing the top ten after the interim step with the final top ten list, the ranking order was different, but only two items on the interim top ten list were excluded from the final list, namely cognitive behavioral therapy, which was thought to be covered by the question about multimodal therapy, and lifestyle advice (which was chosen by four members of the professional group in the interim step, but only one person persisted in wanting it retained as the workshop progressed) (Table 2). Instead the uncertainties about melatonin and psychoeducative therapy were upgraded from rankings of 15 and 19, respectively, after the interim step, to be included in the final top 10.
Some items retained almost the same rank throughout: from the interim step, through the merged list after the group discussions, to the final list, such as the effects of multimodal therapy, parental support programs, and computer-aided working memory training (Table 2). Others shifted more dramatically during the various stages of the process, for example, the effects of methylphenidate medication in substance abusers, supported conversation, and psychoeducative treatment.
DISCUSSION
The results of the workshop show that representatives of consumers and those of health, school, and criminal correctional services can convene and reach consensus on a top ten list of research priorities for treatment of children, adolescents, and adults with ADHD. Despite differing views on some topics, the atmosphere during the workshop was friendly and collaborative. The overall aim of the process was that everyone should agree on the majority of the questions in the final list and could accept the list as a whole, even though not everyone would agree with all ten questions or the specific ranking order. The most difficult task was agreeing on which ten items should be included in the top ten; there was less discussion about the ranking. It is important to emphasize that the most important conclusion reached by the priority setting exercise is to be found in the final list of research questions, rather than in their specific ranking order.
An anonymous evaluation survey conducted after completion of the project disclosed that the members of the working group appreciated the mixture of different perspectives and experiences. The participants also appreciated the opportunity to examine the questions from different perspectives and that the final consensus took into account the central issues of concern to consumers and people closely associated with those diagnosed with ADHD. Both the group session and the plenary session received high reviews in the evaluation.
Some concerns were raised, however, that not everyone was adequately heard during the plenary session and that some of the professionals tended to dominate the discussion. Moderating the plenary discussion was a challenging aspect of the workshop. There will always be some people who are more comfortable than others in presenting their point of view and/or accustomed to being regarded as an authority. To circumvent this, the plenary session could be interrupted by additional discussions in smaller groups or individual voting before decisions are made. This could, however, be more time-consuming and require a 2-day workshop.
Despite some early level of consensus, the discussions during the workshop had a great impact on the ranking order of some of the items (the risk of drug dependence later in life when treated with methylphenidate as a child, the effects of teacher support, if atomoxetine or methylphenidate is more effective as well as the effects of melatonin). The workshop thus played an important role in exchanging experiences and views in the process of reaching consensus about the final list.
One possible criticism of the process of prioritizing research questions and producing top ten lists is that the lists reflect only the opinions of the participating individuals, rather than the larger populations which they represent, potentially resulting in a weak external validity. It is also possible that cultural differences would be a barrier to adoption of the lists in other countries. As in all consensus panels, constructing the panels is a crucial and difficult step, requiring a lot of effort and forethought. It might, therefore, be meaningful to run several panels in parallel, who would all prioritize the same uncertainties to assess the inter-rater reliability, as well as to compare lists compiled in different countries.
Another criticism could be that the priority setting exercise relied merely on scientific uncertainties identified in the HTA report by SBU, and not by a qualitative survey of stakeholders to identify relevant research questions (which is the normal procedure described by JLA). The treatment methods for children with ADHD in the SBU report were selected through a national inventory of methods currently used in Sweden. SBU's project group, consisting of leading experts in the field, added methods used for adults with ADHD, and also consulted with several psychologists and psychiatrists outside of the project group.
Furthermore, there was a reference group consisting of representatives from seven Swedish consumer organizations involved in the project. No additional methods were proposed by the reference group. The consumers involved in the priority setting exercise were invited to submit uncertainties to SBU that were not covered in the HTA report, but no such suggestions were submitted. However, limiting the list of uncertainties to the HTA report could possibly lead to missing questions that were not included in the report but relevant to consumers.
The priority setting process confirmed that there are important unanswered questions about the effects of drugs on ADHD. However, there is also a high demand for answers to questions about the effects of nonpharmaceutical interventions, many of which are not of commercial interest. For example, the SBU report on ADHD included no studies on the effects of teacher support, supported conversation, or psyschoeducative treatment. There are a few published studies on the other interventions, but because the studies are flawed by poor quality or lack of transferability, no definite conclusions could be drawn about the effects of these methods on the core symptoms.
To resolve the uncertainties, high quality and adequately blinded randomized controlled studies are required. The study population should have a clinically verified ADHD diagnosis, according to established diagnostic criteria (DSM-5 or ICD-10). For some research questions, long-term follow-up is important, for example to determine whether medication with methylphenidate during childhood increases the risk of subsequently developing substance abuse. Alternatively, these uncertainties could be bridged by well-conducted observational or register studies. There may be recent publications that could shed new light on some of the uncertainties. For example, a Swedish cohort study published in 2015 (Reference Sundquist, Ohlsson and Sundquist13) included all children born between 1991 and 1995 who were diagnosed with ADHD before the age of 15 (approximately 9,400 individuals). No association was found between stimulant treatment during childhood and subsequent drug use disorder. These new studies should be assessed for bias and included in a systematic review to determine whether the uncertainty has been resolved.
These types of lists provide a unique insight in the patient/clinician perspective. Who is responsible for ensuring that research addressing the questions in the top 10 list is conducted? It is important that research funders and organizations engaging in identifying scientific uncertainties and priority setting exercises collaborate to achieve this. Before opening a themed or commissioned call for grant research, funders could commission an independent organization, such as an HTA organization, to support and administrate a priority setting group as well as perform a current literature review or update an original systematic review. The findings from the priority setting partnership must be included in the literature search, especially if important aspects were missing in an original systematic review. However, because there are several perspectives to be considered (payers, providers, professionals, and patients), lists from different stakeholders should also be synchronized before the research calls.
In Sweden, initiatives have been taken to develop collaborations between SBU and governmental research councils. To gather stakeholders and discuss these issues, SBU has established a steering group consisting of representatives from several governmental agencies and research councils working with healthcare issues. SBU has also described and proposed processes for identifying, prioritizing, and funding research important to users and consumers, to the Swedish Government in the coming strategic research propositions 2016–26. Furthermore, research funders should request that applicants for clinical research grants involve consumers in planning the research project. Such initiatives have been taken in Sweden, for example in a unique joint call by the Swedish county councils and the Swedish Research Council, in which the applicants were required to describe patient involvement in the research projects.
CONFLICTS OF INTEREST
This research received no specific grant from any funding agency, commercial or not-for-profit sectors. The author report no conflicts of interest.
