Background
For almost 20 years, the American Society of Clinical Oncology and American Association of Cancer Research have jointly offered an intensive workshop on clinical trial protocol writing for oncology fellows and junior faculty. Although the workshop has been highly successful, there has been no equivalent program for supportive oncology research. Those interested in supportive oncology were limited to postdoctoral fellowships or highly competitive mentored training grants (e.g., K awards), restricting the accessibility of training in this type of research. Additionally, mentors in supportive oncology research are not available at all cancer centers.
The paucity of training opportunities is particularly unfortunate because the evidence base for the clinical practice of supportive oncology is lacking, especially compared with that for cancer treatment (Abernethy et al., Reference Abernethy, Aziz and Basch2010). Systematic reviews have noted that the limited research available is often characterized by substantial methodological challenges (Hales et al., Reference Hales, Zimmermann and Rodin2010; Zimmerman et al., Reference Zimmermann, Riechelmann and Krzyzanowska2008). Multiple Institute of Medicine reports have emphasized the need for more supportive care research (Committee on Cancer Survivorship, 2005; Committee on Psychosocial Services to Cancer Patients/Families in a Community Setting, 2008; National Cancer Policy Board, 2001), with the 2001 palliative care report specifically citing the lack of trained investigators as a primary reason for having limited evidence. That report asserts that healthcare professionals most interested in this type of research tend to be clinicians with little research experience. Specifically, the report stated that “although they often pose clinically important, reasonable, interesting, and potentially researchable questions…the methods they propose are inappropriate or lacking in scientific rigor.” As supportive care is increasingly integrated into cancer care, the science supporting best practices must keep pace with the rest of oncology treatment.
To conduct a scientifically rigorous supportive oncology study, a new researcher must become familiar with methods and challenges that may differ from therapeutic oncology research and require specialized training. These differences affect all major components of studies. Supportive oncology studies are typically grounded in conceptual models and include hypotheses with theoretical underpinnings, which may be perceived as esoteric or superfluous to individuals without social science training. Second, recruitment can be challenging because motivation may be lower for patients to participate in a supportive care study compared with a clinical trial offering a new cancer treatment. Third, outcomes in supportive care studies often include self-report questionnaires or data collected through qualitative interviews, which require training in their collection and interpretation. Fourth, safety concerns may differ from those of clinical trials, such as the responsibility to address participant-reported potential harm or suicidal ideation in a timely manner instead of grading chemotherapy toxicities. Finally, advanced analytic approaches, such as structural equation modeling to determine the relationships among variables, are often necessary.
To address these concerns, we sought to develop, implement, and evaluate an annual, intensive workshop on research methods in supportive oncology. The goal of the workshop is to facilitate new research in supportive oncology by training junior investigators in a broad range of research methods through mentorship in writing and developing a study protocol for implementation at their institution. This manuscript reports on experiences and data from the first three years of the workshop.
Methods
The workshop was funded by the National Cancer Institute and is hosted by Massachusetts General Hospital (MGH) and the University of Miami. The Dana-Farber/Harvard Cancer Center internal review board determined the program to be exempt from approval because it does not qualify as human subject research.
Development of curriculum
Multidisciplinary groups of experts in supportive oncology research developed the content areas for the workshop in three stages using consensus methods. First, members of the MGH Cancer Outcomes Research Program generated a comprehensive list of areas of competency in supportive oncology research. This group included oncologists, palliative care physicians, psychiatrists, psychologists, and nurses with experience in designing and conducting supportive care trials. Next, the Curriculum Committee reviewed and edited the list of competencies for its completeness. The Curriculum Committee consisted of nationally known oncologists, nurse researchers, psychologists, psychiatrists, and social work researchers within and outside of MGH. Third, before the workshop, an External Advisory Committee of expert supportive oncology researchers outside of MGH reviewed the areas of competency and corresponding lectures. At each stage, the groups reached consensus on which areas were essential to include and which could be excluded.
Recruitment of participants
We tailored the workshop to meet the needs of early career researchers, defined as not having independent funding (e.g., R01 or equivalent grant). Participation in the workshop was competitive with an application process. Eligible participants could be physicians, nurse researchers, social work researchers, psychologists, or other doctoral level trainees and junior investigators in fields related to individual-, community-, and population-level health. All needed to have a doctoral degree or be in the final stages of a doctoral program in which they were conducting their own research. Potential participants submitted an application form, a curriculum vita or biosketch, and an abstract on their proposed project. They were also required to identify a local primary mentor who could help them continue and advance their research projects at their own institutions after the workshop. Early career grantees of the American Cancer Society and National Palliative Care Research Center were invited to apply and given priority.
For the first year of the program, we accepted participants through a rolling admissions process with approval by the workshop principal investigators (PIs; WFP and JST) based on eligibility criteria and assessment of likelihood that the participants would be able to implement their project within the next year. In years two and three of the workshop, we established a formal review committee for accepting applicants. Two faculty members scored each application, and the committee selected participants based on average scores and discussion of application materials. Special consideration was given to the representation of disciplines, geographic regions, and underrepresented groups. Participants received five nights lodging in a hotel and were reimbursed for travel to and from Boston.
Participant preparation
Before attending the workshop, participants completed an online training in Responsible Conduct of Research. We also required that participants complete the Collaborative Institutional Training Initiative program or their institutional equivalent so that they would be able to submit a study protocol to their institutional review boards. The participants were assigned to small writing groups with designated faculty leaders who reviewed participants’ abstracts and provided comments to consider before the workshop.
Workshop schedule
The workshops were held in Boston for six days, starting on a Sunday evening and ending Friday afternoon. Over the course of the week, participants were expected to write either a study protocol or a research proposal to obtain grant funding. The first evening included a welcome dinner and lecture. Monday through Thursday, participants attended two hours of lectures in the morning and two-hour writing groups in the afternoon. Senior researchers with R01 or equivalent funding led the small writing groups of six participants. To ensure a multidisciplinary perspective, each writing group included participants from various disciplines (nurses, physicians, psychologists, social workers, and other clinical researchers). A member of the MGH Patient and Family Advisory Council joined each writing group once during the week as a patient stakeholder. Biostatistical consults, individual meetings with faculty, and optional sessions such as a qualitative coding laboratory and an advanced statistics seminar were scheduled in the afternoons. On Friday, the PIs delivered a final lecture in morning and completed protocols were due by the end of the day.
Evaluation
Evaluation of the workshop is accomplished through an extensive process that includes an assessment of the number of completed protocols by the end of the workshop; evaluations by participants and local mentors; pre- and post-tests of research knowledge; and a one-year follow-up survey. Additionally, a nationally-recognized senior researcher and educator in supportive oncology attended the first workshop, auditing every lecture and one of the writing groups. The auditor took field notes throughout the workshop to identify components that could be improved.
Completed Protocols: The primary immediate outcome for the program was the proportion of participants who submitted a completed protocol to the workshop faculty on the last day of the workshop.
Participant Evaluations: Participants evaluated the lectures daily during the workshop on a scale of 1–4, with 1 being “poor” and 4 being “excellent.” On the last day of the workshop, they were asked to evaluate the writing groups, statistical consults, other activities, and the workshop as a whole.
Self-Evaluation of Research Knowledge and Skills: Before the workshop and one year after the workshop, participants were asked to evaluate components of their research knowledge and skills on a scale of 1–4, with 1 being “poor” and 4 being “excellent.” In Years 2 and 3, an additional evaluation was added on the last day of the workshop.
Pre- and Posttests: Before the start of the workshop and on the last day of the workshop, participants completed a test of knowledge online. Workshop faculty members created potential test items based on the content areas covered by their lectures. The PIs then compiled and reviewed candidate items for appropriateness. Cognitive testing was conducted by administering a draft version of the test to faculty to ensure the clarity of the questions and answers, reduce risk of bias, and ensure representativeness of the content. After the first administration to participants in year 1, we removed items that performed poorly (e.g., those that almost all participants answered correctly on the pretest). In year three, the PIs again reviewed the questions for relevance given changes in content over the first two years.
One-Year Follow-Up Survey: One year after the workshop, participants received an online follow-up survey on their research activities in the year after the workshop. The survey also served as a self-assessment of their research knowledge and skills, using the same items they completed before the workshop. Because access to financial resources can affect the ability to implement protocols written during the workshop, the primary follow-up outcome at one year was the proportion of participants who either implemented their protocol (or submitted their proposal) or wrote a new protocol after the workshop.
Primary Mentor Evaluations: Before and one year after the workshop, the primary local mentors were asked to report their perceptions of the research knowledge and skills of the participants as well as any benefit they may have obtained from the workshop on a scale of 1–5, with 1 being poor and 5 being excellent.
Analyses
Descriptive statistics were compiled using IBM SPSS 22. We conducted t tests or Wilcoxon rank sum tests to analyze comparisons between continuous variables; we used chi-square tests to analyze comparisons of categorical data.
Results
Curriculum
Table 1 summarizes the final curriculum, approved by the Curriculum and External Advisory Committees. Based on evaluations of the first and second years, we reduced the number of lectures in order to have more time for questions and discussion.
Table 1. Workshop curriculum
Participants
We received 195 applications over the three years: 79 for year 1, 63 for year 2, and 53 for year 3. The characteristics of those accepted are shown in Table 2. For the first year, acceptance through rolling admissions led to a situation in which most slots were filled by the week before the application deadline, although the majority of applications were submitted within two days of the deadline. Among the later applicants who were not selected, we offered qualified individuals who would have been accepted earlier in the process admission to the workshop the following year. Ten of those people chose to participate in year 2, making them a natural waitlist control for outcomes. In year 1, we accepted 36 participants. Two participants dropped out close to the workshop date, and two were taken from a waitlist. In years 2 and 3, 38 participants were chosen to account for dropouts.
Table 2. Participant characteristics
ACS, American Cancer Society; NPCRC, National Palliative Care Research Center.
Across all years of the workshop, we accepted applicants from a range of disciplines including physicians (e.g., medical oncologists, palliative care specialists, psychiatrists, surgeons), nurse researchers, psychologists, social work researchers, and other researchers (e.g., exercise physiology/kinesiology, public health, communications). No single discipline constituted the majority of participants in any year of the workshop.
For the majority of protocols, participants aimed to develop or test supportive care interventions such as behavioral, educational, communication, exercise, or health services interventions. Many participants reconceptualized their projects over the first two days of the workshop after obtaining feedback from their mentors and group. In revising their projects, participants generally decided to start the research at an earlier phase, for example by proposing a pilot study to assess feasibility and preliminary efficacy rather than moving forward with a randomized controlled trial of a new intervention.
Evaluation
Completed Protocols/Proposals: In each of the three years, all participants (N = 109/109, 100%) submitted a draft protocol/proposal on the last day of the workshop. In the evaluations, participants rated the value of protocol writing 3.9–4/4 each year. In free text comments, many participants wrote that what they accomplished during the workshop would have taken them months to complete at their home institutions.
Participant Evaluations: Participant evaluations are summarized in Table 3. Overall participant evaluations of the workshop were very favorable each year with high ratings of satisfaction, value, and likelihood of recommending the workshop to a colleague.
Table 3. Participant evaluations of the workshop
Scale 1–4, where 1 is poor and 4 is excellent. SD, standard deviation.
Self-Assessment of Skills at the Immediate End of the Workshop: In years 2 and 3, participants reported significant improvement in their familiarity and confidence in research skills from before and after the workshop (Table 4). However, participants did not report significant changes in their interest in research or likelihood of conducting supportive oncology research in two years, both of which were high before the workshop. We do not have data from year 1 on participants’ self-reported assessment of their research skills at the immediate end of the workshop.
Table 4. Research skills and attitudes assessments
*p < 0.05; **p < 0.01.
Pre- and Posttests: Scores on the pre- and posttests are shown in Table 5. Although the proportion of correct answers increased on the posttest in years 1 and 2, the difference was not significant. In year 3, we observed a significant improvement in the number of correct items.
Table 5. Pre- and posttest scores
SD, standard deviation.
One Year Follow-Up: Thirty of the 36 (83.3%) participants from year 1 completed the one-year follow-up survey. Almost all participants who responded (29/30, 96.7%) either implemented or submitted the protocol they wrote in the workshop or wrote a new protocol/proposal in the year after the workshop. More specifically, 24/30 (80%) implemented or submitted their protocol and 26/30 (86.7%) wrote at least one other protocol or proposal in the year after the workshop. The mean number of protocols that participants wrote in the year after the workshop was 2.4. Participants from year 1 reported significant improvement in their familiarity with and confidence in research skills from before the workshop to one year after the workshop. However, similar to the data from the immediate end of the workshop, participants did not report significant changes in their interest in research or likelihood of still being involved with supportive oncology research in two years, both of which were high before the workshop. Half of the participants (15/30) reported receiving at least one research grant, which was higher than the proportion in the natural waitlist control group (3/10, 33.3%) comparing their first biosketch to their updated biosketch one year later. The total number of grants awarded to participants was 24.
Primary Mentor Surveys: Local primary mentors completed the survey for 24/36 (66.7%) year 1 participants. Similar to the participants' one-year survey, the primary mentors reported significant improvement in the participants' research skills and familiarity with methods from before the workshop to one year after the workshop (Table 4). Primary mentors also did not report significant differences in their perceptions of the participants' interest in research or likelihood of still being involved in supportive oncology research in two years. The local mentors' mean rating of the participants' benefit from the workshop was 4.67/5 (standard deviation = 0.55).
Discussion
We developed and implemented a successful six-day workshop to train early career investigators on skills in supportive oncology research. By training the next generation of supportive oncology researchers, the ultimate goal of the workshop is to increase both the quality and quantity of research in supportive oncology. To date, 109 early career researchers have participated in an annual workshop on research methods in supportive oncology. All participants completed a research protocol by the end of the workshop, and most implemented or submitted their proposals after the workshop. Not only did participants rate the workshop highly in their evaluations, participants and their local primary mentors also reported significant improvements in their perceptions of the participants’ familiarity with research methods and research skill from before the workshop to the end of the workshop and one year later. Comparing workshop participants from the first year with a natural waitlist control group, participants were more likely to receive grants in the year following the workshop.
Of note, we did not observe significant differences in the participants' reported interest in research or likelihood that they would still be doing supportive oncology research in two years from before to after the workshop. Participants’ high ratings of these perceptions before the workshop likely led to a ceiling effect for these items. Nonetheless, the workshop created of a sense of community among supportive oncology researchers, participants, and faculty. Many participants reported that they knew of few, if any, others at their institutions who were interested in leading or initiating supportive oncology research. The six-day workshop provided the opportunity for participants to meet others doing similar work, develop new collegial relationships, and forge potential collaborations with investigators at other institutions. Thus, the workshop created new opportunities for multi-site and multidisciplinary studies, including at least one funded National Institutes of Health R01.
Although the workshop has been successful by our predefined evaluation metrics, we also experienced several challenges over the first three years. One of the most significant challenges was for group leaders to meet the diverse needs of group participants who possessed varying levels of prior research experience. For instance, some participants had already completed their own studies, whereas others were writing a protocol for the first time. With the goal of increasing accessibility of training for this type of research, we committed to including participants with high motivation to conduct research yet minimal research exposure rather than focusing exclusively on those who already had research experience. Although the decision to include very early investigators may yield benefits to the field in the years after the workshop, it may also have affected our one-year outcomes because these participants may be less likely to receive grants and write multiple protocols in the short term.
Another major challenge was that many participants substantially reconceptualized their studies during the first two days of the workshop. Decisions to make fundamental changes in design may have resulted in fewer days to develop a new idea into the protocol and to benefit from scheduled consultations with biostatisticians, survey developers, and so on. However, revised project ideas could also be considered a positive outcome. The workshop helped some early investigators conceptualize a more realistic, feasible, and justifiable project proposal that matched their skill sets while also meeting review criteria set by internal review boards and funding agencies. In years 2 and 3, to accommodate reconceptualizations, we changed the timing of the agenda such that study development lectures occurred at the very beginning of the workshop and biostatistical consults occur toward the middle and end of the workshop.
Finally, although this workshop provided a live forum for trainees and junior faculty to receive training in supportive oncology, the in-person format may have limited accessibility. Although the lectures could be delivered via webinar to increase access to the workshop, it would be challenging to conduct the daily interactive writing groups through video. The immersive experience with frequent meetings and high-intensity engagement requires face-to-face time and could not be replicated through conference calls or video-conferencing. Participants would not have the same opportunities for networking and cultivating professional relationships that are facilitated by an in-person meeting. Additionally, participants reported that a major benefit of the workshop was having six days dedicated to writing away for home and work, thereby minimizing distractions. In fact, one participant described the workshop as a “spa for research.” Furthermore, participants found the communal obligation of needing to produce a deliverable product in only six days to be motivating.
Despite the challenges encountered, the workshop appears to be responding to a need of early career supportive oncology researchers. Every year, the workshop has been oversubscribed and participants report the value of their experience. Armed with new skills, participants return to their institutions across the country with a product to submit to an institutional review board or funding agency. As their professional development continues, we hope the quantity of high-quality supportive oncology research continues to grow. Ultimately, the research will provide the evidence base that will help improve the experience of patients and families affected by cancer.
Acknowledgements
We acknowledge Drs. Jamie von Roenn, Michael Fisch, and John Fromson for their work in assisting with the planning of the workshop. We also acknowledge Drs. Marie Bakitis, James Tulsky, Karen Donelan, Matthew Mimiaga, Angelo Volandes, Melissa Abraham, Inga Lennes, Erin Kobetz, Beow Yeap, Alona Muzikansky, and Samatha Rosenthal for serving as workshop faculty. This workshop was funded by NCI 25 CA 181000.
Conflicts of interest
JAG and LT have received Pfizer research funding. SAS has received royalties from Oxford University Press and Guilford Publications for authored books and has agreements in place with Springer/Humana Press. The remaining authors state no conflicts of interest.
