INTRODUCTION
Cancer research in the National Health Service (NHS) has increased by 10·5% in 3 years since the formation of the National Cancer Research networks (NCRN) in 2000.Reference Sinha 1 This is mainly due to the wealth of evidence that patients benefit from research, and a drive from health professionals to provide evidence-based practice for their patients. This has led to the widespread adoption of single- and multi-centre clinical trials across the United Kingdom, and an increasing culture of research in NHS TrustsReference Sinha 1 with one of the NHS core principles established to provide research for patient benefit. 2
Due to the importance of research there is a positive cultural change driving research locally at NHS institutions, and this is not necessarily only in the form of an NCRN clinical trial. 3 , 4 In addition, local developments such as introducing a new treatment technique or imaging modality are not research, but service development and are also vital to move the service forward. There is a trend for driving innovation in radiotherapy, and a cultural shift in research that maybe due to economic challenges and assuring value for money.Reference Winkler, Duma and Popp 5 – Reference Jones, Cifu, Backus and Sisto 7 Strong research and development (R&D) are important facets of a successful NHS trust. The ideal result is that R&D improves clinical practice. Although it is important to acknowledge and consider the impact on the local radiotherapy service. The initial enthusiasm from clinical staff to embark on a project has to be balanced against the wider implications of resources, costs and other developments.Reference Jones, Cifu, Backus and Sisto 7 , Reference Hearn and Sullivan 8 The impact can be positive, neutral or negative for the patient, the NHS providers or the service commissioners. Seldom would a positive impact apply to all three parties, but for the patient the impact must always be positive. A potential negative impact to the service needs to be evaluated by the multi-disciplinary team (MDT), which includes an oncologist, a research radiographer, physicist, trial officer and radiotherapy managers. A potential impact could be the lack of a facility such as toilets if considering introducing an intervention such as pre-treatment bowel preparation, a decrease in departmental capacity, a loss of revenue or an increase in costs due to the addition of a new medication such as micro-enemas.Reference Jones, Cifu, Backus and Sisto 7 , Reference Hearn and Sullivan 8 Acknowledging these pitfalls can lead to the potential of mitigating or offsetting costs or being clear about potential impacts to allow for the project to proceed. Alternatively, an assessment of impact could lead to the suspension of a project where it is deemed unsafe to proceed due to staffing resources, and this is the right decision to make.Reference Jones, Cifu, Backus and Sisto 7 , Reference Hearn and Sullivan 8 The decision to move a project forward should be made in collaboration across the MDT, with all the information to hand with assumptions clearly defined. However, the information needed to make a decision can be vast. The solution is to summarise the key information and streamline the pathway of a project to completion. Creating a lean process is fundamental to driving a project forward, without initiating unnecessary delays or a stagnation of the service.Reference Bonilla, Pawlicki, Perry and Wesselink 9 – Reference Rich, Bateman, Esain, Massey and Samuel 11
Hence, an assessment tool was required that would assess proposed research, clinical trials and the implementation of new treatment techniques, based on all available evidence-based practice.
A multi-step development method was used along with agreement of objectives among a working group; a patient representative, radiographers, radiotherapy and physics managers. The steps of this method involved the following objectives:
1. The research impact assessment needed to be simple to create and develop.
2. It needed to be easy to navigate for the user and take just 10 minutes to present at an MDT R&D review group.
3. It had to be lean: contain all the impact assessments on one A4 page.
4. It had to be capable of multi-profession input.
5. It had to be capable of multi-profession agreement on key impact assessments.Reference Wilford, Cutler and Williams 12 , Reference Tilling, Sterne, Brookes and Peters 13
Being a ‘paper light’ radiotherapy department the newly named Radiotherapy Research Activity Assessment Tool (RAAT) had to be in an electronic format, with details such as project title, date ranges and an approval tab. The main driver for the electronic format is based on the ease of access, and to meet the requirements of a paperless NHS by 2018.Reference McCluggage 14 , Reference O’Dowd 15 It also needed to include MDT meeting columns, along with details such as scope, benefits to the centre and any other relevant detail. A probability rating was requested by the key stakeholders that would give an indication of project success or failure. Finally, a column to project the recruitment level for trials, and some comment boxes for additional information.
The tool needed to include the flowchart of the project pathway and instructions as separate sheets in the same Microsoft Excel workbook.Reference De Vet, Terwee, Mokkink and Knol 16
METHODS and MATERIALS
First, based on the specification requirements, a scoping review was recommended to find similar tools in clinical practice. Second, the development of the tool was completed by the researcher and a consensus agreement obtained from the working group that included a patient representative, the MDT and consultation with NHS executive directors. At this stage NHS commissioners were not approached. This was followed by a usability assessment and then testing in clinical practice.
Ethical considerations
This study was considered service evaluation by the radiotherapy MDT R&D group at Musgrove Park Hospital. However, the principles of good clinical practice were upheld.Reference Vijayananthan and Nawawi 17
Scoping review method
A scoping review of the literature was undertaken using the following keywords: radiotherapy impact/tool, radiotherapy research assessment tool, radiotherapy impact tool, research impact/assessment tool and quality function deployment (QFD) in cancer/radiotherapy. The following databases were used: Medline, Embase and Cinahl, 18 including conference proceedings.
Scoping review
The search did not reveal anything that met the specific requirements for the impact assessment in radiotherapy, but similar impact tools have been described and studied.Reference Bonilla, Pawlicki, Perry and Wesselink 9 , Reference Vanteddu and McAllister 19 – Reference Gerrish, McDonnell and Kennedy 22 The search gave rise to a vast amount of research assessment and QFD literature, which demonstrated a gap between the macro- (Universities, University Hospital Trusts, research collaborations) and the micro- (A Radiotherapy clinic, NHS Foundation Trusts, Community Hospitals) level institutions. At macro level there is a vast amount of publications and toolkits to assess the impact of research using whole array of methods that justifiably take time to complete to ensure ‘value for money’.Reference Greenberg, Earle, Fang, Eldar-Lissai and Neumann 6 This is especially important when the cost resource of setting up research studies can be in excess of £1 million pounds sterling. However, at the micro level of deciding whether a clinical team, such as radiotherapy, will participate in a clinical trial organised by the macro-level institution, the exchange of resources is less but the implications still need to be considered. After removing some of the macro-level literature and selecting studies that suited the proposed tool, three articles remained using QFD. Bonilla et al.Reference Bonilla, Pawlicki, Perry and Wesselink 9 used an adapted version of QFD, which can translate customers’ (stakeholders) needs into technical specifications and the requirements for developing a radiotherapy service. The tool developed by Bonilla et al. had traits that were transferable to the proposed tool. Similarly, Vanteddu and McAllisterReference Vanteddu and McAllister 19 describe and evaluated a similar approach using QFD to identify processes that are important to both the health-care provider and the patient’s perspective to develop a process to improve health-care processes. Both of these studies shed light on tools developed to asses a form of impact and have merit in the development of the RAAT. In addition, Munoz et al.Reference Munoz, Nembhard and Kraschnewski 20 developed a three-phase QFD process to quantify translational research, and produced a method to generate agreement, develop guidelines, allocate resources wisely, identify benchmarks and form collaborations. Similarly, the RAAT required the same traits and some elements were incorporated. However, Munoz et al.Reference Munoz, Nembhard and Kraschnewski 20 conclude that their method is based on subjective opinion and could be open to bias, and suggest that a larger study is required to reduce the bias effect. This issue was acknowledged and accepted as a potential issue when developing the RAAT.
A grey evidence search was also completed, first by seeking information from a national ‘research radiographer Google group’ forum, and two independent radiotherapy departments, asking if anyone was using a tool to assess the impact of research in their clinics. One recommendation was a publication by Kennedy et al.Reference Kennedy, McDonnell, Gerrish, Howarth, Pollard and Redman 21 and an impact toolkit. Kennedy et al. describes an impact assessment tool for capturing the impact of nurse consultants in clinical practice.Reference Kennedy, McDonnell, Gerrish, Howarth, Pollard and Redman 21 , Reference Gerrish, McDonnell and Kennedy 22 This is a comprehensive tool, and, although it provides some key areas for the proposed topic, it is far too extensive for a 10-minute discussion in a clinical MDT. One of the radiotherapy departments was using a similar tool embedded into Microsoft word, which was mainly used for clinical trials, and provided some useful information for some of the key areas of the proposed tool.
The development of a standardised, consistent, ‘in-house’ tool is proposed, based on the tools described in these studies and the independent radiotherapy department.
The primary endpoint of this work was to develop and implement a RAAT utilising Microsoft Office applications.
A process flowchart was introduced to create a lean pathway that would offer support, channel projects in the right directions and halt those that were deemed to fail.
Starting at the top (Figure 1), a RAAT tool must be completed for all projects and then presented at the radiotherapy R&D group. At this point, the project presented is triaged by the MDT and assigned priority. The project could then proceed to the Beacon Oncology Radiotherapy Group for approval if required, or not as the case maybe. In some cases, the financial implications of the project mean that approval to implement has to be given at Trust directorate level before proceeding.
Figure 1 Development of the project pathway flowchart. Notes: Rectangle=directed action; diamond=consultation; ellipse=decision. Abbreviations: RAAT, Radiotherapy Research Activity Assessment Tool; R&D, research and development; BORG, Beacon Oncology Radiotherapy Group; MEAG, Measurement effectiveness and audit group.
The objectives and requirements were first modelled in Microsoft word using the tick box functions against some of the impact assessment questions. However, the working group decided that the extra functionality requested in the objectives would better suit Excel. In Microsoft Excel, drop-down toolbars were used to select a response to a question, and date picker was used to select a date. Microsoft Excel was found to suit the functions required, and certain macros enabled the numerical assessment of project probability of success or failure (see Figure 2). In order to attain a project probability, numerical quantification had to be assigned to whether a given impact was positive or negative. For example, if an impact on the radiotherapy clinic budget was positive you could assign a score of 10, a neutral impact as 5 and a negative impact as 0 to reflect this subjectivity numerically. Hence, the values were modelled to suit the radiotherapy departments’ balance of priorities from the perspectives of all of the stakeholders: the patient, the staff, the providers.
Figure 2 Development of the Radiotherapy Research Activity Assessment Tool (RAAT).
In addition to the development of the tool, the process flowchart and instructions on how to complete a RAAT were included in the same worksheet. Completed RAATs were assigned numbers starting at 0001 and saved to a particular folder. When approved to proceed to clinical practice they were saved as a PDF in a separate folder. This was put in place as a way to govern the completed RAAT forms, and avoid permanent deletion or unexpected alterations.Reference McCluggage 14 , Reference O’Dowd 15
Usability
The prototype RAAT was preliminarily tested on usability among six MDT members who would be using this in routine clinical practice. A researcher asked all members of the MDT to rate the usability of the RAAT using a ten-point scale (1=poor; 10=very good). A mode score of usability will indicate an initial clinical acceptance of the RAAT knowing that changes will need to be made. To proceed to testing in clinical practice, the usability score had to be ≥6.Reference McHugh 23
Testing in practice
The MDT used the RAAT to decide whether R&D projects should or should not proceed to clinical practice. The aim was to test whether the MDTs initial decision to proceed with an R&D project indicated interrater reliability agreement over time when asked on a two-point scale, 1 for agreement or 2 for disagreement. The interrater reliability of their agreement to proceed with a project was tested at a minimum of 6 months post the initial RAAT assessment. The percentage of interrater reliability method was used to measure reliability of agreement among the MDT as there were no indications that the MDT would guess their agreement, hence not conducting κ interrater reliability testing.Reference McHugh 23 A percentage agreement of above 80% was considered an acceptable level to continue to full implementation in clinical practice as indicated in a publication by McHugh.Reference McHugh 23 Intrarater reliability was not conducted as the RAAT would always be used to assess a project by the MDT, as a collective agreement, rather than autocratically by an individual.
RESULTS
Implementation
A prototype RAAT was developed in an e-form available in Microsoft Excel. The tool included details such as project title, duration of project and approval tab. It also included meeting columns of where the project should be discussed, along with details such as objectives, and benefits to the clinical setting. A probability rating was included giving an indication of project failure or success scored respectively from 1 to 120 (six columns with a potential score of 10 for each). Agreement of the MDT was made verbally and an approval column added to this effect. Finally, a column to project the recruitment level for trials and comment boxes for additional information were added. A prototype of the RAAT was drafted in a document quality management system QPulse®. 24 The radiotherapy R&D MDT gave electronic approval for the RAAT to be used in clinical practice using the QPulse® system late in 2014, and edited again in 2016. Implementation was guided by the Research Radiographer.
Usability testing
Six members of the MDT verbally consented to taking part in the usability assessment in 2014. Verbal consent was obtained as per standard practice for service development and evaluation at the host radiotherapy clinic. The MDT were asked to rate the usability of the RAAT in terms of design and functionality. Essentially, was the RAAT ‘fit for purpose’. The RAAT scored a mode of 6 for usability that was the minimum score required to proceed to the next step, knowing that the RAAT will be edited. Specific changes were made to the RAAT based on the comments in Table 1.
Table 1 Usability rating of the Radiotherapy Research Activity Assessment Tool design on a scale 1–10 (1 is poor, 5 is good and 10 very good)
Abbreviation: MDT, multi-disciplinary team.
Testing in practice
Testing was assessed at radiotherapy R&D MDT meetings over the course of 2 years between 2014 and 2016. Seven members of the MDT verbally consented to take part. In total, 14 R&D projects, ranging from the participation in a clinical trial to the implementation of adaptive bladder radiotherapy, were tested for percentage interrater reliability agreement to initial decision with a minimum of a 6-month interval between meetings. Agreement between seven MDT members resulted in an interrater reliability percentage of 88% (Table 2).
Table 2 Per cent agreement of projects 1–14 for multi-disciplinary team shown as rater 1–7
Discussion
This is the first known publication developing an assessment tool to evaluate the impact of radiotherapy R&D projects in a small micro clinical setting. The assessment tool was initially developed to balance the needs of all the stakeholders, and sustain and improve the local radiotherapy service. The concept of QFD was considered important when developing an in-house tool. A RAAT was developed using Microsoft Excel and implemented in clinical practice. The results have shown that the RAAT is acceptable to the user with a mode score of 6, and the interrater reliability of the RAAT scored 88%, which is defined as an acceptable measure for clinical environments as indicated by McHugh.Reference McHugh 23 The outcomes of this development study are encouraging; however, there are limitations to acknowledge before drawing conclusions. There is a level of subjectivity in how the MDT respondent is asked to rate something where personal opinion is being gauged as concluded by Munoz et al.Reference Munoz, Nembhard and Kraschnewski 20 The effect of subjectivity is diluted by asking more than one MDT respondent;Reference Wilford, Cutler and Williams 12 , Reference McHugh 23 however, seven MDT respondents in the testing phase may not be a large enough sample to generalise to other radiotherapy clinics. The sample was small as the testing was completed during the radiotherapy R&D review meeting to represent standard practice, however, the sample size does have to be acknowledged. Nonetheless, this is evidence to continue to use and develop this tool further at the host radiotherapy clinic. Other limitations are the low number of projects that were assessed using the RAAT. Ideally, a larger number of projects would be included in this development study, however, seven projects were deemed sufficient by the MDT to provide an acceptable assessment in the feasibility phase. Likewise, authors such as Winkler and colleaguesReference Winkler, Duma and Popp 5 evaluated time and resources in radiotherapy finding a small amount data problematic in drawing firm conclusions. Furthermore, the probability of project success or failure seems feasible to practice but was not used as an absolute guarantee by the MDT. Rather it was used to supplement what is known about the project. This functionality requires further development. The design of the RAAT focused on priorities at the development site, with the proviso that future work will investigate the transferability at other clinical sites. Essentially, the instillation of thought process driven by using the RAAT helps the MDT develop a scientific and economical ‘mind-set’.Reference Jones, Cifu, Backus and Sisto 7 The process pathway of R&D projects was created to supplement the RAAT in the form of a process flowchart. An agreed pathway for R&D projects has been beneficial; however, there have been occasions where projects have bypassed the radiotherapy R&D MDT group. This has occurred mainly with clinical trials, owing to the expression of interest in a study. If an interest is not expressed at an early stage, then there may not be opportunity later for the radiotherapy clinic to participate in a clinical trial. However, this means that the MDT R&D group cannot assess the impact of the clinical trial on the radiotherapy service, and potentially have no resources to accommodate the proposed clinical trial. This could inevitably cause delays opening a clinical trial, a breach of targets, and the spoiling of professional reputation. The process pathway has provided a good foundation although it requires further development to overcome the problem with expressing early interest.
The RAAT will be developed further, and will be integrated with Microsoft access to increase the usability. Moreover, it should be noted that although there is a specific emphasis on radiotherapy, this tool could be adapted and developed for other clinical areas. At the host NHS Foundation Trust, there are plans to develop and to test the feasibility of implementation across other clinical areas.
Conclusion
In conclusion, the RAAT seems to be feasible in clinical practice, and provide a useful framework to guide the decision-making process of accepting an R&D project. The tool calls for further testing of usability and long-term implications on all stakeholders. Overall, the RAAT has become an asset to clinical practice at the host radiotherapy clinic, through the focussing of discussion to key considerations and summarising the key implications from trial protocols. There are plans to expand the tool to other clinical areas. The authors recommend this approach to health-care services.
Acknowledgements
The authors would like to thank all staff at Musgrove Park Hospital for their continued innovation and drive for change, and the Clinical School University of Plymouth for supporting this endeavour.
