Introduction
It is often stated that radiotherapy planning is more of an art than a science. This is certainly the case for structure outlining where essentially the user is creating a three-dimensional (3D) model of the tumour and organs at risk (OAR) from a limited two-dimensional (2D) planar dataset. Currently, radiotherapy target structures are ‘drawn’ electronically on individual 2D computed tomography (CT) slices by radiation oncologists to generate 3D volumes that can be used to plan the radiotherapy appropriately. Despite various auto-outlining tools being developed, the process is still labour intensive with recent papers highlighting variability in practice with a mean manual contouring time of 180 minutes for a salivary gland aloneReference Voet, Dirkx, Teguh, Hoogeman, Levendag and Heijmen1 and 108 minutes for a head and neck patient.Reference Thomas, Vinall, Poynter and Routsis2 Even the most cursory examination of the literature will demonstrate this array of timings for different regions and even within the same region. With the current drive towards more intensity-modulated radiotherapy and volumetric-modulated arc therapy planning in departments, it is often the additional structure outlining time that is increasingly time consumingReference Reed, Woodward and Zhang3, Reference Jefferies, Taylor and Reznek4 and is partly responsible for restricting the use of this technology.Reference Thomas, Vinall, Poynter and Routsis2 Drawing on successive CT slices is not only labour intensive but can produce an irregular shaped volume rather than the naturally smooth shapes found in the human body. The accuracy of tumour delineation has been summed up by one authorReference Njeh5 as ‘the weakest link in the search for accuracy in radiotherapy’. Given the major impact of outlining accuracy on potential outcomes for radiation therapy,Reference Allozi, Li and White6, Reference Weiss and Hess7 it is unsurprising that there is a wealth of literature highlighting the problems of structure outlining variability.Reference Zheng, Sun, Wang, Zhang, Di and Xu8–Reference Breen, Publicover and De Silva12 Although many normal structures are relatively easy to identify on CT, target structures in particular require an element of clinical judgement in the outlining process owing to their inherently abnormal appearance and potential for further direct invasion of disease into seemingly normal tissue. The result of this subjective outlining process is clinician-dependent variations that are classed as ‘inter-observer’ variations. Studies have demonstrated that this can successfully be reduced by clinician engagement with dedicated training materials and guidelines,Reference van Herk, Duppen, Remeijer, Burnet, Swift and Khoo9, Reference Khoo, Schick and Plank13 although even recent RTOG guidelines for brachial plexus outlining failed to eliminate variability.Reference Van de Velde, Vercauteren and De Gersem11 A more frustrating finding in the literature is the problem of intra-observer variability,Reference Fiorino, Reni, Bolognesi, Cattaneo and Calandrino14, Reference Remeijer, Rasch, Lebesque and Van Herk15 which compounds the natural inter-observer variation that occurs between different clinicians and frustrates attempts at consistency and accurate quantification of inter-observer variation. It is postulated that part of this variability stems from the relatively crude 2D outlining methods used. Clinicians must make a judgement about how structures will change on superior and inferior sectional images and integrate this into their outlining process. It is anticipated that using a 3D volumetric outlining technique will reduce intra-observer variation in contouring. In turn, a reduction in intra-observer variation will allow for a reduction in error margins that are routinely applied to target volumes. It will also help to quantify inter-observer errors and inform intervention strategies to further reduce error margins.Reference Khoo, Schick and Plank13
Current research in structure outlining is mainly focussed on automating the process using either sophisticated boundary detection tools to locate edges of different structures or stored image data that can be matched to different patients. Although atlas-based auto-segmentation (ABAS) systems are able to outline normal and critical (OAR) tissues with relative accuracy,Reference Chen, Niermann, Deeley and Dawant16 the very nature of cancer growth means that target tissues are likely to have an abnormal appearance and thus will be more challenging for the software to identify. The downfall of ABAS is target delineation and VoetReference Voet, Dirkx, Teguh, Hoogeman, Levendag and Heijmen1 determined that salivary gland treatment based solely on ABAS contours resulted in large underdosage in the region of 7%. In addition, it is vital that outlining of target structures has clinician input in order to provide the essential clinical judgement. All authors agree that ABAS outlines need further editing in order to attain sufficient levels of accuracy to be used clinically. Furthermore, ABAS is only viable for ‘normal’ structures and tumour target structures or abnormal anatomy will still require manual delineation. Any further increase in speed of outlining will thus need to utilise manual input in a more productive manner.
A potentially more fruitful possibility is to create multiple outlines in 3D simultaneously. Structure outlining is the last of the radiotherapy procedures to be conducted in 2D with advances in CT scanning enabling collection of volumetric CT data. Prescription of dose and evaluation of dose limits is also volumetric based. This project aims to enable the radiotherapy outliner to model target structures in true 3D using 3D graphics editing tools and immersive visualisation. McBainReference McBain, Moore and Green17 proposed the first attempt at 3D structure outlining as a replacement for slice-by-slice drawing and discovered a significant (p<0·017) time saving of around 7 minutes/bladder patient. The novel 3D-radiotherapy interactive outlining tool (RIOT) software further develops this concept by allowing the target volume to be generated from a small number of points on orthogonal planes and providing 3D volume-sculpting tools to edit the resulting structure. This paper discusses the results of the initial qualitative evaluation of the 3D-RIOT software. The evaluation was performed as the penultimate phase of the software development process to inform clinical suitability of the final version.
Materials and Methods
User feedback was collated as part of the development phase to ensure clinical suitability, define user training strategies and identify best practice. It should be noted that this evaluation phase precedes the formal clinical evaluation of the finished product.
Qualitative feedback from users was gathered to inform software development. Focus groups are a well-established method of gathering qualitative data to gather opinions and usage data. They encourage dialogue between participants and facilitate collection of rich descriptive and inferential data. Previous workReference David, Williamson and Tilsley18 has shown that this dialogue relating to a shared experience provides participants with ‘permission’ to engage more than a questionnaire-based approach. A loosely structured format was adopted with leading descriptive questions (as seen in Table 1) aiming to gather use and value data, generate suggestions for improvements and initiate further discussion. Follow-up inferential questions encouraged exploration of ideas and established wider underpinning theories relating to application use and rationale for opinions.
Table 1 Focus group questions
Focus group participants were recruited by invitation e-mail from all four local radiation oncology registrars within the studied department and all four responders were selected. Two of the participants were half way through registrar training and two were about to complete; all four had experience in prostate outlining and treatment. Ethical approval for the project was granted by the Metro South Hospital and Health Service HREC and Queensland University of Technology HREC (Reference HREC/14/QPAH/161). Participants were assured anonymity and participation was entirely voluntary. An hour of at-elbow training was delivered to the participants; this comprised an instructor sitting with them, guiding them through the process and familiarising them with the software. They were also provided with a video demonstration and a paper-based user guide. Participants were supplied with a copy of the software and a test patient CT dataset containing a prostate tumour with seminal vesicle involvement. They were then left for a month to experiment with the software before feedback was captured. The software allows the target volume to be generated initially from a small number of points around the whole volume within axial, sagittal and coronal intersecting planes. These are then used to generate a smoothed mesh using MESHLAB. 3D volumetric visualisation allows users to directly manipulate this volume using 3D-sculpting tools until the surface matches the underlying 3D volume. Figure 1 illustrates the three main stages of the process within the application.
Figure 1 Three-dimensional (3D)-radiotherapy interactive outlining tool software process. Abbreviations: CT, computed tomography.
Focus group transcripts and direct observation captured feedback concerning software use, user satisfaction, potential clinical value, training strategies and interface design. Content analysis was facilitated with emerging coding techniques; participant responses were assigned new codes as they arose and collated accordingly. Given the novelty of the volumetric outlining paradigm, a grounded theory approach was used to develop new theories relating to the perceived value and impact. These explored themes were used to refine the application ahead of formal clinical testing.
Results
All four participants provided written responses to a provided proforma and also participated in a 38-minute discussion around the questions. Comments were sought related to the training methods, ease of use, challenges arising, value of 3D and potential clinical use of the software.
The participants found the training to be valuable, although there was an acknowledgement that they were still continuing to learn. The ideal training was reported to be at-elbow hands-on practice with a printed summary to use later; the video was not felt to be advantageous. The sharing of ideas and experiences via collaborative peer learning was valuable.
All participants found the software user friendly, reporting that after an outline had been generated once it was easy to do another one and proficiency was gained rapidly. The point placement was felt to be much faster than traditional outlining methods with participants reporting 1–2 minutes for initial volume creation, although the sculpting was more challenging and time consuming. It was clear that the sculpting had caused difficulties to some participants related to obscuring of the CT by the volume, generation of holes owing to insufficient point placement and 3D orientation. During discussion, it was evident that other participants had been able to solve most issues and this initiated useful suggestions for improvement and training.
All participants reported clear value in being able to use all three planes for outlining. It was interesting to note that they also enjoyed using the software; a common finding with 3D interactive applications.Reference Bridge, Appleyard, Ward, Philips and Beavis19 Despite this, orientation in 3D was evidently a common problem as it represented a new paradigm. Use of the 3D glasses for an immersive experience was generally felt to be unhelpful with participants reporting flickering and the need for direct eye contact with the screen. There was a suggestion that point placing was easier in 3D but that it had limited value for the sculpting.
Overall, all the participants felt that the software was capable of producing clinically acceptable outlines rapidly (when compared with traditional methods) and that the multi-planar capability allowed for improved outlining of the prostate apex. Recommendations for future improvement were provided in relation to 3D orientation tools and improved sculpting visualisation. Aside from feedback concerning the software development, emerging themes from the wider discussion were strongly focussed on the benefits arising from rapid initial volume generation, the difficulties with the transition to 3D orientation, the benefits of multi-planar outlining and the challenges of the 3D volume-sculpting tools. Table 2 contains representative quotes relating to these themes and the issues arising are addressed in the discussion section.
Table 2 Summary of themes
Abbreviations: 3D, three dimensional; CT, computed tomography.
Discussion
Limitations
The findings within this paper are drawn from a small cohort and thus a number of limitations are worthy of note. First, the aim of the work was to evaluate the software in a development stage; further development is ongoing ahead of more rigorous formal testing. Although a structured and impartial approach was adopted for the focus group, facilitation was conducted by the researcher and the Hawthorne effect (where participation in research modifies participant perceptions) may be an influencing factor. The following discussion points, however, relate to wider issues concerning the new paradigm and it is unlikely that participant bias would have affected these themes. The provided ‘test’ patient was another potential influence as the anatomical boundaries were clear and the volume was standard. Furthermore, as participants only had access to a single dataset they would have experienced increased familiarity with the patient compared with a clinical scenario.
Outlining speed
A constant theme throughout the feedback and focus group discussion related to both the rapidity of initial volume creation using the multiple planes and the challenges associated with the sculpting 3D editing tool. Although many of these concerns related to visualisation of both volume and underlying CT, there were also issues related to effective use of the 3D volume editing itself. It is clear that future development will need to focus on improving the editing tools and perhaps provide firm guidelines for the initial point placement phase to minimise the need for editing.
Transition to 3D
All of the participants reported difficulties with 3D orientation resulting from the software’s ability to visualise the volume and CT planes from multiple angles. There were clear feelings of ‘being lost’ and requests for an orientation model or ‘reset’ button were submitted. Related themes concerned the wider 3D orientation and interpretation issues with participants expressing their unfamiliarity with different planes and views. The transition to ‘thinking in 3D’ was a common concern and it was clear that anatomical and CT training had not fully facilitated this ability. With the advent of magnetic resonance imaging-guided radiotherapy 3D anatomical training may be a key addition to the curriculum.
Multi-planar outlining
Although all participants provided positive feedback concerning the ability to utilise different planes within the software, it was clear that there was variation in how this was used. Some relied on axial slices for point placement and other planes for volume editing, whereas one registrar enjoyed using all three planes to produce the initial volume. The seminal vesicles were easily identified on sagittal planes and the coronal was useful for identifying the prostate apex. Further guidance for optimal point placement is being developed ahead of pre-clinical testing.
Volume editing tools
The sculpting tools allowed growth and shrinking of the target volume to match underlying CT data. There were some difficulties visualising both at the same time, although use of transparency tools helped some participants with this. In addition, participants struggled with editing multiple slices at the same time. The potential strength of the 3D sculpting is that adjustment of a contour on an individual slice will result in a graduated change on adjacent slices. This requires the user to identify the point of maximum divergence between volume and CT, and effect the change with that point as the centre. The ‘strength’ of the change also needs to be amended to ensure that only the appropriate adjacent slices are influenced. More guidance with this is clearly required to facilitate future use of 3D editing tools.
Training
It was clear from the discussion that all participants favoured a ‘hands-on’ approach to learning; the provided video was not used at all and written instructions were seen as a reminder. It was also interesting to see the participants affirming the value of a collaborative approach to learning. Peer learning and support was evidently highly valued within this group.
Conclusion
Mesh generation from a small number of points placed on a range of planes is a rapid and effective means of target delineation, although further work is needed to improve multi-slice volume sculpting before more formal pre-clinical testing. Orientation within orthogonal planes and 3D navigation is challenging and may indicate a need for a paradigm shift in anatomy and CT training. Formal clinical testing of the software is now under way and aims to determine how the process compares with the current 2D system in terms of accuracy, speed and user feedback.
Acknowledgements
The authors are grateful for the invaluable support provided by the Radiation Oncology Mater Centre in regard to dataset collection and focus group participants; particularly the direct assistance of Debbie White, Robyn Guidi and Cathy Hargrave. The constant support and technical expertise of the End-to-End Visuals software development team is also gratefully acknowledged. The advice and experience provided by Dr Rob Appleyard was very much appreciated.
Financial Support
The research was supported financially by a PhD Scholarship and project funding from the Queensland University of Technology Science and Engineering Faculty.
