Treatment planning

The primary goal of radiotherapy treatment planning is to design a set of ionising radiation beams that deliver high doses to the tumour, while minimising dose to healthy tissue and vital organs. Large amounts of detailed patient anatomy and functional data are acquired during the planning phase from different imaging modalities such as: CT, MRI, PET and SPECT. With dedicated computer simulation software this data is used to iteratively determine the optimal radiation beam orientations and beam shapes with a resulting radiation dose distribution in the patient around a target volume. The efficient design and final choice of the optimal treatment plan remains a non-trivial task as the radiotherapy professional needs to understand and be able to visualise the dose coverage of anatomic structures in three dimensions, and increasingly four dimensions if reproducibility and time effects are included. Therefore accurate and informative 3D visualisation is required for intuitive and quick evaluation of competing plans. Although the 2D monitors used in conventional radiotherapy planning systems allow for displaying 3D images of the dose distribution together with patient anatomy data, typically only 2D flat surfaces are used. Therefore treatment planners usually view patient anatomy slice-by-slice, in three orthogonal planes (axial, sagittal and coronal), showing images from different modalities side-by-side or using relatively simple overlays.

In radiotherapy, the first use of a stereoscopic display was reported in 1997 by Hubbold et al.,Reference Hubbold, Hancock and Moore⁴ who coupled an autostereoscopic display to a direct volume rendering algorithm. Two sets of preliminary experiments investigated whether subjects could achieve better depth judgements with stereoscopic images than with monoscopic ones and to explore the discomfort caused by aliasing with low-resolution images. Aliasing is a mathematical effect of signal sampling and reconstruction, leading to the appearance of artefacts on digital images reconstructed from under-sampled images. With 2D images the only effect of aliasing is the presence of artefacts, but with stereoscopic images, aliasing artefacts can be present only in one of the two views that have to be combined to give the perception of depth and this can result in user discomfort.

The authors classify their results as preliminary, but observe that the results do demonstrate an overall advantage of stereoscopic over monoscopic viewing of transparent images generated by direct volume rendering. The investigation of the technique applied to radiotherapy data shows an observable improvement in the sense of depth to the image. Their results also showed stereo visualisation to have no benefit in a number of cases. The authors state that there is nothing in the way the visualisation was implemented that clearly explained this. They postulate that some subtle differences in shading on the surfaces may be more important than the stereoscopic disparities in the difficult cases. Judging from the comfort ratings, the results agree with evidence from other studies that effects of spatial aliasing may be to some extent ignored by users when interpreting stereo images.

Use of an autostereoscopic display for robotic radiosurgery planning was described by Schlaefer et al.Reference Schlaefer, Blanck and Schweikard⁵ An autostereoscopic display from SeeReal Technologies GmbH Dresden was used. The two different views of the scene required for stereoscopic viewing were vertically interlaced in the 2D display. In order to generate the user's 3D perception a mask of beam splitters are superimposed onto the display, allowing for two different views from two different positions each corresponding to the observers left and right eyes.

Treatment plans for robotic radiosurgery consist of a large number of beams directed towards the target volume. Software to visualise the resulting 3D dose distribution and the beam directions are implemented using the Visualization Toolkit (VTK).Reference Schroeder, Martin and Lorensen⁶ A hypsometric colour scheme was used to identify hot and cold spots in the target volume (i.e., regions of high and low dose, respectively). An existing treatment plan with 1,200 beams for an intracranial tumour was projected onto the autostereoscopic display to assess the spatial extent of hot and cold regions along with the orientation of the beams. Based on the visual information obtained from the 3D visualisation, 20 beams were manually added to the existing plan in such a way that a large number of cold voxels were hit, but hot voxels were avoided, helping to reduce dose to hot spots and increase dose to cold spots. An inverse planning algorithm was implemented to re-optimise the plan and the result was compared with the original plan. The original plan consisted of 119 weighted beams with a total of 21,763·3 MU, while the plan obtained after adding 20 beams and optimised to discard the less efficient beams, consisted of 123 beams requiring 21,610·7 MU. The manually added beams were all retained with maximum weight by the optimizer algorithm. It was concluded that the visualisation tool was useful to guide optimal beam placement.

An immersive VR simulation environment RTStar (University of Hull, UK), complemented with software enhancing the visualisation and simulation by using 3D stereoscopic data projection and geometric volume analysis, showed benefits to optimise beam orientations for axial 7-field prostate integrated prostate cancer center (IMRT) plans.Reference Shang, Williams, Beavis, Ward, Sims and Phillips⁷ For eight existing prostate IMRT plans the beam geometry was further improved. In the 3D environment most beam angles were modified achieving a better dose homogeneity in the target area (1·9% reduction in global maximal dose). Also rectal and bladder doses were reduced with a 2·3% and 12·9% reduction in maximum dose, respectively. The authors also emphasised that the 3D stereoscopic viewing eliminated the risk of designing a plan that could not be delivered because of a gantry collision with the patient.

The first system to integrate volumetric 3D visualisation with treatment planning in a true 3D planning system was described in three presentations.Reference Gong, Kirk and Zusag^8–Reference Chu, Zhang and Yurkewicz¹⁰ The system combined two commercially available components: the Perspecta Volumetric display System (Actuality Systems, Bedford, MA, USA) and the Philips Pinnacle3 Treatment Planning System (Philips Medical Systems, Madison, WI, USA). The Perspecta volumetric display (Figure 1) works by projecting a sequence of 2D images onto a swiftly rotating omnidirectional diffuser screen enclosed in a polycarbonate resin dome.

Figure 1 The Perspecta System volumetric display. Reproduced with kind permission of the American Institute of Physics from Gong et al.Reference Chu, Zhang and Yurkewicz¹⁰

The treatment plans could be easily transferred between Pinnacle and Perspecta, using Perspecta for display and modification while using Pinnacle for dose calculations. To assist the radiation oncologist during the review of treatment plans, the calculated dose distribution could be rendered in a volumetric 3D display (Figure 2) where anatomical information is visible in a more natural and efficient way than on 2D monitor screens. This enables treatment planners to create complex beam arrangements faster than with 2D monitor screens. In conventional planning the planner iteratively modifies and reviews the 3D beam geometry in relation to organs at risk (OAR) and planning treatment volumes on a 2D screen with 2D image views. This requires manipulation of the 3D image with a pointing device (mouse, trackpad, trackball, etc.), however, with the Perspecta display, the planner only needs to move around the display to change their point of view.

Figure 2 Perspecta autostereoscopic display of images transferred from Pinnacle planning system. (a) Region of interest can be tagged with different colours. In the original article right and left lungs are in green and purple respectively. Tumour is bright green in right lung. Nodal disease is in blue. (b) CT-based image. In the original article different colours are assigned to specific ranges of CT densities (e.g. purple is assigned to bone). Reproduced with kind permission of American Institute of Physics from Gong et al.Reference Gong, Kirk and Zusag¹¹

Quality assurance of the system was considered by Gong et al.Reference Gong, Kirk and Zusag¹¹ Dose at sampled points were checked and found consistent with Pinnacle within 1% or 1 mm. The 3D spatial display of images, contours, and dose distributions exported from Pinnacle to Perspecta were consistent with Pinnacle display. Distances measured by the 3D ruler in Perspecta agreed with Pinnacle. A clinical evaluation was reported in 2009Reference Chu, Gong and Cai¹² with data from 46 patients: 12 brain, ten lung and 11 abdomen/pelvis cases, together with 13 patients from a pilot study. Perspecta plans were considered better in terms of reduced dose to OAR in 28 patients (61%). Lower doses were delivered to critical organs: 34% to the optical chiasm, 17% to the bladder, 10% to the liver, 30% to the kidney and 40% to the lungs. Surprisingly, in 14 patients (30%) Perspecta plans were worse than corresponding plans produced on a conventional planning system, and equivalent in four patients. This was attributed to volumetric 3D planning tools not yet being fully developed and the treatment planners not as familiar with the operation of the Perspecta 3D system as with the conventional planning system. The observation of unfamiliarity with the system does emphasise the need for intuitive user interfaces to effectively process and absorb large amounts of data. Despite this it was claimed that oncologist's evaluation of plans using 3D visualisation was more efficient than using 2D visualisation, because all plan information (target coverage, normal tissue sparing and the locations of hot or cold spots) from all CT slices were available simultaneously. Acceptance and quality assurance aspects and the accuracy and consistency of presenting dose information on Perspecta were also considered. It was suggested that the Perspecta display software (PerspectaRad) could be improved with the ability to commission the display to the user's specific treatment machine to include treatment machine limits.

A VR system for the evaluation of treatment plans was developed by Patel et al.Reference Patel, Muren, Mehus, Kvinnsland, Ulvang and Villanger¹³ This was installed and networked in a radiotherapy conference room at the Haukeland University Hospital. Data were exported from the planning system and fed into the VR application for visualisation. The VR environment consisted of a passive stereo setup made by a semi-rigid back projection screen (BARCO Pas-Cad) and two overlapping LCD projectors (BARCO SXGA 3000 ANSI). Selective views for the right and the left eye were implemented by using circular polarisation filters on the projectors and matched in the user's glasses. The software ran on an up-gradable, low cost and powerful PC graphics system. By using a 2D transfer function CT and dose data were combined, where the dose at the surface of outlined or segmented structures could be rendered with good quality graphical results (Figure 3). In their paper on the clinical evaluation, the authors concluded that the adopted hardware solution was well suited for collaborative multi-disciplinary team sessions. Users can both see each other and the data simultaneously and the degree of nausea that may appear when working in VR is tolerable. The system also provides the opportunity to inform patients about the procedure they are going through in a more informed way than traditional verbal explanations. Since the framework this system is built on already supports tracking, the authors point out it is only a question of availability of resources to be able to have it working in a completely immersed environment. The authors state that a user study quantifying the hypothesised advantage of VR (compared with the existing planning software) would be needed to further explore the potential of this software.

Figure 3 (a) Visualisation of the dose distribution on the surface of a selected CT data volume by making all but the lowest values of the transfer function opaque. (b) Visualisation of the dose distribution on the bony structures is achieved by making areas of high CT values opaque and areas of low CT values transparent. Reproduced with kind permission of Elsevier from Patel et al.Reference Patel, Muren, Mehus, Kvinnsland, Ulvang and Villanger¹³

Butler et al.Reference Butler, Teh and Bell¹⁴ investigated the impact on radiation oncologists’ decision making from presenting information in 2D, 3D and stereoscopic visualisation. Stereoscopic visualisation was implemented on an Apple workstation (8 core 2·8 GHZ processors, 16 GB RAM, NVIDIA Quadro FX 5600 1·5 GB stereo 3D dual link DVI graphics card) and a 24-inch stereoscopic monitor (Planar Systems Inc.) using Osiris software. Plans for ten patients with head and neck carcinoma generated on Pinnacle (Philips) and TomoTherapy (TomoTherapy Inc.) planning systems were evaluated in 2D, 3D and stereoscopic visualisation by three radiation oncologists. The clinicians were asked if the decision-making process was changed as the display progressed from 2D, to 3D, to stereoscopic visualisation. The information provided by stereoscopic visualisation of the relationship of the target to the normal structures, with visualisation of isodose curves with depth perception, was considered clinically significant by the radiation oncologists in all ten cases. Stereoscopic visualisation did not result in changing the dose constraints for any of the plans, although the 3D display provided added assurance that the plans were safe and clinically acceptable. The authors report that in their department head and neck cancer cases are now routinely reviewed with stereoscopic visualisation.

Treatment delivery

Radiotherapy is typically given in daily fractions delivered over a number of weeks. The main challenge is to ensure that, for each fraction of the treatment, the dose delivered to the patient is as close as possible to the planned dose, taking into account body and organ variations. These occur for a host of reasons, of which internal motion because of breathing and tumour shrinkage due to treatment are examples. Checking patient positioning is traditionally based on laser alignment with skin markers and treatment room imaging for verification. More recently, diagnostic quality images acquired in the treatment room immediately before treatment have become available for treatment machines equipped with kV imaging panels and cone beam CT facilities. In routine clinical practice typically 10–15 minutes time slots are scheduled daily for each patients’ treatment. To maintain this workflow, the increasing wealth of imaging information made available from new technologies immediately before and during treatment, needs to be quickly processed by the treatment team. AR techniques have much potential in this respect where they can help to make an optimal use of the setup verification images, improve accuracy of patient positioning or speed up the patient positioning decisions enabling fast but accurate treatment deliveries.

Deutschmann et al.Reference Deutschmann, Steininger, Nairz, Kopp, Merz and Wurstbauer¹⁵ developed a system that enables an overlay of inner structures delineated on CT data (target volumes and OAR) and field boundaries on the X-ray plane in real time (i.e., while fluoroscopy is performed, Figure 4).

Figure 4 Fluoroscopy images matched to projection of outlined structures. Reproduced with kind permission of Springer from Deutschmann et al.Reference Deutschmann, Steininger, Nairz, Kopp, Merz and Wurstbauer¹⁵

The simultaneous display of computer graphics imagery and real material is used to correct patient's positioning errors. More precisely, a projection onto the current X-ray image of 3-D structures not visible in the fluoroscopy because of missing soft-tissue contrast is implemented. Setup deviations between volumetric imaging and simulation were considered for 701 patients. The results of patient position adjustments based on the overlay of CT data and fluoroscopy images were superior to the results based on conventional registration of Digitally Reconstructed Radiographs and Electronic Portal Images. Applying the fast planar imaging technique and 2D-3D registration, translation errors could be corrected. A fast way to easily track rotations on planar images is still to be found.

A method for AR-facilitated patient set-up was proposed by Talbot et al.Reference Talbot, Meyer, Watts and Grasset¹⁶ in a pilot study using an anthropomorphic phantom. The 3D external body contour was obtained from planning CT data. With the phantom positioned on the treatment couch, the 3D body contour from planning CT was superimposed onto a real time video image of the phantom, using AR tracking software (Figure 5). An operator could view the monitor placed outside the treatment room and visually confirm correct positioning throughout set-up and treatment. The performance of the system was investigated by using it to position an anthropomorphic phantom without the aid of additional set-up methods. The translational set-up errors were <2·4 mm and the rotational errors <0·3°. These results demonstrated the feasibility of using AR for patient positioning. The authors state that the developed technique needs further investigation before clinical use

Figure 5 Body contour from CT scan (grey virtual image) and patient's true image (red anthropomorphous phantom). Reproduced with kind permission of Springer from Talbot et al.Reference Talbot, Meyer, Watts and Grasset¹⁶

True 3D display of delivered dose was investigated by Santhanam et al.Reference Santhanam, Willoughby and Kaya¹⁷ They presented a visualisation framework that combines a computer-based simulation of real-time lung tumour breathing motion and dose accumulation with an AR display system (Figure 6). The simulation framework provides visual insights on the variations in the quality of therapy for changes in the patient's breathing conditions from the pattern acquired with the 4D planning CT scan. The display system enhances the clinician's understanding by adding a 3D depth perception of the dose accumulation pattern. The framework is a tool for presenting both preoperative studies and intra-operative treatment efficacy analysis when coupled with a real-time respiration monitor. Evaluation was carried out using six clinical experts and results showed that, using AR compared with a 2D monitor, the experts were more able to efficiently perceive the radiation dose delivered to various aspects of the moving tumour and the surrounding normal tissues. Also, a quicker detection of radiation hot spots that are critical to minimising damage to healthy tissue was observed.

Figure 6 Real-time lung tumour motion (due to breathing) and dose accumulation, displayed on AR active glasses display system. Reproduced with kind permission of IEEE from Santhanam et al.Reference Santhanam, Willoughby and Kaya¹⁷

Abbrevation: Augmented Reality (AR)

Wang et al.Reference Wang, Lee and Fang¹⁸ developed a volume visualisation system with AR interaction, using the Insight Segmentation and Registration Toolkit¹⁹ and the VTK.Reference Schroeder, Martin and Lorensen⁶ Surface comparisons between clinically relevant isodose levels and planning volumes can give more information than conventional dose–volume histograms. A radiotherapy plan for a brain tumour was used to evaluate the software. The authors concluded that the volume visualisation with AR interaction helped the radiation oncologists to observe the under-dosing or over-dosing regions in 3D and to gain insight into the degree of dose inhomogeneity, such as hot or cold spots seen in radiotherapy plans.

A target visualisation system for real-time target verification was reported by Chen et al.Reference Chen, Chang, Liu and Chen²⁰ Image data from ultrasound (US) and CT scans were captured and registered. US-CT image registration was integrated with a human-commanded 6-degree-of-freedom robotic manipulation of US probe and linear accelerator to form an innovative radiotherapy system. Using an automated algorithm, target organs were segmented in CT images, US images were transformed and reconstructed to match each orientation, and image registration was performed in real-time with acceptable accuracy. This image transformation allowed oncologists to visualise CT image-reconstructed targets outside BEV via an US probe positioned non-coplanar to the beams’ plane. Using robotic manipulation allowed oncologists to remotely control the US probe, dynamically track and real-time monitor the coverage of target volumes within a BEV during a simulated beam-on situation. The authors concluded that their target visualisation system might provide a remotely accessible and real-time way to visualise, verify and justify the use of more conformal radiotherapy treatment technologies.

Mass produced devices to interact with clinical data

Technology is changing at a rapid pace and some developments have the potential to profoundly change the way clinical professionals interact with computer-generated data. In particular there is a growing number of adaptations of consumer market technology (game controllers, handheld devices) as an alternative to highly specialised hardware.

Accuray PlanTouch²¹ is the first commercially available software application in radiotherapy that allows oncologists to remotely review and approve radiotherapy plans on the Apple iPad. The application's interface is fully integrated with the CyberKnife planning software. Oncologists can review dose volume histograms, isodose curves, contours and images and approve treatment plans directly from their tablet devices. Treatment planning displays are designed and formatted specifically for the iPad's screen and can be manipulated using the iPad's touch screen capabilities. There are a number of other companies supplying equipment to radiotherapy clinics releasing software for users to review or approve information on handheld devices, for example clinical information that is available during ward rounds.

An example applied to brachytherapy is given by Butler.Reference Butler²² When performing a needle implant for advanced gynaecological malignancies, it is often difficult to predetermine parameters like needle length to target, proximity to bowel and vascularity. To overcome these difficulties laparoscopic guidance is often required. In this example, 3D interactive volumetric display software, utilised by other subspecialties (e.g., cardiovascular interventions), is evaluated to see if it can replace laparoscopic guidance. For a patient with a clinical condition preventing the use of laparoscopic guidance, needle placement utilising the visualisation system as guidance was evaluated. A CT angiography study was fused with a PET imaging study and used to define and refine the target. Before going to the operating room, guidance data (ideal trajectory of needles and other relevant parameters) were predetermined and recorded on an iPad. The iPad was taken into the operating room and used to display the guidance data for additional insight during the intra-operative procedure to complement fluoroscopy, the only other diagnostic imaging available in the operating room. Postoperative CT imaging verified needle placement to be within 2 mm of ideal placement. There were no operating room complications. The author concludes that 3D volumetric reconstructive software can assist the radiation oncologist in preplanning brachytherapy needle placement but, in order to optimise the 3D volumetric reconstruction process, the radiation oncologist needs to understand the geometry of the CT datasets.

Nakata et al.Reference Nakata, Suzuki, Hattori, Hirai, Miyamoto and Fukuda²³ designed a system for 3D and 4D image manipulation using optical tracking AR integrated with a smart-phone. The authors observed that the mouse, the most widely used pointing device on personal computers, was originally designed and best suited for control of 2D cursor movement rather than complex 3D image manipulation. In this work, 3D and 4D images obtained with CT and magnetic resonance imaging were displayed on a PC running Windows 7 (Microsoft). The AR software was based on ARToolKit,²⁴ a video-based tracking technique without a precise video controller UI. In this novel system, the authors used the iPhone or iPod Touch as a remote control device. The functions of this remote control included zooming in or out on the AR object, capturing the PC screen, and playing or pausing the 4D object, and were achieved using a Wi-Fi connection. The system allowed radiologists to browse 3D or 4D images from CT and MR imaging by using an iPhone or iPod Touch to control the PC. AR images required surface rendering, which was achieved by using OsiriX²⁵ imaging software. The surface image data were transferred to a Windows PC with a novel AR viewer developed by using the ARToolKit. The PC was equipped with a web camera allowing recognition of the AR fiducial marker. The software allowed radiologists to manage the AR images using either an iPhone or a conventional two-button mouse as a controller for comparative evaluation. The iPhone or iPod Touch was placed in a plastic jacket with an optical tracking marker printed on the back. The radiologist could move and twist the iPhone or iPod Touch with the optical marker facing the web camera of the PC and the software running on the PC was able to recognise the optical marker. The AR images were shown on the LCD display of the PC with real-time tracking as a superposed model of the optical marker on the background of the real-world view as seen on the LCD monitor. When the radiologist moves the iPhone or iPod Touch, the 3D object on the LCD monitor moves and scales itself at the same time in an intuitive manner. The authors concluded that, although strict comparisons of user interface performance between the AR techniques and a conventional mouse are difficult, AR had high interactivity and 3D image manipulation required no special training. Therefore, performance evaluation of the AR technique was performed without special warm-up trials. They compared the performance of the AR 3D image manipulation method with that of the conventional method. Three different 3D objects were evaluated by 12 different testers. The times for three horizontal predetermined rotations of each 3D object were measured. The average times to perform the rotations with the AR method were statistically shorter than those achieved with the conventional two-button mouse in all three cases.

In research carried out by Gallo et al.Reference Marra, Gallo and De Pietro^26, Reference Gallo, Minutolo and De Pietro²⁷ a novel user interface to provide a direct interaction with medical imaging data in 3D space by off-the-shelf input devices was proposed and evaluated. The interface was implemented as open-source software and integrated into the open-source medical image viewer Medical Imaging TOolkit MITO.²⁸ Both a common mouse and a Wii remote controller were used as input devices (Figure 7).

Figure 7 Direct interaction with medical imaging data in 3D space integrated into the open-source medical image viewer MITO. Reproduced with kind permission of Elsevier from Marra et al.Reference Marra, Gallo and De Pietro²⁶ Abbreviation: Medical Imaging Toolkit (MITO)

The interface featured a novel rotational technique using the geometry itself as the rotation handle. A user study showed that the proposed techniques were easy-to-learn and outperformed the virtual trackball technique in the task of rotating complex-shaped objects.

AR and VR techniques in other domains showing potential application to radiotherapy

In this section a number of AR/VR examples in medicine, or even outside of medicine, illustrating potential applications in radiotherapy, are considered. Registration of the real world, as seen for example through a device's camera, and computer-generated imagery being merged with the scene is a far from trivial task, especially in real time. The examples in this section, however, demonstrate that this can be accomplished with error margins of the order of 2 mm or less. This spatial error margin is accepted in many radiotherapy techniques. A considerable improvement of accuracy has been achieved between the first example considered below dating back to 2002 and the more recent studies.

In 2002, Mitchell et al.Reference Mitchell, Wilkinson, Griffiths, Linsley and Jakubowski²⁹ described a method of image guidance for neurosurgery using the surgeon's binocular depth. For patients with brain tumours, stereoscopic pairs of images of the surface rendering of the head and the surface rendering of the tumour were produced using MRI data. The two pairs of images were colour-coded and combined into one pair of 35-mm slides viewable using an ad-hoc constructed stereoscopic viewer. Registration was achieved by moving the stereoscope in space until the virtual images of the rendered surface of the head coincided with the real head. The stereoscope was then locked in position and the virtual image of the tumour was projected inside the patient's head, allowing the surgeon to locate the tumour. Six clinical cases were considered. A lateral accuracy of 10–15 mm and a depth accuracy of 5–10 mm were achieved.

Another application of AR to conduct minimally invasive orthopaedic surgery was reported by Liao et al.Reference Liao, Ishihara, Tran, Masamune, Sakuma and Dohi³⁰ This paper describes a precision-guided surgical navigation system, that consisted of a combination of laser guidance and 3D autostereoscopic image overlay. Using an integral videography imaging method, images of surgical anatomic structures were superimposed onto the patient without the need for special viewing or tracking devices (Figure 8).

Figure 8 Laser guidance with autostereoscopic image overlay: (a) IV image overlay device and patient/image overlay; (b) alignment of surgical instrument; (c) image-patient registration results and surgical path guidance of laser beams; (d) operational diagram. Reproduced with kind permission of Elsevier from Liao et al.Reference Liao, Ishihara, Tran, Masamune, Sakuma and Dohi³⁰

The image overlay system was integrated with a laser guidance system to improve the placement accuracy of surgical instruments. Experimental evaluations showed that the error in guiding a linear surgical instrument towards a target was within 2·48 mm with a standard deviation of 1·76 mm, and the orientation error was 2·96° with 2·12° standard deviation. This is the same order of spatial accuracy required in modern external radiotherapy.³¹ The authors concluded that the system can support surgeons during their operations and enables them to intuitively identify the insertion path of the surgical instrument. It was also stated that accuracy could be improved by using a display device with a higher pixel density and a higher precision laser guidance device. This would make the system of practical use not only for orthopaedic applications, but also in other medical fields. An application in radiotherapy for the 3D autostereoscopic image overlay systems could be displaying patient's outlined anatomy, planning volumes and planned treatment beams, in the treatment room overlaid to the patient, before treatment as a verification aid.

Tomikawa et al.Reference Tomikawa, Hong and Shiotani³² developed a VR navigation system with open MRI for breast-conserving surgery. The authors report an estimate of the mismatch between VR content and real distance to be in the order of magnitude required by radiotherapy applications. Clear analogies between concepts considered in this study (image registration, surgical margins) and fundamental concepts in radiotherapy (image registration, treatment and OAR margins) suggest possible applications to radiotherapy. In this work dye marking of a breast tumour, serving as guidance for surgical resection, was performed using a real-time 3D VR navigation system. A pilot study using a 3D phantom was carried out for quantitative and qualitative evaluations and a mean mismatch between the navigation system and real distance of 2·01 ±0·32 mm was reported. A study based on two patients was also carried out. Histopathological examinations of the resected specimens of the two patients showed that the surgical margins were free of carcinoma cells.

Kim et al.Reference Kim, Hong and Joung³³ developed a dual surgical navigation system for endoscopic surgery that used VR and AR techniques together to obtain additional depth and visual information for organs. The VR environment was developed to visualise the spatial relationships among the target organs, endoscope and surgical tools. The AR environment was used to display the raw endoscopic images with the nearby organ images overlaid, as obtained from CT and MRI scans, which would otherwise be invisible to the endoscopic probe. Surgeons were enabled to better understand the surgical environment around the target, increasing the safety and accuracy of surgical procedures. Image registration between endoscopic and CT/MRI data was realised using a surface-tracking technique. A virtual model of the endoscope and surgical instruments were displayed into the VR and AR environments based on tracking of the endoscope and instruments position; tracking was carried out using either an optical position sensor or an electromagnetic sensor. Raw endoscopic images are affected by distortion due to camera optics. In order to accurately overlay CT/MRI data on to endoscopic images, camera optics transformation was applied to CT/MRI images. This was realised through camera calibration procedures that allowed the relevant geometric parameters and the lens distortion coefficients to be obtained. Rendering was based on the parameters of the endoscopic camera, so the rendered results mimic the shape and size of the real object, just as it would appear from the endoscopic video camera. In phantom experiments, the translational overall registration error was <2 mm with CT images and an optical position sensor. Higher errors were observed using an electromagnetic tracking sensor and MR images. Correlation between errors and endoscopic camera angles were also observed. The dual navigation system was applied to a cochlear implant surgery for evaluation in a clinical setting. The system was applied to a surgical microscope instead of an endoscope and the clinical application analysis confirmed the feasibility of such a system in the operating theatre. The surgeons who have observed and used the system in the clinical study declared the usefulness of the dual navigation system, considering it to have significant advantages compared with conventional systems.

Gavaghan et al.Reference Gavaghan, Oliveira-Santos and Peterhans³⁴ developed a portable image overlay projector for the visualisation of surgical navigation data and conducted some tests on phantoms to explore the capabilities of the device. Monitor-based visual feedback for image-guided surgery requires the surgeon to perform time consuming comparisons and diversion of sight and attention. Their system utilised a portable image overlay device comprising a navigation computer unit, an infrared-based optical passive tracking system (Vicra, NDI, CA, USA) and touch screens for user interaction and visual display (Figure 9). The optical cameras track known configurations of retro-reflective marker spheres. The system was tested on a range of anatomical models and for planning different surgical interventions (liver, cranio-maxillofacial, orthopaedic and biopsy). The visualisation approach was found to assist in spatial understanding and reduced the need for sight diversion throughout the simulated surgical procedures. The portability of the device and intuitiveness of use suggest an expansion of its application to other parts of medicine, including radiotherapy especially for the patient positioning phase, where monitor-based systems would pose problems of portability and ease of use inside the treatment bunker.

Figure 9 (a) Stereotactic instrument guidance system with integrated image overlay device. (b) (A) Image overlay AR for navigated liver surgery on a patient-specific rigid model and (B), pig liver tissue; (C, D) image overlay AR for navigated cranio-maxillo facial surgical planning; (E) and (F) image overlay AR for navigated orthopaedic tumour resection. Reproduced with kind permission of Springer from Gavaghan et al.Reference Gavaghan, Oliveira-Santos and Peterhans³⁴

Another study reporting an estimate of the registration accuracy between the real scene and AR content is Low et al.,Reference Low, Lee, Dip, Ng, Ang and Ng³⁵ where AR neurosurgical planning and navigation system (the DEX-ray) for surgical excision of meningiomas were implemented. The DEX-ray system is based on the Dextroscope (a stereoscopic 3D pre-operative planning system) and allows the transfer of the Dextroscope planning data into the operating theatre by displaying it on to real-time images, producing in this way a video-augmented presentation of the surgical scene, further enhancing the appreciation of the tumour's location in 3D space. The DEX-ray has an image distortion <0·4 mm in the AR mode and a registration accuracy of 1–3 mm. The AR feature allows for navigation with 3D graphics beyond the visible surface of the surgical site, but yet always in direct context to it, providing a see-through effect and resulting in a more direct understanding of the hidden anatomy relevant to the surgical procedure.

Several architecture-oriented applications of AR implementing visualisation of virtual buildings overlaid on the real scene are found on the web. CityViewAR³⁶ is an AR application designed at Canterbury University, New Zealand, to give a visual reminder of how the city of Christchurch (New Zealand) used to look before the earthquake in 2010. Similar applications, but in the entertainment domain, are reported online, for example by String Labs Limited.³⁷ These last two examples support the view that applications of AR to radiotherapy based on self-tracking capability of tablet devices are feasible. Moreover, for these consumer devices programming techniques are more reusable than for highly specialised devices requiring more low-level programming. This may make the required programming knowledge more readily available, although for some time yet multi-disciplinary collaboration involving specialist developers is likely to be needed to make best use of these tools. The accuracy and reliability achievable by these systems needs further investigation.

Summary and conclusions

The review of novel systems based on AR user interfaces considered in this paper suggests a future where radiotherapy professionals will be able to manipulate 3D and 4D images in a more intuitive and efficient way, possibly anywhere, anytime. This is likely to enable a better use of large amounts of information available with modern diagnostic tools, but more radically may change how collaborative tasks such as clinical case discussions or complex case planning can be performed by allowing experts to be off site.

The reviewed radiotherapy studies point to potential benefits from AR and VR at various parts of the treatment process. Most of the early studies suggest that research in this field will need to address current limitations around operator discomfort, ease of use and sensible selection and accuracy of information to be displayed.

Accuracy of the registration between virtual content and the real scene is reported to be in the order of millimetres or less in recent VR/AR applications to surgery.Reference Liao, Ishihara, Tran, Masamune, Sakuma and Dohi^30, Reference Tomikawa, Hong and Shiotani^32–Reference Low, Lee, Dip, Ng, Ang and Ng³⁵ The required accuracy for most advanced radiotherapy techniques is of the same order of magnitude.³¹ A reduction of the registration error from 5–10 mm to 1–2 mm has been achieved from 2002Reference Mitchell, Wilkinson, Griffiths, Linsley and Jakubowski²⁹ to the present date. If this trend continues the registration error will be made significantly smaller than the spatial accuracy required in radiotherapy for patient positioning and treatment planning, possibly negligible. In that case the potential portability to radiotherapy as setup and verification tools of these and similar techniques found in the literature is feasible in the foreseeable future.

Despite promising results, AR has not taken off in clinical radiotherapy to date, with the exception of teaching and training applications. This may be partly because of the high cost of equipment, explaining the difficulties to develop this into commercial tools. However, the situation is rapidly changing and the cost of high specifications AR and VR capable hardware is considerably decreasing.

Although there was a considerable time lapse between the firstReference Hubbold, Hancock and Moore⁴ and the secondReference Schlaefer, Blanck and Schweikard⁵ study reporting 3D display applications in radiotherapy, since that time the number of publications in this field are steadily increasing, indicating a growing interest from the medical and scientific communities.

The majority of the reviewed studies used costly hardware not widely available commercially, especially holographic displays and state-of-the-art large flat-screen 3D displays and projectors. More recent studies have started to use readily available devices (Wii remote, iPad, iPhone, iPod Touch), where interestingly none of the systems found based on these devices have reported problems of user discomfort, requirements of special training or cost.

The use of tablet and handheld devices (e.g., iPad, iPhone, iPod Touch and Android equivalents) is growing fast and these devices are being rapidly adopted in the medical field, particularly for medical imaging applications. Most tablets also have a built in camera that can be utilised for AR applications. However, computing power on a tablet is limited and the real time registration of the camera image and computer-generated graphics remains a challenge.

In summary, the development of small mass-produced tablet devices coming on the market will allow the user to interact with computer-generated information more easily, facilitating the application of AR and VR to radiotherapy practice. The increased connectivity, making information available anywhere, anytime and enabling virtual presence at remote multidisciplinary team meetings, is likely to significantly change how radiotherapy professionals will work, to the benefit of patients.