Introduction
The provision of healthcare services in developing or resource-poor countries is a difficult and complex problem. One challenge is providing effective medical education. With far greater resource constraints, learners encounter a steep learning curve and are often graduated to greater responsibility without much hands-on practice. Otological surgery is a particular challenge, requiring a very unique set of dexterous skills, including working with a microscope in a very complex and small space. Furthermore, the anatomical complexity of the area places the operator millimetres from potentially dangerous landmarks.
Acquisition of these skills requires exposure and practice. Unfortunately, work-based training opportunities are limited given physician, time, resource and patient constraints. An increased recognition of these constraints highlights the need for alternative forms of medical education. Simulation is a recognised method that is gaining popularity.
It has been shown that simulation can help develop otolaryngology skills prior to application in a clinical environment.Reference Volsky, Hughley, Pierce and Kesser 1 , Reference Clifton, Klingmann and Khalil 2 A virtual high-technology simulator would be unrealistic in a resource-poor setting. A low-fidelity simulator should be cost-effective, avoid the need for maintenance and disposable parts, and be easily portable, making it an ideal simulator for a low-resource area.Reference Abass, Samuel and Odufeko 3
A low-cost, low-fidelity ear surgery simulator, the Ear Trainer, was developed with the specific goal of providing a minimal cost simulator suited to the needs and practicalities of providing training to those in the developing world.Reference Clark, Mitchell, Morris and Westerberg 4 It was designed to allow the simulation of procedures performed down the ear canal, ranging from foreign body removal to middle-ear manipulation tasks, in order to be appropriate for use by a range of healthcare professionals.
Initial face and construct validation studies, performed in the UK, demonstrated that the ear surgery training simulator provides a realistic representation of the ear, and showed that experts perform the tasks better than novices.Reference Clark, Mitchell, Morris and Westerberg 4 As the Ear Trainer has not been validated in its target environment, the current study aimed to assess the Ear Trainer in the developing world setting. This study aimed firstly to assess the face and construct validity of the ear surgery simulator in a low-resource setting, and secondly to evaluate if objective learning can be achieved with use of the device.
Materials and methods
The Ear Trainer simulator was developed by the senior authors through collaboration with the iD Lab at Dalhousie University, Canada. It was designed to be a low-fidelity, robust, realistic and versatile simulator, made at a low cost, with minimal ongoing consumables required. The Ear Trainer consists of a base unit and inserts (Figure 1). The components allow for a replaceable tympanic membrane section, which can be created with cigarette paper or a latex glove, depending on the needs of the task being performed.
Fig. 1 Ear Trainer base unit (a) with assembled middle-ear insert (b).
The study was executed at two primary sites: All Ears Cambodia centre in Phnom Penh, Cambodia, and Mbarara University of Science and Technology in Mbarara, Uganda. Construct and face validity were tested in Cambodia, and objective learning and face validity were tested in Uganda. The office of research ethics at the University of British Columbia, All Ears Cambodia, and the office of research ethics at Mbarara University of Science and Technology, approved this study.
Construct validity assessment
All Ears Cambodia recruits trainees each year to fulfil their primary ear and hearing healthcare clinician role. The provision of basic aural care forms part of their training. This is achieved ‘in-house’. The assessments made for this study were performed four weeks after a new intake of trainees.
Of the tasks previously validated,Reference Clark, Mitchell, Morris and Westerberg 4 foreign body removal with the microscope was considered the most appropriate task type to assess construct validity in this study. Each participant was shown a video of how to perform each task and was given time to practice on the Ear Trainer. Correct set-up of the microscope was ensured (inter-pupillary distance, eye focus, seat height), but not assessed. A time limit of 5 minutes for each task was set, after which time it was abandoned and considered incomplete. It was explained that whilst the tasks did not always mirror real life, they were designed to replicate the hand motions utilised during ear procedures, and in this way provided a simulated but realistic learning experience.
Two foreign bodies were selected for the foreign body removal task. A bead with a central hole was to be removed with the aid of a blunt right-angled hook, and a piece of Blu Tack® putty-like adhesive was to be removed with the aid of a Jobson Horne probe. Both tasks were performed under an operating microscope. The procedure was video-recorded (showing only the participants' hands), and assessed and rated by a blinded observer (LS) using the previously employed rating method discussed below.Reference Clark, Mitchell, Morris and Westerberg 4 Only after all videos had been assessed was each participant's level of experience made known to the observer.
Face validity assessment
Face validity was assessed in both groups of participants (in Cambodia and in Uganda) as it was in the original study.Reference Clark, Mitchell, Morris and Westerberg 4 The degree of realism of the Ear Trainer was assessed with five questions answered using five-point Likert scales (Appendix 1).
Objective learning assessment
Staff from the University of British Columbia otolaryngology department introduced the Ear Trainer to the otolaryngology team at Makerere University and Mbarara University of Science and Technology in Uganda. The University of British Columbia otolaryngology group has a longstanding collaboration with Mbarara University of Science and Technology to build the educational infrastructure of the latter's otolaryngology department.
The study was carried out during a multi-day temporal bone drilling course at Mbarara University of Science and Technology. Residents and staff from the otolaryngology department at Makerere University and Mbarara University of Science and Technology were invited to participate. The Ear Trainer was assessed on 10 participants with variable microsurgical experience, ranging from 1 to 8 years.
Participants were shown a short video clip demonstrating each task. They were then asked to perform the tasks, using the operating microscope, whilst they were video-recorded and timed. Four tasks were performed, as described below.
In the foreign body removal task, a round bead with a central hole was inserted in the ear canal. The task was to remove the bead through a speculum using the multiple tools supplied.
For the myringotomy and ventilation tube insertion task, a speculum, myringotomy knife, alligator forceps and a gently curved pick were supplied, along with an Armstrong tube. The participants were required to correctly place the tube in the tympanic membrane (made of a latex glove).
For the myringoplasty task, a speculum, gently curved pick, alligator forceps and Gelfoam dressing were supplied. In this task, the tympanic membrane was a latex glove end with a hole punched through it; the graft material was cigarette paper. Participants were asked to place the graft lateral to the Gelfoam in the middle ear and medial to the tympanic membrane.
The middle-ear manipulation task was developed to create a task that involved the manipulation of micro-instruments in the middle ear, akin to skills required for ossiculoplasty for example. A middle-ear chamber with a sewing needle inserted was attached. A speculum, alligator forceps, ear suction device and size 4.0 Prolene® suture were supplied. The participants were required to thread the suture through the eye of the needle twice.
Each participant was given 24 hours for self-directed practice and instructed to perform each task 10 times. When the practice was complete, each task was performed and recorded for the post-practice videos within 24 hours. A blinded expert not involved in recording the videos assessed the videos (LS).
Assessment of surgical performance utilised a validated measurement tool that included the Global Rating Scale and Task Specific Checklist. This tool is based on the model of Objective Structure Assessment of Technical Skill, a validated method to assess technical and non-technical skills.Reference Reznick, Regehr, MacRae, Martin and McCulloch 5 The task-specific checklist was adapted according to the limits of this low-fidelity simulator, and a five-point Likert rating scale was used. Further methodical details are available in our initial validation study publication.Reference Clark, Mitchell, Morris and Westerberg 4
Statistical analyses
Video analysis and task scoring enabled assessment of construct validity. A linear mixed model analysis was carried out with two fixed factors: task (Blu Tack or bead) and level of experience (independent practitioner or trainee). This tested the hypothesis that more experienced participants would perform better than novice participants. Objective learning data were analysed using SPSS® statistical software, version 16.0. T-tests were used to evaluate the overall rated score for the videos of the foreign body removal task and the myringoplasty task. As the distribution of the data was skewed for the overall myringotomy and ventilation tube placement scores, and for the time taken to perform the foreign body removal, ventilation tube and the myringoplasty tasks, the non-parametric Wilcoxon signed-rank test was used to compare the pre- and post-practice data. Face validity survey data were combined and presented as mean response scores for each item.
Results
Construct validity findings
The trainee participants in this study, appointed to the role of primary ear and hearing healthcare clinicians by All Ears Cambodia, came almost exclusively from a nursing background. Each trainee (n = 6) had been recruited in the preceding weeks and had approximately only four weeks of experience. During this time, they had received some instruction on ear canal procedures with a headlight, but none had used the microscope. The independent practitioners (n = 5) worked at All Ears Cambodia as academic tutors and clinical mentors for the trainees. Most had used the microscope before, but reported that the majority of their work was performed with a headlight.
A linear mixed model analysis was carried out with two fixed factors: task (Blu Tack or bead) and level of experience (independent practitioner or trainees). The observer's ratings of performance are presented in Table I. The Ear Trainer was able to differentiate between novices (trainees, average of four weeks’ experience) and experts (independent practitioners, average of over nine years’ experience). Compared to the trainees, the independent practitioners were significantly quicker at performing the tasks (p < 0.05), had significantly better overall ratings (p < 0.01) and passed the instrument through the ear canal significantly fewer times (p < 0.01).
Table I Ear trainer foreign body removal task performance as a construct validity measure
Face validity findings
Twenty-four participants completed the face validity questionnaire. The results from Cambodia and Uganda were combined. The mean scores (out of 5) for each validity question are reported below, with higher scores indicating better face validity.
The average score was 3.8 for the question ‘Does the Ear Trainer have dimensions and layout similar to those found in the real ear?’. The average score was 4.6 for the question ‘Do you think that training on such a simulator will help with the acquisition of hand–eye co-ordination when performing tasks using an operating microscope?’ The average score was 4.3 for the question ‘Do you think the skills required to perform the tasks on the simulator are similar to the skills required to perform middle-ear surgery?’. The average score was 3.5 for the question ‘Do you think that the skills required to perform the range of tasks provided by the simulator represent a range of skills required in middle-ear surgery?’. The average score was 3.5 for the question ‘Do you think that the tasks performed on the Ear Trainer are sufficiently difficult so that more senior trainees will perform the tasks better than more junior or inexperienced trainees?’
Objective learning findings
Objective learning was assessed in 10 participants, consisting primarily of otolaryngology residents. The characteristics of the participants are summarised in Table II.
Table II Experience of ugandan participants in whom face validity and objective learning were assessed
Data represent numbers of participants
There was a statistically significant improvement in overall rated scores after practice with the ear surgery simulator for: tympanostomy tube insertion, myringoplasty and the middle-ear manipulation task (p < 0.05) (Figure 2). Similarly, significantly less time (p < 0.05) was needed to perform these three tasks after practice (Figure 3). The individual times taken to perform the tasks show that the majority of participants performed faster after practice (Figure 4). The improvement in performance was most marked in less experienced performers. The overall rated score and time taken to perform the foreign body removal task did not change significantly. The data for individuals reveal, remarkably, that the participants performed well on this task prior to practice.
Fig. 2 Mean overall scores for Ear Trainer tasks. Error bars represent 95 per cent confidence intervals. Higher scores indicate better performance. **p < 0.01; *p < 0.05
Fig. 3 Time taken to perform Ear Trainer tasks. Error bars represent 95 per cent confidence intervals. Lower scores indicate better performance. **p < 0.01; *p < 0.05
Fig. 4 Individual times taken to perform (a) foreign body removal, (b) myringotomy and ventilation tube insertion, (c) myringoplasty and (d) suture manipulation Ear Trainer tasks pre- and post-practice. Dots on the diagonal line would represent no change over time; dots below the line represent an improvement over time.
Discussion
This study demonstrates that the Ear Trainer provides an accurate representation of otological anatomy, as indicated by high face validity scores. The Ear Trainer allows students to perform and learn otological skills in a low-resource setting. The findings mirror the results of a previous study assessing the same simulator in a developed world setting.Reference Clark, Mitchell, Morris and Westerberg 4 In a low-resource setting, using the device improved the overall rated scores and reduced the time needed to perform several microsurgical tasks.
All tasks showed an improvement after practice, except for the foreign body removal task. This result could be explained by performer familiarity with the task, which is probably a result of the high incidence of foreign bodies in the ear canal in the developing world. As shown in Figure 4, only two individuals showed improvements; for the others, there was a ceiling effect, whereby simulator practice no longer had an effect on skill and time to completion. This is consistent with our previous study, where all participants also performed well on this task.Reference Clark, Mitchell, Morris and Westerberg 4
A systematic review published in 2012 reported on 16 otological simulators.Reference Javia and Deutsch 6 The earliest simulator for otolaryngology was reported in 1973, but 49 of the 98 articles were published after the year 2000, indicating a significant increase in interest for medical simulation. The majority were high-fidelity virtual simulators for temporal bone drilling.
Many studies have shown the benefits of simulators, which include the acquisition of tailored skills through direct practice, the provision of a tool for assessment, and the avoidance of clinical complications.Reference Johnson 7 – Reference Tavakol, Mohagheghi and Dennick 9 Such simulators allow trainees to try new tasks relevant to their clinical practice, in an environment where they can make mistakes without risk to the patient, and learn skills that are transferrable to the patient's care.Reference Aucar, Groch, Troxel and Eubanks 10
The simulator used in this study was created for use in a low-resource setting. It was designed on appropriate tenets, including: low cost, zero maintenance and robustness, without the need for expensive replacement parts. The device is a low-fidelity model. Interestingly, low-fidelity physical models have been shown to achieve similar levels of learning as virtual reality simulators.Reference Aucar, Groch, Troxel and Eubanks 10 Our results are in line with previous literature, with trainees demonstrating objective learning with practice on a low-fidelity simulator.Reference McGaghie, Issenberg, Cohen, Barsuk and Wayne 11 Future studies will assess if this skill acquisition translates to improvement in real clinical scenarios.
This study adds to the findings of our original studyReference Clark, Mitchell, Morris and Westerberg 4 in showing the validity of the Ear Trainer when applied to a developing world setting. The face validity results confirm that users perceive the Ear Trainer to represent real-life otological tasks. This is consistent with the findings of the first validation study.
In addition, in accordance with Clark et al., the simulator was perceived to help with hand–eye co-ordination when performing tasks using an operating microscope; this question had the highest face validity score (4.6 out of 5). Of note, the questions that scored the lowest on the face validity questionnaire were those that pertained to applicability for more complex tasks. The challenge point framework states that learning is optimised when the performer is challenged either by the complexity of the task or the complexity of external factors.Reference Guadagnoli and Lee 12 Difficulty on Ear Trainer tasks can be increased by manipulating external factors such as ear canal size, ear canal curvature, or simulated complications such as bleeding.
-
• The provision of medical education in a low-resource setting is a complex problem
-
• Simulation is popular for training surgeons in low-resource settings, with no risk to patient safety
-
• The Ear Trainer is an ear microsurgery simulator for use in low-resource settings by surgeon trainees managing middle-ear disease
-
• Tasks (e.g. foreign body removal, ventilation tube insertion and middle-ear manipulation) are assessed via global ratings and task-specific checklists
-
• The Ear Trainer provides realistic representation, and can differentiate between novice and expert in a low-resource setting
-
• Objective learning results showed that participants’ otological surgical skills improved with Ear Trainer practice
The trainees expressed how useful they found the Ear Trainer. Of those in independent practice, most found the simulator helpful for training. A number of trainees commented on additions that could be made to the simulator. Such suggestions included variation in ear canal size. Such variation is possible. However, for this study, in order to have comparable results, a single ‘type’ of ear canal was used. The device can potentially also be used to improve endoscopic ear surgery skills.
Further research is necessary to validate the Ear Trainer and evaluate objective learning for endoscopic ear surgery. The potential versatility of the simulator is clear, in that different ear canal shapes and tasks can be created easily, and tailored specifically to the trainees. Even in this small study, its value has been shown in roles ranging from primary ear clinicians to senior otolaryngology staff.
Conclusion
This study validates the Ear Trainer as a useful tool for learning otological microsurgical skills in a developing world setting. Participants felt that the simulator had a high degree of realism. Use of the device in a low-resource setting improved microsurgical skills. Future improvements could include variation of ear canal size and tympanic membrane inserts to represent a wider anatomical range.
Appendix 1. Low-fidelity Ear Trainer – face validity assessment
Study number:
Name of evaluator:
Aim
We have produced an ear-training simulator to help teach the skills required to perform otological surgery.
Our specific goals were:
-
• To produce a simulator that is low-fidelity, such that once introduced to a training department, there are no ‘running costs’ or need to provide spare parts (other than items always available, such as latex gloves); these features will make it suitable for use in both high-income and low- and middle-income countries.
-
• To produce a simulator that helps a trainee acquire hand–eye co-ordination when performing skills using an operating microscope.
-
• To produce a simulator that has dimensions and layout similar to those found in real ears, including the differences found in orientation between left and right ear canals.
-
• To produce a simulator with a range of tasks such that it can be a useful training tool for those at different stages of their career.
Questions
With these aims in mind, we would ask you to enjoy using the simulator and then rate the following statements:
-
1. Does the Ear Trainer have dimensions and layout similar to those found in the real ear?
Highly disagree (1) Disagree (2) Neutral (3) Agree (4) Highly agree (5)
-
2. Do you think that training on such a simulator will help with the acquisition of hand–eye co-ordination when performing tasks using an operating microscope?
Highly disagree (1) Disagree (2) Neutral (3) Agree (4) Highly agree (5)
-
3. Do you think the skills required to perform the tasks on the simulator are similar to the skills required to perform middle-ear surgery?
Highly disagree (1) Disagree (2) Neutral (3) Agree (4) Highly agree (5)
-
4. Do you think that the skills required to perform the range of tasks provided by the simulator represent a range of skills required in middle-ear surgery?
Highly disagree (1) Disagree (2) Neutral (3) Agree (4) Highly agree (5)
-
5. Do you think that the tasks performed on the Ear Trainer are sufficiently difficult so that more senior trainees will perform the tasks better than more junior or inexperienced trainees?
Highly disagree (1) Disagree (2) Neutral (3) Agree (4) Highly agree (5)
Background information
Please indicate if you are a trainee or independent practitioner:
-
□ Trainee/OHNS resident. If yes, post-graduate year: ______
-
□ Practising OHNS surgeon
Please indicate the number of years you have practised in the specialty of otolaryngology – head and neck surgery:
-
____________ years
Please indicate if the majority of your work is in otology:
-
□ Yes
-
□ No
Please indicate if you have a specific role in training:
-
□ No role
-
□ Give lectures to medical students and/or residents
-
□ Train residents in surgical skills in OHNS
Please provide any feedback or suggestions that you think might help us to improve this simulator:
The purpose of this questionnaire is to collect information as part of a study to evaluate the effectiveness of the Ear Trainer. Participation is strictly voluntary. By completing this form, you are implicitly agreeing to participate. Any published data will be presented without any identifying markers.
