Risk of bias assessment

Of the studies that performed analysis of model suitability,^{Reference Denadai, Saad-Hossne and Raposo-Amaral12,Reference Kite, Yacoe and Rhodes15,Reference Altinyazar, Hosnuter, Unalacak, Koca and Babucçu17,Reference Agrawal21,Reference Murabit, Anzarut, Kasrai, Fisher and Wilkes23,Reference Oh, Tripathi, Gu, Borden and Wong31,Reference Mallmann, Piltcher and Isolan38,Reference AlReefi, Nguyen, Mongeau, Haq, Boyanapalli and Hafeez40} five studies had a low risk of bias, one had a medium risk of bias, and two studies had a high risk of bias (Table 5). Because of the nature of this systematic review's aims, all studies were included for analysis despite some being ineligible for risk of bias assessment.¹⁰

Table 5. Joanna Briggs Institute checklist for quasi-experimental studies’ risk of bias¹⁰

Local flaps

Nine studies^{Reference Denadai, Saad-Hossne and Raposo-Amaral12–Reference Camelo-Nunes, Hiratsuka, Yoshida, Beltrani-Filho, Oliveira and Nagae20} (Table 3) were identified as training simulators for local random-pattern flaps. Most of the simulated transposition flaps were Z-plasty^{Reference Sillitoe and Platt14–Reference Camelo-Nunes, Hiratsuka, Yoshida, Beltrani-Filho, Oliveira and Nagae20} and rhomboid flaps.^{Reference Denadai, Saad-Hossne and Raposo-Amaral12,Reference Chawdhary and Herdman13,Reference Kite, Yacoe and Rhodes15,Reference Denadai and Kirylko16} Advancement flaps were simulated in three studies^{Reference Chawdhary and Herdman13,Reference Denadai and Kirylko16,Reference Camelo-Nunes, Hiratsuka, Yoshida, Beltrani-Filho, Oliveira and Nagae20} and rotational flaps in four studies.^{Reference Kite, Yacoe and Rhodes15–Reference Altinyazar, Hosnuter, Unalacak, Koca and Babucçu17,Reference Camelo-Nunes, Hiratsuka, Yoshida, Beltrani-Filho, Oliveira and Nagae20} Seven studies were judged to have clear instructions to enable replication,^{Reference Chawdhary and Herdman13–Reference Altinyazar, Hosnuter, Unalacak, Koca and Babucçu17,Reference Loh and Athanassopoulos19,Reference Camelo-Nunes, Hiratsuka, Yoshida, Beltrani-Filho, Oliveira and Nagae20} with four disclosing costs^{Reference Sillitoe and Platt14–Reference Denadai and Kirylko16,Reference Kuwahara and Rasberry18} (Table 3). Denadai et al. assessed a mix of high- and low-fidelity models based on the traditional definitions, and identified no difference in post-test outcomes between models of different fidelity.^{Reference Denadai, Saad-Hossne and Raposo-Amaral12}

Three of the studies^{Reference Denadai, Saad-Hossne and Raposo-Amaral12,Reference Kite, Yacoe and Rhodes15,Reference Altinyazar, Hosnuter, Unalacak, Koca and Babucçu17} measured their model's validity, each demonstrating a Beckman score of 1 for face and content validity. Only two studies^{Reference Denadai, Saad-Hossne and Raposo-Amaral12,Reference Altinyazar, Hosnuter, Unalacak, Koca and Babucçu17} demonstrated a Beckman score of 1 for construct validity (Table 3). Only three studies were suitable for assessment according to the McGaghie's translational outcome assessment scale; two of these studies^{Reference Denadai, Saad-Hossne and Raposo-Amaral12,Reference Altinyazar, Hosnuter, Unalacak, Koca and Babucçu17} scored 2 (measuring changes in performance in a simulation context) and one study^{Reference Kite, Yacoe and Rhodes15} achieved a score of 1 (participant satisfaction). Only one simulator^{Reference Kite, Yacoe and Rhodes15} was validated by expert-level users; meanwhile, five other studies^{Reference Denadai, Saad-Hossne and Raposo-Amaral12,Reference Denadai and Kirylko16–Reference Kuwahara and Rasberry18,Reference Camelo-Nunes, Hiratsuka, Yoshida, Beltrani-Filho, Oliveira and Nagae20} were validated by novice users (medical students to surgical residents). Amongst the studies assessing local flaps, four provided outcome assessments. Two studies^{Reference Kite, Yacoe and Rhodes15,Reference Loh and Athanassopoulos19} performed progress assessments alone, while two studies^{Reference Denadai, Saad-Hossne and Raposo-Amaral12,Reference Altinyazar, Hosnuter, Unalacak, Koca and Babucçu17} analysed progress, reliability and comparative outcomes (Table 4).

Microtia framework

Five simulators (Table 3) for microtia framework^{Reference Agrawal21–Reference Vadodaria, Mowatt, Giblin and Gault25} were identified. Three simulate Brent's framework,^{Reference Erdogan, Morioka, Hamada, Kusano and Win22,Reference Shin and Hong24,Reference Vadodaria, Mowatt, Giblin and Gault25} one Nagata's framework,^{Reference Murabit, Anzarut, Kasrai, Fisher and Wilkes23} and one simulates both Brent's and Tanzer's frameworks.^{Reference Agrawal21} Models utilised either animal by-products,^{Reference Agrawal21,Reference Shin and Hong24} synthetic material^{Reference Erdogan, Morioka, Hamada, Kusano and Win22,Reference Murabit, Anzarut, Kasrai, Fisher and Wilkes23} or vegetable matter^{Reference Vadodaria, Mowatt, Giblin and Gault25} (Table 3). Two simulators^{Reference Agrawal21,Reference Murabit, Anzarut, Kasrai, Fisher and Wilkes23} were deemed high-fidelity while the other three^{Reference Erdogan, Morioka, Hamada, Kusano and Win22,Reference Shin and Hong24,Reference Vadodaria, Mowatt, Giblin and Gault25} were low-fidelity (Table 3). All studies disclosed sufficient reproducible construct methods to enable replication, except Murabit et al.^{Reference Murabit, Anzarut, Kasrai, Fisher and Wilkes23}

Two of the simulators were evaluated by expert-level users.^{Reference Erdogan, Morioka, Hamada, Kusano and Win22,Reference Murabit, Anzarut, Kasrai, Fisher and Wilkes23} Two studies^{Reference Agrawal21,Reference Murabit, Anzarut, Kasrai, Fisher and Wilkes23} included translational assessments according to the McGaghie's assessment scale (Table 4). Validation assessment amongst the studies was limited, with only Murabit et al.^{Reference Murabit, Anzarut, Kasrai, Fisher and Wilkes23} attempting to assess construct validity of the models (Beckman score of 1). None of the studies attempted face or content validation of their models. Similarly, only Murabit et al.^{Reference Murabit, Anzarut, Kasrai, Fisher and Wilkes23} performed progress, reliability and comparative assessments.

Pinnaplasty

One low-fidelity, animal-based model for pinnaplasty^{Reference Uygur, Ozturk, Kwiecien and Siemionow26} was identified (Tables 3 and 4). This study utilised a sheep's head and described the simulated procedure; however, there were no assessments of the model's effectiveness.^{Reference Uygur, Ozturk, Kwiecien and Siemionow26}

Oculoplastic techniques

Simulators addressing oculoplastic techniques such as eyelid laceration repair, eyelid reconstruction, ptosis repair, tarsorrhaphy, blepharoplasty and lateral tarsal strip were identified in five studies^{Reference Uygur, Ozturk, Kwiecien and Siemionow26–Reference Ianacone, Gnadt and Isaacson30} (Table 3). All studies utilised animal models: three porcine^{Reference Zou, Wang, Guo and Wang27–Reference Kersey29} and two ovine^{Reference Uygur, Ozturk, Kwiecien and Siemionow26,Reference Ianacone, Gnadt and Isaacson30} (Table 3). All models were deemed to be low-fidelity. The porcine models did not have a lower eyelid or a lateral bony orbital wall. However, histological assessment did demonstrate a high degree of similarity to human tissue.^{Reference Pfaff28,Reference Kersey29} Similarly, the sheep model has a more angulated orbital floor and low orbital fat pad volume.^{Reference Ianacone, Gnadt and Isaacson30} None of the five studies performed any evaluation of translational outcomes, effectiveness or validity.

Facial nerve anastomosis

Only one model simulated the techniques required for facial nerve dissection and anastomosis.^{Reference Ianacone, Gnadt and Isaacson30} This sheep model was deemed to be high-fidelity because of the close resemblance of the sheep facial nerve to the human facial nerve. No assessments of effectiveness, translational outcomes or validity were performed in this study.

Endoscopic septoplasty and septorhinoplasty

Six animal simulators,^{Reference Dini, Gonella, Fregadolli, Nunes and Gozzano34–Reference Gardiner, Oluwole, Tan and White39} three synthetic three-dimensional (3D)-printed simulators,^{Reference Zammit, Safran, Ponnudurai, Jaberi, Chen and Noel32,Reference Zabaneh, Lederer, Grosvenor and Wilkes33,Reference AlReefi, Nguyen, Mongeau, Haq, Boyanapalli and Hafeez40} and one mixed synthetic and animal simulator^{Reference Oh, Tripathi, Gu, Borden and Wong31} were identified for endoscopic septoplasty and septorhinoplasty (Table 3). All synthetic 3D-printed simulators^{Reference Oh, Tripathi, Gu, Borden and Wong31–Reference Zabaneh, Lederer, Grosvenor and Wilkes33,Reference AlReefi, Nguyen, Mongeau, Haq, Boyanapalli and Hafeez40} were based on computed tomography scans of human facial skeletons. Of the animal models studied, five were ovine^{Reference Dini, Gonella, Fregadolli, Nunes and Gozzano34,Reference Dini, Gonella, Fregadolli, Nunes and Gozzano36–Reference Gardiner, Oluwole, Tan and White39} and one study utilised chicken sternal cartilage for cartilage grafting.^{Reference Weinfeld35} The mixed model^{Reference Oh, Tripathi, Gu, Borden and Wong31} utilised porcine cartilage mounted on plastic. Four models simulated endoscopic septoplasty.^{Reference Touska, Awad and Tolley37–Reference AlReefi, Nguyen, Mongeau, Haq, Boyanapalli and Hafeez40} All studies disclosed detailed instructions on construction to enable replication, but only four studies^{Reference Zammit, Safran, Ponnudurai, Jaberi, Chen and Noel32,Reference Touska, Awad and Tolley37,Reference Gardiner, Oluwole, Tan and White39,Reference AlReefi, Nguyen, Mongeau, Haq, Boyanapalli and Hafeez40} disclosed the costs involved (Table 4).

Of the 10 studies evaluated, only 3 (2 endoscopic septoplasty simulators^{Reference Mallmann, Piltcher and Isolan38,Reference AlReefi, Nguyen, Mongeau, Haq, Boyanapalli and Hafeez40} and 1 septorhinoplasty simulator^{Reference Oh, Tripathi, Gu, Borden and Wong31}) performed outcome assessments (Table 4). Mallmann et al.^{Reference Mallmann, Piltcher and Isolan38} performed comparative, progress and reliability assessments while evaluating an endoscopic septoplasty simulator; the remaining two studies^{Reference Oh, Tripathi, Gu, Borden and Wong31,Reference AlReefi, Nguyen, Mongeau, Haq, Boyanapalli and Hafeez40} performed only a reliability assessment. A comprehensive assessment of construct validity (Beckman score 2) was performed in two studies,^{Reference Oh, Tripathi, Gu, Borden and Wong31,Reference AlReefi, Nguyen, Mongeau, Haq, Boyanapalli and Hafeez40} which used multiple assessment parameters. Face and content validity assessments were less robust, with all three studies that performed analyses^{Reference Oh, Tripathi, Gu, Borden and Wong31,Reference Mallmann, Piltcher and Isolan38,Reference AlReefi, Nguyen, Mongeau, Haq, Boyanapalli and Hafeez40} achieving Beckman scores of 1. Translational outcomes scores of 1 were achieved by all three of these studies.^{Reference Oh, Tripathi, Gu, Borden and Wong31,Reference Mallmann, Piltcher and Isolan38,Reference AlReefi, Nguyen, Mongeau, Haq, Boyanapalli and Hafeez40} Eight studies^{Reference Zammit, Safran, Ponnudurai, Jaberi, Chen and Noel32–Reference Dini, Gonella, Fregadolli, Nunes and Gozzano34,Reference Dini, Gonella, Fregadolli, Nunes and Gozzano36–Reference AlReefi, Nguyen, Mongeau, Haq, Boyanapalli and Hafeez40} were deemed high-fidelity, while the other two^{Reference Oh, Tripathi, Gu, Borden and Wong31,Reference Weinfeld35} were low-fidelity (Table 3).