Introduction
In a 2015 report, the Institute of Medicine (IOM; Washington, DC USA), now known as the National Academy of Medicine (NAM; Washington, DC USA), stated that the field of Emergency Medical Services (EMS) exhibits signs of fragmentation; an absence of system-wide coordination and planning; and a lack of federal, state, and local accountability.Reference McCoy and Schultz 1 The NAM recommended clarifying what roles the federal government, state governments, and local communities play in the oversight and evaluation of EMS system performance, and how they may better work together to improve care. Using a systematic review and environmental scan, this project, funded by the Michigan Department of Health and Human Services (MDHHS; Lansing, Michigan USA), aimed to review the existing peer-reviewed and grey literature on EMS oversight and EMS quality measurement from a systems perspective to determine gaps in knowledge, measures of high-quality oversight, and to fill the knowledge gap identified by NAM.Reference Balshem, Stevens and Ansari 2 Its research questions are two-fold: (1) what aspects of EMS systems are most measured in the peer-reviewed and grey literatures, and (2) what do these measures entail for EMS oversight?
Prehospital Care and Public Health
Emergency Medical Services is often described as the intersection between public health, public safety, and health care.Reference Delbridge, Bailey and Chew 3 According to the 2011 National EMS Assessment, there were over 31 million EMS responses and close to 23 million EMS transports in the United States (excluding Louisiana, Illinois, Michigan, Ohio, Oregon, and Rhode Island).Reference Mears, Armstrong and Fernandez 4 The importance of EMS is demonstrated by the volume of care it provides and in its relationship with the overall health care system.Reference Shah, Bazarian and Lerner 5 A 2010 study of EMS use by older adults in the state of North Carolina found that over 38% of all EMS arrivals to the emergency department were for patients over 65, and over 60% of patients over 85 arrived by EMS.Reference Platts-Mills, Leacock, Cabanas, Shofer and McLean 6 This relationship between prehospital EMS and hospital-based emergency medicine will likely grow as the population of Americans aged 65 years or older is projected to double during the next 25 years, and as the result of the recent growth of Mobile Integrated Health Care.Reference Aldeen, Hartman and Segura 7 - Reference Ortman, Velkoff and Hogan 9 Moreover, the critical conditions commonly treated by EMS, such as ST elevation myocardial infarction (STEMI), cardiac arrest, trauma, and stroke, are among the leading causes of death in the US.Reference Johnson, Hayes and Brown 10 The public health impact of these conditions, the volume of care EMS provides, and its relationship to hospital-based acute care providers make necessary the review, development, and application of standardized quality measures for EMS care. This would not only impact EMS quality and patient outcomes, but public health as well.
Despite the common licensing of agencies and providers at the state level, EMS systems have regional variability, differing greatly depending on geographic location, resources, health system involvement, variation in EMS scope of practice, laws, and oversight entities. As a result, the types of EMS systems are often suggested to be as numerous as the systems themselves.Reference Zachariah and Pepe 11 The different types of providers in the US are listed in Table 1.Reference Perry and Busch 12 Ultimately, these combinations of various providers and facilities come together to provide three basic medical functions: patient stabilization, evacuation, and distribution.Reference Pepe, Copass, Fowler and Racht 13 Although structures and resources vary, the fundamental components and purposes of any system are essentially the same, allowing for the application of common quality measures.
Abbreviation: EMS, Emergency Medical Services.
The goal of this systematic review and environmental scan is to assess the current state of EMS quality measurement from a systems perspective and to evaluate characteristics and practices associated with high-quality EMS oversight. This work complements the EMS Compass Initiative, a national effort funded by the National Highway Traffic Safety Administration (NHTSA; Washington, DC USA) Office of EMS and led by the National Association of State EMS Officials (NASEMSO; Falls Church, Virginia USA). The EMS Compass engages stakeholders to develop quality measures in the following six domains: clinical process/effectiveness, patient and family engagement, patient safety, care coordination, population/public health, and efficient use of health care resources. 14 The EMS Compass is aimed at reaching consensus indicators but is not yet at the stage of recommending best practices in EMS oversight.
Emergency Medical Services Oversight in the US
There is variability in oversight of prehospital care within the US as a whole and within each state. Generally, there are multiple overlapping levels of oversight. Each day, EMS care is provided under protocols with direct medical control available and often required for specific situations. Protocol development also varies considerably. In some states, there is a single set of state-wide protocols for all prehospital providers; typically, these are created in a committee consisting of physicians and EMS providers, often volunteers. State EMS regulatory officials participate pursuant to state regulations. In other states, protocols are developed at a local level, and in some cases, an individual agency delivers care under its own physician medical director.
Finally, there is also significant variation in the medical director role. All but one state requires each individual agency to have a medical director.Reference Slifkin, Freeman and Patterson 15 In some states, the individual service medical director is directly responsible for creating the protocols and delegating practice to the EMS providers. In others, the role of the medical director is mentoring, process improvement, and education as part of a larger state and/or regional system.Reference Schooley and Horan 16 There are EMS agencies that have physicians in the field, particularly with the growth of EMS as an accredited sub-specialty. Field physicians provide real-time mentoring and review of runs. Current technology allows concurrent review of run data, including assessment of cardiopulmonary resuscitation (CPR) quality during resuscitation.Reference Ko, Chen, Lin, Ma and Lin 17 In many states, there are roles for physicians at a regional level, usually a county, group of counties, or other designated area. The regional entities often do the work of protocol development and may track quality indicators. Often, destination protocols designating which hospitals are approved destinations for specific conditions like stroke or STEMI are also developed.
Quality Measurement
The Centers for Medicare and Medicaid (CMS; Baltimore, Maryland USA) define quality measures as “tools that help us measure or quantify health care processes, outcomes, patient perceptions, and organizational structure and/or systems that are associated with the ability to provide high-quality health care and/or that relate to one or more quality goals for health care.” 18 The review and compiling of existing EMS quality measures is an essential step in identifying and filling gaps in the understanding of what constitutes high-quality EMS and EMS oversight. In a 2009 report, NHTSA recommended indicators of system performance through the following service functions and community attributes: system design and structure, human resources, clinical care and outcome, response, finance/funding, quality management, and community demographics. 19 While considering EMS in terms of service functions is essential, understanding the concepts of EMS oversight and care provision is also critical to identifying gaps in quality measurement.
The fundamental goal of quality improvement is to improve patient care. For prehospital care, quality measures are applied to multiple overlapping components of the EMS system, as outlined in Table 2. Oversight includes those entities that hold EMS providers accountable to providing appropriate care.Reference Kuehl 20 The agency medical director is the first layer of oversight;Reference Pepe and Curka 21 other entities include local, regional, or state-level government agencies or institutions.Reference Pepe and Curka 21 , Reference Stone, Seaman and Bissell 22 The agency level is a private or public, non-profit or for-profit organization that provides EMS services in a defined area.Reference Sobo, Andriese, Stroup, Morgan and Kurtin 23 The next level of EMS care is the personnel level, which for the purposes of this review are one or more EMS providers (eg, an individual emergency medical technician [EMT] or an EMS team).Reference Bayley, Weinger, Meador and Slovis 24 - Reference Lossius, Røislien and Lockey 26 Systems of care refers to a single disease or injury entity that uses pre-defined quality measures focusing on that particular patient diagnosis.Reference McLean, Maio and Domeier 27 , Reference Ware 28 Quality measures are often organized according to the Donabedian Model, which distinguishes measure types into three categories: (1) structure, (2) process, and (3) outcome.Reference Donabedian 29 In the most basic sense, structure refers to the institutions and providers in which care takes place, while process refers to what is done to the patient, and outcomes are the results of this care. Quality measures of all three types can be applied to an aspect of prehospital care.
Abbreviations: EMS, Emergency Medical Services; EMT, emergency medical technician; STEMI, ST elevation myocardial infarction.
Methods
Applying the Donabedian typology of quality measurement to this model of EMS care, the peer-reviewed and grey literature were searched through a systematic review and an environmental scan to identify gaps in EMS system quality measurement and EMS oversight.
Systematic Review Data Sources and Searches
The following databases were searched between March 2016 and May 2016 by title and abstract (1) PubMed (National Center for Biotechnology Information, National Institutes of Health; Bethesda, Maryland USA); (2) Web of Science (Thomson Reuters; New York, New York USA); (3) SCOPUS (Elsevier; Amsterdam, Netherlands); and (4) EMBASE (Elsevier; Amsterdam, Netherlands; Figure 1). The search was restricted to English language articles published between 1966 and 2016 (Table 3). Articles that did not include quality measures or discussions of quality measurement were excluded.
Systematic Review Data Extraction and Analysis
As PRISMA guidelines suggest, to reduce selection bias, inclusion and exclusion criteria and variables were defined before review (Appendix Table 1; available online only).Reference Moher, Shamseer and Clarke 30 , Reference Liberati, Altman and Tetzlaff 31 Title review was performed by three authors (RT, MA, MC) to eliminate duplicate titles and titles not referring to EMS. If it was unclear from the title of the article, the abstract was retrieved and the same review process was applied.
Articles that were retained after title and abstract review were compiled into Dedoose (Manhattan Beach, California USA), a qualitative analysis tool.Reference Dedoose 32 At this stage, the sample was divided between two reviewers (MC, RT) and an initial review was conducted to determine whether the articles were on the topic of EMS and quality measurement (Appendix Table 1; available online only). This step included discussions between the three reviewers (RT, MA, MC) to ensure each was using the same method for reviewing the articles. The third reviewer (MA) re-reviewed all coding to assure consistency between the primary reviewers (MC, RT). After this stage, 17 articles were excluded for not being on the subject of EMS or with no description of quality measurement.
Using a data extraction template (Appendix Table 2; available online only), the retained articles were analyzed on several levels, including: article type (original, review, and/or conference proceeding); quality measure type (structure, process, or outcome); quality measure level (oversight, agency, personnel, and patient); and finally, the specific measure itself. Following review and coding, the extraction form and codes were reviewed by two authors (RT, MA) to ensure consistency and agreement.
Environmental Scan Data Sources and Searches
According to the Agency for Healthcare Research and Quality (AHRQ; Rockville, Maryland USA), “an environmental scan examines unpublished literature and publicly available program information.”Reference Porterfield, Hinnant, Kane, Horne, McAleer and Roussel 33 This type of literature is referred to as “grey literature,” which according to the commonly accepted Luxembourg definition is literature that is “produced on all levels of government, academics, business, and industry in print and electronic formats, but which is not controlled by commercial publishers.”Reference Farace and Frantzen 34 Incorporating grey literature into a systematic review through an environmental scan is an approach promoted by AHRQ as a way of reducing publication bias found in traditional literature reviews. 35 The Internet is often used as a platform for disseminating grey literature by a wide range of organizations, such as government and nongovernment organizations, research centers, health institutes, and nonprofit organizations, contributing to a proliferation of this source of data.
To evaluate grey literature relevant to EMS quality measurement and oversight, a list of the federal EMS-related agencies and key professional societies was compiled with input from federal and state subject matter experts (Table 4). Each agency or organization’s website was visually searched for sub-pages labeled with the terms: quality improvement, reports, assessments, white papers, briefings, or synonymous terms. For each agency or organization’s website, if it supported website search function, separate searches were run for the following terms: [“medical direction,” “medical control,” “prehospital care”, “quality”, “measure”]. The search was restricted and targeted to English language literature produced between 1996 and 2016.
Abbreviations: DHS, Department of Health and Human Services; DOT, Department of Transportation; EMS, Emergency Medical Services.
Environmental Scan Data Extraction and Analysis
Following each search, the search results were visually scanned by title for relevance to EMS oversight and quality measurement; this was determined by requiring the titles to contain the following or synonymous terms: EMS, paramedic, prehospital, out-of-hospital, ambulance, quality, performance, measurement, and improvement. Appendix Table 3 (available online only) presents the grey review data results.
Results
Systematic Review
Of the 58 studies that met the pre-defined criteria, 48 were original research, 10 were reviews, 46 included process measures, 36 included outcome measures, while just 19 included structure measures. Most studies applied quality measures at the individual prehospital provider level (40), followed by agency (28), system of care (28), and oversight (5). Counts for the number of articles in the final sample that include a specific measurement level measure type (structure, process, and outcome) are presented in Table 5.
a These frequencies refer to measures, each article may apply multiple measures but application of type or level is counted no more than once per article.
Environmental Scan
From the 10 targeted websites, 115 grey literature resources met the pre-defined criteria, 34 were EMS oversight from the federal government, 57 were from state EMS leadership and management organizations, four were identified at the local level, 13 from clinical membership organizations, and seven from EMS educator organizations (Table 4).
Discussion
The goal of this systematic review and environmental scan was to assess the current state of EMS quality measurement from a systems perspective and to evaluate characteristics and practices associated with high-quality EMS oversight. In the peer-reviewed literature, it was found that most studies measured quality at the individual EMS provider level while agency-level quality measures were typically aggregations or averages of these measures applied to the entire EMS agency (eg, average response time or protocol compliance rates). At the system of care level, quality measures were typically related to patient survival or condition-specific clinical measures (eg, time to return of spontaneous circulation). Oversight was the least covered level of EMS care, consistent with NAM’s cited lack of an evidence base for EMS oversight and suggesting that this research area needs to be developed further. In contrast to the peer-reviewed literature, the grey literature is ahead, containing more information on EMS oversight and quality improvement (QI) and can serve as a foundation for guiding the research and policy agenda for best practices in EMS oversight and quality measurement.
In general, the main function of oversight is to set practice standards and hold provider agencies and their personnel accountable for providing appropriate care. In the peer-reviewed literature, measurement at the oversight level was generally related to medical direction and the needed infrastructure for agencies to ensure protocol compliance and report on quality measurement. Medical direction of daily EMS care is provided indirectly by protocols. The protocols always have an option of direct medical control and certain conditions, such as the performance of certain procedures or administration of certain medications, may require direct medical control. Protocols include guidelines for assessment of complaints and specific treatment of certain conditions. When protocol deviations occur, the reasons are generally reported to be subjective assessments of patient need; for instance, when a patient with nausea and epigastric pain is treated as an abdominal complaint when they are actually having an acute myocardial infarction (AMI).Reference Cushman, Zachary Hettinger, Farney and Shah 36 , Reference Greer, Williams, Valderrama, Bolton, Patterson and Zhang 37 Other common protocol deviation includes incorrect drug dosage, particularly in children. Oversight may support or discourage staff discretion and make clear when deviation is and is not appropriate, thus allowing for objectively necessary deviations from protocol.Reference Henderson 38 - Reference Aufderheide, Yannopoulos and Lick 40 No articles assessing the quality of direct medical control were found.
Emergency Medical Service agencies typically deliver care according to the protocols set forth by the oversight entity, but they are also responsible for maintaining the quality of care provided by their personnel. This is often done by chart review, which has been shown to reduce the number of cases requiring remediation, increase the proportion of charts rated as clinically acceptable, reduce the proportion of misplaced endotracheal tubes, and increase the appropriate administration of aspirin.Reference Munk, White, Perry, Platt, Hardan and Stoy 41 It is possible that chart review alone may simply force improved documentation without meaningful improvement in patient care and is best used as a part of a program of focused assessment including simulation and skills testing.
The peer-reviewed literature suggests that oversight requires not only involvement in QI through medical direction, but also through creating quality monitoring and improvement infrastructure such as state-wide QI programs.Reference Dunford, Domeier and Blackwell 42 There is variability in EMS agency commitment to QI; some agencies devote many more resources that others. This is best done through a uniform dataset and consistent reporting language from local EMS agencies to oversight authorities.Reference Dunford, Domeier and Blackwell 42 , Reference Mears, Pratt and Glickman 43 Mandating uniformity in data reporting may allow for more transparent oversight and quality assurance. There have been efforts towards this goal, most prominently the National EMS Information System (NEMSIS), which serves as a standardized repository of prehospital EMS data, but it is dependent on good quality data being entered into the system.Reference Dawson 44
Also, the peer-reviewed literature indicated that oversight can provide the leadership and coordination necessary for QI. For example, Kingsbury et al recommended standardized and coordinated approaches to STEMI care to help decrease the variation in the use of resources, improve coordination of care and access to percutaneous coronary intervention, and as a result, improve patient outcomes.Reference Kingsbury, Natarajan, Forsey, Oakes and Bakar-Irwin 45 Many states have adopted systems of care programs for disease and injury states using this approach. The development of inter-organizational relationships between agencies and acute care providers can also improve patient outcomes and personnel satisfaction.Reference Fischer, Kamp and Riesgo 46 Collaboration between agencies and hospitals through regular communication and coordination has been shown to be associated with lower AMI mortality rates.Reference Landman, Spatz, Cherlin, Krumholz, Bradley and Curry 47
In both the grey and peer-reviewed literature, it was found that traditional EMS quality measurement has focused almost exclusively on response times.Reference Myers, Slovis and Eckstein 48 , Reference Murphy, Wakai, Walsh, Cummins and O’sullivan 49 Although response times may indicate high-quality processes and structures, evaluating best practices in structures and processes in their own right is necessary because response times may be a function of factors outside the EMS system itself, such as population density and transportation infrastructure.Reference Aringhieri, Carello and Morale 50 Even so, response times or quicker time to treatments, such as earlier CPR or defibrillation, have been shown to be correlated with improved patient outcomes in a number of studies,Reference Chen, Ma and Chen 51 - 53 but not all.Reference Callaham and Madsen 54 Quality improvement outside of response times are essential though; these may include QI approaches such as the use of clinical safety charts, education sessions, and leadership engagement in QI which have been shown to improve EMS agency performance on key indicators and the recording of QI data.Reference Ebbs, Middleton, Bonner, Loudfoot and Elliott 55
Based on evidence from the peer literature review, job satisfaction and organizational commitment of EMS personnel were found to be associated with stronger organizational communication, improved productivity, and reduced turnover.Reference Carriere and Bourque 56 The length of time an EMT has been with an agency also predicts performance on condition-specific measures such as time to intubation, proper CPR, increased patient survival after out-of-hospital cardiac arrest, and reduced patient reported pain.Reference Ailsby 39 , Reference Carriere and Bourque 56 , Reference Dyson, Bray, Smith, Bernard and Finn 57
Peer-reviewed articles on the quality measurement of EMS personnel included studies on the impact of teamwork and team composition on performance. These studies found that communication and team composition are important to reducing errors and improving efficiency. Consistent, formal channels to voice concerns and opinions can support effective communication.Reference Williams, Rose, Simon and Consortium 58 Regarding the composition of ambulance teams, those teams that included Advanced Life Support (ALS)-capable paramedics were found to provide higher quality treatment compared to Basic Life Support (BLS)-only units.Reference Chen, Ma and Chen 51 - 53 , Reference Krarup, Lassen and Clemmensen 59
Articles that included patient-level quality measures were perhaps the most diverse due to patient outcomes being the primary outcome of most research in prehospital care.Reference Bayley, Weinger, Meador and Slovis 24 , Reference Ailsby 39 , Reference Aufderheide, Yannopoulos and Lick 40 , Reference Dunford, Domeier and Blackwell 42 , Reference Chen, Ma and Chen 51 , Reference Braun 60 - Reference El Sayed 62 Many proposed or explored measures were either specific to a narrow type of treatment (eg, cardiac arrest or STEMI) or limited to description of a single population or small group of patients and their outcomes.Reference Bayley, Weinger, Meador and Slovis 24 , Reference Ailsby 39 , Reference Aufderheide, Yannopoulos and Lick 40 , Reference Dunford, Domeier and Blackwell 42 , Reference Chen, Ma and Chen 51 , Reference Citerio, Galli and Cesana 52 , Reference Callaham and Madsen 54 , Reference Coster, Irving, Turner, Wilson, Phung and Siriwardena 61 , Reference Cady and Scott 63 Although the articles covering patient-level quality were diverse, the patient-level measures themselves were narrow in scope, such as patient satisfaction and other patient outcomes (eg, survival to hospital admission or discharge).Reference Dunford, Domeier and Blackwell 42 , Reference Chen, Ma and Chen 51 , Reference Callaham and Madsen 54
The emphasis on response times noted in the peer-reviewed literature is expected in prehospital quality measurement given that they are relatively easy to measure and report and have been found to be correlated with patient outcomes.Reference Chen, Ma and Chen 51 - 53 There is a lack of research and quality measurement of the attributes of EMS systems that promote better patient outcomes. To reduce this gap, the application of organizational studies to prehospital EMS care is recommended. The application of organizational studies to health care delivery has a rich history in the hospital setting.Reference Flood and Fennell 64 Its application allows for empirically based explanations and expectations for the organization and behavior of EMS staff, team dynamics, agency leadership, and overall effectiveness.Reference Aiken, Sochalski and Lake 65 - Reference Newhouse 67 Its application to the study of prehospital care will be a maturation of the field academically and professionally by establishing formal knowledge of prehospital care organization, with the potential of further developing the professional identity of EMS, much as it did for the fields of medicine and nursing.Reference Mick and Wyttenbach 68 - Reference Freidson 73
The grey literature, being more practitioner-focused complimented the academic literature’s lack of practice-oriented information regarding EMS oversight and systems-level quality measurement. Sources of grey literature related to EMS practice and quality measurement were derived from leading field experts and government agencies responsible for quality performance and oversight (eg, National EMS Advisory Council [NEMSAC; Washington, DC USA], Federal Emergency Management Agency [FEMA; Washington, DC USA], and state and regional EMS oversight authorities).
The grey literature review revealed the value of improved EMS performance and oversight in improving health care delivery systems. A multi-level, multi-sectoral systems approach offers federal, state, and regional EMS authorities with practice models for comprehensive EMS reform, and evidence-based guidelines for promoting and implementing prehospital care and evaluation. A number of resources provided key principles and practical resources for designing and implementing high-quality performance and oversight measures for EMS systems.
The grey literature suggests that high-quality oversight entails: (1) a systems-approach that involves multiple stakeholders of EMS practice (eg, federal and state agencies, educational and professional credentialing programs, regulatory authorities, EMS agencies, and hospitals); 19 , 74 - 78 (2) implementation of disease-specific (eg, cardiac arrest or stroke) QI measures;Reference Goodloe, Thomas and Blackwell 79 (3) establishing network building opportunities; 80 (4) adopting national QI standards; 81 (5) developing strategic plans and coordinated state-wide initiatives for continuous QI; 81 , Reference Hormozdyaran 82 (6) developing self-assessment tools for regional EMS authorities related to key areas of oversight and performance; 83 (7) instituting accreditation, training, and credentialing standards; 78 , 84 and (8) strong agency and leadership commitment to QI.Reference Hormozdyaran 82 , 85 The relative lack of these findings in this systematic literature review suggests that empirical study into these aspects of EMS oversight and their impact on patient outcomes is needed, which can inform best practices for their implementation and quality measurement.
Limitations
Limitations
Common limitations to systematic reviews include publication bias and exclusion of non-English articles and are applicable here.Reference Higgins and Green 86 The grey literature does not have publication bias; however, by definition it has not undergone peer review, suggesting the possibility for less rigorous methodology or less empirical evidence for its recommendations. Although this review followed PRISMA guidelines, where applicable, the non-interventional nature of much of the peer-reviewed and grey literature related to this topic made assessing the quality of the literature inapplicable.Reference Liberati, Altman and Tetzlaff 31
Implications
Engaged oversight and deliberate practice of QI by agencies can improve the quality of EMS care. This can be done through the development of practice standards and accountability processes; standardization and coordination of care to reduce variation; QI collaborations between health care providers and care settings; positive relationships between oversight and providers; and maintaining quality monitoring infrastructures, such as databases. Limited quality measurement of EMS oversight is an important gap in the peer-reviewed literature. The grey literature is ahead in this realm and can guide the policy and research agenda for EMS oversight quality measurement. A better understanding of the structural, organizational, and procedural characteristics that lead to successful EMS oversight can help establish best practices in prehospital care delivery and help improve patient outcomes.
Conclusion
To improve the quality of EMS and patient outcomes, the establishment of best practices in prehospital care is critical. Therefore, the attributes of high-quality EMS systems must be determined and developed into reliable, valid, and standardized measures. This goal will require an in depth understanding of how EMS care is delivered. Most existing EMS quality measures are operational and are related to processes which are essential but leave much to be uncovered in the area of organizational structure and oversight. The grey literature is ahead in this area and can guide the policy and research agenda for EMS oversight quality measurement and structure.
Acknowledgements
The authors would like to acknowledge the valuable contributions of Gregg S. Margolis, Director of the Division of Healthcare System Policy at the US Department of Health and Human Services (Washington, DC USA) for content expertise and manuscript review.
Author Contributions
RT and MA conceptualized the project, designed the review process, and contributed to manuscript writing, editing, and article review. MC contributed to the review of titles and abstracts, article review, and manuscript writing and editing. RK and ML contributed to writing and editing of the manuscript. KW and JS contributed to reviewing and editing of the manuscript.
Supplementary Material
To view supplementary material for this article, please visit https://doi.org/10.1017/S1049023X17007129