Taxonomic identification of faunal remains recovered from archaeological excavations is a requisite first step to attaining many of zooarchaeology's analytical goals (e.g., Gifford-Gonzalez Reference Gifford-Gonzalez2018; LeFebvre and Sharpe Reference LeFebvre, Sharpe, Giovas and LeFebvre2018). The identification process requires first ascertaining which anatomical traits of skeletal parts are the result of variation between species (inter-taxonomic variation) and which reflect variation within a species (intra-taxonomic variation). The former traits allow zooarchaeologists to assert that a particular distal left tibia represents a yellow-bellied marmot (Marmota flaviventris) as opposed to some other species of similar body (and skeletal) size. Zooarchaeologists have spent countless hours scrutinizing skeletons of known taxonomic identity to determine which skeletal traits are taxonomically diagnostic and which ones are not. In North American zooarchaeology, this process began in the late nineteenth century (Eaton Reference Eaton1898) and has extended continuously up to the present (e.g., Adams and Crabtree Reference Adams and Crabtree2012; Balkwill and Cumbaa Reference Balkwill and Cumbaa1992; Brainerd Reference Brainerd1939; Brown and Gustafson Reference Brown and Gustafson1979; Gilbert Reference Gilbert1973; Jacobson Reference Jacobson2003; Lawrence Reference Lawrence1951; Lubinski and Hale Reference Lubinski and Hale2018; Olsen Reference Olsen1964).
Twenty years ago, Gobalet (Reference Gobalet2001) asked four experts to identify the same zooarchaeological sample of fish bones. He found considerable variability across investigators in both the number of specimens identified and the taxonomic identifications made. These results were not surprising to us given the history of (not) reporting taxonomic identification protocols in zooarchaeology. Gobalet's (Reference Gobalet2001) results suggest that different analysts employ different skeletal traits or traits of varied diagnostic value to make identifications. Although rarely cited, we believe, as Driver does, that Gobalet's study “should have been a wake-up call to the discipline” (Reference Driver2011:37).
Beyond the well-acknowledged problem that limited access to sufficient comparative materials can significantly reduce identification accuracy (see references in Lyman Reference Lyman2019), Driver described the key issue as an assumption that the methods and anatomical traits used “for identification are sufficiently well tested that one does not need to justify most identifications, except in relatively rare circumstances” (Driver Reference Driver1992:39). Importantly, he noted that “we do not systematically test the quality of our identifications using ‘blind’ tests. . . . Consequently we have no idea of the accuracy of our methods” (Driver Reference Driver1992:40–41). He added, “While most zooarchaeologists would probably agree that most identifications are probably accurate, they have no empirical or theoretical basis for this claim” (Driver Reference Driver1992:41).
This argument could as easily be made today as in 1992 (e.g., Lau and Kansa Reference Lau and Kansa2018; Wolverton Reference Wolverton2013). Given that anatomical traits used for identifications are rarely published, there can be little peer review or testing of them, although some may occur at conferences and workshops and on listservs. In our combined 60+ years of experience, however, no one has questioned our taxonomic identification techniques or requested that we describe the anatomical traits used to make identifications. We worry that this is typical of the discipline. Since accurate identifications provide the basis for much of zooarchaeology, and insofar as paleozoological identifications of species are having increasing influence on conservation biology decisions (e.g., Barnosky et al. Reference Barnosky, Hadly, Gonzalez, Head, Polly, Michelle Lawing, Eronen, Ackerly, Alex, Biber, Blois, Brashares, Ceballos, Davis, Dietl, Dirzo, Doremus, Fortelius, Greene, Hellmann, Hickler, Jackson, Kemp, Koch, Kremen, Lindsey, Looy, Marshall, Mendenhall, Mulch, Mychajliw, Nowak, Ramakrishnan, Schnitzler, Das Shrestha, Solari, Stegner, Allison Stegner, Chr, Stenseth, Wake and Zhang2017; Dietl and Flessa Reference Dietl and Flessa2017; Lyman and Cannon Reference Lyman and Cannon2004; Wolverton et al. Reference Wolverton, Nagaoka and Rick2016), an inaccurate identification could have implications beyond archaeological inquiry. Three examples illustrating uncertainty about identification protocols and the changing status of some skeletal traits from taxonomically diagnostic to nondiagnostic are provided in Supplemental Text 1. It is in large part because of such cases that we, Driver (Reference Driver1992), and Gobalet (Reference Gobalet2001) believe that skeletal traits thought to be diagnostic among taxa of similar size and morphology should be tested.
Identification Protocols: Best Practices and Examining Success
We begin by characterizing the identification process and defining concepts we use to evaluate an identification protocol. The terms “reference” and “comparative” specimens designate bones of known taxonomy wherein the taxonomy was determined by standard biological criteria among living organisms. Zooarchaeologists have long used such collections to develop means to identify ancient faunal remains. There are today no recommended best practices or generally accepted standards for reporting identifications—that is, which skeletal parts were identified and how those specimens were identified (Clason Reference Clason1972; Driver Reference Driver, Francis and Poplin1982; Lawrence Reference Lawrence and Matolcsi1973; Wolverton Reference Wolverton2013). Furthermore, there tends to be minimal description of the identification protocol used in a particular study (e.g., Baker and Shaffer Reference Baker and Shaffer1999; Driver Reference Driver1992, Reference Driver2011; Emery Reference Emery and Emery2004; Lupo Reference Lupo2011; Lyman Reference Lyman, Maschner and Chippindale2005; Stewart Reference Stewart, Grupe and Peters2005). Authors of recent zooarchaeology textbooks and articles state identification “methods” should be described and published, particularly listing the reference collections and published skeletal guides consulted (e.g., Beisaw Reference Beisaw2013; Driver Reference Driver1992; Gobalet Reference Gobalet2001; Hesse and Wapnish Reference Hesse and Wapnish1985; LeFebvre and Sharpe Reference LeFebvre, Sharpe, Giovas and LeFebvre2018; Nims and Butler Reference Nims and Butler2017; O'Connor Reference O'Connor2000; Rea Reference Rea1986; Reitz and Wing Reference Reitz and Wing2008), but the suggestion is not consistently made, even in textbooks (e.g., Davis Reference Davis1987; Gifford-Gonzalez Reference Gifford-Gonzalez2018; Hillson Reference Hillson1992; Rackham Reference Rackham1994). Some authors have urged analysts to publish the skeletal traits used to make identifications (e.g., Butler Reference Butler2011; Butler and Lyman Reference Butler and Lyman1996; Driver Reference Driver, Francis and Poplin1982; Gobalet Reference Gobalet2001; Graham and Semken Reference Graham, Semken, Graham, Semken and Graham1987; Klein and Cruz-Uribe Reference Klein and Cruz-Uribe1984; Lawrence Reference Lawrence and Matolcsi1973; Lyman Reference Lyman, Maschner and Chippindale2005, Reference Lyman2011a; Wolverton Reference Wolverton2013; Wolverton and Nagaoka Reference Wolverton, Nagaoka, Alves and Albuquerque2017). This suggestion, however, is only followed when an identification is particularly interpretively significant or difficult (Bochenski Reference Bochenski2008), such as the distinction of remains of domestic sheep (Ovis aries) from those of domestic goat (Capra hircus) (e.g., Balaase and Ambrose Reference Balaase and Ambrose2005; Boessneck Reference Boessneck, Brothwell and Higgs1969; Halstead et al. Reference Halstead, Collins and Isaakidou2002; Haruda Reference Haruda2017; Payne Reference Payne1985; Salvagno and Albarella Reference Salvagno and Albarella2017; Wolfhagen and Price Reference Wolfhagen and Price2017; Zeder and Pilaar Reference Zeder and Pilaar2010).
However, reporting that the University of California Berkeley Museum of Vertebrate Zoology skeletal collection was consulted and that Gilbert's (Reference Gilbert1973) guide was examined reveals little about the actual identification protocol followed—the techniques and anatomical traits used. Anatomical traits can be morphological (shape), metric (size), or meristic (frequency, as in the dental formula). Techniques can involve visual comparison of a zooarchaeological specimen with some (typically unreported) number of reference skeletons of similar size and morphology (which species were examined and why) or other approaches, such as measuring skeletal dimensions for statistical comparison (which dimensions were measured and why, and how many reference specimens of each of which taxa were measured). Best practice should be “transparency in identification methodology” (Lau and Kansa Reference Lau and Kansa2018:34), including everything just mentioned.
We use general science concepts to evaluate the identification process. “Validity” concerns the question, Are we measuring what we think we are measuring, or does the measurement technique provide an empirical indication of the concept of interest (Carmines and Zeller Reference Carmines and Zeller1979; Lastrucci Reference Lastrucci1967; VanPool and Leonard Reference VanPool and Leonard2011)? “Reliability” (replicability, repeatability) concerns the question, If we measure something twice, do we get the same answer, or, on repeated trials, do we obtain the same results (Carmines and Zeller Reference Carmines and Zeller1979; Lastrucci Reference Lastrucci1967; Zar Reference Zar1996)? Two aspects of reliability and validity (Golafshani Reference Golafshani2003; Heale and Twycross Reference Heale and Twycross2015; Whittemore et al. Reference Whittemore, Chase and Mandle2001) are important with respect to testing taxonomic identification protocols. The first is the “stability” of results (Heale and Twycross Reference Heale and Twycross2015) across repeated testing, known as “test–retest” (Roberts et al. Reference Roberts, Priest and Traynor2006). A test–retest method has been suggested by zooarchaeologists (Nims and Butler Reference Nims and Butler2017; Wolverton and Nagaoka Reference Wolverton, Nagaoka, Alves and Albuquerque2017), but reidentifying zooarchaeological specimens identified a few weeks ago does not test validity or accuracy, because the true identities are unknown. The second aspect of reliability and validity is “equivalence,” or consistency of results across multiple respondents (Heale and Twycross Reference Heale and Twycross2015). We think of the equivalence of results across multiple investigators as “usability.” By this, we mean, Is an identification protocol readily applied by different investigators, especially researchers who were not involved in development of the protocol? And does use of a protocol by multiple investigators influence the validity and/or reliability of the identifications?
“Precision” is defined as the closeness of repeated measurements of the same specimen (Sokal and Rohlf Reference Sokal and Rohlf1969; VanPool and Leonard Reference VanPool and Leonard2011; Zar Reference Zar1996). In this sense, a precise identification would be one made by several investigators of the same specimens as, say, Lepus americanus (snowshoe hare). That does not necessarily mean the identification is correct or accurate; “accuracy” means how close a measurement is to the true value (Sokal and Rohlf Reference Sokal and Rohlf1969; VanPool and Leonard Reference VanPool and Leonard2011; Zar Reference Zar1996).
Finally, “resolution” concerns the grain of a measurement—think dots per square inch in a photograph, where a greater number of dots means finer grain and greater resolution (a “sharper” or “higher-definition” image). A measurement of 0.01 mm is of greater resolution than a measurement of 0.1 mm, and a species-level identification is of greater resolution than a genus-level identification (Findley Reference Findley, Hester and Schoenwetter1964; Gifford-Gonzalez Reference Gifford-Gonzalez2018). In both cases, a fine-grained or high-resolution measurement (identification) contains all of the information of a coarse-grained or low-resolution measurement, but the latter only includes some of the information of the former.
Mindful of these concepts, we suggest criteria for the most rewarding testing of identification protocols. Testing would most simply employ protocols with explicitly described techniques and anatomical traits. If it has not been used much, it will not have entered the standard operating procedures of zooarchaeologists who might find it of value, and it will not have provided invalid identifications at a worrisomely high rate. Furthermore, it will be easy to correct any ambiguities or inaccuracies in the protocol discovered during the test. The researcher who developed the protocol should be involved in the testing, along with researchers of varied levels of experience with identifying ancient faunal remains. The developer will have intimate knowledge of the protocol and be able to correct problems detected by those researchers without such knowledge. Including researchers with varied levels of experience ensures that both novices and experienced researchers can use the protocol. The protocol should be tested with reference specimens of known taxonomy so that its validity can be evaluated. The test reference specimens should be of individual organisms not used to develop the protocol to maintain independence of the test from development of the protocol. An additional benefit would attend such independent testing because a greater range of intraspecific variability would likely be involved, particularly if the test specimens originate in populations different from the specimens used to develop the protocol (Polly and Head Reference Polly, Head and Elewa2004).
We do not expect all such tests to be published, but we think it is appropriate to describe one such test here in the hopes that the testing method will be mimicked for many allegedly taxonomically diagnostic traits. We also hope the results of such tests will be incorporated into a freely accessible online database. This suggestion follows one made by Lawrence (Reference Lawrence and Matolcsi1973) to create and maintain a hard copy of such a thing—a suggestion not followed due to, as Lawrence recognized, its impracticality. The suggestion of an online database was made more recently by Barr (Reference Barr2008), and it should now be both possible and practical (see also Lyman Reference Lyman2019).
Materials and Methods
We implemented a test of the protocol proposed by Lubinski and Hale (Reference Lubinski and Hale2018) for identifying stylohyoids of North American artiodactyls, hereafter referred to as the LH protocol. This protocol provides an ideal test case. First, its recent development means that it has not yet been widely applied, and consequently, that potentially incorrect identifications have not been produced. Second, its recency means weaknesses have not yet been detected. Third, the LH protocol involves an explicit dichotomous key made up of detailed descriptions of anatomical traits—both taxonomically diagnostic and not diagnostic—and a particular order for considering those traits, which lends itself to rigorous testing. Fourth, testers included one of the protocol originators (PML) and users who had nothing to do with developing the protocol and who have not previously used it, both an experienced zooarchaeologist (RLL) and a novice (MPJ). Our test exemplifies an ideal model for testing any identification protocol.
The artiodactyl stylohyoid (1) is large enough for consistent archaeological recovery (measuring 4–19 cm in length for adult specimens); (2) is commonly found at North American archaeological sites (up to half as common as the most abundant skeletal element at some sites); (3) often exhibits evidence of butchery and is sometimes manufactured into pendants; and (4) is not well known by many zooarchaeologists (no North American artiodactyl skeletal key includes this element), so documenting it will contribute to zooarchaeological knowledge (Lubinski and Hale Reference Lubinski and Hale2018). The stylohyoid is the largest bony element in the hyoid complex of artiodactyls in the Antilocapridae, Bovidae, and Cervidae. It is a paired bone in the throat below the mandibles. The element (Figure 1) is anchored at either end with cartilaginous rods, from the posterior, dorsal process to the temporal bone of the cranium (Saber and Hoffman Reference Saber and Hoffman1985), and from the anterior end to remaining hyoid-complex elements, which are small and indistinct in these taxa.
Figure 1. Example stylohyoid shown in (a) dorsal view, and (b) lateral view. This is a left bighorn sheep (Ovis canadensis) specimen (University of Washington Burke Museum 81695). It measures 69 mm in total length (anterior end to dorsal process). (Color online)
Lubinski and Hale (Reference Lubinski and Hale2018) recorded metric and morphological traits of nearly 500 reference hyoids of 13 species (10 genera, three families) of North American artiodactyls. They determined which traits seemed to distinguish taxa and estimated the rate of accurate identifications using the traits. Some traits allowed only distinguishing large from small artiodactyls; other traits permitted Bovidae to be distinguished from Cervidae; still other traits were diagnostic of genera; and some traits allowed identification to a particular species. Some congeneric species (e.g., Ovis aries, O. canadensis, O. dalli) could not be distinguished, but it seemed that most anatomically complete and skeletally mature stylohyoids could be identified at least to taxonomic family. No independent test of the validity and reliability of the stylohyoid traits was performed. Furthermore, no researcher who had not been involved in developing the protocol employed it to identify taxonomically known specimens, so there was no test of its usability.
To test the validity, reliability, and usability of the LH protocol, we obtained stylohyoids of known taxonomy from four zooarchaeology reference collections: California State University, Chico (CSUC); Eastern Tennessee State University (ETSU); Iowa State University (ISU); and Indiana University (IU). None of these test specimens had been used by Lubinski and Hale (Reference Lubinski and Hale2018) to develop this protocol. Lubinski received collection loans and removed specimens with significant breakage or adhering connective tissue that would complicate measurement, yielding a test sample of 77 stylohyoids representing 11 species of North American artiodactyls (Table 1). This sample derives from 54 different animals and includes 23 bilateral pairs, allowing evaluation of left-right identification consistency. It also includes at least one animal from each of the eight taxonomic groups that can be identified with the protocol: Antilocapra americana, small bovid, small cervid (Odocoileus sp.), large bovid, Alces alces, Bos taurus, Cervus elaphus, and Rangifer tarandus.
Table 1. Specimens in Study Sample.
a The Lubinski and Hale (Reference Lubinski and Hale2018) identification protocol does not allow for distinction of these three small bovid species, so they are combined here.
b The Lubinski and Hale (Reference Lubinski and Hale2018) identification protocol does not allow for distinction of these two small cervid species, so they are combined here.
The three test analysts represent a range of experience in identification of mammalian skeletal elements and with stylohoid identification. Lubinski has approximately 25 years of experience and no previous training or collaboration with Lyman, and he developed the stylohyoid identification protocol. Lyman has approximately 40 years of experience and had never before attempted to identify artiodactyl stylohyoids to species, in part due to the few specimens of this element in comparative collections he has used, and because no taxonomically diagnostic criteria were known to him. Johnson has approximately three years of experience but none in stylohyoid identification. Neither Lyman nor Johnson helped develop the LH protocol, which is why they could evaluate its usability. None of us was provided with the taxonomic identities of the test specimens until after we each independently identified them using the LH protocol.
Lubinski recorded the two morphological attributes and 10 osteometrics called for in Lubinski and Hale (Reference Lubinski and Hale2018:Table 5), recording metrics to 0.01 mm using digital calipers. He then followed the eight steps of the key in their Table 5 to make identifications, which involve two morphological traits and five of the osteometrics. After identification was complete, Lubinski sent the 77 test specimens first to Lyman and then to Johnson, with no guidance other than illustrations from Hale (Reference Hale2016:Figure 11) regarding how to complete the measurements.
Upon receipt of the 77 test specimens from Lubinski, Lyman practiced the identification protocol using 12 reference stylohyoids representing four species and curated in the University of Missouri Zooarchaeology Comparative Collection (five Antilocapra americana, two Cervus elaphus, four Odocoileus sp., and one Ovis aries). In every case, an accurate identification was determined. Feeling confident in his having learned the protocol correctly (and therefore believing the usability of the protocol to be high), he applied the protocol to the 77 test specimens. All metrics Lyman recorded were measured to the nearest 0.02 mm using dial calipers.
Johnson first practiced the protocol using seven specimens representing six species in the Central Washington University Zooarchaeology collection (one Antilocapra americana, two Bos taurus, one Capra hircus, one Cervus elaphus, one Odocoileus sp., one Ovis aries). Having success with these, he applied the protocol to the 77 test specimens. He recorded measurements to the nearest 0.01 mm using digital calipers.
Although all analysts used the LH protocol, we did not apply it identically. Lubinski used all criteria (except dorsal curvature, discussed below) for the eight steps of the key in Lubinski and Hale's Table 5 (Reference Lubinski and Hale2018) to make identifications. This means he faced the possibility of conflicting results in each step. When there was a conflict, he made a conservative identification as suggested by Lubinski and Hale (Reference Lubinski and Hale2018:373). For example, if in Step 7 Lubinski found 7a to key to Alces/Cervus, whereas 7b keyed to Rangifer, he would exit the key and list the result as unknown large cervid. Johnson also used all criteria except dorsal curvature and employed Lubinski and Hale's Table 5 (Reference Lubinski and Hale2018). When a conflict was noted, he was not certain how to proceed, so the criterion from Table 4 was employed, and the other criterion was ignored. Lyman used only one criterion for each step of the key, following Lubinski and Hale's Table 4 (Reference Lubinski and Hale2018), and had no conflicting results in a step to consider.
Results
We each predicted taxon for 72–74 of the 77 test specimens (Table 2). All of us omitted two specimens with unfused angles, two of us omitted two specimens as unfused or slightly broken, and two of us omitted one pathological specimen. This provided a final sample of 72–74 individual bones and 21–22 paired sets. Identification accuracy varied due to differences described above in how we applied the protocol. Given that some of Lubinski's identifications were to lower-resolution (higher) taxonomic groups than distinguishable by the LH protocol, we consider both instances in which there is a perfect match in the three investigators’ identifications, and consistent matches in which the three investigators’ identifications are not identical but are taxonomically consistent (e.g., two of us identified a specimen as Cervus elaphus, and the third identified it as large cervid).
Table 2. Results of Blind Test Identifications.
a Listed as Facility: Specimen Number - Side (if both sides included).
b As reported by loaning curation facility.
c This specimen was keyed as small bovid but was noted as pathological, and no identification would have been made if it had been an archaeological specimen.
If the LH protocol is reliable, taxonomic identifications made of left and right sides of paired elements will match, and taxonomic identifications made by each analyst of individual specimens will match. If the LH protocol is valid, taxonomic identifications should, regardless of investigator, match known taxonomy (if not resolution). If the LH protocol is usable, Lyman's and Johnson's identifications should closely match Lubinski's identifications as well as the known taxonomy.
Results indicate that the LH protocol is reliable; identifications were consistent across analysts for nearly all test specimens (Table 3). There were no mismatches in identifications between left and right sides across the three analysts. All identifications are consistent between Lubinski and Lyman, and they differ from Johnson for just two specimens (3%). Results also indicate that the LH protocol is valid but not 100% of the time. All of us misidentified a few specimens (see below), and Johnson misidentified two specimens correctly identified by the other two analysts (Table 3). Finally, Johnson and Lyman's results suggest that the LH protocol is quite usable; both successfully applied it to small pilot sets of specimens to learn the protocol, and then accurately identified 92%–94% of the test specimens.
Table 3. Evaluating Blind Test Identification Success.
a Perfect match in identifications—that is, at the same taxonomic resolution (e.g., all identifications were for Cervus elaphas).
b Identical match or agreement at a different taxonomic resolution (e.g., one identification was for Cervus elaphas, and the other for unknown large cervid).
Our identifications matched known species in the majority of cases. Counting differences in resolution that are consistent (e.g., large cervid vs. Cervus elaphus) as correct, there were 68–70 correct identifications—a 92%–96% success rate. One case was a partial mismatch. Lubinski identified specimen # 70 as large cervid and Lyman and Johnson identified it as Cervus elaphus; IU records list it as Alces alces.
In three cases, all analysts’ identifications failed to match the “known” species. We identified specimen # 6 as Antilocapra americana, whereas CSUC records list it as Odocoileus hemionus. We identified # 15 as Cervus elaphus, but ETSU records list it as Bos taurus, and we identified # 71 as Cervus elaphus, whereas IU records list it as Alces alces. These may be failures of the LH protocol's validity, or they may be errors in the records associated with the comparative collections. Such problems can occur as a result of labeling errors or incomplete removal of bones from dermestid beetle colonies used to skeletonize carcasses (e.g., Grayson and Maser Reference Grayson and Maser1978), so the stylohyoids are collected with bones of a different animal. Lubinski has noted indisputable taxonomic recording errors in several collections, such as four stylohyoids labeled 21715 from the University of Wisconsin Zoological Museum and four stylohyoids labeled 28721 from the Slater Museum of Natural History. That the potential exists for misidentified reference specimens has been noted before (e.g., LeFebvre and Sharpe Reference LeFebvre, Sharpe, Giovas and LeFebvre2018; Lyman Reference Lyman2019), but it has not been sufficiently emphasized. Accuracy in identification of archaeological specimens will depend on the accuracy with which the consulted reference specimens are identified.
Discussion
Our test of the LH protocol is thorough because it employs a large sample of stylohyoids from more than 50 animals (although there are few large artiodactyl specimens), and the test specimens were not included in the sample used to design the LH protocol. Results demonstrate that the protocol is reliable—identifications were replicable between anatomical sides and analysts. The LH protocol is also valid and accurate: 92%–96% of the taxonomic identifications were correct per investigator. The high level of agreement in results based on metric attributes indicates that any inter-analyst differences in measurement are not significant (e.g., Breslawski and Byers Reference Breslawski and Byers2015). And the usability of the LH protocol is good—if not perfect (see below) because its major aspects are spelled out explicitly. Consequently, both a highly experienced analyst and a novice produced a majority of correct identifications.
There were significant differences between analysts in identification resolution, a component of validity and a subject of some debate. Put simply, is it better to have low-resolution (e.g., genus-level) identifications with low rates of misidentification or high-resolution (e.g., species-level) identifications with higher rates of misidentification? Some advocate the former (e.g., Chaplin Reference Chaplin1971; Gilbert Reference Gilbert1973; Lawrence Reference Lawrence and Taylor1957, Reference Lawrence and Matolcsi1973; Peres Reference Peres, VanDerwarker and Peres2010). Others do not necessarily advocate the latter but seem less concerned about the rate of inaccurate identification (e.g., Ziegler Reference Ziegler1973). Gobalet (Reference Gobalet2001:380) seems to imply that accurate species-level identification is of less importance for a zooarchaeologist interested in human-behavioral questions than for a biologist interested in biogeography. To us, the implied acceptance of inaccurate identifications is ill-advised, regardless of the research question.
Lubinski's application of the LH protocol led to accurate low-resolution identifications, whereas Lyman and Johnson's led to inaccurate high-resolution identifications. In our view, low-resolution identification is preferable to misidentification because there is less potential for errors. It is undeniable that differences in taxonomic level of identification affect comparability of results between analysts, a component of usability. But it is also true that incorrect taxonomic identifications can range from having a minor influence on interpretations (e.g., Lyman Reference Lyman2012) to a fairly major influence on what we think happened in the past (e.g., Lyman Reference Lyman2011b). An example of the latter is that the paleoenvironmental implications of bird bones inaccurately identified as turkey (Meleagris gallopavo) from the Great Basin of western North America were negated once the identifications were corrected to sage grouse (Centrocercus urophasianus; Grayson Reference Grayson1977).
The high degree of reliability and validity suggests that the LH protocol is usable, although differences in our applications led to different degrees of resolution of identification and slightly different misidentification rates. In performing this blind test, Lubinski also realized that Step 4a (S- or C-shaped curvature in dorsal view) of the LH protocol was not easy to use. Based on the test sample of stylohyoids, this trait appeared to vary in a more continuous way than in an unambiguously distinct way, and it should be deleted from the key. None of the three analysts used this trait in this study. Other improvements to usability are to note typographical errors for Step 6a and 6b in Table 5 of Lubinski and Hale (Reference Lubinski and Hale2018), which should have no additional steps for large bovids and direct the user to Step 7 for large cervids, and to provide the photographs of how to take metrics from Hale's (Reference Hale2016:23–24) thesis.
This test has demonstrated inter-analyst variability in the use of an explicit key-based identification protocol. Despite our goals to each follow the directions in Lubinski and Hale (Reference Lubinski and Hale2018), we did not employ the protocol identically. This reflects two things. First, as a developer of the LH protocol, Lubinski was familiar with it and used it in the manner he believed to be the most correct. Second, as non-developers, Lyman and Johnson were less familiar with the protocol, so they tested their understanding of it with a pilot study. Finding success, they adopted it to identify the test specimens. In hindsight, they learned a lesson worth emphasizing: study closely any identification protocol with which you are not intimately familiar and, if possible, consult the developer of that protocol as to the correctness of your understanding of it. We differed in how we operationalized the protocol, and our resulting identifications varied accordingly. If viewed in terms of identification resolution, the differences in results are notable, with 92%–94% identical matches between Lyman and Johnson's identifications and known taxonomy, and 86% with Lubinski's identifications. On the other hand, if viewed in terms of mismatches between identifications and known taxonomy, regardless of resolution, the variability in accuracy is quite modest—6%–8% compared to 4% inaccurate. In our view, this constitutes an acceptable level of agreement and underscores the overall validity, reliability, and usability of the LH protocol.
Earlier we expressed our preference for what are probably accurate low-resolution identifications (taxonomic genus or family) over potentially inaccurate high-resolution identifications (species). Others may prefer the opposite. Whichever is preferred, our test underscores the need for explicit description of protocols, including the anatomical traits thought to be taxonomically diagnostic, and whether high- or low-resolution identifications were attempted.
Conclusions
Gobalet recommended that “identifications should be accepted only when there has been a discussion of the comparative methods used and the criteria used in the discriminations” (Reference Gobalet2001:385). We have expanded Gobalet's recommendation by outlining a specific procedure for testing identification protocols—both their analytical techniques and their anatomical traits. Following the wisdom of Lawrence (Reference Lawrence and Matolcsi1973), we see the critical nature of taxonomic identification as a warrant for explicit description in reports of the reference specimens and illustrated skeletal keys consulted, as well as the analytical techniques and skeletal traits used to make identifications. We also urge the creation of an open-access database of skeletal traits believed to be taxonomically diagnostic. It is a huge task to state identification criteria for every specimen, and editors are not likely to accept such criteria in published works, even for a limited set of the most problematic and interpretively significant identifications. We therefore suggest starting small, by focusing on more difficult identifications, explicitly testing anatomical traits, and describing those traits used to make identifications. The creation of an open-access database of skeletal traits would facilitate this process, providing a place for publication, peer review, and discussion. The database should allow editing when a trait is found to be difficult to employ (has low usability) or when it does not always result in accurate identifications (has low validity; e.g., Barr Reference Barr2008; Lyman Reference Lyman2019). Comments regarding the resolution provided by an anatomical trait should be part of the discussion of each one proposed to be taxonomically diagnostic. A trait naming system would also be helpful so that future zooarchaeologists might be able to list identifiers such as “Trait MA-106.” A nomenclatural system like the one developed by Driesch (Reference Driesch1976) for osteometry, with which zooarchaeologists the world over are familiar, suggests that such a system is not implausible (its practicality is unclear).
It is our hope that the sort of test described here will be undertaken and reported going forward so that problems such as those described by Gobalet (Reference Gobalet2001) can be avoided in the future. Like Wolverton (Reference Wolverton2013) before us, we hope to see a marked improvement in zooarchaeological quality control. Nims and Butler's (Reference Nims and Butler2017) recommendation for continuous evaluation of the identification protocol being used over the duration of a project is a superb additional suggestion to enhance the probability of correct taxonomic identifications and reduce inter-observer variation. Much of modern zooarchaeology rests on high-resolution taxonomic identifications. Insofar as those identifications are weakly substantiated and poorly documented, so too are any interpretations based on them.
Acknowledgments
Loans of comparative stylohyoid bones were provided by Laura Scheiber (Indiana University), Matt G. Hill (Iowa State University), Chris Widga (Eastern Tennessee State University), and Matt O'Brien (California State University, Chico). This manuscript was improved in light of comments provided by the editor and four anonymous reviewers.
Data Availability Statement
A Microsoft Excel spreadsheet containing all of the original measurements generated in this study is available from Pat Lubinski at Central Washington University.
Supplemental Materials
For supplemental material accompanying this report, visit https://doi.org/10.1017/aaq.2020.45. For supplemental figures showing how to take stylohyoid measurements, see Figure 10 from Hale (Reference Hale2016:23–24), available at https://digitalcommons.cwu.edu/etd/344/.
Supplemental Text 1. On Taxonomic Identification and the Changing Status of “Diagnostic” Skeletal Traits.
