Discontinuous markables

One of the distinctive features of arrau is the support of discontinuous markables—markables built out of non-continuous material. Discontinuous chunks are problematic for many corpus annotation formats Amoia et al. (Reference Amoia, Kunz and Lapshinova-Koltunski2011), and thus many guidelines developed for various linguistic phenomena allow for labeling continuous constituents exclusively.

Discontinuous markables, however, are common in dialog, for instance, in cases of so-called collaborative completions (Poesio and Rieser Reference Poesio and Rieser2010) illustrated by (5), where the mention an orange screw with a slit is constructed out of utterances 1.2 and 1.3.

1. (5)

   

For this reason, Müller included the functionality for the annotation of discontinuous markables in the mmax2 annotation tool (Müller and Strube 2006), developed to support his research on anaphora resolution in dialog (Müller Reference Müller2008), where spans can be arbitrary sequences of tokens. However, discontinuous markables also provide a way to include in a markable all information provided by the text, for example, in cases of coordination where the two coordinated nps share some information, illustrated by (6). In this example, the two names Anna Snezak and Morris Snezak are coordinated, but the last name Snezak is only repeated once. Discontinuous markables make it possible to include both the segments of text marked as part 1 and part 2 in the same markable. Similarly in (7).

1. (6) …after owners [^part1 Anna]^part1 and Morris [^part2Snezak] ^part2…

2. (7) So he doesn’t have to play [^part1 the same Mozart]^part1 and Strauss [^part2 concertos]^part2 over and over again.

Discontinuous markables are typically ignored in anaphora resolution: state-of-the-art mention detection systems always output continuous chunks; the publicly available SemEval and conll coreference scorers (Pradhan et al. Reference Pradhan, Luo, Recasens, Hovy, Ng and Strube2014) assume numbered brackets as mention boundaries that cannot encode discontinuous fragments. To make arrau usable for these purposes, whereas the markable can be discontinuous, minimal spans cannot be. This way, all the markables in arrau can be aligned to contiguous sequences of tokens.

Referring and non-referring markables

Most anaphorically annotated corpora focus on referring markables, or mentions proper: markables that refer to discourse entities and participate in anaphoric relations. This decision, primarily motivated by reasons of cost, makes it however difficult to train models able to recognize and interpret non-referring markables—nominal expressions that do not refer a discourse entity. It has been shown, however, that filtering out at least some types of non-referring expressions can improve the performance of a coreference resolver (Uryupina et al. Reference Uryupina, Kabadjov, Poesio, Poesio, Stuckardt and Versley2016b). In order to develop such a classifier for a corpus like OntoNotes in which non-referring expressions are not annotated, separate classifiers are required—for example, Björkelund and Farkas (Reference Björkelund and Farkas2012) trained a pre-filtering classifier for non-anaphoric it, you and we on the OntoNotes data.

In arrau, all nominal expressions are treated as markables, including non-referring nominal expressions. The annotation scheme and guidelines are based on those developed for gnome, where the lftype attribute (κ = .73) was used to distinguish between referring expressions proper (called terms in gnome) and several types of non-referring interpretations of NPs, including expletives (8), predicatives (9,10), quantifiers (11) and coordinations (12) Poesio (Reference Poesio2000a, Reference Poesio2004b). In arrau, coders are asked, first of all to classify markables as referring or non-referring. If a markable is classified as referring, coders are then asked if that expression is discourse-old or discourse-new (Prince Reference Prince, Thompson and Mann1992), and in the first case, to identify its antecedent (see Section 2.4). If the markable is classified as non-referring, coders have to either assign it to one of the gnome categories of non-reference, or label it as idiomatic (13), or as an incomplete or fragmentary expressions (14).

1. (8) And [there]^{non–referring}’s a ladder coming out o of the tree and [there]^{non–referring}’s a man at the top of the ladder

2. (9) It see it seems to be [a busy place]^{non–referring}

3. (10) 1 ml of the prepared solution for injection contains 0.25 mg ([8 million IU]^{non–referring}) of Interferon beta-1b.

4. (11) [Most of the analysts polled last week by Dow Jones International News Service in Frankfurt, Tokyo, London and New York]^{non–referring} expect the US dollar to ease only mildly in November.

5. (12) Mr. Sutton recalls: “When I left, I sat down with [[Charlie Rangel], [Basil Paterson] and [David]]^{non–referring}, and David said, ‘Who will run for borough president?

6. (13) so that would um if we left at six in the morning would that make [sense]^{non–referring} six (mumble)

7. (14) U: okay then um okay then originally we need to have um the one boxcar go to [oranum]^{non–referring} go to Corning from Elmira

The choice of marking quantifiers and coordination in arrau as non-referring is possibly the most controversial decision we took. The quantifier Most of the analysts polled last week by Dow Jones international in (11) is marked as non-referring. Similarly, whereas we asked coders to mark individual noun phrases (Charlie Rangel, Basil Paterson and David in (12)) as referring markables that can participate in anaphoric relations, the embedding coordinate np is marked as non-referring. These decisions mean that any expression anaphorically related to that quantifier cannot be marked as such. However, plural anaphora to antecedents introduced by coordination can be annotated, as discussed in Section 2.4. In the case of quantifiers, the decision was motivated by the high disagreement that we observed among our coders when left free to mark a quantifier as either referring or non-referring. For the case of coordination, the reasons were more complex; we discuss them when explaining how plural anaphora is handled. Both decisions might be reconsidered in a future release of arrau.

Table 2 shows the distribution of various types of non-referring markables in the entire corpus and in the four individual domains, overall and for each type of non-referring markables. As could be expected, the distribution of non-referring expressions is genre-specific. Thus, the two domains with spontaneously generated no-curated texts (TRAINS and PEAR) have a large number of incomplete or fragmentary expressions, virtually non-existent in RST and gnome documents. Idioms are common in all the genres except gnome—a collection of medical leaflets written in a very formal language. Predicative non-referring expressions, especially appositions, are more common in news.

Table 2. Distribution of non-referring markables in arrau

Genericity

The guidelines for genericity adopted for the gnome corpus were developed to distinguish generic uses of nominal expressions (as in Dogs bark) from non-generic cases (as in I saw dogs in the street). Developing reliable guidelines for this type of annotation proved quite a challenge, and two schemes were conceived before developing one achieving sufficient reliability. The first scheme attempted to capture the type / token distinction—a similar distinction to that between generic and specific entities made in the ace-2 coding scheme—but this type of judgment proved difficult to agree on in particular with mentions referring to substances such as oil or chemical components of medicines such as oestradiol, as illustrated in (15). The result was that this simple scheme only achieved a very modest level of reliability (κ = .33).

1. (15) Not that [oil]^generic suddenly is a sure thing again.

A second scheme was then developed in which a new value, undersp-generic, was introduced as the value to be used for all references to substances.Footnote ^j The new scheme achieved better reliability, but still only κ = 0.55. The biggest remaining problem were quantifiers (including definites and indefinites). Our annotators found it very hard to agree on whether a quantified np used (non-generically) to quantify over a specific set of individuals at a particular spatio-temporal location, as in Many lecturers went on strike (on March 16th, 2004), should be marked as generic or not. A third and last scheme was therefore developed, in which separate values were introduced for each type of quantifier, as well as new guidelines, according to which the annotation of the genericity attribute is carried out following a decision tree going from the easiest cases to the more complex ones. Coders are first asked to check whether the nominal is in the scope of an explicit operator such as a conditional like if (as in (16)) or an individual quantifier such as every or most (iquant) (as in (17)) or a temporal quantifier like always or once (as in (18)) a modal (as in (19)) or an instruction (as in (20)). In these cases, the nominal is not marked as generic, but as being in the scope of the appropriate operator. If no such explicit quantifier/operator is present, coders are asked to check whether the nominal refers to semantic objects whose genericity is left underspecified, such as substances (e.g., gold), as in (21) seen before or in (22). Finally, the annotator is asked whether the sentence in which the markable occurs is generic, and in this case, to mark the nominal as generic-yes if it refers generically, as in (23), or generic-no otherwise. With these instructions, reasonable intercoder agreement was finally achieved (κ = .82) (Poesio Reference Poesio2004b).

1. (16) New York State Comptroller Edward Regan predicts a $ 1.3 billion budget gap for the city’s next fiscal year, a gap that could grow if there is [a recession]^generic.

   (operator-conditional)

2. (17) Mr. Uhr said that Mr. Petrie or his company have been accumulating Deb Shops stock for several years, each time issuing [a similar regulatory statement]^generic.

   (operator-iquant)

3. (18) In addition , once [money]^generic is raised , [investors]^generic usually have no way of knowing how [it]^generic is spent.

   (operator-tquant)

4. (19) They argue that their own languages should have [equal weight]^generic, although recent surveys indicate that the majority of the country’s population understands Filipino more than any other language.

   (operator-modal)

5. (20) Use [alcohol wipes]^generic to clean the tops of the vials move in one direction and use one wipe per vial.

   (operator-instruction)

6. (21) Not that [oil]^generic suddenly is a sure thing again.

   (underspecified-substance, RST)

7. (22) 1 ml of [the prepared solution for injection]^generic contains 0.25 mg ( 8 million IU ) of [Interferon beta-1b]^generic.

   (underspecified-substance, gnome)

8. (23) In its report to Congress on [international economic policies]^generic, the Treasury said that any improvement in the broadest measures of trade, known as the current account.

   (generic-yes)

Genericity was already marked according to these guidelines in the first release of arrau (Poesio and Artstein Reference Poesio and Artstein2008), but its annotation was only partially checked. One of the main revisions carried out for the second release of the corpus was a systematic check that the annotation of this attribute was consistent with the guidelines. The distribution of generics and quantifiers in the separate arrau domains resulting from this verification is shown in Table 3. In total 2252 mentions were annotated as generic (2% of the total number of markables), 3167 as being bound by some other operator (3%), and 1.4% as underspecified.

Table 3. Distribution of generic mentions in arrau

Bridging anaphora

Annotating—indeed, identifying—bridging anaphora in a reliable way is a difficult task (Poesio and Vieira Reference Poesio and Vieira1998; Vieira Reference Vieira1998), which is one of the reasons why so few large-scale corpora for anaphora include this type of annotation (apart from our own work, we are only aware of few attempts to do so; see Section 4.4 for a discussion of this work and (Poesio et al. Reference Poesio, Pradhan, Recasens, Rodriguez, Versley, Poesio, Stuckardt and Versley2016) for additional discussion of larger corpora some of which also include anaphora). The arrau guidelines for bridging anaphora are based on a series of experiments that started with the work of Vieira (Reference Vieira1998) and Poesio and Vieira (Reference Poesio and Vieira1998) and continued in the gnome project (Poesio Reference Poesio, Bruneseaux, Romary and Walker2004b). Vieira and Poesio attempted to annotate the full range of bridging references as discussed, for example, in Passonneau (Reference Passonneau1997) and Poesio et al. (Reference Poesio1999), but only achieved very poor agreement. In gnome, attempts were made to identify a subset of the relations that could be annotated reliably (Poesio Reference Poesio2004b), finding that by limiting the annotation to three types of relations: element-of as in (24), where the middle is a bridging reference to the middle of the three horizontal zones; subset as in (25), where Polygonal openwork rings incorporating an inscription in (u2) is a bridging reference to two gold finger rings in (u1) based on an inverse subset relation; and a generalized possession relation poss covering both part-of relations as in (26) and general possession relations, as in (27). The element relation was also used to annotate certain types of other anaphora, as in (28).

1. (24) The sixteen panels are each divided into [three horizontal zones]₁, [the middle]_→1 containing a letter

2. (25) (u1) [Two gold finger-rings from Roman Britain (2^nd–3^rd century AD)]₁.

   (u2) [Polygonal openwork rings incorporating an inscription]_→1 are a distinctive type found throughout the Empire.

3. (26) (u1) [These “egg vases”]₁ are of exceptional quality

   (u2) basketwork bases support [egg-shaped bodies]_→1

   (u3) and bundles of straw form [the handles]_→1

4. (27) (u1) [The Getty museums microscope]₁ still works,

   (u2) and [the case]_→1 is fitted with a drawer filled with the necessary attachments.

5. (28) (u39) [The two stands]₁ are of the same date as the coffers, but were originally designed to hold rectangular cabinets.

   (u42) [One stand]_→1 was adapted in the late 1700s or early 1800s century to make it the same height as [the other]]_→1.

Poesio et al. found that coders following the gnome guidelines achieved good precision but low recall on identifying bridging references (Poesio Reference Poesio2004b). When asked to mark mentions as either discourse-new, discourse-old, or bridging according to the gnome definition of bridging, coders agreed on the type of relation for bridging references in 95.2% of the cases, but each of them only spotted about 1/3 of bridging references on average, and typically different bridging references, so that only 22% of bridging references were marked as such by all annotators.

The arrau Release 1 guidelines followed the gnome guidelines, but with an extension and a simplification. Annotators were asked to mark a mention as related to a particular antecedent if it stood to that antecedent in one of the relations identified in gnome (indeed, the same examples were used), and in addition, if they stood in two additional relations (but without testing the reliability of this annotation):

• other, for other nps, broadly following the guidelines in Modieska (Reference Modieska2003);

• an undersp-rel relation for “obvious cases of bridging that didn’t fit any other category.”

In arrau Release 1, however, coders were not asked to specify the relation—effectively, any associative bridging reference was considered a case of “underspecified relation.” In arrau Release 2, the annotation of bridging references was revised for the rst domain only and coders were now asked to mark the relations only in that domain. The resulting statistics about bridging references in arrau Version 2 are shown in Table 4. A total of 5512 bridging references were marked, but a classification of the relations was only provided for the 3777 bridging references identified in the rst domain. In the table, we write P+S+E+O+U as category for the bridging references in the other domains, currently not classified. We intend to provide a classification of these bridging references, as well as re-checking the existing classifications, in Release 3 of the corpus, currently planned for 2018.

Table 4. Distribution of bridging references in arrau

Plural anaphora

Till recently, no data-driven studies were attempting to model plural anaphora specifically, except for the simplest cases of plural reference to a plural antecedent, as in (29).Footnote ^l

1. (29) (u1) from Avon going to Dansville pick up [the three boxcars]₁

   (u2) go to Corning load [them]₁ and …

This is because some types of plural reference are intrinsically difficult, both for annotation and resolution. We believe therefore that a dataset annotated for plural anaphora in a principled way will open several challenging research possibilities.

One example of the more complex forms of plural anaphora is plural reference to sets of objects introduced by listing their elements, as in the following toy examples.

1. (30) a. [Mr. Luzon and his team]₁, however, say [they]₁ aren’t interested in a merger.

   b. Mr. Luzon agreed with his team that [they]_? aren’t interested in a merger.

Anaphoric annotation schemes that do require coders to mark plural reference to antecedents introduced by coordination do so by assuming that the coordination Mr. Luzon and his team in (30a) (an actual example from the rst portion of arrau) introduces a discourse entity, and asking coders to link they to that entity. Indeed, this is the approach that was followed in gnome. This approach will not however work for the very similar (30b) (our own), since in this example there is no longer a constituent for Mr. Luzon and his team—so they becomes a discourse new mention with no antecedent. The approach to annotating plurals adopted in arrau was based on the belief that these two very similar cases of plural reference should be treated in the same way. In arrau, we annotate plural anaphors to sets of individually introduced entities as bridging references to each member of the corresponding set encoding an (element-of) bridging relation. Thus, in (30a) as well as in (30b) “They” is linked to both “Mr. Luzon” and “his team” individually. Note that such annotation allows for a more uniform interpretation of plural reference to individually introduced entities.

Discourse deixis

The term discourse deixis was introduced by Webber in Webber (Reference Webber1991) to indicate the reference to abstract entities which have not been introduced in the discourse through a nominal expression,Footnote ^m as in the following example from the trains corpus, where that in utterance 7.6 refers to the plan of shipping boxcars of oranges to Elmira.

1. (31)

   

Discourse deixis in its full form is a very complex form of reference, both to annotate (Artstein and Poesio Reference Artstein, Poesio, Schlangen and Fernandez2006; Dipper and Zinsmeister Reference Dipper and Zinsmeister2012) and to resolve (Marasović et al. Reference Marasović, Born, Opitz and Frank2017). Very few anaphoric annotation projects have attempted annotating discourse deixis in its entirety (Artstein and Poesio Reference Artstein, Poesio, Schlangen and Fernandez2006; Dipper and Zinsmeister Reference Dipper and Zinsmeister2012; Kolhatkar Reference Kolhatkar2014); more typical is a partial annotation, as in the work of Byron and Navarretta, who annotated pronominal reference to abstract objects (Byron and Allen Reference Byron and Allen1998; Navarretta Reference Navarretta2000); in OntoNotes, where event anaphora was marked (Pradhan et al. Reference Pradhan, Ramshaw, Weischedel, MacBride and Micciulla2007); and in the work of Kolhatkar Kolhatkar (Reference Kolhatkar and Hirst2014), which focused on so-called shell nouns. As a result, very few systems have attempted resolving this type of anaphors (Eckert and Strube Reference Eckert and Strube2000; Byron Reference Byron2002; Kolhatkar and Hirst Reference Kolhatkar and Hirst2012; Marasović et al. Reference Marasović, Born, Opitz and Frank2017)

Discourse deixis was one of the “difficult cases of anaphora” on which the arrau project focused, and a number of annotation experiments were conducted (Artstein and Poesio Reference Artstein, Poesio, Schlangen and Fernandez2006), resulting in guidelines according to which

1. 1. A coder specifying that a referring expression is discourse old is asked whether its antecedent was introduced using a phrase (mention) or segment (discourse segment)

2. 2. Coders choosing segment as the type of antecedent have to mark a sequence of (predefined) clauses

Artstein and Poesio (Reference Artstein, Poesio, Schlangen and Fernandez2006) point out that measuring disagreement on this type of annotation requires making a number of assumptions, and that figures of α Krippendorf (Reference Krippendorf2004) ranging from 0.45 to 0.9 can be achieved depending on which assumptions are made.

The statistics about discourse deixis in arrau Version 2 are shown in Table 5. A total of 1633 cases of discourse deixis were identified. It is worth noting how the trains sub-domain contains more than half the total cases of discourse deixis even though it is less than half the size of the rst sub-domain. (We intend to re-check the annotation in Release 3 of the corpus, currently planned for 2018.)

Table 5. Distribution of discourse deixis in the subdomains of arrau

Anaphoric ambiguity

A number of studies have shown that anaphoric expressions both in dialog and text can be ambiguous (Poesio and Reyle Reference Poesio, Reyle, Bunt and van der Sluis2001; Poesio et al. Reference Poesio, Sturt, Arstein and Filik2006b; Versley Reference Versley2008; Recasens et al. Reference Recasens, Hovy and Martí2011). A classic illustration is Example (32), from the trains corpus (Poesio and Reyle Reference Poesio, Reyle, Bunt and van der Sluis2001). The pronoun it in (u2) could refer equally well to engine E2 or the boxcar at Elmira. Studies carried out as part of arrau showed that such examples were fairly common in the trains corpus, and that different coders would interpret them differently (Poesio and Artstein Reference Poesio, Artstein and Meyers2005a; Poesio et al. Reference Poesio, Sturt, Arstein and Filik2006b). Other studies have shown that occurrences of it can be ambiguous between an expletive and a discourse deixis interpretation (Gundel et al. Reference Gundel, Hedberg and Zacharski2002).

1. (32) (u1) M: can we.. kindly hook up … uh … [engine E2]₁ to [the boxcar at.. Elmira]₂

   (u2) M: +and+ send [it]_1,2 to Corning as soon as possible please

The arrau coding scheme accommodates this. Referring markables can be marked as ambiguous between a discourse-new and a discourse-old interpretation; discourse-old mentions can be marked as ambiguous between a discourse-deictic and a phrase reading; and both phrase and segment mentions can be marked as ambiguous between two distinct interpretations. The annotated corpus contains examples of ambiguous anaphoric expressions from text as well, as in the following example.

1. (33) Criticism of [the Abbie Hoffman segment]₁ is particularly scathing among people who knew and loved the man. <…> Both women say they also find it distasteful that [CBS News is apparently concentrating on Mr. Hoffman’s problems as a manic-depressive]₂. “[This]_1,2 is dangerous and misrepresents Abbie’s life,” says Ms. Lawrenson, who has had an advance look at the 36-page script.

In (33), the anaphoric mention “This” is ambiguous between “the Abbie Hoffman segment” (identity anaphora) and “CBS News is apparently concentrating on Mr. Hoffman’s problems as a manic-depressive” (discourse deixis).

The extent of ambiguity in anaphoric interpretation found using the arrau scheme was analyzed in a study reported in Poesio and Artstein (Reference Poesio, Artstein and Meyers2005a). A total of 18 subjects were asked to annotate dialogs from the trains subdomain of arrau with a scheme allowing them to mark for ambiguity. Poesio and Artstein reported that a minimum of 10% of markables in the trains corpus were marked as explicitly ambiguous. They also found however that a much higher percentage of markables, up to 40%, were implicitly ambiguous—i.e., were annotated differently by different subjects. In Poesio and Artstein (Reference Poesio and Artstein2005b) methods for computing agreement in a scheme allowing for ambiguity were proposed, based on developing extended distance metrics for α (Krippendorf Reference Krippendorf2004; Artstein and Poesio Reference Artstein and Poesio2008). Values of α between .58 and .67 were reported depending on the type of distance metric used and the choice of markables.

Statistics about anaphoric ambiguity in arrau Version 2 can be found in Table 6. The first column of the table shows the category of the first interpretation of the ambiguous markable: discourse old (either phrase or segment), discourse new, or non-referring. The second column shows the second interpretation indicated by the coder: again discourse old (phrase) but with a different antecedent, discourse new, discourse deixis, or non-referring. A total of 234 cases of ambiguous markables were identified, which is a very small fraction of the around 100,000 markables in arrau Version 2; the results of Poesio and Artstein (Reference Chiarcos and Krasavina2005a) suggest however that this figure substantially underestimates the actual extent of ambiguity, at least by a factor of 4. The majority of these ambiguities (75%) are between two discourse old interpretations with different antecedents, but there are also several cases of DN/DO ambiguity and DO/DD ambiguity. We also note how in all cases of ambiguity the first interpretation chosen is discourse old; this is because the instructions explicitly require coders to choose DO as first interpretation if the ambiguity is between a discourse-old interpretation and some other interpretation.

Table 6. Distribution of ambiguity in the subdomains of arrau

Referentiality

As mentioned above, the annotation schemes for coreference used in the best-known resources for English (the muc and ace corpora, OntoNotes) do not require the annotation of non-referring expressions, or even singletons. But several of what we have called the “new wave” of linguistically motivated corpora, particularly those with a syntactic definition of mention, do, although typically (non-)referentiality is indicated in a more indirect way than in arrau. In tüba/dz (Hinrichs et al. Reference Hinrichs, Kübler and Naumann2005), for instance, an expletive attribute is used to mark pleonastic instances of the impersonal third person singular pronoun es (it). No other types of non-referentiality seem to be marked. In ancora (Recasens and Martí Reference Recasens, àrquez, Sapena, Mart, Taulé, Hoste, Poesio and Versley2010), mentions are automatically extracted from the syntactic tree, and an entityref attribute is used to mark referring nps, so non-referring mentions can be identified although again the representation is not quite so explicit as in arrau.Footnote ^s

Genericity

When the arrau annotation started, we were only aware of one attempt at marking the genericity status of mentions apart from our own efforts in gnome—the annotation of the entity-class attribute in ace-2, with values generic and specific— but there has been some more work in this area since.

The ace-2 Entity Detection and Tracking Guidelines do provide instructions for distinguishing generic from specific mentions, relying heavily on examples to address the problems we encountered in gnome. We do not know however whether these difficulties were in fact solved as we are not aware of any results regarding the reliability of these guidelines. This annotation was nevertheless used to train one of the first models of automatic genericity classification (Reiter and Frank Reference Reiter and Frank2010). Herbelot and Copestake (Reference Herbelot, Copestake, Featherston and Winkler2008) developed an interesting scheme, strongly rooted in the literature on genericity in formal linguistics (Carlson and Pelletier Reference Carlson and Pelletier1995), and still attempting to capture genericity and specificity but treating them as two separate two dimensions of classification, as done in Carlson and Pelletier (Reference Carlson and Pelletier1995). The first version of the scheme provides a label for generic entities (gen), which would be the equivalent of the arrau label generic-yes; one for non-generic, specific entities (spec); and one for non-generic, non-specific entities (non-spec). In addition, the label amb is used for references ambiguous between generic and non-generic reading (like undersp-gen in the arrau scheme), and a label group for references to subgroups of a generic entity. This version of the scheme achieves a similar reliability to that of the second version of the gnome scheme for genericity. The authors then developed a second set of guidelines, based on the same scheme but providing detailed instructions for a number of special cases; with this second set of guidelines they manage to achieve a reliability of κ = 0.74. In the Prague Dependency Treebank, all nominals are marked as generic or specific, and coreference relations are only marked between nominals with the same category (generic or specific). Recently, a systematic analysis of coreference with generic nps was carried out by Nedoluzhko (Reference Nedoluzhko2013). Finally regarding manual annotation, we shall mention the recent and very interesting work by Friedrich et al. (Reference Friedrich, Palmer, Sorensen and Pinkal2015) who annotated clauses and subjects for genericity, a type of annotation that would be a very useful preliminary step towards the annotation of genericity of mentions in arrau. Friedrich and colleagues reported an average agreement of around κ = .56.

Corpora annotated for bridging references

At the time the arrau annotation started, there had not been many other attempts to annotate bridging reference apart from our own work as part of the Viera / Poesio corpus (Poesio and Vieira Reference Poesio and Vieira1998) and gnome corpus (Poesio Reference Poesio2004a),Footnote ^t but there have been a number of efforts, since many of which have attempted to annotate a broader range of the bridging relations identified in the early literature (Passonneau Reference Passonneau1997; Davies et al. Reference Davies, Poesio, Bruneseaux and Romary1998; Vieira Reference Vieira1998).

One of the most ambitious such efforts in terms of coverage of relations is the work by Gardent and Manuélian as part of the annotation of the DeDe corpus (Gardent and Manuélian Reference Gardent and Manuélian2005). Gardent and Manuélian annotate a range of bridging relations including, apart from the part relations encoded in arrau, a more general circumstantial relation covering a variety of relations. The annotation was also carried out using mmax2 and the markup scheme is very compatible with that used in arrau. No agreement results were however reported as far as we’re aware.

Possibly the most extensive effort towards annotating bridging carried out in parallel with the annotation of arrau is the annotation of bridging coreference in the Prague Dependency Treebank (Nedoluzhko et al. Reference Nedoluzhko, Mírovský, Ocelák and Pergler2009a). Nedoluzhko et al. distinguish, apart from part and subset relations,

• A funct relation covering function-value relations, as proposed in mate (Davies et al. Reference Davies, Poesio, Bruneseaux and Romary1998)

• A new contrast relation covering relations between opposites (People don’t chew, it’s cows who chew)

• A more general underspecified group rest, which is used for capturing other types of bridging references such as event argument.

Nedoluzhko et al. measured interannotator agreement using a combination of F1 values (for the antecedent) and κ (for the relation), achieving F1=.59 for the antecedent, and κ = .88 for the relation.

Another substantial annotation effort was carried out by Hou, Markert and Strube (Markert et al. Reference Markert, Hou and Strube2012), who annotated HSNotes, a corpus of 11,000 nps in 50 texts taken from the wsj portion of OntoNotes for information status, building on the scheme by Nissim et al. (Reference Nissim, Dingare, Carletta and Steedman2004) but also annotating the anchors of 663 bridging nps. Their scheme also expands on the definition of mediated from Nissim et al. by also including other anaphora among the bridging references, as well as the funct relation. An interesting analysis of the differences between the notion of bridging reference annotated in arrau and that annotated in HSNote can be found in Roesiger (Reference Roesiger2018).

Finally, we should mention that there has been quite a lot of research on bridging reference in corpus linguistics, which, while not producing usable corpora, did involve developing annotation guidelines—a notable example being the work by Botley (Reference Botley2006).

Discourse Deixis

Annotating discourse deixis is another task not tackled in all large-style anaphoric annotations, but there have been a number of efforts both preceding, parallel with, and subsequent to the effort in arrau and the already discussed studies of agreement in discourse deixis annotation (Artstein and Poesio Reference Artstein, Poesio, Schlangen and Fernandez2006).

Prior to arrau, we will mention the seminal work by Byron, who annotated pronominals and demonstratives in the trains-93 corpus, including abstract objects (Byron and Allen Reference Byron and Allen1998), and used the data to develop the first resolver of references to abstract objects we are aware of Byron (Reference Byron2002); Navarretta, who in parallel with Byron carried out similar studies of abstract reference in Danish and Italian (Navarretta Reference Navarretta2000, Reference Navarretta and Johansson2008); and by Eckert and Strube (Reference Eckert and Strube2000), who analyzed references in dialogs to both concrete and abstract objects. We will also mention the illuminating work by the corpus linguists Gundel, Hedberg, Hegarty, and associates on reference to “clausally introduced entities” (Gundel et al. Reference Gundel, Hegarty and Borthen2003, Reference Gundel, Hedberg and Zacharski2002), that was an important influence on our own work.

The most notable effort carried out in parallel with the work on arrau is the work in OntoNotes on annotating event anaphora, an important category of reference to abstract objects.

Following the first work in arrau on annotating discourse deixis, there appeared a number of studies that attempted to annotate a comparable subset of the phenomenon in other languages. Most notably among these efforts are the work in ancora on discourse deixis in Catalan and Spanish (Recasens Reference Recasens and Johansson2008), and the work by Dipper and Zinsmeister (Reference Dipper and Zinsmeister2012) on abstract anaphora in German. More recently, a very systematic analysis and annotation of another subset of discourse deixis, so called shell nouns, has been carried out by Kolhatkar (Reference Kolhatkar2014) and Kolhatkar and Hirst (Reference Kolhatkar and Hirst2012).

Ambiguity

Anaphoric ambiguity is a very understudied phenomenon and there have been hardly any other attempts to create a corpus in which the ambiguity of expressions is marked, with two exceptions. The coreference annotation carried out by Krasavina and Chiarcos (Reference Krasavina and Chiarcos2007) as part of the work on the Potsdam Commentary Corpus (Stede Reference Stede2004) is the only other coreference annotation scheme we are aware of that asks coders to mark ambiguity. The guidelines produced by Chiarcos and Krasavina (Reference Chiarcos and Krasavina2005) require coders to use the ambiguity mention attribute to indicate ambiguous mentions, and the type of ambiguity: for instance, ambig-ante if the mention is clearly discourse old but it’s not clear what the antecedent is, or ambig-expl for instances of es (“it”) that could be interpreted either as anaphoric or as expletives.

We must admit however that the results of Poesio and Artstein (Reference Poesio, Artstein and Meyers2005a) convinced us that this aspect of the arrau guidelines clearly needs rethinking, and in our research since we have taken a completely different direction, aiming to capture implicit ambiguity rather than explicit, as in arrau. Indeed, this aim was one of the primary motivations behind the Phrase Detectives project Poesio et al. (Reference Poesio, Chamberlain, Kruschwitz, Robaldo and Ducceschi2013), which has developed a Game-With-A-Purpose to elicit from players multiple interpretations for anaphoric expressions (25 on average and as many as 32 in some cases). The Phrase Detectives game uses a simplified version of the arrau coding scheme, but all interpretations are stored, and preliminary analysis suggests that around 40% of mentions have at least two interpretations selected by at least two players. A first, small subset of the Phrase Detectives corpus was recently released via ldc (Chamberlain et al. Reference Chamberlain, Poesio and Kruschwitz2016). More recently, this line of research has led us to start the dali project,Footnote ^u in which besides carrying out the development of more Games-With-A-Purpose to study anaphora, methods to compare and analyze these interpretations will also be developed.

Datasets

Three separate datasets were made available for the three distinct tasks. All three datasets were in the format used for the evalita-2011 evaluation campaign (Uryupina and Poesio 2013), which in turn was derived from the tabular conll-style format used in the semeval 2010 shared task on multilingual anaphora (Recasens et al. Reference Recasens and Martí2010). (See Poesio et al. Reference Poesio, Grishina, Kolhatkar, Moosavi, Roesiger, Roussel, Simonjetz, Uma, Uryupina, Yu and Zinsmeister2018 or the shared task page for further details on the format.) Three of the arrau sub-corpora were used: pear, rst and trains.

Evaluation Scripts

Three evaluation scripts were developed for the three tasks.

The coreference evaluation script developed by Moosavi and Strube, which, in turn, builds upon the official conll implementation (Pradhan et al. Reference Pradhan, Luo, Recasens, Hovy, Ng and Strube2014), was modified to produce the scorer for Task 1. We will refer to this script as ‘the extended coreference scorer’.Footnote ^w The extended scorer, when run excluding non-referring expressions and singletons and ignoring MIN information, evaluates a system’s response using the same metrics (indeed, a reimplementation of the same code) as the standard conll evaluation script, v8 (Pradhan et al. Reference Pradhan, Luo, Recasens, Hovy, Ng and Strube2014).Footnote ^x When required to use MIN information, the extended scorer follows the muc convention, and considers a mention boundary correct if it contains the MIN and doesn’t go beyond the annotated maximum boundary. When singletons are to be considered, singletons are also included in the scores (all metrics apart from muc can deal with singletons). Finally, when run in all-markables mode, the script scores referring and non-referring expressions separately. Referring expressions are scored using the conll metrics; for non-referring expressions, the script evaluates P, R and F1 at non-referring expression identification. The extended coreference scorer is available from Moosavi’s github at https://github.com/ns-moosavi/coval.

The evaluation script for Task 2 is based on the evaluation method proposed in (Hou et al. Reference Hou, Markert and Strube2013). The script separately measures precision and recall at anchor entity recognition (e.g., whether set_3 is the right coreference chain) and at anchor markable detection (i.e., whether markable_308 is the appropriate markable of set_3). Note that whereas the identification of the anchoring entity is considered correct whenever the right coreference chain is identified, irrespective of the particular anchor markable chosen, the identification of the anchor markable is strict, i.e., it is only considered correct if the same markable as annotated is found.

Finally, the evaluation script for Task 3 computes the Success@N metric proposed by Kolhatkar and Hirst (Reference Kolhatkar and Hirst2014)) and also used by Marasović et al. (2017). success@n is the proportion of instances where the gold answer—the unit label—occurs within a system’s first n choices. (S@1 is standard precision.)

Task 1: Markable detection

One of the important differences between corpora for anaphora / coreference is the definition of mentions (or markables, in this case). In order to compare the difficulty of markable detection in arrau with that of mention extraction OntoNotes, we ran two markable extractors on both corpora: a few versions of a mention extractor based on the Stanford core pipeline, and our own implementation of an lstm architecture for markable detection (see Poesio et al. Reference Poesio, Grishina, Kolhatkar, Moosavi, Roesiger, Roussel, Simonjetz, Uma, Uryupina, Yu and Zinsmeister2018 for details). Two versions of this markable detection were run on the OntoNotes dataset, one optimized for F1, one for recall. The results are shown in Table 12.

Table 12. Markable detection in arrau and OntoNotes

The results suggest that markable extraction in arrau is considerably easier than mention extraction in OntoNotes. This might be due to the differences in markable definition, since singletons and non-referring nps have to be excluded in OntoNotes. But the accuracy gaps might also be a result of the domain differences between OntoNotes and arrau. To test this we tested the Stanford pipeline on the wsj portion of the OntoNotes test set. The highest scores on the wsj portion is obtained by the rule-based version of the pipeline, and is lower (43.1% F1) than that for the entire set. This suggests the difference in performance are due to the more straightforward notion of markable used in arrau.

Task 1

The Stanford CORE deterministic coreference resolver (Lee et al. Reference Lee, Chang, Peirsman, Chambers, Surdeanu and Jurafsky2013) was run on the rst subset of the dataset for Task 1 as a baseline, using the division into training, development and test built in the shared task for this subdomain. The system was run both on gold and on predicted mentions, and evaluated first using both the conll official scorer and the extended coreference scorer ignoring singletons and non-referring markables, then including those.

The first 10 lines of Table 13 show the results by the Stanford Deterministic Coreference Solver when run over gold markables, scored using both the extended coreference scorer and the conll official scorer excluding both singletons (4161 markables) and non-referring markables (1391)–i.e., the same conditions as in the standard conll evaluations. In these conditions, the extended coreference scorer and the conll official scorer obtain the same scores modulo rounding. The following lines in Table 13 show the results when including singletons in the assessment; for this evaluation, the Stanford deterministic coreference resolver was made to output singletons instead of removing them prior to evaluation. When non-referring markables are included as well, the results for referring expressions remain identical, but in addition, the scorer outputs the results on those separately. (The Stanford deterministic coreference resolver does not attempt to identify non-referring markables, hence all values are 0.)

Table 13. Baseline results on Task 1. Gold markables

The first conclusion that can be obtained from this table is that the results achieved by the Stanford resolver on gold markables on this dataset are broadly comparable to the results the system achieved on gold markables (a conll score of 60.7). The second observation is that the system appears quite good at identifying singletons, as its conll score in that case is over ten percentage points higher—in other words, the system is very much penalized when running on the conll dataset.

Table 14 shows the results obtained by the Stanford deterministic coreference resolver when evaluated on predicted markables instead of gold markables. These are the results that are more directly comparable with those obtained by this system in the conll 2011 shared task. We can see a substantial drop in conll score, from 58.3 on predicted markables in the conll 2011 shared task to 43.2 on predicted markables with the Task 1 dataset. Most likely, that indicates that some degree of optimization to the characteristics of conll dataset was carried out in the system even though the system is not trained.

Table 14. Baseline results on Task 1 with predicted mentions, without MIN information

Finally, Table 15 shows the effect of using the MIN information. As can be seen from the table, this results in five extra percentage points.

Table 15. Baseline results on Task 1 with predicted mentions, using MIN information

Task 2

One aspect of anaphoric interpretation for which there were no previous results with arrau is bridging reference. One group from the University of Stuttgart participated in this subtask (Roesiger Reference Roesiger2018). We summarize here the results; for further detail, see the paper.

Roesiger developed two systems, one rule-based, one ML-based. The results obtained by these systems on all three subdomains are summarized in Table 16. The three columns present the result of the two systems at the tasks of (i) attempting to resolve all gold bridging references; (ii) only producing results when the system is reasonably convinced; and (iii) identifying and resolving bridging references. These results appear broadly comparable to those obtained by Hou et al. (Reference Hou, Markert and Strube2013) over the ISNotes corpus as far as the rst and trains domain are concerned, but much lower for the pear domain–although given the small number of bridging references in this domain (354) not too much should be read into this. See Roesiger (Reference Roesiger2018) for some interesting hypotheses regarding the differences between the two corpora.

Table 16. Roesiger’s results on Task 2 for all domains