2. State of the art in the early 1990s

In the early 1990s, NLP – leaving aside speech processing and machine translation (MT) for the moment – was dominated by two main strands. One was the parsing of natural language (usually English) texts using large, hand-coded grammars together with linguistic resources, often hand-coded as well. The other was Information Extraction (IE), essentially template filling, following the model of the Message Understanding Conferences (MUC) (Grishman and Sundheim Reference Grishman and Sundheim1996), in which pre-specified templates (essentially structured database entries) were populated by analysis of (again, usually English) natural language texts. These were generally shallow analysis systems, without any attempt at full syntactic parsing. There was also a third body of work on the modelling of dialogue, using methods drawn from models of interaction and belief, but avoiding the methods of ‘chatbots’, which were simpler systems outside academic research but which often performed well. There were significant bodies of work on Speech Processing and on MT. Related to these work had started to develop on machine learning (ML) or statistical methods (SM), and formal evaluations of NLP systems had started to be undertaken.

This section reviews each of these topics briefly.

No conclusions about the relative importance of these individual themes should be drawn from the length of coverage of these topics in this section. Nor should any conclusion be drawn about the scale of work (in terms, e.g., of papers or research projects) in the mid-1990s or at other times from the length of coverage here. The level of detail here is driven in part by the expertise of the authors, and in part in order to establish a foundation for our review of the current state of the art in later sections. In particular, the relatively lengthy coverage we give of Jelinek’s MT work is because of its role in creating the current focus on data-driven work and SM in the field.

2.6 Concluding remarks on the 1990s

We begin by presenting a tabular analysis of the papers which appeared in Volume 1 of NLE to provide a somewhat anecdotal overview of the state of the art of the field in 1995.

Several papers were excluded from the analysis as either being survey papers on particular topics or not being amenable to this analysis. It is included here principally for completeness and to provide a basis from comparison in the discussion of the current state of the art in Section 3.

In contrast to Table 2, representing the current state of the art of NLE, Table 1 shows that in 1995 almost all papers published by NLE relied on the use of larger linguistic structures or information: sentences, clause, phrases or at the very least words with parts-of-speech assignment were the product of the analysis.

Table 1. Shallow vs Deep techniques of 20 recent NLE articles

^aIncludes tokenisation, stemming, etc.; ^bgeneration.

Table 2. Shallow vs Deep techniques of NLE Volume 1 articles

^aIncludes tokenisation, stemming, etc.; ^bactually phrase parsing.

In conclusion, in 1995, there were the first signs of the ML revolution appearing in NLE, especially in MT. A more scientific model of research, dominated by relatively large shared test data sets and other resources had started to emerge. But much work was still based on small-scale hand-crafted grammars, software and other resources, which was little closer to practical application with real users than the systems had been around 20 and more years previously, in the 1970s.

3. The state of the art 25 years later

In contrast to the state of the art of NLE in the early 1990s, we are now at the end of the second decade of the 21st century and want to pick out three new, or at least much more prominent, aspects of the field that were not so prominent 25 years ago.

First is the widespread, practical use of NLE products by millions, possibly billions, of people worldwide. We note, in particular, the very widespread use of MT in Facebook, Google and elsewhere. Much of the criticism of the ALPAC report in the 1960s (Pierce et al. Reference Pierce, Carroll, Hamp, Hays, Hockett, Oettinger and Perlis1966) remains valid. These systems cannot produce the quality of translations produced by highly skilled human translators on a consistent basis. They often produce gobbledygook, especially for languages where they have been less well trained and for things like fast-moving street slang on social media, and other specialist languages, but few human translators can produce good translations in these circumstances. However, fully automatic MT systems do provide sufficient quality for many practical tasks: to assess the general tone and drift of reaction to a social media post; to help assess the relevance of a patent for human translation and technical analysis; to access the content of a scientific article where the reader’s expert knowledge compensates for translation errors; and so on. Furthermore, there is evidence that the performance of MT systems is sometimes better than available human translations in these practical settings, a circumstance the authors of the ALPAC report foresaw as a possibility, despite their focus on scientific and technical translation in a very different world. However, this possibility probably leads to the redundancy of such human translators, so the evidence is hard to gather (see Läubli and Orrego-Carmona Reference Läubli and Orrego-Carmona2017).

Further, many operational systems in search, sentiment analysis, and a number of other areas rely on technologies which would have been regarded as topics for NLE research 25 years ago, but are now in widespread use (although most if not all remain topics for active research). Probably most notable is the widespread use of named entity recognition and tracking.

Another is chatbot technology. By the mid-2000s, this technology had started to became mainstream via the annual Loebner competition (Floridi et al. Reference Floridi, Taddeo and Turilli2009) and is now the inspiration behind modern applications such as Siri and Alexa. Some commentators believe there will be as many copies of these as there are humans within decades. The vast majority of users simply take these commercial products for granted, whereas those of us who have some concept of the detailed operations and the developments over the last quarter century or so which have led to them are often left amazed by the progress and the successes of our peers.

Second, as in many other fields, the current dominance of ML is noteworthy. Supervised, semi-supervised and unsupervised ML usually over very large corpora has become the dominant paradigm. Some thinkers always expected this to be the case. Sparck Jones (Reference Sparck Jones1986) discussing work she originally undertook in the 1960s is a notable example. Although the terminology used is very different from current usage: unsupervised learning is referred variously by her as ‘the Theory of Clumps’ or ‘automatic classification’, yet the relationship between the ideas is clear (Wilks and Tait Reference Tait2005). As an illustration of this point, all but one paper in NLE 24(5) leant significantly on ML techniques, and even that paper might be argued to make some use of ML.

Recently, Deep Learning has become a widely adopted within the NLP community. Deep Learning is a coverall term for a range of SM and ML methods which are characterised by the use of hidden layers or other stack-like arrangements of individual learning systems. Generally, these hidden layers are thought to capture some sort of generalisation or abstraction not easily captured by simpler ML architectures, and thereby to address sparse data issues amongst other things. Significant successes have been reported on a number of long-standing problems (e.g. in parsing, Andor et al. Reference Andor, Alberti, Weiss, Severyn, Presta, Ganchev, Petrov and Collins2016), but there remain many problems to be overcome before the true impact of Deep Learning on NLE can be assessed (Manning Reference Manning2015; Cho Reference Cho and Mitkov2018).

Third, the widespread use of word n-gram, skip-gram and related models at the expense of deeper structures is striking. Much work in the early 1990s was based on the assumption that successful applications required accurate and robust analysis based on the grammatical structures in natural language utterances. Jelinek’s insight – which was Colby’s for conversational analysis two decades earlier – that deeper structural analysis might not be essential for useful MT has proved applicable to a range of other applications. Though Jelinek, as we noted, eventually accepted that his MT systems needed to learn grammatical and lexical information beyond word-string-to-word-string matching. It seems that learned pattern matching across strings of words (or morphologically reduced word lemmas) is adequate for many practical purposes. A similar movement has taken place in modelling conversation: whereas chatbot systems were for decades excluded from NLP research as superficial and skimming only surface phenomena in input, they have now developed to a point of sophistication where they are both in widespread commercial use – in web-based help systems – and are being melded to ‘deeper’ analysis systems that attempt to understand more fully what is being said to them.

To illustrate this move from attempting to extract deeper structures from text to the use of shallower structures, we analysed 20 recent articles published in NLE. The survey was undertaken in late 2018. Twenty papers were selected from NLE Volumes 24 (up to Issue 5) plus 23(6), and to make up the numbers, the first paper from 23(5). Volume 24(3) was omitted from the analysis. It was a special issue on Language for Images, and the Special Issue selection policy meant it was an even less representative sample of the field than regular issues. Also excluded were survey articles (which often cover older as well as more recent work); papers on text generation and papers which focus on specific topics, like tokenisation methods rather than end-to-end applications.

So from this limited survey, 75% of papers published in the relevant issues of NLE use bag of words, n-grams or skip grams. About 50% of the papers use those methods alone. About 35% use chunking (into phrases, clauses or in some cases less linguistically motivated units) or part-of-speech tagging. About 30% of the published papers aim to produce structures at the clause or sentence level. About 35% of the papers use a combination of techniques.

A striking feature is the variety of languages covered in these recent papers. They include Turkish, Arabic, Yoruba, Chinese and a variety of Romance languages including less widely spoken ones like Galician. In contrast, in NLE Volume 1, all but three of the papers concerned English (exclusively or very predominantly). The exceptions addressed French, Swedish (both in a paper substantially focused on English) Russian and Hebrew. However, note that many fewer papers were published in Volume 1 than in Volume 24: 14 in Volume 1 as opposed to 32 in Volume 24.

One might speculate about why these techniques based on bags of words, n-grams or skip grams are proving so popular in practically orientated work. One possibility is that much of the meaning of an utterance is carried in the lexical elements, and that linguists of an earlier era overestimated the extent to which syntax, structural semantics and pragmatics contributed to the use of language in practical settings. Another possibility is that language is inherently redundant, and in practice, information in the lexical channel reinforces and supplements information in the deeper structural channels (e.g. relying on grammatical relations), so the structure as opposed to the words themselves can often be ignored. A third is that current applications of NLE are biased towards those in which these n-gram-based techniques work well. A fourth is that the development of techniques which utilise the shallow technologies has been stimulated by the structure of the field at the expense of other directions.

There is no doubt that open competitions, based on a shared task, with fairly clear evaluation metrics, shared data and often other shared resources, have advanced the field; the doubts some have are mainly as to whether the isolated, testable tasks add together to give advance in more holistic applications like MT or conversational analysis. European research funders have tended to shy away from such competitions, seeing them as a waste of resources. But even after over 30 years, the model pioneered by MUC continues to have an important influence on research in the field.

In particular, the shared tasks represented by MUC and its successors have led to the easy availability of freely available data sets which are essential for the development of ML-based systems.

However, as well as the limitations of isolated, testable tasks, as opposed to end-to-end applications, another doubt about the open competitions is whether the metrics used to evaluate success have been biasing technological development in ways which inhibit innovation. For example, the BLEU metric (Papenini, Rouskos, Ward and Whu Reference Papenini, Rouskos, Ward and Whu2002), widely used in MT evaluation and elsewhere, operates by comparing the output of a test system with a gold standard example text on an n-gram by n-gram basis. One has to question whether continued widespread use of BLEU as a measure of MT system quality inhibits work which may translate using paraphrase or radical rewording, compared to work which translates word by word and n-gram by n-gram, especially in the light of doubts about its correlation with human assessment of MT quality (Callison-Burch, Osborne and Koehn Reference Callison-Burch, Osborne and Koehn2006).

Nonetheless, the probability is that the move away from deeper analysis has been driven more by the nature of language used in practical day-to-day settings than any other factor.

4. So what has changed?

As noted in the previous section, the widespread use of word (or lemma) n-grams and skip grams is still a surprise to many established researchers. Although this trend had started to emerge in the 1980s outside the information retrieval (IR) community, it was not widely adopted by the NLE community until the first decade of the 21st century, stimulated in part by interactions and crossovers between work in the IR and NLP communities.

In the 1960s and 1970s, there was an ongoing debate within IR as to whether document retrieval systems using statistics derived from word usage in documents and cross-document relationships (like citations) could match or exceed the performance of knowledge-based system using controlled vocabularies, taxonomic indexing schemes and other hand-crafted knowledge sources, familiar in related but rather different forms to the library science and NLP communities. By the 1990s, it had been convincingly demonstrated that practical IR systems (like internet search engines) could be constructed using primarily word-based statistics, and this evidence had started to excite interest in a number of areas, especially ones of interest to both the IR and NL communities, like MT and word sense disambiguation.

Indeed, it was often the case that systems based on word grams or skip grams were seen initially as baseline systems against which ‘smarter’ systems could be compared. It just proved harder to beat the baseline than the authors expected. The direct influence on MT developments come from the automatic speech recognition community through Jelinek, as we noted, but indirect influence on NLP and AI came at the same time from the IR community through researchers like Sparck Jones and others.

The field has become significantly more scientific, in the sense that it is much more dominated by evaluation regimes in which a number of systems are measured against some agreed benchmark, and that advance was driven almost entirely by the US funding agencies. This has encouraged better formulated hypotheses about ways to improve systems and performance and more-or-less objective assessment of whether performance has in fact been improved. As noted previously, one can have various doubts about the effect of this change to a more scientific methodology has had in the development of overall system performance and for some specific technologies. However, published papers in NLE are now, at the time of writing in 2018, much more likely to report facts expressed as proof or disproof of hypothesis which can be reproduced, than was the case in the mid-1990s.

Formal evaluation campaigns have on the whole improved dialogue in the field. If one goes back beyond the genesis of NLE to the 1970s when both the current authors were already in the field, much published material went little beyond what might be labelled feasibility studies (see, e.g., Schank and Colby Reference Schank and Colby1973). This style, and the poorly engineered nature of the systems produced, combined with the limitations of the then available computer power, and the poor availability of large-scale data, inhibited progress in the field for many years.

Today, the widespread availability of open source tools of a wide variety of kinds – a product very largely of the insistence on group cooperation by the US funding agencies – makes it possible to rapidly bolt together end-to-end applications and to produce specific variations of systems and to observe the effect on performance of changing them.

Of course, true reproducibility remains an elusive goal as in many other fields. But great progress has been made over the last 25 years.

5. What has not progressed as wemight have expected?

In the 1990s, there was a general expectation that much future language work would be based on widespread use of pre-existing dictionaries. Pioneering work in this area had been undertaken by Michiels in Liege in the 1980s (Michiels Reference Michiels and Jones1983) and this was followed by Boguraev, Carroll, Briscoe, Carter and Grover (Reference Boguraev, Carroll, Briscoe, Carter and Grover1987), Boguraev and Briscoe (Reference Boguraev and Briscoe1989), Wilks, Slator and Guthrie (Reference Wilks, Slator and Guthrie1996), and many others. Much of the work was based on the Longmans Dictionary of Contemporary English (Proctor Reference Proctor1978), although a number of other dictionaries were used. There was an untested assumption that dictionaries were where meaning was located so let’s go and use them. So it was widely expected that the future of the field lay with the use and development of these dictionaries, and semantic data extracted from them. However, this has proved not to be the case.

In practice, the dominant lexical resource has been WordnetFootnote ^b and the related projects for other languages (Fellbaum and Miller Reference Fellbaum and Miller1998; Bond and Paik Reference Bond and Paik2012), which is paradoxical given that Wordnet was designed as a psychological rather than a computational tool.

One reason for the eventual dominance of Wordnet was the legal problems created by the commercial publishers who felt they needed to control the intellectual property embodied in the machine-readable version of their dictionaries, and the consequent problems with licensing and reuse this created. Wordnet specifically allows such usage.

Another reason commercial dictionaries were not as widely used in NLE as might have been expected was the somewhat divergent interests of the natural language engineers and the lexicographers when the dictionary needed to be updated and extended.

A related area in which progress has not been made in the direction generally expected in the early 1990s is the use of existing reference works. It had been expected that NLE systems in the 21st century would depend significantly on what one would now call fact bases mined from standard references like the Encyclopaedia Britannica. This has not happened in part because, just as with dictionaries, such publications have been supplanted to a large extent in general use by internet resources like Wikipedia. Resources that can be updated, verified and used by anyone have won out over closely controlled ‘walled-in’ resources. Despite this, our ability to ingest these open human usable resources into a computer system and use them effectively for purposes like question answering has progressed to a much more limited extent than many expected in the mid-1990s and it remains an active area of research (see, e.g., Choi, Seo, Chen, Jia and Berant Reference Choi, Seo, Chen, Jia and Berant2018). However, Google’s Knowledge GraphsFootnote ^c, derived from Wikipedia, have actually proved a useful source of real-world knowledge for a range of applications including their commercial dialogue system SIRI.

It has proved much more challenging to deal with notions of truth or veracity in practical language than most language engineers envisaged 25 years ago. It was simply not envisaged in the mid-1990s that determining whether a newspaper article was reporting facts or merely repeating fake news would became a major future task of NLE systems.

Finally, there has been much less progress in some important and pervasive phenomena of real language use than one might have expected. For example, perhaps most notably, there has been less progress with metaphor processing than some would have hoped for 25 years ago. After a gap of several years in 2011, the US funding agency iARPAFootnote ^d did restart major funding of metaphor research at a number of sites and some important work emerged in a number of languages (English, Spanish, Russian and Farsi). Cross linguistic comparisons were made and benchmarked and some useful progress was demonstrated. However, no attempt was made to link the work directly to MT or other practical language processing tasks, so metaphor remains a pervasive and central unsolved issue of MT and language engineering generally.