2. Related studies

Automatic language identification is a well-explored problem in literature. Studies have focused on different level of granularities – document-level language identification (Cavnar and Trenkle (Reference Cavnar and Trenkle1994); Yang and Liang (Reference Yang and Liang2010)), sentence-level language identification (Carter et al. (Reference Carter, Weerkamp and Tsagkias2013); Reference Wang, Bojja and KannanWang et al. (2015)) and word-level language identification (Das and Gamback (Reference Das and Gamback2014); Reference Rijhwani, Sequiera, Choudhury, Bali and MaddilaRijhwani et al. (2017)). While most early language identification studies focus on the document-level language identification of regular text like news articles, historical documents, etc., over the last few years, the attention has shifted towards short noisy text in different social media platforms like discussion forums (Reference Abainia, Ouamour and SayoudAbainia et al. (2016)), Facebook (Reference Sarma, Sanasam and GoswamiSarma et al. (2019)), Twitter (Reference Zubiaga, San Vicente, Gamallo, Pichel, Alegria, Aranberri, Ezeiza and FresnoZubiaga et al. (2016)), etc. While sentence-level language identification has also been addressed for social media text, with the ever-growing popularity of social media platforms, word-level language identification has, in particular, garnered much attention and is an active area of research in the current times. The discussion in this section, therefore, focuses on word-level language identification.

Based on the resources used, word-level language identification approaches used can broadly be categorised into (i) classification using dictionaries, (ii) classification using code-mixed annotated data, (iii) classification using monolingual corpora and (iv) classification using transliteration.

A dictionary-based approach for language identification has been used in Barman et al. (Reference Barman, Das, Wagner and Foster2014) where the language of the word is classified based on its frequency of occurrence in multiple language dictionaries. Dictionary look-ups for language identification have also been used in Nguyen and Doğruöz (Reference Nguyen and Doğruöz2013). However, considering the variations of word forms in social media content and unavailability of dictionaries for transliterated text, dictionary look-up is not an efficient approach for language identification in terms of coverage as well as performance.

The use of code-mixed data annotated at the word level with the corresponding language information to train word-level classifiers is one of the most commonly used approaches. Classification-based approaches like SVM, Naive Bayes, and sequence classification-based approaches like CRF have been used in Barman et al. (Reference Barman, Das, Wagner and Foster2014) and Gundapu et al. (2018) to train word-level classifiers. CRF-based approaches have also been used in Nguyen and Doğruöz (Reference Nguyen and Doğruöz2013), (Xia (Reference Xia2016), Sikdar and Gambäck (Reference Sikdar and Gambäck2016) and Chittaranjan et al. (Reference Chittaranjan, Vyas, Bali and Choudhury2014). A CRF classifier combined with post-processing heuristics has been used in Banerjee et al. (Reference Banerjee, Kuila, Roy, Naskar, Rosso and Bandyopadhyay2014). In addition to addressing word language identification, authors in Das and Gamback (Reference Das and Gamback2014) also introduce a metric called Code-Mixing Index that shows the level of mixing between different languages in a given text. Authors in Vyas et al. (Reference Vyas, Gella, Sharma, Bali and Choudhury2014) also use a CRF-based classifier for word-level language identification, where they also address other tasks like back transliteration, normalisation, and part-of-speech tagging. Word-level language identification and prediction of code-switching points have also been addressed in Piergallini et al. (Reference Piergallini, Shirvani, Gautam and Chouikha2016). Sequence classification using RNN has been used in Samih et al. (Reference Samih, Maharjan, Attia, Kallmeyer and Solorio2016). A multichannel convolutional neural network (CNN) combined with a bidirectional long short-term memory (BiLSTM)-CRF module has been used in Mandal and Singh (Reference Mandal and Singh2018). Sequence to sequence models has also been used in Jurgens et al. (Reference Jurgens, Tsvetkov and Jurafsky2017). Instead of sequence models, authors in Zhang et al. (Reference Zhang, Riesa, Gillick, Bakalov, Baldridge and Weiss2018) propose a two-stage model wherein in the first stage, a distribution over the languages for a given word is predicted, followed by a decoder in the second stage which along with the language distribution predicted in the first stage also takes into consideration the global constraints over the entire sentence.

A drawback of using word-level annotated code-mixed data for training classification models is that they are expensive to obtain. Considering variations in word forms in social media text owing to factors like code-mixing, phonetic typing and spelling variations, in the absence of large-scale annotated data, it may not be possible to capture word and language variations in a highly multilingual environment. Therefore, in an effort to reduce the dependence on annotated data, classification using monolingual corpora in the respective languages has been proposed. Authors in King and Abney (Reference King and Abney2013) use a collection of monolingual texts and employ weakly supervised and semi-supervised methods for word-level language identification in multilingual documents. Authors in Rijhwani et al. (Reference Rijhwani, Sequiera, Choudhury, Bali and Maddila2017) also use unsupervised models using monolingual corpora and HMM for word-level language identification. Instead of using real code-mixed datasets, authors in Gella et al. (Reference Gella, Bali and Choudhury2014) also create a synthetic code-mixed language identification dataset from a collection of monolingual text. However, a drawback associated with using monolingual corpus for word language classification is that it is not easy to obtain a clean monolingual corpus for transliterated text. Considering that a large section of Indian social media conversations is in transliterated form, these approaches may not be feasible in many scenarios.

Most of the studies discussed above make use of only word-level information for language identification. However, character-level information has also been found useful. Capturing character- level information is associated with two advantages. First, it can be useful in handling unseen word variations. Second, it is most likely that character sequences of different languages will have different structural properties which can be used to disambiguate languages of words. Character embeddings and subword unit information have been used to capture this information in Jaech et al. (Reference Jaech, Mulcaire, Hathi, Ostendorf and Smith2016) and Mave et al. (Reference Mave, Maharjan and Solorio2018). In a slightly different approach, authors in Singh et al. (Reference Singh, Sen and Kumaraguru2018) use a transliteration model to transliterate romanised script to Devnagiri script. They use RNNs to train a character language model for each language. Given an annotated corpus, the output of the language models is combined with other features to train a word-level language classifier. In addition to character n-gram information, phonetic information has also been used for word language identification in Das et al. (Reference Das, Mandal and Das2019).

Apart from individual studies on word-level language identification, several survey articles (Jauhiainen et al. (Reference Jauhiainen, Lui, Zampieri, Baldwin and LindÉn2019) and Reference Garg, Gupta and JindalGarg et al. (2014)) dedicated to language identification also exist in literature. A number of shared tasks (Solorio et al. (Reference Solorio, Blair, Maharjan, Bethard, Diab, Ghoneim, Hawwari, AlGhamdi, Hirschberg and Chang2014) and Reference Molina, Rey-Villamizar, Solorio, AlGhamdi, Ghoneim, Hawwari and DiabMolina et al. (2016)) have also been organised to address word-level language identification problems. Apart from word-level language identification, other closely related studies include analysing different aspects of code-switched data (Reference Rudra, Sharma, Bali, Choudhury and GangulyRudra et al. (2019)), prediction of code-switching points from multilingual communications (Reference Papalexakis, Nguyen and DoğruözPapalexakis et al. (2014)), predicting foreign language usage in social media posts (Reference Volkova, Ranshous and PhillipsVolkova et al. (2018)), language informed modelling of code-mixed data (Reference Chandu, Manzini, Singh and BlackChandu et al. (2018)), predicting the presence of matrix language (Reference Bullock, GuzmÑn, Serigos, Sharath and ToribioBullock et al. (2018)) and predicting likelihood of word borrowing in social media (Reference Patro, Samanta, Singh, Basu, Mukherjee, Choudhury and MukherjeePatro et al. (2017)). Recently subword-level language identification has also been addressed in literature. Although it is beyond the scope of this study, some of the recent studies around intra-word code-switching involve the work in Mager et al. (Reference Mager, Cetinoglu and Kann2019) and Nguyen and Cornips (Reference Nguyen and Cornips2016).

While most of the existing studies report quite satisfactory performance, they have certain disadvantages associated that have also been pointed out in the above discussion. First, developing word-level annotated data is expensive both in terms of time and effort and subject to human error. Second, most methods that avoid the use of word-level annotated data use other resources like dictionaries, monolingual corpora, transliteration tools, etc. Therefore, these approaches are only applicable to languages that are well equipped with these resources. Also, very often, these resources are available in the native scripts of the languages. The use of transliterated text for posting content on social media makes these resources unusable. This paper, therefore, focuses on presenting a method for word-level language identification that uses minimal resources such that it can quickly be adapted for any language. This also makes the proposed approach particularly suitable for low-resource languages. The proposed approach discussed in Section 3.3 is motivated by the observations in Sarma et al. (Reference Sarma, Singh and Goswami2018).

3. Proposed methodology

This section describes the proposed model to address word-level language identification over code-mixed social media text in a multilingual environment. The objective is to build a dynamic switching mechanism between two classifiers that consider the word in isolation (described as global semantic similarity) and the word with its contextual information (described as local contextual similarity) (Reference Sarma, Singh and GoswamiSarma et al. (2018)). Before discussing the details of the proposed methods, we briefly discuss the nature of the dataset used in this study to help the reader to understand the proposed method with ease. Word-level language identification has considered datasets annotated in two different levels: annotated at the word level and annotated at the sentence level. Word-level annotation is expensive compared to sentence-level annotation. Therefore, word-level language identification using word-level annotated data may not be suitable for low-resource languages. Like in our earlier study (Reference Sarma, Singh and GoswamiSarma et al. (2018)), this paper considers the same sentence-level annotated dataset. While using sentence-level annotations, the vocabulary set may be divided into two disjoint subsets resolved set $\mathcal{R}$ and unresolved set $\mathcal{U}$ . The resolved set $\mathcal{R}$ are those words that occur only in sentences of a particular language. Therefore, they can be assumed to belong to the same language as the language of the sentences in which they are occurring. The unresolved set $\mathcal{U}$ , on the other hand, are those words that occur in sentences of multiple languages. Therefore, the language of these words cannot be confirmed using the sentence-level language information alone. So, the task of language identification can be considered as the task of identifying the underlying languages of the words in the unresolved set $\mathcal{U}$ using the words in the resolved set $\mathcal{R}$ . The words in the resolved set $\mathcal{R}$ are used as training samples in building classifiers. The classifiers proposed in Sarma et al. (Reference Sarma, Singh and Goswami2018), that is, considering word in isolation and considering contextual information of words are considered as the baseline classifiers in this study. Details of these classifiers are briefly discussed below.

3.2. Word-level language classification using contextual information (local contextual similarity)

In a multilingual environment, context (preceding words and succeeding words) often helps in identifying the language of a target word, that is, the language of the context and target words may often be the same. Contextual information is particularly important when the languages under consideration contain shared vocabulary, whether due to inherent similarities between languages or due to other phenomena such as phonetic typing or creative writing. Therefore, in the local context similarity-based classifier, context information is also considered while classifying the language of the target words. The target word is represented by a vector which is the concatenation of the vector representations of the target word and words in the context in order of their occurrences.

Like in global similarity, vector representations are obtained by training word embedding models over a large corpus of code-mixed text. Classification frameworks including CNN, BiLSTM, SVM and LR are built over the concatenated vector representations and trained using words in $\mathcal{R}$ as training samples. The detailed experimental set-up of these classifiers is discussed in Section 4.1. This approach is called the local contextual similarity-based approach because the classification is based on the target word as well as the local context in which the target word is occurring.

Both the above classification approaches are associated with respective advantages and disadvantages. The global semantic similarity-based approach, being based only on the embedding vector of the target word, all occurrences of the same word are assigned the same language label. While this approach works well when words are borrowed or embedded into sentences of other languages, this approach fails when the word is valid in multiple languages. On the other hand, considering the contextual information helps in correctly identifying languages of words that are valid in multiple languages. In a noisy multilingual environment, it is desirable that the model learns to prioritise between the words in isolation and the contextual information depending upon a given input. The proposed model discussed below attempts to achieve this by using a feed-forward neural network combined with the output from the individual classifiers which we refer to as the baseline classifiers with respect to the proposed framework.

Figure 1. Proposed word-level classifier combining output word-level classifiers built using word in isolation (C1) and using contextual information (C2).

3.3. Proposed classification framework

The proposed approach uses a feed-forward network to dynamically switch decisions between the global semantic similarity and local contextual similarity for any given input. Figure 1 shows the proposed model architecture. Let $\mathcal{C}1$ represents the global semantic similarity-based classifier and $\mathcal{C}2$ represents the local contextual similarity-based classifier. The output from $\mathcal{C}1$ and $\mathcal{C}2$ are vectors representing the probability distribution over the languages for the target word. Let these probability distributions be represented by $\textbf{o}_1$ and $\textbf{o}_2$ , respectively. Let $\textbf{x}$ be the feature vector representing the target word. This $\textbf{x}$ could be the textual content of the input. It can also be other non-text-based features representing the target word and its context. The different representations of $\textbf{x}$ are discussed in detail in Section 3.4. The input vector $\textbf{x}$ is passed to a fully connected feed-forward neural network with one hidden layer. Let $W_1$ be the parameter matrix between the input layer and the hidden layer and $W_2$ be the parameter matrix between the hidden layer and the switch layer. The output vector of the switch layer y is normalised with a sigmoid operation. The expressions for the forward pass are shown below:

(1) \begin{equation}\textbf{h} = tanh(W_1^T\textbf{x}+ \textbf{b}_1)\end{equation}

(2) \begin{equation}\textbf{y} = \sigma (W_2^T\textbf{h}+ \textbf{b}_2)\end{equation}

The vector $\textbf{y}$ in Equation (2) acts as the switch, the purpose of which is to switch the output of the classifier between $\textbf{o}_1$ and $\textbf{o}_2$ based on the given input x. This is shown in Equation (3) where $\odot$ represents element-wise product:

(3) \begin{equation} \textbf{z} = softmax((1-\textbf{y})\odot \textbf{o}_1 + \textbf{y}\odot\textbf{o}_2)\end{equation}

If $|\textbf{y}|$ is high, that is, the values in $\textbf{y}$ are close to 1, the first component of the sum in Equation (3) reduces to 0 and the final output of the classifier is biased towards $\textbf{o}_2$ . Similarly, if $|\textbf{y}|$ is low, that is, the values in $\textbf{y}$ are close to 0, the final output of the classifier is biased towards $\textbf{o}_1$ . Therefore, the objective of the proposed framework is to train the network in such a manner that the values in $\textbf{y}$ range in between 0 and 1 according to whether the final desired output should be biased towards $\textbf{o}_1$ or $\textbf{o}_2$ . The proposed model consists of three components – the two baseline components and the feed-forward network that acts as the switching mechanism between the baseline components. Instead of training the three components end-to-end, each of the components is trained separately, and the outputs from the pre-trained baseline models are selected based on the output of the proposed switch network, that is, y. This enables the proposed framework to be used with any baseline component. For learning the parameters of the switch network, the representation vector of the input text x is considered as the only input to the network. The output vectors $\textbf{o}_1$ and $\textbf{o}_2$ of the component classifiers are used while estimating the model output z as defined in Equation (3). Let $\tilde{\textbf{z}}$ be the ground truth vector (one hot vector) associated with the input $\textbf{x}$ . Cross-entropy between $\tilde{\textbf{z}}$ and z has been used as the loss function for learning the model parameters $W_1$ and $W_2$ as defined below:

(4) \begin{equation} E = - \sum_{i=1}^C\tilde{z}_{i}log(z_{i}) \end{equation}

where C is the number of classes, that is, the number of languages. The training procedure for the component classifiers are discussed in Section 4.

3.4. Input representation

The goal of the proposed framework is to dynamically switch between the baseline classification outcomes depending on the given input. This study proposes different features that can be used to represent a given input text to train the proposed framework to make an effective switching decision. These features are discussed below. These features are generated as a part of the feature selection module shown in Figure 1. Figure 2 describes in detail the processing of the input text to generate input vectors of different components. The generated input vector x is fed as input to the switch network. The following subsections describe the generation of the vector x which is used as input to the switch network.

Figure 2. Generation of feature vectors from input text using different components.

3.4.1. Content features

The objective of the content features is to use textual information to identify the language of a word. The textual information considered in this case comprises the target word and the surrounding words. Therefore, the input vector x, while considering content features, is the same as that of the context-based component classifier, that is, the target word along with its context (preceding and following words) is fed as input to the proposed classifier. The intuition is that the word, along with its context, will be able to capture switching characteristics between the two classifiers. Pre-trained word embeddings obtained by training a word embedding (skipgram) model over a large code-mixed corpus are used to represent the input. While feeding as input to the classifier, the word embeddings of the target word and the words in the context are concatenated in the order of their occurrence (Figure 2(a)).

3.4.2. Neighbourhood based features

The content features make use of the textual information directly for identifying the language of a target word. However, the noise present in the social media data, like irregular and ambiguous word forms, leads to limitations in the performance of the text-based features. Therefore, the goal of the neighbourhood-based features is to extract non-textual information from the data that can overcome the limitations of the text-based features.

The neighbourhood-based features proposed in this study attempts to encode the semantic relation of the words with other words in the corpus into a fixed-sized vector. Therefore, the neighbourhood vector is a fixed-sized vector representing structured information about the input in contrast to the content features that are represented by the word embeddings that accommodate detail semantic information about the word and its context.

This study proposes two neighbourhood-based features as discussed below. The intuition is that the neighbourhood characteristics of a word can be an important indicator of whether the word is a borrowed word or is a word that is valid in multiple languages. These approaches are discussed in detail below:

• • Local neighbourhood: The local neighbourhood-based information tries to encode the semantic information between the target word and its local context (preceding and following words). The intuition is that the semantic distance of the target word with its context can be a good indicator of whether the target word and the words in the context belong to the same language. In other words, if a target word belongs to the same language as its context, its distance from other words in the context will be lesser compared to when the target word belongs to a different language and is embedded or borrowed into the current context. Accordingly, the classification output can be biased towards the prediction obtained considering the contextual information or prediction obtained considering the word in isolation. This approach is therefore called the local neighbourhood-based approach because it considers the local neighbourhood of the target word.

The local neighbourhood is represented by a vector comprising the minimum, maximum and average distance of the target word from the words in the context. The distance is given by the cosine similarity between the embedding vectors of the words. This is shown in Figure 2(b). The intuition is that a word will be semantically more similar to words in the same language than words in other languages. Therefore, when a word is borrowed or embedded into a sentence of another language, its distance from the context would be higher compared to instances when the target word and the context words are in the same languages.

Let $\textbf{t}_i$ be the embedding vector of the $i^{th}$ word in a sentence. Then, the $t_{i-j}$ and $t_{i+j}$ denote its $j^{th}$ left and right neighbours, respectively. The input vector x to the switch network is defined by a vector with three elements as follows :

(5) \begin{equation} x_0 = \min(\{ s |s = cosine(\textbf{t}_i,\textbf{t}_{i+j}))~ \forall_j, -k \leq j \leq k, j \neq i\}) \end{equation}

(6) \begin{equation} x_1 = \max(\{ s |s = cosine(\textbf{t}_i,\textbf{t}_{i+j}))~ \forall_j, -k \leq j \leq k, j \neq i\}) \end{equation}

(7) \begin{equation} x_2 =\frac{1}{2k} \sum_{\substack{j=-k\\j\neq i}}^k( cosine(\textbf{t}_i,\textbf{t}_{i+j})) \end{equation}

where k denotes the number of words in the left and right neighbourhood.

• • Global neighbourhood: The global neighbourhood tries to encode the semantic information of the target word with respect to its neighbours in the global embedding space. The global neighbourhood of the target word is represented by a vector that represents the k-nearest neighbours of the target word in the global embedding space. The intuition is that if a word is valid across multiple languages, the neighbourhood of this word in the global embedding space will consist of words of different languages. On the other hand, if a word is valid only in a single language and its occurrences in sentences of other languages are the result of embedding/borrowing, the global embedding space of such words will comprise all words of the same language. Therefore, while determining the language of a target word, if the neighbourhood of the target word is dominated by words in a single language, the output from the classifier considering the word in isolation should be given preference. On the other hand, if the neighbourhood is composed of words from different languages, then while determining the language of the word, the output from the classifier considering contextual information should be given priority. The global neighbourhood vector x of a word t is the distribution of the languages in its neighbourhood. Let $\mathcal{V}_k$ be the set of top k most similar words to the target word t and let $\mathcal{L}$ denote the set of languages under consideration. Then the elements of the feature vector x that denotes the global neighbourhood of the target word t consists of the number of words in each language in the set $\mathcal{V}_k$ . For example if $|\mathcal{L}|$ =4, then the feature vector representing the global neighbourhood of the term t is given by x = ( $x_1$ , $x_2$ , $x_3$ , $x_4$ ) where $x_i, i=1 ... 4$ , is the number of words in $\mathcal{V}_k$ that belong to language $L_i$ . The feature vector x is then used as input to the switch network. As discussed in Section 4, the switch network is built considering only words in the resolved set $\mathcal{R}$ .

3.4.3. Combination of content and neighbourhood-based features

The aim is to represent an input by a combination of its content features and neighbourhood features. The idea is to explore the advantages of combining the detailed semantic information represented by the content features with the fixed-sized structural information represented by the local and global neighbourhood. We explore two ways of combining the features. In the first approach, all three sets of features, that is, the text features, the local neighbourhood features and the global neighbourhood features are merged in the hidden layer. In the second approach, the above sets of features are merged in the hidden layer following a selective feature dropout mechanism proposed in Zhang et al. (Reference Zhang, Riesa, Gillick, Bakalov, Baldridge and Weiss2018). In the original paper (Reference Zhang, Riesa, Gillick, Bakalov, Baldridge and WeissZhang et al. (2018)), selective feature dropout is implemented by randomly setting the feature group values to zero with 50% probability. However, in the current study, the feature dropout strategy is implemented by alternatively setting a very high dropout (90%) between each feature group and the hidden layer during the training iterations.

4. Dataset and experimental set-up

This study uses the same dataset reported in Sarma et al. (Reference Sarma, Singh and Goswami2018). This dataset is a collection of code-mixed transliterated Facebook comments collected from a highly multilingual environment using Facebook Graph API. A total of 409,168 user messages have been collected, of which 28,986 messages have been manually annotated with the corresponding language information in the sentence level. Details of this dataset are shown in Table 1. It consists of four languages: Assamese, Bengali, Hindi and English. While identifying Facebook channels for generating the dataset, the following considerations have been taken into account.

• • The dataset is collected from a highly multilingual environment where there are chances of users participating in multilingual conversations

• • Languages under consideration are such that there is overlapping vocabulary between different languages

• • All languages share the same script (Roman script)

Table 1. Description of the dataset annotated at the sentence level

Two annotators familiar with all the languages are deployed for annotation. The same set of words are annotated by both annotators. In the end, samples that receive the same annotation by both annotators are retained. The following guidelines have been followed while annotating the messages:

• • Messages that contain all words of the same language or that contain words borrowed or embedded from other languages without affecting the underlying language sense are considered monolingual sentences and are assigned the corresponding language label.

• • Messages that comprise two or more sentences/fragments in different languages are considered multilingual and are not considered for annotation. For example, the sentence Movie acchi thi [Movie was good], is considered a monolingual sentence (Hindi in this case) and the sentence Story was good but movie bohot lambi thi [Story was good but movie was very long] is considered a multilingual sentence.

• • Messages that do not belong to either of the four languages are labelled as unknown and are not considered in the current study

Using the sentence-level dataset, as discussed in Section 3.3, Resolved and Unresolved word sets are created. Table 2 shows the number of words in resolved and unresolved sets. The words in the resolved set are used to build the baseline component classifiers.

Table 2. Description of word-level annotations obtained using sentence-level annotations

Table 3. Description of the dataset used to train and evaluate the proposed switch network

Further, to train the switch network, a small number of words from the resolved set $\mathcal{R}$ are chosen for manual annotation. The goal for manual annotation is to correct any wrong labels in $\mathcal{R}$ that may have occurred while propagating the language information from sentence level to word level. Furthermore, if both the baseline components result in the same predicted label, then the decision of the switch network is not crucial. The prediction from any one of the baseline components may be chosen as the final prediction. However, if the predictions from both the baseline components are different, then the switch network must learn to make the correct choice between the baseline components based on the input characteristics. Therefore, while choosing training samples for the switch network, only samples where there is a disagreement between the outcomes of the two classifiers are chosen. An evaluation dataset annotated at the word level is created to evaluate the performance of the proposed framework. The evaluation dataset is described in Table 3. It is important to note that considering that the challenge is to disambiguate languages of words in the unresolved set $\mathcal{U}$ , the evaluation set only consists of words from $\mathcal{U}$ . The annotation guidelines for creating the word-level annotated dataset are as follows:

• • Words are annotated according to the context. The same word in different context may belong to different languages. For example, the word ache in the phrases thik ache and stomach ache are annotated as Bengali and English, respectively.

• • Named entities and universal expressions cannot be assigned any language. Instead they are assigned two different class labels ne and amb, respectively. These words are currently excluded from the evaluation set.