3.1 Low-dimensionality statistical embedding

LDSE is the generalisation of the LDR method (Rangel et al. Reference Rangel, Rosso and Franco-Salvador2016b) where skewness, kurtosis and moments (Bowman and Shenton Reference Bowman and Shenton1985) are used to measure the distribution of weights for each class. The intuition behind both methods is that, in an annotated corpus, the probability of each term to belong to each of the classes should be different. If we use weights to represent such probability, we may assume that the distribution of weights for a given document should be closer to the weights of its corresponding class.

We obtain the tf-idf (Salton and Buckley Reference Salton and Buckley1988) matrix Equation (1) for the terms of the documents D in the training set. Each row represents a document $d_{i}$ and each column represents a term $t_{j}$ belonging to the vocabulary T. Each $w_{ij}$ represents the tf-idf weight for the term $t_{j}$ in the document $d_{i}$ . The last column $\delta(d_i)$ represents the assigned class c from the set of all classes C to the document $d_{i}$ :

(1) \begin{equation}\begin{bmatrix} w_{11}&w_{12}&...&w_{1m} & \delta(d_1) \\ w_{21}&w_{22}&...&w_{2m} & \delta(d_2)\\ ...&...&...&...& \\ w_{n1}&w_{n2}&...&w_{nm} & \delta(d_n)\\\end{bmatrix}\end{equation}

As formalised in Equation (2), for each term t and each class c, we define the term weight W(t, c) as the ratio between the weights of the documents belonging to the class c and the sum of all weights for that term.

(2) \begin{equation}W(t,c) = \frac{\sum_{d\in{D}/c=\delta(d)}w_{dt}}{\sum_{d\in{D}}w_{dt}}, \forall{d\in{D}, c\in{C}}\end{equation}

A document d is represented as shown in Equation (3), with as many dimensions as the number of terms in the document multiplied by the number of classes:

(3) \begin{equation}d = W(t,c) =\\ \{W(t_1,c_1), W(t_2,c_1), ..., W(t_t, c_1),\\ W(t_1,c_2), W(t_2,c_2), ..., W(t_t, c_2),\\ ...,\\ W(t_1,c_c), W(t_2,c_c), ..., W(t_t, c_c)\}\\ \sim \forall{\ t\in{T}, c\in{C}}\end{equation}

In order to reduce the dimensionality of the representation, we obtain descriptive statistics from the previous distribution of weights. Heitele (Reference Heitele1975) pointed out three fundamental concepts regarding random variablesFootnote f: their distribution, mean and variability. Moments are based on a generalisation of the average. Hence, they are generic indicators of the distribution. They represent the arithmetic mean of a specified integer power of the deviation of the variable from the mean. In this sense, two distributions are equal if all their infinite moments coincide. Thus, we can assume that the more similar their both distributions are, the more similar their moments are. For the distribution of weights for each class c, we obtain the following measures statistical embedding (SE) shown in Equation (4): minimum, maximum, average, median, first and third quartiles (Q), (Gini Reference Gini1971) indexes (G) to measure the distribution skewness and kurtosis, and the first 10 moments (M). Based on that, documents are represented using Equation (5):

(4) \begin{equation} SE(W) = \{min(W), max(W), avg(W), median(W),\\ Q_{1}(W), Q_{3}(W), G_{1}(W), G_{2}(W), M_{2..10}(W)\} \end{equation}

(5) \begin{equation}d = SE(W(t, c)) \sim \forall{\ t\in{T}, c\in{C}}\end{equation}

To better illustrate the previous formulas and their practical application, we used the LDSE method to represent the documents of a corpus annotated with two classes. This corpus is the Portuguese subset of the PAN-AP17, which will be explained later and for which the average feature avg(W) is plotted in Figure 1. This figure confirms that both classes can be easily separated.

Figure 1. Portuguese subset of the PAN-AP’17 represented with the avg(W) feature of LDSE. The x-axis represents each of the terms in the corpus. The Y-axis represents the average weight for each term in Brazilian (blue) or Portuguese (red) varieties.

We experimented with several machine learning algorithms (Bayesian, Logistic, Neural Networks, Support Vector Machines, Trees and Rules-based, Lazy, and Meta-classifiers) implemented in WekaFootnote g. After that, we selected the best performing ones on the training data in each case.

3.2 Corpora

In this section, we describe the corpora used in this research work. First, we describe the PAN-AP’17 corpus which covers four languages and their varieties. This corpus allowed us to demonstrate the suitability of LDSE to address language variety identification, taking into account also the authors gender. Then, we describe the ARAP-Tweet corpus (Zaghouani Reference Zaghouani and Charfi2018a) which allows us to evaluate the use of the LDSE method for more fine-grained identification of Arabic varieties taking into account the authors’ age and gender.

3.2.1 PAN-AP’17

PAN LabFootnote h at Conferences and Labs of the Evaluation ForumFootnote i focuses on different forensics linguistics tasks: author identification (Kestemont et al. Reference Kestemont, Tschuggnall, Stamatatos, Daelemans, Specht, Stein and Potthast2018), profiling (Rangel et al. Reference Rangel, Rosso, Montes-y-Gómez, Potthast and Stein2018) and obfuscation (Hagen et al. Reference Hagen, Potthast and Stein2018). Given a certain document, the aims are to infer who is the author that wrote it as well as the authors’ demographic traits. Obfuscation is the opposite task to author identification. It aims at making the identification of authors based on their writing style impossible. PAN provides an opportunity for the research community to validate and compare the state-of-the-art methods and technologies for the three forensics linguistics tasks mentioned above.

The focus of the 2017 Author Profiling shared task was on gender and language variety identification in Twitter. The PAN-AP’17 corpus includes four languages: ArabicFootnote j, English, Portuguese and Spanish. For each language, several varieties were considered as shown in Table 1. For each variety, tweets geolocated in the capital cities (or the most populated cities), where this language variety used were collected. Unique users were selected and annotated with their corresponding variety. A dictionary with proper nouns was used to annotate the users’ gender. Moreover, we manually inspected their profile photo to improve the annotation quality. Finally, for each user, 100 tweets were collected from her/his timeline. The corpus was divided into training/test following a 60/40 proportion, with 300 authors for training and 200 authors for test per gender and variety. More information on this corpus is available in the shared task overview paper (Rangel et al. Reference Rangel, Rosso, Potthast and Stein2017).

Table 1. PAN-AP’17 corpus, covering four languages with their corresponding varieties and the cities selected as representative of such varieties

3.2.2 ARAP-Tweet

ARAP-Tweet is a corpus that was developed at Carnegie Mellon University Qatar (Zaghouani Reference Zaghouani and Charfi2018a) in the context of the ARAP project. The total number of tweets in this corpus is above 2 millions (exactly 2,032,539) and the total number of words is above 18 millions (exactly 18,582,436). Across all dialectal varieties of this corpus, the average number of tweets per user is 684 and the average number of words per tweet is 9.

Arabic dialects have been generally classified by regions such as in Habash (Reference Habash2010), who classified the Arabic major dialects into North African, Levantine, Egyptian and Gulf. Similar dialectical varieties were also used at PAN following Sadat et al. (Reference Sadat, Kazemi and Farzindar2014). However, dialect variation within regions could be significant. For example, the Tunisian dialect is different from the Moroccan dialect even though they belong to the same North African/Maghreb region. Therefore, fine-grained annotated Arabic language resources are required. ARAP-Tweet is a corpus that provides fine-grained dialectal Arabic tweets annotated with age and gender information. It contains 15 dialectical varieties corresponding to 22 countries of the Arab world. For each variety, a total of 102 authors (78 for training and 24 for test) were annotated with age and gender, maintaining balance for both variables. Three age groups are distinguished: Under 25, Between 25 and 34, and Above 35. The included varieties, as well as the regions they belong to, are shown in Table 2. Further information on this corpus is available in Zaghouani (Reference Zaghouani and Charfi2018a,Reference Zaghouani and Charfib).

Table 2. ARAP-Tweet corpus: language varieties and regions

Figure 2. Comparative results of the three best performing teams in the Author Profiling shared task at PAN 2017 versus LDSE. The best performing team (Basile et al. Reference Basile, Dwyer, Medvedeva, Rawee, Haagsma and Nissim2017) obtained the highest result in Arabic and Spanish. The second best performing team (Tellez et al. Reference Tellez, Miranda-Jiménez, Graff and Moctezuma2017) obtained the highest result in English and Portuguese.

4.1 Classification results

Figure 2 shows the results obtained by the three best performing teams at PAN 2017 together with the results we obtained with LDSEFootnote k. Results are shown for the four languages as well as the average among them. At PAN, the best accuracy results in Arabic and Spanish were achieved by Basile et al. (Reference Basile, Dwyer, Medvedeva, Rawee, Haagsma and Nissim2017), who obtained 83.13% and 96.21%, respectively. They also obtained the best overall result in the shared task (91.84%). In case of English and Portuguese, the best accuracy was obtained by Tellez et al. (Reference Tellez, Miranda-Jiménez, Graff and Moctezuma2017), with 90.04% and 98.5%, respectively. Overall, they had the second best result in the task (91.71%). Basile’s team approached the task with combinations of character, tf-idf word n-grams and SVM. Similarly, Tellez’s team used SVM with combinations of bag-of-words. The third best performing team was Martinc et al. (Reference Martinc, Skrjanec, Zupan and Pollak2017), who used logistic regression with combinations of character, word, POS n-grams, emojis, sentiments, character flooding and lists of words per variety, achieving an average accuracy of 90.85%. It is worth mentioning that also deep learning approaches (e.g., recurrent neural networks, convolutional neural networks, as well as word and character embeddings) were used by other participants but they did not lead to the best results.

In Figure 2, the results obtained by LDSE are also shown. The figure shows that LDSE achieves the best results for Portuguese (99% vs. 98.5%) and Spanish (96.36% vs. 96.21%), while it achieves the second best results for Arabic (83% vs. 83.13%) and English (89.94% vs. 90.04%). Overall, LDSE has the best performance with an average accuracy of 92.08% versus the second best performance of 91.84%. As shown in Table 3, there is no statistical significance between the best results at PAN and the ones obtained by LDSE, which confirms its competitiveness with the state-of-the-art approaches.

Table 3. Significance (p-values) when comparing LDSE results with the three best performing teams in the Author Profiling shared task at PAN 2017 (^*0.05; ^**0.01)

4.2 Confusion among varieties

The error among varieties of the same language is analysed using confusion matricesFootnote l as shown in Figures 3, 4, 5 and 6, respectively, for Arabic, English, Portuguese and Spanish.

Figure 3. Confusion matrix for Arabic varieties with LDSE on the PAN-AP’17 corpus.

Figure 4. Confusion matrix for English varieties for LDSE on the PAN-AP’17 corpus.

Figure 5. Confusion matrix for Portuguese varieties for LDSE on the PAN-AP’17 corpus.

Figure 6. Confusion matrix for Spanish varieties for LDSE on the PAN-AP’17 corpus.

As shown in Figure 3, the maximum confusion in Arabic varieties is from Gulf to Egypt (14.25%), followed by Maghreb to Egypt (12.75%), whereas the lowest confusion is from Egypt to Levantine (0.5%). The rest of the errors are between 2.25% (from Egypt to Gulf) and 6% (from Levantine to Gulf). The highest accuracy was obtained for the identification of Egyptian Arabic (93%). Together with the lowest confusion seen previously, these results show that this variety is the less difficult to be identified. Conversely, the Gulf and Maghreb varieties are the most difficult ones to identify, with accuracies of 76% and 77%, respectively, and with the highest confusions to other varieties. Finally, the results obtained for the Levantine variety are higher than the average (86% over 83%). These results are similar to the ones obtained by the PAN participants, where both the Egyptian and Levantine Arabic varieties were the less difficult to identify.

Figure 4 shows the LDSE confusion matrix among English varieties. The highest confusion is from Ireland to Great Britain (6.25%), United States to Canada (6%), Canada to United States (5.25%) and Great Britain to United States (4.5%). Some of these errors correspond to varieties geographically close or that even share geographical borders. The other errors are lower than 4.5%, with almost no error in cases such as New Zealand to Canada (0.75%), Canada or Great Britain to New Zealand (0.5%), Ireland to Canada (0.5%), Ireland to New Zealand (0.25%), New Zealand to Ireland (0%) and United States to New Zealand (0%). Considering the previous insights, together with the highest accuracy obtained (94.75%), we conclude that the New Zealand variety is the less difficult English variety to identify. The second less difficult English variety is Irish English (89.25%), and all the rest range between this maximum value (of 89.25%) and the minimum value obtained for Great Britain (84.5%). Similarly to what was observed already at PAN, we conclude that the geographically closer the two English varieties are, the higher the confusion between them is.

As shown in Figure 5, the results for Portuguese are very high and almost without errors, which is in line with the results achieved by the PAN shared task participants. There is no confusion from Brazil to Portugal varieties, and only 2% of the Portuguese variety is confused with the Brazilian one. This gives an accuracy of 100% for identifying Brazilian Portuguese, which is the less difficult Portuguese variety to be identified. The accuracy is 98% for the Portuguese variety of Portugal.

In case of Spanish, the confusion matrix among varieties is shown in Figure 6. It can be observed that all the Spanish varieties have similar results, ranging from 95% to 97.25%, with no significant difference among them. The highest error is from Peru to Spain (7.5%), Chile to Argentina (5%), Peru to Argentina, Chile and Colombia (2.5%), and the rest are lower to 2%. Except in the case of Peru and Spain, we can conclude again that the geographical proximity of varieties may affect the confusion between them.

Finally, in Table 4, we summarise the differences between the lowest and highest accuracy obtained for each language both for the best participant at PAN and for LDSE. The last column shows the difference between PAN and LDSE. In case of English and Spanish, LDSE is significantly more stable than the systems at PAN. This is also true for Portuguese but without statistical significance. In case of Arabic, LDSE is significantly more variable. However, we can argue in favour of this variability due to the very high accuracy obtained for the Egyptian variety (93%, about 10% higher than the best performing team at PAN).

Table 4. Difference between highest and lowest accuracies per variety language, both for the best participant at PAN in that language and LDSE. The last column shows the difference between them (^* indicates a significant difference)

4.3 The impact of the gender on the language variety identification

In this section, we compare the systems at PAN to LDSE with respect to the impact of gender on the language variety identification. In Table 5, we compare the LDSE results to the average results of the systems at PAN, as reported in Rangel et al. (Reference Rangel, Rosso, Potthast and Stein2017). In Table 6, we compare LDSE to the best performing system per language at PAN 2017Footnote m. Both tables show that it is more difficult to properly identify the variety in case of males except for Spanish. We also observe that at PAN, these differences are significant in case of Arabic and Portuguese. Especially in the case of Arabic, the difference decreases from 7.06% to 2.50%.

Table 5. Language variety identification accuracy per gender (^* indicates a significant difference) comparing LDSE to the average of all the systems at PAN

Table 6. Language variety identification accuracy per gender and language (^* indicates a significant difference) comparing LDSE to the best performing system at PAN

When comparing LDSE to the best performing system per language at PAN (in Table 6), we can see that the difference decreases in the case of Arabic, English and Spanish, whereas it remains the same in the case of Portuguese. It is noteworthy that in the case of Arabic and Spanish, the decrease is statistically significant, from 5.75% to 2.50% and from 1.00% to 0.43% for both languages.

In Figure 7, the errors per gender for each variety are shown in detail. In case of Arabic, we can observe that the maximum error occurs with the Gulf variety for males (31%), followed by the Maghreb variety for females (28.5%). This coincides with the analysis of the confusion matrix, where we concluded that the Gulf and Maghreb varieties are the most difficult to identify. Errors per gender for both Egypt and Levantine varieties are more well balanced, even though it is remarkable that in case of Egypt, females are a little bit more difficult to be identified (1%). In case of English, there is the same number of varieties with a higher number of errors in one gender or the other. For example, in the case of Australia, New Zealand and United States, there are more errors in case of females, whereas the contrary occurs with Canada, Great Britain and Ireland. Finally, the highest difference occurs with Canada (8%). With respect to Spanish, except for Colombia and Mexico, there are more errors for males. In Spanish, the differences are smaller (2%) and the performance per gender is more balanced than in English and Arabic. In the case of Portuguese, all errors occurred with the variety from Portugal, with 3/4 of the errors belonging to females.

Figure 7. Percentage of errors per gender for each language variety (PAN-AP’17 corpus).

5.1 Classification results

Figure 8 shows the LDSE results when using the ARAP-Tweet corpus. We experimented with five machine learning algorithms: BayesNet, Multilayer Perceptron, Simple Logistics, SVM and Random Forest.

Figure 8. Accuracy obtained by LDSE with five different machine learning classifiers on the ARAP-Tweet corpus.

We obtained the best accuracy result with Multilayer Perceptron (88.89%), followed by Logistic Regression and Support Vector Machines (87.5%), Random Forest (86.94%) and Bayesian Networks (86.11%). However, these differences are not statistically significant. In the next sections, we will use Multilayer Perceptron. It is worth to mention that for this experiment, we selected only 100 tweets per author in order to maintain a comparable scenario to PAN, at which LDSE achieved 83% of accuracy.

5.2 Confusion among varieties

If we analyse the confusion matrix among the varieties of Figure 9, we can see that most errors occur with the Saudi variety (63% of accuracy), followed by the Qatari variety (71% of accuracy). The average accuracy was 88.89%. The following Arabic varieties were the less difficult to distinguish: Egypt (100%), Libya (100%), Morocco (100%), Sudan (100%), Iraq (96%), Lebanon Syria (92%), Palestine Jordan (92%), Tunisia (92%) and Yemen (92%). Together with Saudi Arabia and Qatar, the most difficult Arabic varieties to identify are those of Kuwait (83%), Oman (83%) and UAE (83%).

Figure 9. Confusion matrix for Arabic varieties for LDSE on the ARAP-Tweet corpus.

The highest error occurs from Saudi Arabia to UAE (17%), varieties from two neighbouring countries. Similarly, most errors occur within the same region. For example, within Gulf, there is confusion at classifying from Qatar to UAE (8%), Saudi Arabia (4%) or Yemen (8%), as well as from Kuwait to Oman (8%), Qatar (4%) and UAE (4%). Similarly, within the Levant region, there is confusion from Palestine Jordan to Lebanon Syria (4%), or to close countries albeit these are in another region. This is the case, for example, from Palestine Jordan to Saudi Arabia (4%) or Saudi Arabia to Lebanon Syria (4%). Similarly to PAN, the highest confusion occurs within the Arabic variety of the Gulf region, whereas the highest accuracy was obtained for the identification of the Egyptian variety (100%).

5.3 The impact of the age and gender on the language variety identification

In this section, we analyse the impact of the authors’ age and gender on Arabic language variety identification. Figure 10 and Table 7 show the distribution of errors depending on the authors’ age and gender.

Table 7. Distribution of the errors depending on the authors’ age and gender (ARAP-Tweet corpus)

Figure 10. Distribution of the errors depending on the authors’ age and gender (ARAP-Tweet corpus).

We observe that the percentage of errors in case of female authors (62.5%) is much higher than in case of males (37.5%). Concretely, there is a difference of 25%. This is also true in case of the age classes Under 25 and Between 25 and 34, where the difference is 15%. However, the opposite occurs for the age class Above 35, where the errors in case of males are 5% higher than in case of females. In case of females, the highest error occurs with the age class Under 25 (27.5%) and the lowest with the age class Above 35 (12.5%), with a significant difference of 15%. Conversely, the highest error in case of males occurs with the age class Above 35 (17.5%), whereas the lowest one occurs with the age class Between 25 and 34 age class (7.5%), with a highly significant difference of 10%. Taking into account only the age ranges, the highest error is in case of the class Under 25 (40%), with a significant difference of 10% over the other two classes. We can conclude that Arabic varieties included in ARAP-Tweet are less difficult to identify when the author is male (37.5% of error) or belongs to the age classes Between 25 and 34 and Above 35 (30% of error), and especially when the author is male in the age class Between 25 and 34 (7.5% of error).

It is noteworthy that the obtained distribution of errors per gender for this corpus is the contrary to the error distribution obtained for the PAN-AP’17 corpus. In that latter corpus, the proportion of errors between females and males was approximately 46% versus 54%. This significant difference in error distribution can be explained by the different methodologies followed to build the two corpora. In case of ARAP-Tweet, the corpus was collected from Twitter and then perfectly balanced with respect to gender and age classes, whereas in case of PAN-AP’17, the retrieved tweets followed a real scenario distribution with respect to age groups (e.g., it included more people Above 35 than Under 25). Furthermore, in case of the PAN-AP’17, the collected Twitter authors had their geolocalisation activated. Probably, this option depends on the users age (e.g., younger people could be more conscious about their privacy and therefore deactivate this option more often).

In Figure 11, the error per age and gender is shown for each Arabic variety (only varieties with errors). The highest error occurs with males in the class Above 35 in the case of Tunisia (6.98%), followed by Kuwait (4.65%) also for the same age group and gender, and Qatar (4.65%) for males in the age class Between 25 and 34. The remaining errors with males occur mainly in the age classes Under 25 and Above 35, with a frequency of 2.33% each. In the case of females, the highest errors for Oman variety (6.98%) occur in the classes Under 25 and Between 25 and 34. For Qatar, the highest errors (6.98%) occur for the class Under 25. For Saudi Arabia, the highest errors occur in the case of the classes Between 25 and 34 (6.98%) and Under 25 (4.65%). We should highlight the case of Kuwait and UAE that has an average error of 17% since there is an age class with no errors: Under 25 in case of Kuwait and Above 35 in the case of UAE. Finally, it is worth to mention that in most Arabic varieties, there are no errors for males in the class Between 25 and 34 (except Qatar with 4.65% and UAE with 2.33%).

Figure 11. Percentage of errors per age and gender for each language variety (ARAP-Tweet corpus).

5.4 The effect of the corpus size

Since the ARAP-Tweet corpus contains a variable number of tweets per author, we analysed the effect of this number on the variety identification task using the same machine learning algorithms as described previously. In Figure 12, we observe that the accuracy of all classifiers improves when the number of tweets increases except in the case of Simple Logistics, whose behaviour becomes erratic from 700 tweets. The average accuracy increases from 87.38% to 94.79% (with the exception of Logistic Regression from 700 tweets). This is an average improvement of 7.41% which is statistically significant. The best performing algorithms are Multilayer Perceptron and Random Forest. In order to be consistent to what was done previously, we used Multilayer Perceptron for the following experiments. With this classifier, the accuracy increased from 88.89% to 95.28%. This is a statistically significant improvement of 6.39%. Therefore, we can conclude that the more the tweets are in the corpus, the better is the classifiers performance.

Figure 12. Accuracy when the number of tweets increases (ARAP-Tweet corpus).

Figure 13 shows the improvement of each increment in the number of tweets per author in steps of 100. To simplify the visualisation, we only show the Multilayer Perceptron and the average of all the algorithms excluding Logistic Regression. We observe that the trend in both cases is clearly downward and tends to zero. On average, the highest decrease is from 300 to 400 tweets, whereas in case of Multilayer Perceptron it is from 600 to 700, and the slope is softer.

Figure 13. Accuracy improvement for each increment in the number of tweets per user in steps of 100 (ARAP-Tweet corpus).

Figure 14 shows a decrease in accuracy when using less than 100 tweets. We observe that the trend is descending, slow at the beginning and faster when the number of tweets decreases from 30. The average among classifiers decreases from 87.39% to 53% (i.e., by 34.39%), which is highly significant. In case of the Multilayer Perceptron, the decrease is higher from 88.89% to 40.83%. However, this decrease in accuracy is not significant until the number of tweets is reduced to 40 for both the average classifier and Multilayer Perceptron.

Figure 14. Accuracy when the number of tweets per user decreases (ARAP-Tweet corpus).

This analysis is important from the viewpoint of a real scenario because retrieving contents from Twitter and processing large amounts of tweets are both costly. Therefore, it is important to balance the quality with the cost, and to select the optimum number of tweets at which the accuracy improvement is not significant.