2.1 Formal Definition

We now turn to formally define multi-label classification. For reference, we provide a summary of the mathematical notation used within this section in Appendix Table C.1 of the Supplementary Material. The multi-label paradigm can be more formally defined as follows: let $\mathcal {X} \in \mathbb {R}^d$ be a d-dimensional input space and the label space with q possible classes be $\mathcal {Y} = \{0,1\}^q$ . Similar to the standard classification task defined above, the goal is to learn a mapping function—in this case, defined as $ h~:~ \mathcal {X} \xrightarrow {}\mathcal {Y}$ —from the training set $\mathcal {D} = \{x_i,Y_i~ |~ 1 \leq i \leq m \}$ . For each instance $(x_i,Y_i)$ , the input $x_i = (x_{i1},...,x_{id})$ is a d-dimensional vector and the output is $Y_i$ is a q-dimensional set of labels. Therefore, for each new unseen example of $x \in \mathcal {X}$ , the multi-label classifier $h(\cdot )$ predicts a set of labels $h(x) \subseteq \mathcal {Y}$ .

For supervised text-as-data classification problems, the output of a multi-label model can thereby be seen as a real-valued function $ f~:~ \mathcal {X}~\text {x}~\mathcal {Y} \xrightarrow {} \mathbb {R}$ , where $f(x,y)$ can be interpreted as a compatibility or confidence function that evaluates how compatible or likely $y \in \mathcal {Y}$ is the correct label of x. The classifier function $h(\cdot )$ can be obtained by taking the output with largest compatibility score $h(x) = \underset {y \in \mathcal {Y}}{arg~max} f(x,y)$ or by using a thresholding function $ t~:~ \mathcal {X}~ \xrightarrow {} \mathbb {R}$ such that $h(x) = \{y | f(x,y) \geq t(x),y \in \mathcal {Y}\}$ .

Based on the formal definition presented above, it is evident that traditional supervised classification problems can be viewed as a simplified version of multi-label learning, where each target has only one (binary or nominal) label. The generality of multi-label classification makes this task much more difficult to solve, given that each directional inter-label relationship must be accounted for as a predictive feature in a manner that accounts for permutations. That is, in order to accommodate inter-label relationships, one ideally needs to not only apply a classifier once to each label (as is presently done within political science), but rather to classify each label once for every possible ordering (i.e., subset) of all remaining labels—since these remaining labels, and the order by which they themselves are classified, now have bearing on each label’s subsequent prediction. In this regard, the key challenge of multi-label classification is thus the output space, which grows exponentially as the number of labels increase. For example, a problem with 5 binary labels has 32 different label subsets. If we increase this number to 15 binary labels, the number of possible combinations grows to 32,768.

To address these shortcomings, researchers have identified several ways to efficiently leverage the relations among labels within multi-label classification tasks. Two longstanding strategies include accounting for (i) pairwise correlations between any two labels (Ueda and Saito Reference Ueda, Saito, Becker, Thrun and Obermayer2002; Qi et al. Reference Qi, Hua, Rui, Tang, Mei and Zhang2007) or (ii) rankings between relevant and irrelevant labels (Elisseeff and Weston Reference Elisseeff and Weston2002; Brinker, Fürnkranz, and Hüllermeier Reference Brinker, Fürnkranz and Hüllermeier2006). In comparison to BR, these alternative approaches better manage the trade-off between performance and computational cost within multi-label classification tasks.

However, these approaches encounter problems when the relationships among labels become more complex than simple pairwise associations—which is oftentimes the case for real-world social science data. This is especially the case for many efforts to code quantities of interest from political texts. For example, with regards to the country-year human rights application presented below, CIRI’s separate, nonmutually-exclusive human-rights labels for state torture, political imprisonment, extrajudicial killings, and disappearances (Cingranelli and Richards Reference Cingranelli and Richards2010) are likely to be highly interdependent facets of states’ overarching strategies of repression, as opposed to simply being linked via pairwise associations. In such cases, multi-label alternatives that accommodate the influences of all labels when predicting each label (Yan, Tesic, and Smith Reference Yan, Tesic and Smith2007; Ji et al. Reference Ji, Tang, Yu and Ye2008; Cheng and Hüllermeier Reference Cheng and Hüllermeier2009), can achieve superior performance, albeit at the expense of higher computational costs and more constrained scalability.

2.2 Categorization of Multi-Label Learning Algorithms

Methods for multi-label classification can be divided into two general categories: problem transformation methods and algorithm adaptation methods. The former tackles the multi-label classification problem by reformulating this classification problem into other tasks, such as binary classification (Read et al. Reference Read, Pfahringer, Holmes and Frank2009) or multi-class classification (Tsoumakas and Vlahavas Reference Tsoumakas and Vlahavas2007). On the other hand, algorithm adaptation methods tackle the multi-label problem by adapting learning algorithms to directly deal with multi-labeled data (Zhang and Zhou Reference Zhang and Zhou2007), oftentimes in manners that better account for the label associations in one’s data. We first discuss algorithm adaptation below, before returning to problem transformation, and then a broader summary of all approaches.

2.3 Summary of Approaches

We provide (i) an overview of the computational costs associated with each multi-label approach in Appendix D of the Supplementary Material and (ii) a summary of our (tentatively favored) problem transformation approaches, alongside the earlier-described BR approach, in Figure 1. Since these latter methods are based on transforming the multi-label problem into binary or multi-class classification, the training procedure is identical to standard supervised learning algorithms.Footnote ⁴ Likewise, these methods’ flexibility in incorporating different base classifiers of one’s choosing helps to ensure comparable underlying interpretability relative to BR on this dimension. That being said, ECC, LP, and RAkEL’s reliance on varying degrees of ensembling does raise practical challenges for these three approaches’ interpretability, relative to simpler BR frameworks. Given that multi-label text-as-data problems are primarily oriented toward the measurement and accurate prediction of labels for future use—rather than explanation—this tradeoff in improved accuracy for some loss in interpretability is preferable for many researchers.

Figure 1 Relationships among problem transformation approaches.

With the above caveats in mind, we note that many of the multi-label approaches reviewed above have exhibited good-to-excellent performance in text-as-data contexts in past comparisons of multi-label methods (Madjarov et al. Reference Madjarov, Kocev, Gjorgjevikj and Džeroski2012). Yet, in terms of relative performance, none of these multi-label methods has emerged as consistently superior to the others.Footnote ⁵ In light of this, our empirical applications compare the performance of a wide array of multi-label and standard supervised classification approaches. However, in order to do so, special consideration must first be given to choices of comparison metrics, in light of the multi-label context being considered. We, hence, now turn to discussing model comparison in multi-label contexts, before turning to our applications in full.

3.1 Example-Based Metrics

Hamming Loss is an example-based metric that computes the fraction of misclassified labels for each observation. It considers both prediction errors (when the prediction is incorrect) and omission errors (when the label is not predicted at all), where lower values are more optimal. For example, a hamming loss of $10\%$ means that $90\%$ of all labels were classified correctly. It can be formally defined as:

(1) $$ \begin{align} L_{\mathrm{Hamming}}(Y, \hat{Y}) = \frac{1}{n_{\text{labels}}} \sum_{k=1}^{n_{\text{labels}}} \mathbf{1}(\hat{y}_k \not= y_k) \end{align} $$

where $n_{\text {labels}}$ is the total number of labels, $\mathbf {1}$ is the indicator function, $\hat {y}$ is the kth predicted label and y is the actual label.

Subset Accuracy is a stricter metric than Hamming Loss. Instead of evaluating the fraction of correctly classified labels, Subset Accuracy only considers a prediction as correct if all of an observation’s predicted labels are identical to its true label set:

(2) $$ \begin{align} Subset_{Acc}(Y, \hat{Y}) = \frac{1}{m} \sum_{i=1}^{m} \mathbf{1}(\hat{Y}_i = Y_i), \end{align} $$

where m is the total number of examples in the test set, $\mathbf {1}$ is the indicator function, $\hat {Y_i}$ is the ith predicted label set, and $Y_i$ is the ground-truth label. This metric can thus be interpreted as reporting the percentage of all observations that have all labels correctly classified, with higher values on this metric being more optimal.

3.2 Label-Based Metrics

For label-based metrics, we consider Macro- and Micro-F1 scores. Each is based on precision and recall.Footnote ⁶ Precision indicates the proportion of predicted positives that are truly positive while recall denotes the proportion of actual positives that are classified correctly. Macro-F1 is the harmonic mean of the precision and recall averaged across all labels. The precision-macro ( $p_{\mathrm {macro}}$ ) and recall-macro ( $r_{\mathrm { macro}}$ ) are defined as follows:

(3) $$ \begin{align} p_{\mathrm{macro}_j} = \frac{\mathrm{TP}_j}{\mathrm{TP}_j + \mathrm{FP}_j}, \end{align} $$

(4) $$ \begin{align} r_{\mathrm{macro}_j} = \frac{\mathrm{TP}_j}{\mathrm{TP}_j + \mathrm{FN}_j}, \end{align} $$

where $\mathrm {TP}_j,\mathrm {FP}_j,\; \mathrm {and}\; \mathrm {FN}_j$ denote the true positive, false positive and false negative rate for the label j respectively and q is the number of labels for each example. Macro-F1 can be defined in terms of these metrics as

(5) $$ \begin{align} \mathrm{Macro{-}F1} = \frac{1}{q}\sum^q_{j=1} \frac{2 ~ p_{\mathrm{macro}_j}. ~ r_{\mathrm{macro}_j}}{p_{\mathrm{macro}_j} + r_{\mathrm{macro}_j}}. \end{align} $$

On the other hand, for Micro-F1 both precision and recall are defined differently:

(6) $$ \begin{align} p_{\mathrm{micro}_j} = \frac{\sum^q_{j=1}\mathrm{TP}_j}{\sum^q_{j=1}\mathrm{TP}_j + \sum^q_{j=1}\mathrm{FP}_j}, \end{align} $$

(7) $$ \begin{align} r_{\mathrm{micro}_j} = \frac{\sum^q_{j=1}\mathrm{TP}_j}{\sum^q_{j=1}\mathrm{TP}_j + \sum^q_{j=1}\mathrm{FN}_j}. \end{align} $$

Correspondingly, Micro-F1 is defined as follows:

(8) $$ \begin{align} \mathrm{Micro{-}F1} = \frac{2 ~ p_{\mathrm{micro}_j}. ~ r_{\mathrm{micro}_j}}{p_{\mathrm{ micro}_j} + r_{\mathrm{micro}_j}}. \end{align} $$

Micro-F1 calculates the metrics by counting the total number of true positives, false negatives, and false positives. That is, the scores of all labels are aggregated to compute the metric. On the other hand, Macro-F1 calculates the metrics for each label independently, and then averages them.

3.3 Ranking-Based Metrics

Ranking Loss evaluates the fraction of label pairs that are incorrectly ordered. As such, it considers only the relative rankings (i.e., orderings) of one’s label predictions—in terms of which labels exhibit higher versus lower predicted probabilities in that label set—and the correspondence between these rankings and a true label set. It can be defined as:

(9) $$ \begin{align} R_{\mathrm{loss}} = \frac{1}{m} \sum^m_{i=1} \frac{1}{|Y_i||\bar{Y}_i|} |\{(y',y'' | f(x_i,y') \leq f(x_i,y''), (y',y'') \in Y_i ~ \text{x} ~ \bar{Y}_i \}|, \end{align} $$

where $\bar {Y}$ is the complementary set of $\mathcal {Y}$ in $\mathcal {Y}$ . In this case, Ranking Loss is interpreted such that the lower the Ranking Loss, the better the performance. In a similar manner to AUC in single-label prediction contexts, one strength of Ranking Loss is its reliance on relative orderings of label predictions rather than on arbitrary thresholds (e.g., 0.5). This ensures that predictive evaluations via Ranking Loss will be less sensitive to factors that lead to consistently (high or low) predictions relative to a chosen threshold.Footnote ⁷

To illustrate this, imagine a set of true labels $Y_{1_{\mathrm {true}}} = [1,0,0],Y_{2_{\mathrm { true}}} = [1,1,0]$ and a corresponding set of label predictions given by a chosen classifier of $f_{1_{\mathrm {pred}}} = [0.4,0.1,0.2],f_{2_{\mathrm {pred}}} = [0.9,0.8,0.6]$ . Using a threshold rule of $0.5$ (1 if $f(\cdot ) \geq 0.5$ , 0 otherwise), we would obtain a relatively uninformative Hamming Loss of 33% and Subset Accuracy of 0%, as $Y_{1_{\mathrm {pred}}} = [0,0,0],Y_{2_{\mathrm { pred}}} = [1,1,1]$ . On the other hand, the resulting Ranking Loss would be zero (i.e., perfect).

4.1 Mexican ATI Requests

Our first application examines ATI requests made to the Mexican federal government during the period 2003–2015. ATI requests in this context have been previously analyzed in efforts to assess the degree to which Mexican citizens use this ATI system to hold their government publicly accountable (Berliner, Bagozzi, and Palmer-Rubin Reference Berliner, Bagozzi and Palmer-Rubin2018), or to understand government responsiveness (Almanzar, Aspinwall, and Crow Reference Almanzar, Aspinwall and Crow2018; Berliner et al. Reference Berliner, Bagozzi, Palmer-Rubin and Erlich2021). We provide additional background on Mexico’s ATI system in the Supplementary Material. The textual content of our ATI requests contains requesters’ open-ended descriptions of their desired information, as entered into an online ATI request system’s primary request field, supplemental information field, and attachments field. Attachments were webscraped, converted to machine readable text via optical character recognition, and then combined with other request text fields to form our primary textual entries of interest. Altogether, this process produced a sample of 1,025,953 requests for our consideration.

We are interested in a wide variety of traits associated with each of these request texts, pertaining to qualities such as the use of legalistic and technical language, the number of distinct pieces of information requested, and the appropriateness of the request for the targeted agency. Developing accurate and fine-grained measures of attributes like these will enhance understandings of the nature and dynamics of citizen demand for information, and of the request-specific determinants of government responsiveness—both in general and as it varies across agencies and time. We accordingly drew a random sample of 4,925 requests—stratified by year—from our full sample of request texts. Six Mexico City-based coders coded these request texts for distinct ATI-request traits. For this paper, we retained 26 total (ATI-request trait) labels for classification. Our original request traits—and overall human coding approach—are discussed in the Supplementary Material.

With this sample of 4,925 human coded requests, we then trained all aforementioned classifiers on this sample, classifying all 1,021,028 remaining (non-human coded) ATI requests for each of our 26 labels. All relevant text processing steps that were applied to the raw texts prior to classification—and the hyperparameters used for each model—are described in the Supplementary Material. The overall level of correlation for our 26 labels is not high, suggesting that this application is a “hard test” for the potential benefits of multi-label classification. On average, the correlation between our pairs of hand-coded labels is 0.06 with the lowest and highest pairwise correlations being 0 and 0.40, respectively.

We next evaluate the value added of the multi-label framework (i.e., of considering inter-label relations for our document-level labels). In order to do so, we compare the results obtained from four plausible approaches to handling multi-label data. The first pair of general approaches that we consider are BR and ML-kNN. Recall that neither of these two approaches consider label relations. By contrast, the second pair of general multi-label approaches that we consider (CC and LP) do consider label correlations.

Within each of these general approaches, we then implement and consider several of the extensions described further above. Turning first to the CC approach, we consider its basic implementation and also a version where the label ordering was permuted. For the latter CC approach, the final output was composed of the average of each permuted chain, ECC.Footnote ⁸ Similarly, for the LP approach, we considered both its standard definition and the RAkEL extension. The base classifier for each of these methods was a standard logistic regression.

For the ML-kNN, we only considered the standard version since its proposed extensions are not yet readily available. On the other hand, we evaluate four different varieties of BR classifiers. The first, presented as standard BR, used a simple random forest with the same parameters applied to all labels. For the second BR variant, we perform a grid search to find the best performing classifier and corresponding hyperparameters for each label. This approach is labeled as BR optimized below.

For the third BR variant, BR optimized threshold, we kept the same classifier for all labels. However, instead of using the standard 0.5 classification threshold, we optimized the threshold for each label by selecting 20 different splits of the training data and selecting the threshold value for each label that maximized the F1-score. Finally, to more directly address class-imbalance in some labels, we used an oversampling technique to generate synthetic samples from the minority class, known as SMOTE (Blagus and Lusa Reference Blagus and Lusa2013). Using this technique, we trained the final BR classifier that we consider, hereafter referred to as BR optimized SMOTE.

The multi-label results obtained from all classifiers considered are summarized in Table 2. The algorithms used were based on the implementations provided by Python’s scikit-multilearn package (Szymański and Kajdanowicz Reference Szymański and Kajdanowicz2017) and sklearn (Pedregosa et al. Reference Pedregosa2011)Footnote ⁹ . Each model was evaluated using 80% of the data for training and 20% for testing with 10 different splits. Further details on hyperparameter selection for each algorithm considered appear in Appendix Table E.2 in the Supplementary Material.

Table 2 Results for 10 different data splits.

The top three performers for each metric are highlighted. Full details appear in Table E.1 of the Supplementary Material.

Turning to Table 2, we find that ECC achieves the best performance among all tested classifiers. That is, we can determine from Table 2 that ECC routinely outperforms our alternate approaches, and can also observe that ECC is a top-three performer across all metrics used. This finding supports our contentions regarding the importance of taking into account relationships between labels for multi-label classification of political text-as-data. The complete results for all metrics considered here can be found in Table E.1 of the Supplementary Material and are consistent with this summary interpretation. These results suggest that ECC also exhibits the lowest average deviation when compared to other algorithms. Additional discussion of the predicted proportions for each label is likewise presented in the Supplementary Material.

For the four BR classifiers considered in Table 2, the optimized version achieved the lowest Hamming Loss whereas the SMOTE version achieved the highest F1-Macro. The latter finding underscores our earlier contentions regarding F1-Macro being a poor metric for class imbalanced data. Regarding the former finding, we can note that each of the classifiers used in the BR optimized-component of this application was selected based upon the accuracy score for its corresponding label individually. Therefore, these BR classifiers were optimized by minimizing the Hamming Loss. Regarding the remaining classifiers in Tables 2 and E.1, we can further observe in this case that the results for ML-kNNFootnote ¹⁰ are remarkably worse than all other models.

We can thus conclude that multi-label approaches that give careful consideration to the relationships between one’s labels tend to outperform several less label-aware approaches that are more common in the political science literature. To this end, our CC approaches tended to exhibit higher Subset Accuracy than BR and comparable results for other metrics. This finding, and those outlined above, suggest that multi-label classification allows researchers to achieve superior labels within multi-label classification tasks, even in contexts where the correlation between these multiple labels is relatively low. This provides strong support for the multi-label approach in a real-world text-as-data context. Our next application offers researchers further guidance on when, specifically, multi-label methods are likely to be more or less effective.

4.2 Country-Year Human Rights Practices

Our second application allows us to evaluate the benefits of using multi-label algorithms in the context of a separate political science text-as-data domain. For this application, we specifically compare the best performing multi-label algorithm from our first application (ECC) to BR using a set of extant human rights texts and labels. Our human rights data include a combination of (i) country-year textual reports on human rights practices and (ii) overlapping labels of states’ annual human rights practices, as human-labeled from these same texts by the CIRI data project (Cingranelli and Richards Reference Cingranelli and Richards2010). As noted in the introduction, automating the measurement of human rights abuses from the human rights texts is an area of growing interest in political science (Fariss et al. Reference Fariss2015; Greene, Park, and Colaresi Reference Greene, Park and Colaresi2019; Murdie, Davis, and Park Reference Murdie, Davis and Park2020; Park, Greene, and Colaresi Reference Park, Greene and Colaresi2020a,Reference Park, Greene and Colaresib). Our application of multi-label methods to these data thus stands to advance the state of the art for these automated endeavors.

The text-as-data component to this application is a corpus of U.S. State Department Country Reports on Human Rights Practices for the years 1981–2011.Footnote ¹¹ Each document corresponds to the State Department’s assessment of a particular country’s annual domestic human rights practices during the previous calendar year. We match these reports to 14 indicators, from the CIRI project, of specific types of human rights violations and practices at the country-year level. These indicators were human-coded by the CIRI project from the U.S. State Department texts outlined above, and encompass categories of human rights abuse that range from targeted killings and torture to violations of women’s political rights and freedom of speech. To simplify interpretation, we (i) dichotomize each (originally ordinal) CIRI variable and (ii) then reverse-code each dichotomized variable such that higher values imply worse (as opposed to better) human rights performance. We provide further details on these CIRI variables in the Supplementary Material, Appendix F.

In total, there are 4,756 reports, each with 14 binary indicators for (country-year) violations of different human rights wherein a 0 denotes “no violations” of a given violation type and 1 denotes a human rights violation of a certain type. As noted above—and unlike our first application—these labels were directly drawn from an existing human-coded country-year dataset (i.e., CIRI), rather than from human-labeling of our own. These human rights labels are furthermore much more highly correlated than the previous ATI Requests application. On average, the correlation between each current label is 0.33, with a maximum value of 0.60 and a minimum 0.06.

With respect to the text features considered, we begin with the full document term matrix (DTM) of raw term counts from our human rights reports of interest. This DTM format is consistent with formats used by past text-as-data research into these same State Department human rights reports (Fariss et al. Reference Fariss2015), and, in our case, follows the preprocessing steps applied by Bagozzi and Berliner (Reference Bagozzi and Berliner2018) to a similar corpus of human rights reports. After implementing these preprocessing steps, our final DTM contains 2,445 unique stemmed unigrams as features.

In addition to these DTM-features, the ECC algorithm also allows us to incorporate the label correlation information between our CIRI human rights violation target measures within our multi-label classification tasks. We anticipate that the latter information will be highly relevant, given that—as mentioned in Section 2—many of the specific CIRI violations considered here (e.g., disappearances, killings, and torture) are likely to arise (and be correlated with one another) under a common strategy of state repression. We are, hence, interested in quantifying the added improvement that is gained by accounting for these interdependencies. Within these evaluations, this second application also (uniquely) varies the set of retained features that we use in classification, so as to assess whether our findings with respect to the ECC’s added improvements vary in relation to the number of features that a given researcher has available for use in classification.

We thus evaluated the performance of ECC and BR with respect to the classification of our 14 binary human rights violation categories when including different subsets of DTM features, which are chosen at random. We posit that if a classifier has enough information from its original features to adequately classify all target labels, there is probably very little to gain in leveraging the relationships between target labels during classification.Footnote ¹² On the other hand, if there are insufficient features to properly classify all target labels, the additional information available to the researcher via any empirical correlations between each multi-label label will likely improve prediction significantly. This suggests that our multi-label approach will improve in relative performance over BR as the number of available features for classification declines.

We utilize our human rights data within a series of experiments to evaluate the above contentions. These experiments corresponded to our evaluation of ECC and BR model performance across 10 different data splits (80% for training and 20% for testing) for each feature level considered. The number of features ranged from all features available to as low as 20% of the total number of features drawn at random. In evaluating the ECC and BR approaches, we consider three base classifiers in each case: support vector machine (SVM) with linear kernel, random forest, and logistic regression.

The results from these experiments are presented in Figure 2, considering five of the model performance criteria outlined above: Subset Accuracy, Hamming Loss, F1-Macro, F1-Micro, and Ranking Loss. For all results plotted in Figure 2, BR and ECC base classifiers were chosen according to best overall performance. The best base classifier for BR was an SVM with linear kernel, whereas for the ECC the best performing base classifier was a random forest. Our conclusions are comparable when we standardize the base classifier across BR and ECC.

Figure 2 Performance of binary relevance (BR) and ensemble classifier chain (ECC) for the proposed metrics. Shaded regions denote the standard deviation of the bootstrap splits. For the metrics reported in the left-hand panels, higher values imply better performance. For the metrics reported in the right-hand panels, lower values imply better performance.

Examining the performance criteria in Figure 2, we can first observe that ECC routinely outperforms BR no matter the model performance metric or percentage of retained features considered. As in the case of Mexican ATI requests, this result provides strong support for the use of ECC (and hence multi-label methods) in contexts where a researcher is faced with multi-label data. Further, when only a small percentage of features are available (20%), the gap in performance between ECC and BR is the widest. For instance, this gap is $11.09\%$ for Subset Accuracy with 20% of features available. This result implies that the desirability of ECC over the more commonly used BR is especially pronounced when researchers are faced with a multi-label problem but have a limited number of features for prediction. As the number of available features increases, we find that the difference in performance between ECC and BR becomes less and less notable.

Hence, when features are abundant, the benefits of ECC become less salient, suggesting that researchers whose prediction tasks already include an exceptionally large number of (relevant) features may find BR to be sufficient. By contrast, researchers faced with a more limited set of features—in terms of total number, relevance, or related traits (e.g., sparsity)—are likely to especially benefit from using ECC or other multi-label methods. These trade-offs notwithstanding, as both applications suggest, multi-label methods typically exhibit advantages over BR approaches even in cases where one’s feature set is relatively large and even when one’s multi-label targets exhibit a relatively low level of correlation.

4.3 Monte-Carlo Simulations

The applications presented above assess the performance of a wide range of multi-label classifiers in two distinct empirical settings and across different levels of available features. However, these applications do not provide insight into the relative performance of these approaches under differing (i) scenarios of available training data or (ii) end-use cases for one’s classified labels. Accordingly, our Supplementary Material further compares our best performing multi-label approach (ECC) to BR across a series of Monte-Carlo simulations.

We find that ECC outperforms BR for every classification metric considered—and especially for Subset Accuracy and Hamming Loss. Across three auxiliary regression set-ups, we then confirm in this context that ECC’s superior classification performance also ensures that the parameter estimates obtained when subsequently using ECC’s classified labels as regressors and/or regressands are superior in accuracy and coverage to those recovered by BR—and increasingly so as one’s available training (test) data decrease (increase).