II. Ranking Model Selection

We first introduce three candidate models for analyzing wine-tasting data. Suppose M judges are instructed to assign ordinal scores to N wines, with the levels and meanings of the ordinal scores being the same for all judges. We assume that each wine to be evaluated has an underlying continuously valued latent (nonobserved) quality, which determines the true ordering of all wines. The observed scores, regardless of the original ordinal scores or transformed ranks, are estimates of the latent wine quality. In addition, the ranking based on score average or rank average allows each rater to have an equal impact on the aggregate score, implicitly assuming that judges have a homogeneous rating pattern.

Let y _ij be the observed ordinal score assigned by judge j on the ith wine and θ _i (i = 1, · · · , N) be the underlying quality of wine i. Then the model using the original score is

(1) $$y_{ij} = \theta _i + e_{ij}, \; e_{ij} \sim N\left( {0,\; \sigma ^2} \right),$$

where measurement error e _ij made by judge j on evaluating wine i is assumed to have a normal distribution with mean zero and a common variance σ ². The second model is based on the ranks. Let z _ij be the rank assigned by judge j on the ith wine, where it can be transformed from the original score y _ij . The model has a form similar to Model (1),

(2) $$z_{ij} = \theta _i + e_{ij}, \; e_{ij} \sim N\left( {0,\; \sigma ^2} \right).\; $$

The third model is the ranking model by Cao and Stokes (Reference Cao and Stokes2010). The biggest difference from the above models is that this ranking model allows judges to have different rating patterns. The other difference worth mentioning is that the other two models assume that the ratings (regardless of whether they are the original scores or ranks) are numerical measures of wine quality. However, the ratings are ordinal values, and the differences in the ratings do not necessarily represent the true differences in wine quality. In the third model, the ordinal nature of the data is fully considered, whereby judges are assumed to assign scores by first estimating the wine quality and then comparing it with category cut-offs, which are the same for all judges. In addition to the above notation, let x _ij be the estimate of θ _i by judge j, giving the third model the following format:

(3) $$x_{ij} = \alpha _j + \beta _j \theta _i + e_{ij}, \; e_{ij} \sim N\left( {0,\; \sigma _j^2} \right)$$

and

$$y_{ij} = s \Leftrightarrow c_{s - 1} \lt x_{ij} \le c_s. $$

That is, measurement error e _ij made by judge j on his or her assessment of wine i is assumed to have a normal distribution with mean zero and judge-specific variance $\sigma _j^2 $ . The quantities c _s denote the common cut-offs for all judges, α _j is the bias parameter for judge j, and β _j is the discrimination parameter. A generous judge tends to give a higher average score to all wines, producing a positive α _j , while a stringent judge has a negative α _j . A competent judge's discrimination parameter β _j is positive, indicating that the criteria used by judge j are consistent with the wine's latent quality θ _i . A small β _j value suggests that the judge assigns ratings in a small range, and a large β _j value suggests that the judge assigns ratings that are more separated to distinguishable wines. Parameter $\sigma _j^2 $ describes the amount of inconsistency in a judge's scoring pattern. The larger the value of $\sigma _j^2 $ , the more randomness in the judge's evaluation. Readers can refer to the paper for more details on fitting the model.

We use the deviance information criterion (DIC) to compare the three models. The DIC (Spiegelhalter, Best, Carlin, and van der Linde, Reference Spiegelhalter, Best, Carlin and van der Linde2002) is an extension from the AIC, the classical Akaike information criterion. It is a measure of predictive power based on the trade-off between model fit and complexity. Like the AIC, lower DIC values indicate stronger models. A generally accepted notion is that differences of more than 10 rule out the model with the higher DIC. A difference of less than 5 indicates that the two models are very comparable in terms of model fit. Using the Paris 1976 red-wine-tasting data, the DIC values for the three models are 296.8, 297, and 259, respectively. Using the Princeton 2012 red-wine-tasting data, the DIC values for the three models are 243.1, 246, and 207.1, respectively. Both real datasets show the same conclusion on the model comparison: Model 1 and Model 2 provide similar model fit to the data, while Model 3 yields much better model fit than the other two models. This is not surprising; whether using the original score or the rank value, Model (1) and Model (2) both assume that the judges have homogeneous rating patterns (i.e., similar bias, discrimination, and ranking error). Model 3 is much more flexible in allowing different rating patterns for different judges, thus producing a much better model fit to both datasets.

III. Ranking Methods Comparison

Based on the model-selection result, we use Model (3) to simulate data following the setup of the Paris 1976 red-wine tasting (i.e., 10 wines rated by 11 judges). Specifically, we use the estimates of judge bias, discrimination, and variation from the real data analysis, and then we use Model (3) to generate y _ij values. Based on the simulated data, we compare three ranking methods (score average, rank average, and Shapley ranking) in terms of their ranking accuracy. To compute the Shapley rankings, we use a similar simulation design to that of Ginsburgh and Zang (Reference Ginsburgh and Zang2012), wherein three cases are considered. The first case, denoted as Shapley (1), assumes that each judge would have chosen three wines; the second case, denoted as Shapley (2), starts with the wine with the highest score for each judge and then goes down until it reaches a gap of 2 points; the third case, denoted as Shapley (3), assumes that the subset of wines that each judge finds meritorious includes the wines with scores that are higher than a certain cut-off (e.g., 15 points).

We compare the ranking results of the three methods based on two criteria. One is the squared-error loss, which calculates the sum of squared differences (Cao, Stokes, and Zhang, Reference Cao, Stokes and Zhang2010) between the estimated ranks and the true ranks. It is a suitable measure when accurate ranking of all wines is of interest. The other is the percentile loss, which only considers whether the wines are correctly put in a certain subset. In this study, we consider whether the selected best wine is indeed the best wine.

We generate 1,000 datasets and calculate the values from both loss functions. Table 1 summarizes the results. Note that for both loss functions, a smaller value indicates that the method provides a more accurate ranking result. The simulation study shows that the ranking based on score average is generally more accurate than the one based on rank average. More specifically, the score-average-based ranking is about 25% more accurate than the rank-average-based ranking when accurate ranking of all wines is of interest and 30% more accurate when the goal is choosing the very best wine among all wines. Shapley ranking, in general, has inferior performance compared with the other two methods.

Table 1 Simulation 1 (Judge Variation is the Same Among Ranking Methods)

Note that in practice, under Shapley ranking, judges do not need to rate or rank all the wines. Instead, they only need to think in terms of two groups, which is psychologically very different and imposes less burden during wine tasting. The previous simulation setup does not take the less evaluation burden on wine judges under Shapley ranking into consideration. In the next simulation study, we consider a different simulation setup to account for this feature of Shapley ranking. Model (3) includes three judge characteristics: bias, discrimination, and random variation. We assume that under Shapley ranking, bias and discrimination do not change much, because those two reflect judges’ personal rating styles. However, the amount of variation may change. Under Shapley ranking, the evaluation burden is much less, allowing judges to demonstrate higher accuracy (i.e., less random variation) in selecting the subsets that they find meritorious. Under this assumption, we reduce the standard deviation of measurement error in Shapley ranking to 80% of the previous level, while the other two methods have the same standard deviation as before. The results are presented in Table 2. Under the squared-error loss, the score average still is the best, and Shapley ranking remains the least competitive. This result is reasonable, because the goal of Shapley ranking is not to rate all the wines but to divide them into two groups, with wines in each group being indistinguishable. Under the percentile loss, Shapley (2), which starts with the wine with the highest score for each judge and then goes down until it reaches a gap of 2 points, performs best. Recall that the percentile loss only considers whether the wines are correctly put in a certain subset, which is consistent with the Shapley ranking method. Shapley (2) beats the score average and the rank average in this setting thanks to the smaller judge variation. Shapley (2) also outperforms the other Shapley ranking schemes, because its setting is most consistent with dividing the wines into two groups, with a clear gap of 2 points separating them; the other two schemes use arbitrary rules (i.e., each judge would have chosen three wines or scored higher than 15) to identify the two groups.

Table 2 Simulation 2 (Judge Variation in Shapley Ranking is 80%)