1. Background and motivation

Vertical segregation is pervasive in academia, in particular it is evident in the different shapes of the academic hierarchical structure by sex. In the case of economics in Italy, on which we focus in this work, women are distributed in a pyramidal shape, with few full professors at the top (19 percent of women economists), more associate professors in between (35 percent), and the relative majority constituted by assistant professors at the base (46 percent); whereas for men the hierarchical structure looks like an overturned pyramid: full-professors represent the highest share (44 percent), followed by associate professors (31 percent) and, in the lowest percentage, researchers (25percent).^{Footnote 1}

Within the European Union, to analyze the proportion of women in the top academic positions the European Commission recommends the use of the Glass Ceiling Index (GCI). The index ranges from 0 to infinity; a value of 1 indicates that there are no differences between men and women in terms of their chances of being promoted to full professorship, and the higher the value the stronger the glass ceiling effect. Considering all disciplines, in 2017 the GCI amounted to 1.62 in Italy, and 1.93 in economics. The index for economics is even higher than in traditionally male dominated fields (e.g. with 1.64 in mathematics and computer sciences), and apparently this strong gender bias in economics is not a typically Italian phenomenon (for e.g. the case of the USA, Fourcade et al., Reference Fourcade, Ollion and Algan2015).

For Italian economists there has been some progress towards reducing the glass ceiling in the recent past: the share of women among full professors increased from 7.7 percent in 2002 to 15.8 percent in 2017. But in the face of a flurry of reforms and institutional innovations, inequalities persist and even some reversals of the recent positive trend are possible, as we show in this paper.

Indeed, following an international trend emphasizing meritocratic and supposedly objective quality-based criteria, Italy introduced a system in which funding allocation for universities and personnel recruitment and promotion is founded on centralized research evaluation increasingly based on bibliometric measures. We focus here in particular on the effects of a 2010 reform of Italian universities’ system (Law 240 of 2010) that introduced a centralized “national scientific qualification” (ASN, from the acronym of the Italian “Abilitazione Scientifica Nazionale”) as a necessary requirement for anybody wishing to compete for associate or full professor positions.

Due to the scale of the procedure (involving every single candidate for tenure in the country) and its praiseworthy transparency, the ASN has already occasioned a growing literature (reviewed in section 3). In this context the case of economics is especially interesting because the procedure uniquely exhibited features typical of both peer review and bibliometric methods. Indeed, as it had already happened in a previous national research evaluation procedure, though at that time focused on the evaluation of research centers and university departments, many evaluators perceived economics as a discipline lying somewhere between the natural and the social sciences, and departed from the practice of the other social sciences by giving significant more weight to journal rankings within their assessment of single publications (see Bertocchi et al., Reference Bertocchi, Gambardella, Jappelli, Nappi and Peracchi2015; Ancaiani et al., Reference Ancaiani, Anfossi, Benedetto, Blasi, Carletti, Cicero, Ciolfi, Costa, Colizza, Costantini, di Cristina, Ferrara, Lacatena, Malgarini, Mazzotta, Nappi, Romagnosi and Sileoni2015; Baccini and De Nicolao, Reference Baccini and De Nicolao2016, Reference Alberto and De Nicolao2017a, Reference Baccini and De Nicolao2017b).

Considering the outcomes of the first two rounds of the ASN for economics (Corsi et al. Reference Corsi, D’Ippoliti and Zacchia2018) document discrimination of heterodox and women economist candidates for tenure. In this work we focus on the outcomes of the qualification for the highest academic position only, and explicitly compare candidates within the two systems, before and after the 2010 reform. Since up to the reform, qualifications were only determined on the basis of peer review at the local level, a comparison between the two systems allows us to investigate the possible effects of bibliometrics in particular in terms of gender diversity. From this point of view, a crucial issue is not just how many women make it to the top of the academic ladder, but also which ones and how. It thus emerges from our analysis that increasingly women start publishing on topics that were less studied by women before in order to conform to a standardized profile of economist, with possibly relevant consequences in terms of pluralism within the discipline.

As noted in “Citation Patterns in Economics and Beyond: Assessing the Peculiarities of Economics from Two Scientometric Perspectives,” “methodologically guided evaluations in general and evaluative scientometrics in particular, reactivity induces actors to anticipate evaluation criteria. This anticipation in turn affects the behavior of the subjects or institutions evaluated” (Aistleitner et al. Reference Aistleitner, Kapeller and Steinerberger2017, 8-9). Along these lines, we interpret differences between the observed behavior of candidate economists before and after the 2010 reform as evidence of a reaction on their side to the new rules, because the reform was announced and widely debated before it was gradually implemented. Moreover, the general direction of the reform, in particular the heavy reliance on bibliometrics, is part of a more general process that started in the early 2000s with the first nationwide evaluation of research centers, and it followed a path common to several other countries, such as the UK or the Netherlands. Thus, our work could be considered as an application to the case of gender diversity in economics, of the broader issue of performativity of research evaluation.

2. Reaching the top: the evolution of recruitment in Italian universities

There are three main ranks of an academic career in Italy: Ricercatore Universitario, roughly corresponding to an assistant professor or lecturer, has become a three-year renewable temporary position after the 2010 reform; Professore Associato, corresponding to an associate professor, is now the lower-ranked tenured position; and Professore ordinario, corresponding to a full professor, is the highest ranked tenured position. All researchers and professors are classified as belonging to one field (called scientific-disciplinary sector, there are 367 in all); every field is grouped into one of 14 so-called scientific areas. For every position, teaching and research duties as well as wages are defined by national laws.

The rules on hiring and promoting university professors have changed in the country over the past decades, moving from national to local competitions in 1998 (Law 210, 1998), and lately again to a mixed system of national qualification (so-called ASN) followed by local competitions, introduced in 2010 (Law 240, 2010). Two rounds of the new ASN took place in 2012 and 2013, and after a break they were resumed in 2016. Since specific regulations governing the process were partially changed between 2013 and 2016, we focus here on the first two rounds only.

Until 1998, full professors were selected in a national competition held every four to five years, mainly when the requests of new positions by universities was sufficiently large. In 1998, new rules have been introduced moving the selection process from national competitions to local ones in order to give universities more control over their own recruitments. Each department, conditional to their budgeting constraints, could now open vacancies and appoint the members of the selection committees, five for each competition (of which one was to be an internal member, usually coming from the same department holding the competition). Each selection committee could proclaim up to three winners (two from 2002), but success in the process only implied gaining the status of “appointable” (idoneo), being then left to a vote by the whole Department if and who among the appointable candidates would actually be hired (or promoted, if an internal candidate). The department could choose to appoint one of the winners or neither of them (but in this case the department could not open new positions for two years). Anyway, those who passed the selection but were not appointed could be hired by any other Italian university within the next three years. Due to this disconnect between competition for an opening and the actual hiring of a candidate, the post-1998 recruitment system produced an increase in the number of competitions but also significant uncertainty for candidates.

In 2008 there has been another significant change in the rules governing competitions for full professorship in what we refer to here as the old system, with the substitution of the appointment of the selection committee members for a random draw among full professors who declared their availability at the national level (Decree-Law n. 180, 2008). Selection committees were now composed of two external randomly chosen commissioners, and an appointed internal one. This procedure aimed to avoid the creation of ad personam committees and to expand the circle of “gate keepers.”^{Footnote 2} The new mechanism of random selection had a significant effect from a gender perspective: 44.4 percent of competitions for full professorship in economics held in 2008 had at least one woman commissioner, while the average share in the pre-2008 competitions was 34.7 percent (see Table 1).^{Footnote 3}

Table 1. Gate Keepers in full professorship competitions: 2001-2008

Note: we consider the competitions for full professorship in Italy in the field SECS-P01 held from 2001 to 2008; in 2007 there have not been competitions in this field.

Crucially, in 2010 (with law 240, 2010) a new reform of the selection procedures instituted a system of national scientific qualification (ASN), based for the first time on peer review “informed” by bibliometric indicators defined by law. According to the new selection procedures (first implemented in 2012), only researchers who obtained a national scientific qualification can now compete at the local level for a vacancy at an Italian university. The ASN is granted with a unified, national procedure and is a necessary condition for employment as tenured faculty in Italy, though it is not a guarantee of employment. Indeed, hiring and promotion remain based on the single Departments’ opening of vacancies that are filled by means of local competitions. These local competitions are adjudicated by a three-member committee, with one internal and two external members. So even in the new system there is a disconnect between being declared appointable (i.e. holding a qualification) and being hired. For reasons of consistency we ignore here the actual hiring and promotion and rather focus on the qualification part of the procedure.

The ASN evaluations are held separately for each scientific area. In each field the candidates are evaluated by a committee of five members: four of the them are full professors at Italian universities, and one is a foreign commissioner from an OECD country. The commissioners are randomly selected from a pool of full professors who meet certain scientific productivity standards and who volunteered for the task. The qualification is based on candidates’ CVs and publications only, and no tests or interviews are foreseen. For the evaluation of publications, in practice the dominant criterion (Corsi et al. Reference Corsi, D’Ippoliti and Zacchia2018), the ASN system provides two different sets of bibliometric indexes: one for the natural sciences (the so-called “bibliometric areas”), and one for the social sciences and humanities (the so-called “non bibliometric areas”). For economics (defined as a “non bibliometric area”), candidates are evaluated based on three parameters:

1. (i) the number of articles published in high quality scientific journals (so-called A-list^{Footnote 4}) in the ten years before the application;

2. (ii) the number of book chapters and journal articles in any scientific outlet; and

3. (iii) the number of scientific monographs.

The Italian National Agency for Evaluation of University and Research (ANVUR) provides each selection committee with the following:

• - quantitative thresholds concerning all the three parameters above; until 2013, these thresholds were determined by the estimated median number of publications (in a certain time span^{Footnote 5}) by the full professors employed in Italian universities in the same field;

• - a list of journals defined as “A-list”, and one of journals defined as “scientific”;

• - a report on every candidate with the number of her/his publications, separately listed according to the three parameters (for candidates in the natural sciences, selected citation indexes and the corresponding thresholds).

In the debate around the 2010 reform law, there often was an understanding that in order to qualify at the ASN it would be necessary to pass at least two of the three thresholds. However, the selection commissions had (and have used) full autonomy in their evaluation, and the thresholds were only to be considered as reference points. The only actual requirement was a positive evaluation by a qualified majority of four positive votes out of the five committee members (this requirement was recently reversed to a simple majority by a judgement of Italy’s administrative main court). In economics, the bibliometric indicator on the number of articles in A-list journals proved in practice of decisive relevance (Corsi et al., Reference Corsi, D’Ippoliti and Zacchia2018), thus confirming an international trend whereby government-sponsored journal rankings prove increasingly consequential in economics (see next section).

Moreover, a critical issue raised by the ASN system is the uncertainty about the new selection procedures, caused by the lack of regularity in the calls for applications. As a matter of fact, the selection procedure has not occurred on a regular basis so far; there have been two yearly rounds in 2012 and in 2013, and five quarterly rounds (with slightly different criteria) between December 2016 and April 2017. At the time of writing, while a new call for potential commissioners has been published, it is still unknown what the future pace of ASN rounds will be after 2019. This means that it is impossible for Italian researchers to plan their career paths sufficiently in advance.

However, so far the debate has mostly revolved around issues of candidates’ selection and fairness in gender balance. In particular, the increased number of women involved in the process as evaluators, possibly caused by the introduction of random drawing introduced in 2008 and confirmed in 2010, inspired a number of studies on the impact of the gender composition of selection committees.^{Footnote 6} Abramo et al. (Reference Abramo, D’Angelo and Rosati2016), Bagues et al. (Reference Bagues, Sylos-Labini and Zinovyeva2017), Checchi et al. (Reference Checchi, De Poli and Rettore2018), and De Paola and Scoppa (2015; Reference De Paola and Scoppa2017) study the effect of evaluators’ gender on the probability of women candidates to withdraw from the selection process, and/or their probability of being appointed to professorship. While using different empirical techniques, these works have not come to a consensus on the possible effects of the gender composition of evaluation committees, let alone on the direction of such effects.

More in general, the ASN has been used by several researchers as a sort of natural experiment. De Paola et al. (Reference De Paola, Ponzo and Scoppa2018) study the introduction of the new recruitment and promotion system to find that it did not increased women candidates’ probability of being promoted to associate or full professorship. As mentioned above, Corsi et al. (Reference Corsi, D’Ippoliti and Zacchia2018) show that – since women exhibit lower bibliometric indicators on average – evaluation procedures based on equal statistical measures produce unequal results. In the first rounds of the ASN for economics, women candidates as full professor have on average 4.1 A-list journal articles (which they identify as the crucial indicator) as compared to men’s 5.3.

In fact, concerning this last point on the fairness and consequences in terms of gender equality of the adoption of bibliometrics as opposed to peer review, the ASN allows contributing to a wider literature, summarised in the next section.

3. Gender biases in research evaluation

There is a wide debate on the pros and cons of research evaluation methods, especially concerning the respective advantages of bibliometrics and peer review.^{Footnote 7} In briefly summarising the literature we must limit ourselves here to economics, with particular attention to the case of Italy.

Considering the effects of research evaluation methods in terms of diversity and/or their potential discriminatory outcomes, no consensus has as yet emerged. Traditionally, the focus has been on the consequences of evaluation methods in terms of contents and scientific developments, e.g. the potential discouragement of radically new path-breaking ideas (Wang et al. Reference Wang, Veugelers and Stephan2017), multidisciplinary methods (Rafols et al. Reference Rafols, Leydesdorff, O’Hare and Nightingale2012; Hicks et al. Reference Hicks, Wouters, Waltman, De Rijcke and Rafols2015), or research orientations and fields pursued by a minority of researchers in the respective disciplines (López-Piñeiro and Hicks Reference López-Piñeiro and Hicks2015; Kapeller and Steinerberger Reference Kapeller and Steinerberger2016).

In the case of economics, several studies focused on Anglo-Saxon countries (e.g. for the UK: Lee Reference Lee2006, Lee et al. Reference Lee, Pham and Gu2013; Harley and Lee Reference Harley and Lee1997; for the USA: Fourcade et al. Reference Fourcade, Ollion and Algan2015; for Australia: Bloch Reference Bloch2010), on German-speaking countries (Grimm et al. Reference Grimm, Puhringer and Kapeller2018), France (Chavance and Labrousse 2016), and Italy (Corsi et al. Reference Corsi, D’Ippoliti and Lucidi2010, Reference Corsi, D’Ippoliti and Lucidi2011; Baccini Reference Baccini2016). They highlight a gradual decline in the diversity of approaches among academic economists, and the increasing homologation of research towards the mainstream paradigm, as a consequence of evaluation methods increasingly based on bibliometrics. It is generally found that sub-fields of economics have different preferred dissemination outputs (i.e. more books and book chapters in the history of economic thought, as opposed to nearly exclusive reliance on journal articles for econometrics); researchers active in the different fields of economics have different research interests or preferences for interdisciplinary approaches; and the various fields are made up of scientific communities of various sizes. Both in the UK and Italy this discussion intersects that on the use of journal rankings (which many consider an inappropriate way to use bibliometric methods, if applied to the evaluation of single journal articles: HEFCE, 2015), because these lists tend to be extremely biased towards the mainstream. In the case of Italy, Corsi et al. Reference Corsi, D’Ippoliti and Zacchia2018 document how out of 454 journals included in the A-list for the ASN, only 12 (2.6 percent) are also in the ranking of heterodox journals by Cronin et al. Reference Cronin, Lee, McConnell and Dean2010. Moreover, in the A-list only two journals are explicitly engaged with gender, sexuality and/or feminist approaches.

However, fairness in terms of gender equality has increasingly been investigated within the literature on the bias of peer review (Marsh et al. 2009; Reference Marsh, Jayasinghe and Bond2011). One problem is that for both the evaluation of journal article submissions and applications for funding and/or hiring or promotion (the two aims for which peer review is more used), peer review is often single or double blind, making it impossible for researchers to consult public datasets on referees’ and applicants’ sex.^{Footnote 8} Perhaps as a consequence, results have been mixed and often country- or discipline specific (Tamblyn et al. Reference Tamblyn, Girard, Qian and Hanley2018).

A meta-analysis of 21 studies on peer review within fund grants applications found that the estimates of the gender effect vary substantially from study to study, but in general men have statistically significant greater odds of receiving grants than women, by about 7 percent (Bornmann et al. Reference Bornmann, Ruediger and Hans-Dieter2007). However, Marsh et al. Reference Marsh, Bornmann, Mutz, Hans-Dieter and O’Mara2009 use a more sophisticated statistical method, finding no significant gender effect.

A common finding is that editors select more male reviewers and this pattern is more pronounced for male editors (Buckley et al. Reference Buckley, Sciligo, Adair, Case and Monks2014; Primack et al. Reference Primack, Maron and Campos-Arceiz2017). This seems to be due to a predominance of senior male active researchers, combined with more invitations going to senior researchers; however, women also tend to show a slightly higher propensity to decline invitations to review (Lerback and Hanson Reference Lerback and Hanson2017).

In many instances, the apparent predominance of male reviewers does not seem to lead to higher rejection rates for female authors (Primack et al. Reference Primack, Ellwood, Miller-Rushing, Marrs and Mulligan2009; Lerback and Hanson Reference Lerback and Hanson2017). In contrast, Murray et al. Reference Murray, Siler, Lariviére, Wei, Collings, Raymond and Sugimoto2018 find a small but statistically significant advantage for men authors within the peer review of 7,192 recent submissions to the biosciences journal eLife. They find that such gender inequity was greatest when the team of reviewers was all male, while mixed-gender teams lead to more equitable peer review outcomes.

In the Italian case, Jappelli et al. Reference Jappelli, Nappi and Torrini2017 analyzed a non-disclosed sample of publications evaluated with both bibliometric methods and peer review within the “research quality evaluation” (VQR, from the Italian acronym), a nationwide official research assessment exercise in which each tenured researcher submitted their best three or four publications. Jappelli et al. find that in their sample on all disciplines bibliometric evaluation does not penalize women with respect to men, whereas peer review might, and that in general bibliometric evaluation proves to be more favorable to women than peer review evaluation, regardless of the reviewers’ sex.

Independently of their finding or not of gender-based discrimination, most studies report a lower number of submissions from women. For the case of economics, Hengel Reference Hengel2018 reports that female authors are held to higher standards within peer review and suggests that women may internalized this trend opting for a lower number of higher quality initial submissions.

Concerning specific discrimination in hiring, an increasing number of surveys gather information about the prevalence of gender bias in academia. For example, Carr et al. Reference Carr, Ash, Friedman, Szalacha, Barnett, Palepu and Moskowitz2000 for the USA or Knights and Richards Reference Knights and Wendy2003 for the UK find that women are likely to perceive gender-based discrimination in their academic environment. More recently Howe-Walsh and Turnbull Reference Howe-Walsh and Turnbull2016 observe how, although the United Kingdom universities adopted policies to mitigate gendered practices, women respondents perceive direct and indirect discrimination mainly in the recruitment and selection process and in the lack of recognition of their professional successes.

An argument that led some countries (such as Italy or the UK during the 2000s^{Footnote 9}) to increasingly recur to bibliometric indicators is that they are objective. Being defined ex ante and independently of the specific person (or publication) under evaluation, bibliometric indicators by design remove direct discrimination, i.e. the application of different rules to different cases. However, bibliometric methods could bring about issues of indirect discrimination (Larivière et al., Reference Larivière, Chaoqun, Gingras, Cronin and Sugimoto2013, p. 212), i.e. the unfair application of uniform rules to different cases. From a gender perspective, a crucial issue is that women publish fewer articles, partly because of heavier teaching and administrative burdens (documented for Italy by Baccini et al. Reference Baccini, Barabesi, Cioni and Pisani2014). There is also evidence that men engage more in self-citing (for the case of economics: Ferber and Brün, Reference Ferber and Brün2011; King et al., Reference King, Bergstrom, Correll, Jacquet and West2017) and that more in general publications with female authors receive fewer citations on average, even after controlling for other author and publication characteristics (HEFCE 2015; Abramo et al., Reference Abramo, D’Angelo and Rosati2016). In the case of Italy-based economists, D’Ippoliti (Reference D’Ippoliti2018) shows that social network dynamics, such as being co-authors, working in the same institutions etc., significantly determine an author’s number of citations, and that women and younger authors are at a disadvantage due to their lower network centrality. These issues have led some authors (e.g. Corsi et al., 2010; Reference Corsi, D’Ippoliti and Lucidi2011, for the case of Italy) to propose the development of standardization procedures of the bibliometric indexes to be used for evaluation.

As mentioned, in applying bibliometric indicators for research evaluation, in practice journal-level indexes rather than publication-level metrics are frequently used. Studies have demonstrated that the journal ratings of a department’s publications are the strongest predictor of the results obtained in the 2008 UK’s RAE (Brooks et al., Reference Brooks, Fenton and Walker2014) as well as the subsequent REF (Stockhammer et al., Reference Stockhammer, Dammerer and Kapur2017), although journal ratings were not formally used in the evaluation. Similarly, Corsi et al. (Reference Corsi, D’Ippoliti and Zacchia2018) find that a government-determined list of “A-ranked” journals crucially shaped the results of the evaluation of researchers described in the next section. Yet, the use of journal rankings in particular has been criticized (Moed Reference Moed2007), for example because the distribution of citations is highly skewed making journal-level averages little indicative of single papers’ visibility. This issue too is relevant from our point of view, as e.g. for the UK case Brooks et al. (Reference Brooks, Fenton and Walker2014) find that women on average received lower scores according to some journal ratings lists, because at least in the case of business studies there are important differences in the rated quality of journals in which men and women publish.

Overall, the evidence on biases in bibliometric indexes is more clear-cut than that on peer-review, but there is no agreement on the extent to which such biases should be regarded as problematic and if they should be compensated within research evaluation exercises (HEFCE, 2015). In part as a response to these worries, as mentioned the UK, which had traditionally acted as a champion of large-scale centralised research evaluation among European countries and is still the most studied case, moved to a system formally based on peer review only. In Italy, current regulation considers bibliometric indexes as appropriate for certain disciplines (mostly the natural and life sciences) while reserving peer review for others (humanities and social sciences). However, in the ASN the system was in fact a mixed one for the so-called “non-bibliometric areas,” encompassing both peer review and journal rankings, as described in the previous section.

These two debates, on gender-based discrimination and on pluralism and scientific paradigms, are not totally disconnected. Indeed, there is empirical evidence that women and other minorities hold specific views. In the case of economics, Albelda Reference Albelda1997, Stastny Reference Stastny2010, and May et al. Reference May, McGarvey and Whaples2014; Reference May, McGarvey and Kucera2018 find systematic differences between men’s and women’s research interests and economic policy preferences. It could thus be the case that increasing diversity of backgrounds could bring about higher diversity of scientific perspectives too (e.g. Forget Reference Forget, Mary, Robert and Evelyn1995; May et al. Reference May, McGarvey and Kucera2018). However, some authors consider this to be an essentialist argument if taken to imply a causal relation whereby more women in academia would automatically translate into greater diversity of views (Christensen Reference Kimberley2001, 108). For example, Zacchia Reference Zacchia2017 finds that a progressively more competitive environment in the last few decades induced women economists to adopt a strategy of “homologation” towards the same research interests of their male colleagues, in order to increase their chances to achieve a tenured academic position. Therefore, in the empirical analysis that follows we do not consider just the share of women who qualified at the ASN in comparison to the previous system, but we also investigate possible changes in the profile of these successful women.

4. A comparison of successful candidates for full professorship

We collected information on all candidates to 23 competitions for full-professorship in economics held before the introduction of the new recruitment procedures in 2010,^{Footnote 10} half of which held between 2001 and 2003, and half held in 2008, after the introduction of the random selection of commissioners. We also gathered data on all candidates for a qualification as full professor in economics within the first two rounds (in 2012 and 2013) of the ASN system.

In order to allow for a comparability between the sorts of systems, before and after 2010, for the pre-reform competitions we define as “successful” all candidates who were declared “appointable” and not only those who were subsequently actually employed. This way, there is a clear analogy between the statuses of the winners in the two systems (Jappelli et al. Reference Jappelli, Nappi and Torrini2017). However, it should preliminarily be pointed out that as of today 48 percent of the winners in the pre-2010 competitions considered in our analysis went on to become full professors of economics (not necessarily in the university that advertised the position), while the share decreases to 24 percent for those who qualified in the first two rounds of ASN. Such difference could be due to the shorter time span since the results were announced of the first rounds of the ASN.^{Footnote 11}

Information on the candidates before the 2010 reform was obtained from the final reports compiled at the conclusion of each competition by the selection committee members. Given data availability constraints, we were able to collect data only on 23 competitions, constituting 53 percent of all competitions for full professorship in economics in the 2001-2008 period. In particular, we consider 12 selections held in 2008 (66 percent of all competitions held in 2008) and 11 competitions held in the years 2001-2003 (44 percent of the total in the selected years).^{Footnote 12} For the ASNs, data about all candidates, such as their CVs, lists of publications, bibliometric thresholds values, and final result (if they qualified or not) has been freely available online on a specific ASN dedicated for a period of 90 days from the end of each qualification.

In total, we collected data on the profiles of 522 candidates, of which 22.8 percent were women. As shown in Table 2, in our sample 177 economists were candidates for a full professorship before the reform (of which 24.3 percent were women), and 345 after it (22.1 percent women).

Table 2. Gender differences among successful candidates to full professorship in pre- and post-reform competitions

For all candidates we collected all publications recorded in EconLit, a rich database maintained by the American Economic Association, in the ten years before the competition in which they took part. From this source we gathered metadata on each publication, such as the keywords, JEL codes,^{Footnote 13} the number of co-authors, the type of publication and the publication outlet (journals and publishers). We use EconLit because it is one of the most complete databases for journal articles, book chapters, books and working papers in all fields of economic research. It has a wide geographical coverage and it is updated monthly since 1969. For all candidates to the 2012 and 2013 rounds of the ASN, we integrated the information from EconLit with that provided in the CVs published on the ASN website.

As shown in Table 2, women experience greater difficulties in reaching top positions than men. They represent 22 percent of successful candidates in the pre-reform competitions, and even fewer, only 15.6 percent, in the ASN.^{Footnote 14} Looking at the success rate (number of winners over total candidates) by sex, we also find that the gender gap increased between the two periods.

It is not possible to attribute this deterioration in women’s success rates and gender gaps to the changed qualification rules only. Indeed, between the pre-2008 and the post-2010 periods fiscal policy in Italy became more restrictive, implying inter alia relevant cuts to the universities’ budgets. Accordingly, high-ranking positions became more scarce and, as pointed out, e.g. by De Paola et al. Reference De Paola, Ponzo and Scoppa2018, gender discrimination appears to be more pronounced when the available slots are more limited. However, extant literature reviewed in the previous section suggests that qualification rules may have uniquely impacted on the typical profile of men and women winners. This may both be interesting per se, and provide hints on the role of qualification criteria in shaping the observed outcomes.

5. The changing profile of successful candidates for full professorship

As shown in Table 2, candidates who qualified for full professorship tended to be older in the new system. In the pre-reform competitions, the median age for women who were able to break the glass ceiling was 43 years old (vis-à-vis 42.5 for men) while in 2012-2013 the median age was 47 years for women and 45 for men. On average, women succeeded in achieving full-professorship almost two full years after their male colleagues.

Over time, the scientific production of successful candidates has significantly increased. Since Cole and Zuckerman 1984, gender differences in productivity among academics is usually reported in the literature as a key element influencing gender inequality in academic careers (Levin and Stephan Reference Levin and Stephan1998; Xie and Shauman Reference Xie and Shauman2003; Fox Reference Fox2005; Leahey Reference Leahey2006; Fox et al. Reference Fox, Carolyn and Bao2011). As reported by Abramo et al. Reference Abramo, D’Angelo and Caprasecca2009 for the natural and life sciences, we find evidence of a progressive reduction of the gender productivity gap among Italian researchers. As Table 2 shows, this is true in economics as well: over the years the median number of publications has increased more for women than for men. In the ASN, the median is 24 publications for women and 27 for men.

We then integrate information on the sheer productivity with a measure of visibility, considering the H index.^{Footnote 15} We find a reduction in the gender gap among the successful candidates: women economists who succeeded in the ASN have the same or even higher H indexes than their male colleagues.

In sum, women economists appear to show an increasing ability to adapt to the “rules of the game” to succeed in academic competitions, by increasing their visibility both in terms of scientific production and citations. This higher adaptation implies a progressive homogenization in their choice of types of publication, with the growth of journal articles and working papers (see Table 2) at the expense of books and chapters in collective volumes. Only 2 percent of the publications by ASN candidates are monographs and 13 percent are collective volume articles, significantly lower than the figures observed for competitions held before 2012. The decrease in productivity in terms of books and book chapters is higher for women than for women, and this again may suggest that women suffer from a higher pressure to conform to a standard that could make them more visible in terms of the bibliometric indicators used for research evaluation.

Finally, considering the content of publications, we focus on research topics and consider the JEL codes recorded for each publication in EconLit.^{Footnote 16} We consider the first section of each JEL (henceforth “aggregate JEL”), represented by a single letter, thus distinguishing 20 major sub-fields of economics.

Figure 1 reports the aggregate JEL codes of all publications authored by men and women economists in the two periods considered. It shows a common trend among the candidates to the ASN towards a sharp reduction of articles on the History of Economic Thought, Methodology, and Heterodox Approaches (denoted by the JEL code B). This contraction is more evident for women; in fact, women exhibit the highest variability in research preferences between pre- and post-reform competitions. Those women economists who passed the national qualification for full professorship exhibit a smaller gap in research field preferences with respect to their male colleagues, with a higher production in Macroeconomics and Monetary Economics (E), Labor and Demographic Economics (J), and Economic Development and Innovation (O); conversely, they authored fewer publications in Public Economics (H), Industrial Organization (L), and Economic History (N). The main exception to this trend is women winners’ higher interest in Financial Economics.

Figure 1. Gender differences in research fields: distribution of publications’ JEL codes pre-2012 and 2012-2013.

To quantify the degree of heterogeneity in the research fields of winning candidates before and after the reform, we compute a Duncan segregation index. The index is defined as: ${S_f} = {1\over 2}\mathop \sum \nolimits_{i = 1}^n \left| {{m_i} - {f_i}} \right|$ , where m _i and f _i respectively represent the percentage of men and women in a particular field (denoted by i). The Duncan index reports the proportion of women who would have to swap fields with a person of the other sex in order for both sexes to be represented in each field exactly in proportion to their representation in the whole sample. Therefore, a value of 0 indicates that the distribution of men and women across fields is the same, while a value of 100 percent would imply that women and men are active in completely different research fields. In our data, the Duncan index amounts to 39.7 percent before the reform of the university system, and 21.5 percent thereafter.

Overall, these results clearly mark a trend: the choices of research topics and publication venues by those who want to reach the top of the academic career tend to converge towards a uniform profile, and there is a tendency to homologate to the standardized, highly visible profile of the (male-like) successful candidate.

6. Determinants of success in breaking the glass ceiling before and after the reform of the university system

The descriptive statistics described in the previous section suggest a higher tendency/pressure in the recent years for women to conform to a highly visible profile in terms of bibliometric indexes. In this section, we aim to test this hypothesis by means of a multivariate analysis, inquiring whether diversity both in terms of researchers’ identity, and of research methodology and themes proved an asset or a liability in the competitions. In other words, we try to establish whether the introduction of the new recruitment system, heavily based on bibliometrics, tends to make the glass ceiling more unbreakable for women and the less mainstream economists.

We run probit regressions separately examining the determinants of the probability of successfully qualifying in a competition to full professorship held either before or after the 2010 reform. The determinants of success are assumed to be individual characteristics such as gender and age, and a set of variables that characterize the candidates’ publications in terms of quantity, type, impact and content. Specifically, we consider the production rate by looking at the total number of publications in the ten years before the competition, as well as what proportion of publications is composed of journal articles, books and book chapters. We also study candidates’ academic production in terms of:

1. (i) Visibility: measured by the H index on Google Scholar for each candidate in the ten years before the competition.

2. (ii) Co-authorship: the mean number of co-authors of each candidate’s publications.

3. (iii) Variety of interests: the average number of different JEL codes used to describe each candidate’s articles.

4. (iv) Intellectual diversity: we identify economists that research outside the mainstream, considering the cases of heterodox economists and historians of economic thought.

For the last characteristic, we use the identification of heterodox economists in Italy developed in Corsi et al. Reference Corsi, D’Ippoliti and Zacchia2018, while for historians of economic thought we use the relevant JEL codes.^{Footnote 17}

As shown in Table 3, we find that the determinants of success in the competitions held before the reform are often different with respect to those in the ASN. Preliminarily, it should be noted that most of the variables considered do not appear to contribute to understanding the determinants of qualifying as full professor before the reform. An intercept-only model (analogous to guessing an individual’s chances of success only knowing the average success rate) would allow correctly predicting the outcome of roughly 77 percent of the pre-reform sample. This figure is not significantly increased when adding observable characteristics of the individual or of her publications, and a Wald test of joint statistical significance of these variables would lead us to reject the hypothesis that they are indeed relevant. The exceptions are a dummy variable denoting if the individual ever published in the history of economic thought, until 2011 positively contributing to her chances of success, and the numbers of monographs she authored, which also exhibits a positive sign. Finally, in certain specifications (both before and after the reform) the number of book chapters authored by an individual could decrease rather than increase her chances to qualify as full professor. Since book chapters attach less prestige than other publication outlets, this finding seem to echo that by an experiment by Powdthavee et al. Reference Powdthavee, Riyanto and Knetsch2017, who found that the inclusion of lower rated journals in a putative candidate’s CV had a statistically significant negative impact on other economists’ assessment of the author’s scientific standing.

Table 3. Determinants of the probability of success for candidates for full professorship, probit estimation: pre-2012 and 2012-2013 competition

Notes: heteroskedasticity-robust standard errors in parentheses. *** p < 0.01, ** p < 0.05, * p < 0.1

In contrast, in the ASN rounds both individual and publication-level characteristics are important predictors of a candidate’s chances of success. This first contrast may lend support to the official position that inspired the reform, whereby success in the previous system was related to a complex of career and individual characteristics often difficult to see and understand for outsiders, whereas the new system increases transparency of rules and outcomes.

However, a major sign of this increased transparency is the negative and statistically significant coefficients of age and the woman dummy variable. This implies that irrespective of all other individual and publication characteristics for which we control, in the new post-reform system being older and/or a woman is associated to an unexplained residual disadvantage in the competition for qualification as full professor. Effectively, the observed lower success rate for women under the new rules can only partly be explained by their lower productivity or the visibility of their publications, and a sheer glass-ceiling effect seems to constrain their chances of success. This result does not seem to be limited to economics, as Marini and Meschitti Reference Marini and Meschitti2018 reach similar conclusions for other disciplines within Italy’s ASN.

Concerning publication characteristics, the results of the probit regressions show that the relevance of being the author of a book has changed drastically in the two systems: before the 2010 reform, writing books had a significantly positive effect, while it turns out to have a significantly negative effect in the 2012-2013 ASN. This is compatible with evidence for other countries and disciplines, that the introduction of bibliometric methods entails a significant bias in favor of journal articles, at the expense of books in particular. However, in the case of Italy’s ASN the sheer number of journal articles does not seem to exert a significantly positive impact on candidates’ chances of qualifying: this is most likely a consequence of the highly different weight attributed to journals in the A-list (Corsi et al., Reference Corsi, D’Ippoliti and Zacchia2018).

In the ASN system, other variables that typically correlate with higher bibliometric indexes become relevant to successfully qualify to full professorship: visibility, in terms of the H index, and writing with many co-authors acquire a significant positive effect after the 2010 reforms. However, it may be even more relevant to note that with the introduction of the ASN not only the quantity and visibility of publications become relevant, but their content and method too. Thus, having a large spectrum of research interests or a heterodox approach penalized candidates to the ASN. Having some interest in the history of discipline, which played a positive role until 2010, ceased to exert any impact thereafter. There is thus a real risk that selecting specific bibliometric indicators, by imposing a single set of disciplinary values embedded in these indicators, may consciously or not imply the imposition of distinct academic and cultural aims and specifically the imposition of a univocal supposedly right way of doing economics. Being associated to the recruitment and promotion of research staff, this is affecting and will shape the future generation of academic economists in the country, suppressing intellectual diversity within the discipline.

Furthermore, the finding that overall bibliometric indexes of productivity and visibility correlate with candidates’ chances to qualify as full professors has a gender dimension too. Indeed, considering only publication-level characteristics and ignoring candidates’ demographics, due to women’s lower bibliometric indicators the probit model (reported in column 5 of Table 3) predicts a 46.4 percent average probability to qualify for male candidates, and a lower one, 43.2 percent, for female candidates. If we run the same estimation separately for men and women, further divergence arises as men would have a predicted probability to qualify of 49.8 percent on average, whereas women of 31.3 percent.^{Footnote 18} Therefore, not only do women exhibit lower bibliometric indicators, but in the context of the peer review assessment by the selection committee for economics, women’s performance in terms of these indicators was even less highly prized.

In conclusion, the mixed “informed peer review” system of the ASN system does not allow us to precisely disentangle how much of the observed glass ceiling could be due to the use of bibliometric indicators and how much to peer review. However, evidence on economics allows us to infer that both methods in fact conduced to a disadvantage for women candidates.