THE CHALLENGE OF RESEARCH ASSESSMENT FOR SCHOLARS IN THE HUMANITIES, SOCIAL SCIENCES AND LAW

The assessment of scholarly publications is an integral part of the scientific process and is one of the primary and pervasive functions in the development of scientific knowledge. Before identifying the main methodological options for evaluating the quality of research outputs in the humanities, social sciences and law, it is useful to specify the variety of situations involved in the research assessment which can be broadly classified into the followingFootnote ¹:

• • Purpose of the evaluation (funding, salaries, recruitment, academic career);

• • Subjects involved in the assessment (individuals, departments, universities, institutions);

• • Evaluation method (peer review, bibliometrics criteria, mixed methods);

• • Objects of evaluation (journal articles, monographs, conference proceedings, patents, other research products…).

In recent years the issue of evaluating research outputs has played a central role in higher education, also driven by public policies implemented by the European Union countries following the Bologna DeclarationFootnote ² in 1999 and the Lisbon StrategyFootnote ³ in 2000. A number of initiatives, both nationally and internationally, have been undertaken and guidelines have been adopted that emerged from the evaluation of research programs at European level. Many European universities are launching initiatives to encourage ‘research excellence’. The goal of the Exzellenzinitiative of German universitiesFootnote ⁴ and the Initiatives d'excellence in French higher educationFootnote ⁵, is to compete with top universities in the world. Similarly, research quality evaluation systems in UK higher education institutions, such as the Research ExcellenceFootnote ⁶Footnote ⁶ Framework – REF (the latest results of which were published in December 2014) and the Standard Evaluation Protocol – SEP in the NetherlandsFootnote ⁷ are significant examples of well-established systems. In particular, the Higher Education Funding Council for England (HEFCE)Footnote ⁸ has created a Steering Group to evaluate the evidence on the potential role of performance metrics in the next instalment of the REF. HEFCE is to review the role of metrics in the assessment and the management of research, by considering how well metrics can be used across different academic disciplines to assess the excellence of research undertaken in the higher education sector. However, the measurement and assessment of research quality still faces some strong opposition from non-bibliometric areas where quantitative methods are not, in a way, directly applicable to humanities, social studies and law.

In fact, the evaluation of research in such areas of study is a particularly delicate matter. This depends on various factors such as the heterogeneity of research studies and scientific production, the specificity of the type of material, the lack of objective data for poor coverage of these disciplines in international databases, different citation styles, number of authors. Added to this, is the consideration that even among individual areas of the humanities, social sciences and law there are profound differences. In particular, the production is very varied and each type of product has its specific weight in each area. It seems that social and legal scientists choose monographs as their preferred material for disseminating new knowledge, while relying on chapters of books – it must be said – to further add items to their CV.

Furthermore, a crucial aspect when it comes to evaluating the humanities, social sciences and, in particular, law concerns the meaning of the concept of ‘internationalization’ that does not mean writing in English nor be cited in international databases: this misleading concept has been criticised by many. The humanities, social sciences and law are inherently linked to national languages, some concepts are even untranslatable and many research areas are, in a sense, strictly ‘local’. The real international feature is to be identified in the composition of the editorial and scientific board, co-authorship, level of prestige and importance of the journal in the field for the community of scholars out of the national context, even if written in a national language. Safeguarding the “biodiversity” by using national languages and heterogeneous bibliographic genres, is very important as the only way to preserve the richness and specificity of humanities, social sciences and law.

At an operative level, studies and practice make use of two methods that are not alternative, but rather they complement each other in the evaluation of humanities, social sciences and legal research.

• • Qualitative methods based on peer review, allows for a detailed analysis of each text, but requires a large number of independent experts with no conflicts of interest. The experts' judgment is strongly influenced by the journal in which the paper is published.

• • Quantitative methods based on bibliometrics. Bibliometric analysis is seen as a practical and even cost-effective pursuit. There are three types of bibliometric indicators: quantity indicators, which measure the productivity of a particular researcher; quality indicators, which measure the performance of a researcher's output; structural indicators, which measure connections between publications, authors, and areas of research.

In addition to these methodologies, the socio-economic impact of scientific research deserves careful consideration. This method of research assessment is based on the assumption that the socio-economic influence of research has to be analysed and understood as well, in order to make a proper evaluation of the research itself. In fact, this evaluation method implies the involvement of stakeholders who are interested in the implementation of the research, using methods other than those internal to the scientific and scholarly community. In particular, these methods require that stakeholders are involved both in the decision-making of research evaluation and in the process of the allocation of fundingFootnote ⁹. Over the past few years, as part of the national programs dedicated to research assessment, increasingly structured methodologies have been developed for the evaluation of the socio-economic impact of research. A first example of such an evaluation process is found in the Dutch Standard Evaluation Protocol, in which the number of publications is no longer an independent criterion for the performance of research groups. Like the British, the Dutch are putting much more emphasis on societal relevance than in the previous assessment protocols. The Dutch Protocol takes into account the following elements:

1. 1. The quality of research in society: it mainly concerns policies and effects of the interaction of research with stakeholders.

2. 2. The impact of research in society: it regards the effects produced on specific procedures in society (for example, new protocols, regulations, laws), which lead to changes in the behaviour of specific stakeholders and institutions.

3. 3. Research exploitation: this refers to research activities that lead to the realization of results available for products, processes and services.

Of course, the assessment of the socio-economic impact of research is not without its criticsFootnote ¹⁰. In fact, it is very difficult, as regards basic research, to identify such impact. This type of research can give a return only decades later, while applied research, especially industrial research, can be immediately evaluated for its achieved results and socio-economic effects. Another criticism comes from those who argue that the impact assessment is highly detrimental to the humanities, social sciences and law where the socio-economic effects are often not easily identifiable. The main risk in these areas is represented by a substantial cut in research funding. These issues underlying the assessment of the socio-economic impact of research, however, should not be an obstacle to the introduction of new evaluation practices, oriented to the study of the societal impact of research.

The university system is capable of legitimising itself and creating a strong consensus in society, if it can open up to the outside world and prove that through scientific research the society in general, and the economic and production system, can develop. For these reasons, it becomes increasingly important to promote research evaluation methodologies that allow the results achieved to be recognised even outside of academia. The experiences made in the anglo-saxon world (ie. the REF in UKFootnote ¹¹ and the Excellence in Research for Australia – ERAFootnote ¹²) show that in recent years research evaluation is moving towards innovative practices involving, in an ever wider way, stakeholders and society as a whole. The challenge that research evaluation's systems will face in the near future is likely to improve the tools for assessing the socio-economic and cultural impact of research, limiting the inconsistencies and weaknesses of the system.

In summary then, a proper practice of research evaluation should always be based on an appropriate combination of quantitative indicators (including the bibliometric ones), analysis of peers (peer review) and the socio-economic impact of research. These three elements together represent the core functions of the process of growth and dissemination of scientific research. Peer review seems to confirm itself as the leading tool for effective evaluation in humanities, social sciences and law, while quantitative data can be used as a corollary to a qualitative approach in the informed peer review, thus providing experts who analyse the contributions also with reference profiles. Nevertheless, it is worth emphasizing that there are objective difficulties due to the impossibility of comparing judgments, as well as difficulties connected to niche disciplines which are represented by a reduced number of scholars and therefore of experts. Therefore, the importance of transparency of purpose, criteria and results is fundamental, keeping in mind that evaluation is an in-progress, improvable and perfectible taskFootnote ¹³.

THE EVALUATION OF LEGAL SCIENCE

This area of law has always refused evaluation practices based on quantitative methods, such as bibliometrics, or the use of quantitative indices based on citation analysis for assessing research quality. In fact, legal science has been traditionally assessed within national legal systems, according to qualitative methods rather than bibliometrics, outside of any international contextFootnote ¹⁴. Under the influence of hard sciences, researchers in this area are now ‘forced’ to follow the evaluation model of the latter. Listed below are the specific features of this discipline that make it a suitable area to study and analyse not only for its complexity, but also for its strong interdisciplinarity.

• • Variety of users

It is often difficult to draw a dividing line between the publications focused on research, training, practice.

• • Coverage

Legal research activity addresses issues concerning local, national, European and international law.

• • Language of publication

While it is assumed that English is the lingua franca for the disciplines of Science, Technology, & Medicine (STM), in most social science domains and in the law area, publishing in languages other than English is quite commonFootnote ^15,Footnote ¹⁶. As a matter of fact the publishing language of law is mostly the national language. Summaries in another language (even in a vernacular language like English) are rare, except in legal journals dealing with European and international law.

• • Lack of adequate bibliographic databases

Legal publications are rarely present in international bibliographic citation databases (Web of Science, Scopus, European Reference Index for the Humanities and the Social Sciences – ERIH PLUS, Liste des revues en sciences humaines et sociales de l'Agence d'évaluation de la recherche et de l'enseignement supérieur – AERES, JournalBase …) due to the use of the national language, absence of an adequate auditing procedure and, at times, poor international relevance of the topics covered.

• • Legal scientific knowledge ages in time

While in the STM domains scientific knowledge become obsolete within 3 to 4 years, in the law domain contributions and in particular books do influence the field sometimes for decadesFootnote ¹⁷.

• • Single authorship

Legal scholars often conduct research individually or publish in small teamsFootnote ^18,Footnote ¹⁹.

• • Plurality of bibliographic genres

In addition to monographs and journal articles, research results can be published in the form of annotated judgements, comments on legislation and reviews. There is also a large number of reprints of monographs and a steady increase of blogs devoted to specific areas of the lawFootnote ²⁰.

• • A strong link with practice

Many scientific journals host contributions from practitioners (ie. from judges, professionals, civil servants) or articles on operative issues, that in other scientific disciplines are likely to have a different destination.

• • Meaning of referencing

Legal science lacks a tradition of a systematic citation analysis. Most times legal researchers look for completeness in references by giving an overview of all contributions in the field (articles, contributions in books, books), with no distinction as to quality. Therefore, references are not necessarily representative of a scientific contribution's quality, but also depend on other factors like availability of sources by the researcher and the presence of the contribution in legal databases. The absence of received citations is not an indication of the absence of influence and /or the quality. References serve primarily to demonstrate the amount of reading done, rather than the quality of cited material. Most of the time references are an indicator of diligence in research and therefore also a way to assess its goodness; in this context, citing serves mainly to those who cite, not to those who are cited. This procedure alters the meaning of the “citation unit” which is the foundation of bibliometrics in the hard sciences. Moreover, the significance of citation, specifically in the legal field, is controversial. There can be different classifications of citation according to its functions (positive, informative, negative). This implies a difficulty in interpreting and weighing these different kinds of citation.

Despite these characteristics that make the law difficult to assess, it is worth studying whether it is possible to combine, in various ways, bibliometric indicators and the use of peer review in order to assess legal research qualityFootnote ^21,Footnote ²².

Both these methodologies offer attempts towards a legal evaluation approach that is, in a way, comparable with that of the hard sciences. At operative level in many European countries the practice of drawing up lists of scientific law journals is well established. In the United StatesFootnote ²³ and AustraliaFootnote ²⁴ effective ranking systems of scientific law journals have been set up over the years. These tools are of great help in the assessment of legal results and can be used to monitor research on a more continuous basis and to predict the outcome of future peer-reviewed assessments. However, a method based primarily, or exclusively, on lists of journals or on publishing houses is likely to be identified as a static (very difficult to update) and rigid (as predetermined ex ante) system. There is a danger that the ranking of journals and publishers can be decided not so much on the actual value and success of the periodicals, but on the power of scientific societies.

IN PARTICULAR: THE LEGAL MONOGRAPH

In the field of legal research, monographs play a dominant role as complex products resulting from a prolonged research activity. In comparison to law journals, monographs give references to a broader set of materials: not only to journal articles, but monographs, unpublished documents and grey literature. Furthermore, they frequently make reference to works of other disciplines, not necessarily to works in the same field. Publishing monographs is not an extravagant choice; indeed it is a necessary choice that meets the requirements of legal scientists. Monographs can adequately describe the context under study, analyse all the variables, giving reference to all the documents that properly provide evidence of what is asserted by the authorFootnote ²⁵. Therefore, a correct and appropriate evaluation process in law area of study cannot ignore this kind of publication. As in other non-bibliometric disciplines, the assessment of books in the field of law is a very difficult activity, and although the best way to assess the quality of a work would require its individual reading, unfortunately this method is impractical, both in terms of cost and timeFootnote ²⁶.

Evaluating legal monographs by creating lists of different types of books, publishers or series is much more difficult than assessing the quality of journalsFootnote ²⁷. This is due to a number of biases in the evaluation procedure, such as the presence at the same publisher house, of ‘series’ of different weight and prestige, the extreme specialization of some works that, despite their value, have no place in major publishing houses. Unlike scientific journals where qualitative methods (such as lists of quality reviews) are used, for monographs no structured evaluation experience of legal research quality has yet been realised.

Nowadays in the major Western countries, there is no experience of classification of publishers and of monographs in the field of law. There are many reasons for this such as the impossibility of creating a classification of all the publishing houses of the world. In this scenario, adequate criteria that can support the evaluation of legal monographs would be highly recommended. Such criteria should be based on the combination of quantitative methods (for example, the citation counts) and of a uniform classification based on series or on some kind of ranking of publishing houses. Added to this, some relevance should be given to process indicators (prestige of the scientific committee of a series or journal ...), product indicators (reviews in international and national journals) and dissemination indicators (availability in libraries).

In the effort to face some of these problems, an Italian research project on the role of legal monographs has been financed by ANVUR (National Agency for the Evaluation of Universities and Research Institutes)Footnote ²⁸. The project, started in February 2015, aims at trying to define uniform criteria for the assessment of research monographs in the field of law by raising awareness among the Italian legal scientific community, making the evaluation process as impartial as possible. In particular, the project aims at:

1. 1. analysing the evolution of the forms of publication in the legal field;

2. 2. establishing the variables and indicators that have a larger consensus among legal scientists to assess the quality of scientific books;

3. 3. reviewing the international initiatives in the field;

To achieve these objectives, a survey, both at national and international level, will be conducted to explore the quality criteria that legal scientists consider relevant for the assessment of monographs. The international survey will be conducted with the support of the League of European Research Universities (LERU)Footnote ³⁰. Furthermore, a comparative analysis will be carried out of the experiences of a number of European countries (France, UK, The Netherlands) with particular attention to available lists and classification of publishers, types of monographs, and weighting methodologies according to the type of contribution. The final result will be a set of best practices designed to help in the assessment of Italian legal monographs to be proposed to the national legal academic community.

THE CONTRIBUTION OF OPEN ACCESS TO RESEARCH EVALUATION

The European Commission has established Open Access (OA) as a priority for the European Research Area (COM(2012) 392)Footnote ³¹, adopting it as a principle. Recently, it has extended and reinforced its OA policy by requiring each beneficiary of the current EU Framework Programme for Research and Innovation (Horizon 2020 - H2020), to ensure OA to all peer-reviewed scientific publications resulting from funded researchFootnote ³². Moreover, in the framework of the European Commission's Digital Agenda for EuropeFootnote ³³, and the signature of a Memorandum of Understanding (MoU)Footnote ³⁴ between the European Commission and the European University Association (EUA)Footnote ³⁵, the EC has committed EUA to encourage universities to implement OA policies at institutional level also for the purpose of research evaluationFootnote ³⁶. In such a European strategic framework, the role of OA, as a system of production, dissemination and use of research results, is decisive in the research evaluation procedure.

The tangible benefits of OA policies are mainly related to the fact that self-archiving of the full-text of research outputs can support the process of evaluation of scientific research products allowing a complete control over intellectual production as a whole. Another benefit concerns increased visibility of the research outputs; disseminating works using OA gives maximum impact to the intellectual activity. Research results are available in advance of publication, the prestige and the number of citations as well as funding opportunities can rise accordingly. The more the work is retrieved, the more it may be read, therefore getting multiple citations as compared to the more limiting print or electronic formats available on a publisher's websiteFootnote ³⁷. An article published in an open access repository immediately after its creation, can speed up the process of dissemination of knowledge and, therefore, the relevant citation receivedFootnote ³⁸. Moreover, a paper published in a journal often represents the end-point of a communication chain which started, perhaps from a speech at a conference. Open access repositories allow for the exploitation of all stages of this chain, the growth of the reputation of the author (with the benefits of promotional opportunities for the author themselves!) and the future impact of the contribution.

There are many open access projects that support the research assessment process. The most important is undoubtedly arXivFootnote ³⁹, an open e-print archive with over 100,000 articles in physics, 10,000 in mathematics, and 1,000 in computer science. Similar experiences are the Social Science Research Network (SSRN)Footnote ⁴⁰ devoted to the rapid dissemination of scholarly research in the social sciences and humanities and PhilSci-ArchiveFootnote ⁴¹, an electronic repository that publishes pre-print papers in the field of philosophy of science. Another repository in a different subject is the information network Research Papers in Economics (RePEc)Footnote ⁴², which offers working papers, books and journal articles made available by the Munich Personal RePEc Archive (MPRA)Footnote ⁴³ and by publishers involved in the project.

However, in the specific area of law there are no similar experiences for OA. Although the Gold and Green OA routes are consistently growing in specific fields of law (i.e. international and environmental law), nonetheless OA is still complementary to traditional legal publishing. The practice of sharing research data is still at an early stage and commercial legal oligopolies of scholarly communication are so far quite powerfulFootnote ⁴⁴. To promote and encourage the evolution of OA for advanced scholarly communication a real innovation in terms of the assessment practices of legal scienceFootnote ⁴⁵ is needed; also adopting mechanisms of assessment already effective in some areas of research such as the ‘open control’ approach. This consists of ‘exposing’ manuscripts on the internet for a given time, with the possibility of them being commented on by all interested readers, before a decision is made by the editorial board to publish them.