Is the reviewer able to assess an interdisciplinary paper?

Before agreeing to assess a manuscript, reviewers should ask themselves whether they feel competent enough to do so.Reference Goldbeck-Wood^7, Reference Rosenbaum¹⁵ This is likely to be a thorny issue for interdisciplinary submissions, particularly for innovative approaches. In this case, few scientists will be equally familiar with all of the fields spanned by the article. Thus, many reviewers will only be able to provide a partial assessment from the point of view of their own discipline or methodological approach.Reference Hoppin^8, Reference Lee^14, Reference De Schutter¹⁶ In this situation, the reviewer can still provide a useful report, but should make it clear in the confidential comments to the editorial team that she has commented on, for example, the mathematical methodology, and left, for example, the biological framework to be checked by someone else.

There are further reasons to accept to review an interdisciplinary paper even if the reviewer is not an active researcher in all the relevant fields. One reason is to get to know about new interdisciplinary developments in advance and with much more depth than just by quickly scanning the abstract of the published paper. Secondly, reviewing an interdisciplinary paper is an exercise in thinking out of the box and is thus good practice in sharpening one’s own creative thinking. Moreover, the editor might be looking for an opinion from outside the interdisciplinary field and thus for a more detached view on the analysis. Although being few in number, specialists of, say, the economics of space tourism, will tend to be enthusiastic about the topic and eager to promote it. Paraphrasing Smith,Reference Smith¹⁰ this is why an editor who wants to debunk a paper on constructivist alchemy sends it to a mathematical chemist, not to an alchemist, a constructivist or a combination of the two.

In general, assessing interdisciplinary papers requires personal characteristics that also favour the pursuit of interdisciplinary research (e.g. open-mindedness, flexibility, curiosity, lateral thinking).Reference Bruce, Lyall, Tait and Williams¹ Even if the reviewer is not an active researcher in the particular interdisciplinary field of the submission, it can help if she has already worked in an interdisciplinary way, even if in other fields. Although this remains to be investigated and exceptions will be common, it could be that, other things being equal, speaking more than one language and/or having lived in different countries, by providing more opportunities for intercultural experiences, could be associated with better skills in assessing interdisciplinary research.Reference Lasater, Ray, Driever, Rosenfeld and Bradley¹⁷

Is the paper really interdisciplinary?

If the submission claims to be interdisciplinary (either explicitly in title, abstract and/or text, or implicitly, by having been submitted to a journal publishing interdisciplinary papers), then it is important that the reviewer makes sure that this is indeed the case. Since interdisciplinarity is generally seen as a good thing,Reference Klein¹⁸ it could be surreptitiously used by authors to increase their chances of acceptance.

A reviewer should thus ask: is this paper really making use of the theories and/or methods of several fields? Is it integrating them or merely juxtaposing them? In the latter case, the analysis would probably be better described as multi- rather than inter-disciplinary, although use of these terms is probably not consistent outside of the literature on multi- and inter-disciplinarity.Reference Ramadier^19, Reference Booth²⁰ Therefore, reviewers should check that the papers correctly identify their contribution as being inter-, multi-, or even trans-disciplinary. Should the article transcend the fields by creating a hitherto unused approach, it might be not just inter-, but trans-disciplinary.Reference Bruce, Lyall, Tait and Williams¹ This is not the case if the authors only use the methods of one field to address a problem normally tackled in another discipline. Sometimes interdisciplinary approaches become so common that they create a new field (e.g. bio-informatics, molecular ecology, network epidemiology). One of the signs that these have become established boundary fields is when articles on these topics stop claiming explicitly to be interdisciplinary and journals specialized in the new interdisciplinary fields appear.

Is the interdisciplinary paper sound?

Whether or not there is an explicit claim of interdisciplinarity, the judgement of whether an interdisciplinary manuscript should be published or not has to rest on other features. A poorly executed analysis needs revision even if it is a classic example of interdisciplinary research. This may, however, turn out not to be trivial to detect: in newly forming interdisciplinary fields, the boundaries between rigorous and poor science may become fuzzy. After all, this is the point of interdisciplinary research: questioning established practices and introducing new ways of looking at scientific problems.

How to assess scientific integrity in these cases? Questions such as whether the statistics follow the best current practices, the results appear to be plausible, and the authors did what they claim to have doneReference Smith¹⁰ remain relevant for interdisciplinary papers. Different scientific fields may have different standards as to what makes a good contribution to the literature,Reference De Schutter^16, Reference Klein²¹ but there are general features (e.g. a long-awaited solution to an unsolved problem, comprehensive empirical evidence/theoretical framework, balanced discussion of previous relevant literature) which are shared among most fields. To gauge the scientific soundness, a reviewer may also make use of ancillary features of the article, such as redundancy in the text, quality of the figures, presence/absence of motivation and evaluation for the study. However, can the quality of the overall reasoning of a manuscript be used to infer the soundness of an interdisciplinary approach? Logical leaps, muddled sentences, inconsistencies are all unfortunate features of a submission, but may not be compelling evidence against performing and reporting an innovative analysis, as they can all be avoided in a revised manuscript.

On the other hand, given that journals are nowadays flooded with submissions,Reference Goldbeck-Wood^7, Reference Hochberg, Chase, Gotelli, Hastings and Naeem²² features such as poor use of statistical tests, failure to motivate the choice of a particular technique and absence of a framework of related studies justifying the question investigated can be fatal even to the most original interdisciplinary study. If an article is clearly interdisciplinary and sound, so that it deserves publication after problems in the presentation have been solved, then reviewers can be very helpful by pointing out, for example, discrepancies between abstract and text, lack of clarity of particular sentences, alternative explanations for the results or unwarranted conclusions.Reference Bourne and Korngreen⁹ In the following, we discuss in the context of interdisciplinary contributions some key criteria generally used in the assessment of mono-disciplinary submissions.

Is the interdisciplinary paper original?

Novel hypotheses, methodology, results and/or conclusions are important features of submissions, and may make such work likely to be influential, if coupled with addressing a non-trivial issue. Interdisciplinary papers are likely to be original, but are not necessarily so. There can be interdisciplinary papers that repeat or confirm interdisciplinary analyses already present in the literature, whether the authors knew about these previous studies or not. A reviewer of an interdisciplinary submission is thus well advised to check the literature for analogous studies. In some fields, making sure that similar analyses have not already been done can entail a literature search spanning decades. For novel interdisciplinary approaches, reviewers have to be particularly up-to-date with recent papers, which can mean having to scan the online lists of papers in press of relevant journals, since these papers will tend not yet to be present in databases. It can also be helpful looking at previous papers from the same group of authors.Reference Hoppin⁸

Although one can never be sure that something has not already been done, there are now tools that enable one to search the scientific literature in ways which were undreamt of only a few decades ago (Web of Science, ISI Proceedings, Medline, Scopus, Google Scholar, DBLP).Reference Müller, Kenny and Sternberg^23–Reference Lufrano and Staiti²⁸ A diligent reviewer will thus make use of more than one database. It might be easier to scan the literature in certain interdisciplinary fields if these fields have just been launched. However, in some cases, similar approaches may exist that are not called the same thing, so that reviewers of interdisciplinary manuscripts need to be creative with the keywords they use. Despite the growing proportion of scientific studies published in English,Reference Valkimadi, Karageorgopoulos, Vliagoftis and Falagas²⁹ in certain cases it might still be worthwhile searching in the scientific literature of other languages.

Is the interdisciplinary paper likely to inspire further research?

It might be years before it becomes clear whether a paper was influential or not, but journals are still interested in publishing papers that are likely to inspire further research. This is because of the tyranny of the impact factor, which provides a positive feedback between influential articles in a given journal and more influential articles in later volumes of the same journal. Original interdisciplinary papers are likely to be influential, as they open up new perspectives and ways of doing science.

This can make journals wishing to increase their standing eager to publish interdisciplinary submissions. At the same time, given the high rejection rates of many journals and the pressure to publish, authors may try to exploit this situation by exaggerating their claims of interdisciplinarity. A reviewer thus needs to double-check carefully that a paper which claims to be interdisciplinary really is so, not just for the sake of assessing the correctness of such a claim, but also because interdisciplinarity is likely to inspire further research. Of course, this may not be sufficient, as influential papers also need to be sound, original and address a problem on which many people are likely to go on working.

Just as for non-interdisciplinary submissions, the likelihood that an interdisciplinary paper will be influential depends on whether it presents major results or not. Elegant and technically correct but useless interdisciplinary work; positive, but minor, multidisciplinary contributions to knowledge; and neither elegant nor useful, but not actually wrong transdisciplinary analyses are all typically less likely to produce an outstanding and long-remembered publication with enthusiastic support from reviewers.Reference Smith¹⁰ A problem with assessing interdisciplinary submissions, having at the back of the mind the likelihood that they will be influential, can be that this might produce bias in the literature. It is known that submissions that report the presence of significant differences are more likely to be published and to be cited than those that fail to do so.Reference Begg and Berlin^30, Reference Easterbrook, Berlin, Gopalan and Matthews³¹ If reviewers judge more favourably exciting (as opposed to unsurprising) interdisciplinary papers in the same way as reviewers of non-interdisciplinary papers appear to do,Reference Jasienski³² the file drawer problem (positive results are more likely to be published than negative ones) will also be replicated for the interdisciplinary literature.

Is the interdisciplinary paper of broad interest?

A novel and influential paper is in many cases a paper of broad interest. This is because it will be read (and cited) widely. However, a paper may be influential in a certain community without necessarily becoming known in other fields or by the public at large. Interdisciplinary papers have the advantage of spanning more than one discipline, thus multiplying the potential number of readers. Still, interdisciplinarity does not necessarily guarantee broad interest. A paper in, for example, ethnobotany may be specialized for readers of that interdisciplinary field and of little appeal even to pure ethnologists or botanists.

For an interdisciplinary paper to be of broad interest it needs to be required reading to researchers in the two or more parent disciplines, and to be of general interest to other scientists. Today, this is a tough requirement, given the magnitude of the scientific literature produced yearly in any field of inquiry. Many journals of high reputation now routinely (and not very imaginatively) motivate their rejections with the too narrow interest of the submission. Interdisciplinarity can help in broadening the audience of a paper, but may still not be enough to justify acceptance in such journals. As interdisciplinarity becomes more common, it inevitably loses some of its sparkle. Reviewers for journals with high rejection rates thus need to watch out whether interdisciplinarity is accompanied by other features such as novelty, appropriateness and generality of the approach. They should also try to predict whether the paper would appeal to a readership found across the union of the multiple fields as opposed to the intersection.

Is the paper appropriate for the journal?

There is a feeling among scholars of interdisciplinarity that studies at the boundary of scientific fields have more trouble in finding an appropriate journal than do mono-disciplinary submissions.Reference Fry³³ This is not so much the case for established interdisciplinary fields such as astrobiology, bioinformatics, conservation biogeography or clinical biochemistry, but can be so for those recently formed or merging perspectives from distantly related fields (e.g. remote sensing and linguistic anthropology). In the latter case, scientists may find it hard to publish the results of their interdisciplinary collaborations because a journal for their new field does not yet exist and the journals of the two or more parent disciplines do not recognize such articles as appropriate.

Reviewers of interdisciplinary submissions should try to balance the requirements of specialized journals with the advantages provided by new perspectives and cooperation with other fields. If an interdisciplinary contribution is original, well-researched and likely to lead to publicity in the press, publication should be recommended even if the journal to which it was submitted does not explicitly welcome studies merging perspectives with neighbouring or distant fields. At the same time, a submission that claims to be at the interface between, for example, the life and the physical sciences is only appropriate for a journal that publishes such submissions if it contains both perspectives. Reviewers should ensure that such a contribution is well balanced, with no predominance of one of the two perspectives.

Is the paper understandable?

Interdisciplinary articles are at risk of not being understood by practitioners of the two (or more) parent fields, as they integrate new methods and concepts. After all, one of the reasons most scientists tend to remain active in a certain field rather than jumping from, for example, software engineering to renewable energy policy or from plant genetic resources to internal medicine is that there is a cost inherent in such career moves. This involves getting to learn new bodies of knowledge with their associated underpinning theories, outstanding issues and shared vocabularies. Interdisciplinary science is a way of building bridges between scientific fields and can thus enhance communication where this was absent or rare. However, hybridization between scientific fields can also create a new jargon inaccessible to the uninitiated.

Reviewers should make sure they understand what is written in an interdisciplinary manuscript. If they cannot follow its vocabulary, chances are high that readers will also not make much out of the published article. Although it is not the task of a reviewer to rewrite a draft for the authors, suggestions as to which parts can be better formulated and what general issues make understanding of a particular interdisciplinary submission difficult are all features of constructive reviews.Reference Hoppin^8, Reference Smith¹⁰ A reviewer coming from one of the two (or more) disciplines involved is obviously in a better position to assess and, if needed, improve the parts of the manuscript relevant to his or her discipline(s). However, that reviewer should still read the remaining parts of the text so as to ensure that at least the gist of it can be understood by a reader not entirely familiar with the subject.

Are the references interdisciplinary?

A balance between the two (or more) parent disciplines also needs to be achieved in terms of the cited references. These should not just reflect one or the other field. Similarly, the references cited in a paper on, for example, emergency medicine and developmental economics should span a similar temporal period for both fields, without being all recent for one and all from the 1970s for the other. In established interdisciplinary fields at the interface between, for example, the mathematical and the biological sciences, this issue may not present itself, as there will be no need to go back to the single disciplines given the availability of previous studies in the interdisciplinary field.

In general, just as for non-interdisciplinary submissions, a reviewer of interdisciplinary articles should comment on whether the authors have made reference to appropriate work, and whether there are too many (or not enough) references. However, interdisciplinary reviewers who wish to maintain their anonymity (e.g. Smith) should take care to avoid suggesting that the authors cite more papers by Smith et al. This can be a problem in the case of interdisciplinary manuscripts if the field is still very young and there have only been a few previous studies using a particular approach. In this case, the recommendation that the paper by Smith should be referred to might not actually indicate that the reviewer of the submitted paper was Smith.