Academic life has always been concerned with evaluation and the creation of hierarchies among the good, the better, and the best – students, teachers, researchers, publications, institutions. Even before the invention of grades, examinations, or the research university, students flocked to Plato's Academy in ancient Athens, Abelard's logic classes in medieval Paris or Vesalius's anatomy demonstrations in Renaissance Padua, drawn thither by intellectual reputation. Yet many scholars and scientists around the globe believe themselves to be subject to a burden of evaluation that outstrips precedent, both as judges and judged: national and international university rankings, pressure from politicians in state-funded educational systems to prove that tax-payers are getting value for their money, the availability of more grant money from more sources, the growing number of students who apply to some form of post-university and post-doctoral training have all conspired to multiply demands for letters of recommendation, referee reports, promotion reviews, and service on grant panels. At the same time, suspicions about how well academic evaluation fulfills its avowed goals of rewarding merit have thickened. Studies alleging bias along racial, gender, or class lines, the lack of objective criteria, crony-ism, caprice or downright corruption have undermined confidence in the integrity of the review system.
In her study of how a handful of interdisciplinary panels in the United States (with a brief, sidelong glance at France) award grants and fellowships, sociologist Michèle Lamont addresses these concerns about what is happening to academic evaluation. She is less concerned with how much than with how well it is done, but she is alert to how the former indirectly influences the latter: panelists with mounting piles of applications to review and discuss under severe time constraints can devote less critical attention to each case. With the permission of the American Council for Learned Societies, the Social Science Research Council, the Woodrow Wilson National Fellowship Program, “a Society of Fellows at an elite university”, and an anonymous foundation, she conducted interviews with 49 panel members who agreed to be interviewed (mostly by phone) on condition of anonymity and full disclosure of the purpose of her research. She also interviewed the program officers who prepared and attended (but not as voting members) the panel meetings of their respective foundations. Despite her central concern with the pursuit and indeed bare existence of academic excellence and her own experience as a participant in other panels, Lamont firmly declines to give her own assessment of the accuracy of panelists’ views about what had happened during their deliberations: “I did not aim to establish whether respondents’ accounts of their actions corresponded to their observed behavior. Instead, I analyzed their representations of their behavior, together with statements about the quality of scholarship, as part of their broader construction of excellence” (p. 263).
Lamont's analysis of her interview material focuses on the criteria of academic excellence explicitly mentioned by panelists and the degree to which such criteria are compatible with other factors, such as a concern for diversity, “moral qualities” (including determination, humility, and authenticity) and the influence of what she calls “idiosyncratic” or “evanescent” (though they are anything but transient) judgments of elegance, flair, and, occasionally, a certain resemblance to the panelist's own research predilections. Unsurprising (especially to readers who may themselves have served on such panels) are her findings that various disciplines cultivate their own distinctive epistemic cultures, that definitions of excellence are strongly correlated with these cultures, that panelists generally defer to each other's disciplinary expertise, that they cheerfully ignore the guidelines specified by the granting foundations, that chance plays a not insignificant role in outcomes (e.g. whether applications are considered in alphabetical order), and that the inflation of the currency of praise has rendered letters of recommendation all but useless in evaluation.
Nor will eyebrows be raised to learn that almost all the panelists interviewed believe that the process was fair, if not perfect, and that usually (in Lamont's own words) “cream rises to the top”. Yet as Lamont rightly points out, identification with both process and outcome is consequential: it motivates panelists to serve and to serve well, conscientiously preparing their dossiers and earnestly debating the merits of applications when they might have spent a quiet weekend at home or in the library. Panelists find their reward in the pleasure of intellectual debate (several of the interviewees were delighted to have a tour of the freshest research in other disciplines from eminent scholars), esprit de corps, and the satisfaction of a worthy job well done.
More counterintuitive, at least to those who view such evaluation processes as a divvying up of spoils or a contest among power-hungry barons to enlarge their fiefdoms, are the strongly felt and, if the testimony of the interviews is to be believed, strongly enforced norms that prohibit acting on the basis of self-interest. This extends to promoting one's own students and discipline; panelists who do not voluntarily recuse themselves in cases of potential conflict of interest incur the sharp criticism of their fellows and will in all likelihood not be invited back by the program officer to take part in next year's panel. All the panelists are aware that they are evaluating each other as well as the applications for research funding, and most comport themselves accordingly. Longwinded, name-dropping, ill-prepared, bullying panelists are not unknown, but their views are ever more sharply discounted by their colleagues as the long weekend in a windowless conference room wears on. Conversely, erudition, sagacity, analytic acuity, consistency, conscientiousness, and respect for others command esteem and often deference in cases where opinions diverge.
It is the signal achievement of the format and ethos of the interdisciplinary grant panel that incentives, both obvious and subtle, mostly encourage good behavior rather than bad, in the sense of fostering fairness. As Lamont shrewdly observes, it is easier to be evenhanded and magnanimous in situations in which participants will not have to live with the consequences: in contrast, for example, to tenure decisions in their home departments, the panelists receive no feedback about how their decisions pan out – an astonishing lapse on the part of the funding bodies, if Lamont's small sample is anything to go by.
Although Lamont emphasizes the diversity of disciplinary “epistemic cultures” in the humanities and social sciences and identifies four distinct “epistemological styles” (constructivist, comprehensive, positivist, and utilitarian) in which panelists justify their evaluations to one another, there seems to be remarkable convergence in at least the vocabulary her interviewees used to identify their criteria of academic excellence. “Originality” and “significance” are mentioned by over 80 percent of the humanists, historians, and social scientists – in contrast to “feasibility”, mentioned in only about 50 percent of the panelists across the board. Humanists value “clarity” somewhat (68 percent), historians more (80 percent), and social scientists not very much (41 percent). Due to the small sample sizes and the aggregation of disciplines as diverse as art history and philosophy under divisions such as “humanities”, within which there may have been wide dispersion, these figures are at best suggestive. But what they suggest is at least apparently at odds with Lamont's claims about disciplinary epistemic cultures elsewhere in the book. For example, she repeatedly attributes the high degree of consensus among historians concerning academic quality to a “shared definition of good craftsmanship”. Just what counts as “good craftsmanship” is not specified, but it seems reasonable to assume that it has something to do with methods of research and mustering of evidence. Yet only 55 percent of the historians interviewed cite “methods” as a criterion of excellence, in contrast to 72 percent of the social scientists. This may just be an artifact of an open-ended interview style and strategically vague categories. But vagueness is not incorrigible, especially when interviewer and interviewee are highly articulate and finely discerning, as in this case. Here and elsewhere one wishes that Lamont had pressed her interviewees for more detail about exactly what they meant by their criteria, beyond simply tallying the number of times each was mentioned. Without a sharper focus on content, it is difficult even to understand the panelists’ perceptions of academic excellence, much less whether those perceptions correspond to the actual outcomes of the peer review process.
Guarantees of fairness are not the same thing as guarantees of excellence. A process that scrupulously polices sources of potential bias – whether on the basis of religion or race, discipline or kinship, gender or institution – is at best a necessary but not a sufficient condition for a smoothly functioning meritocracy. Indeed, it is easy to imagine situations in which discrimination is countered by a welter of bureaucratic rules that interfere with the exercise of informed judgment and thereby pit fairness against judgment. Lamont offers a nuanced and perceptive account of the alleged tension between values of academic excellence and diversity – often opposed to one another in recent American debates over affirmative action – in the deliberations of the panelists.
She points out that diversity has many dimensions (her interviewees worried more about institutional than about ethno-racial or gender diversity) and that it “is perceived [within American academia] as an intrinsic good, leading to a richer academic experience for all and a broader production of talent for society as a whole” (p. 213). Diversity becomes constitutive of, not opposed to, excellence because it opens up promising new areas of research and recruits talented researchers. She is however less sensitive to the spurious opposition between rules and judgment, objective and subjective, the quantitative and the qualitative that muddies much discussion about academic evaluation. There is, for example, a vast difference between interpretation as a cognitive skill wielded by humanists in their reading of sources and the professional narcissism (Lamont calls it “homophily”) that impels scholars to privilege unthinkingly research that resembles their own. Yet because neither can be specified by objective rules, she (and apparently some of her interviewees) lump both together as “subjective” and “personal”. This Manichean thinking makes it almost impossible to use the words “evidence” and “connoisseurship” (or “personal” and “expert”, or “cognitive” and “moral”) together – even though it is the essence of judgment to combine these qualities.
Lamont sees her study as exploratory: she writes that she hopes to stimulate more research on the topic of academic evaluation and also to prod more self-examination among the evaluators about their assumptions and criteria. However, she does not shy away from some policy implications, particularly in the exportation of American evaluation formats to other academic systems. She has valuable things to say about the peculiarities of the American higher education system, with its great variety of institutional forms, heterogeneous student bodies, and relentless making of hierarchies, which European and Asian funding agencies keen on adopting American evaluation procedures wholesale ought to heed. With these audiences in mind, she may have chosen to stay close to her empirical material. Much of the study's fascination lies in her deft mustering of quotations to make and illustrate her points. She takes methodological inspiration from the ethnographic methods of Erving Goffman and the pragmatism of John Dewey. Theory is sparse, perhaps deliberately so: she invokes Weberian impersonal rules and Durkheimian rituals, but neither category does much work in making sense of her interview material. Instead, she prefers classifications and descriptions (e.g. her four epistemological styles) that would be readily recognizable to her interviewees, even if they dissented in the details. One nonetheless longs for a more sustained, less commonsensical reflection on the faculty of judgment itself and why it has become so intrinsically problematic in contemporary academic culture.
