Hostname: page-component-745bb68f8f-l4dxg Total loading time: 0 Render date: 2025-02-11T08:13:34.694Z Has data issue: false hasContentIssue false

How (not) to measure institutions: a reply to Robinson and Shirley

Published online by Cambridge University Press:  16 October 2012

STEFAN VOIGT*
Affiliation:
Institute of Law and Economics, University of Hamburg, Hamburg and CESifo, Munich, Germany
Rights & Permissions [Opens in a new window]

Abstract:

This is a short reply to the comments by James Robinson and Mary Shirley on my paper ‘How (not) to measure institutions’.

Type
Comment
Copyright
Copyright © Millennium Economics Ltd 2012

First, I want to thank both James Robinson (Reference Robinson2013) and Mary Shirley (Reference Shirley2013) for their insightful and constructive critique. Since there are many points on which we agree, I want to be brief and pick up one point made by Robinson, one by Shirley and one that both of them make.

Robinson (Reference Robinson2013) reminds us that dealing with institutions by necessity implies dealing with endogeneity. This is why – according to him – the distinction between institutions on the one hand and policies on the other is unimportant. But this would only seem to follow if all kinds of endogeneity should be treated exactly alike. And on this, I would beg to differ. If some variables only change over decades or even centuries while others can be substantially altered within months, then this difference can be crucially important for political choices (no matter whether we prefer to call them institutional or policy choices). Assuming that conflicting institutions are costly in the sense of making development more difficult, it would seem that – at least in the short run – political choice should aim to minimize such conflicts. If some institutions are more prone to political choice than others, then policies should focus on these to minimize costly conflicts between various types of institutions. Using slightly different words, Roland (Reference Roland2004) makes a very similar point.

Robinson is not afraid of endogeneity. Having established some of the most ingenious instruments around, he might have reason not to. Here, his claim is that subjectivity may, as argued in my contribution, indeed be problematic but that econometrics would come to our rescue because subjectivity would only be another form of endogeneity and ‘if you have a valid instrumental variable then you can solve the problem’. As is well known, really convincing instruments are extremely scarce. Robinson's proposal does not really seem to solve the problem but merely moves it one step up: if we have a valid instrument, the quality in the measurement of institutions could, indeed, become secondary, but why should it be easier to find an adequate instrument than measuring institutions in an objective way?

Shirley (Reference Shirley2013) maintains that the epistemic potential of cross-country regressions has been largely exhausted. The interesting questions of the future would be different. And she continues to give some examples like ‘why are some countries able to create independent judiciaries and others not?’ I perfectly agree that this is an interesting question. Indeed, I have even done some work exactly on this question (Hayo and Voigt, Reference Hayo and Voigt2007). But no matter whether we rely on cross-country regressions or other approaches, answering this question presupposes that we are able to measure institutions in a valid and reliable way.

The point that Shirley and Robinson both make deals with the most adequate tools used to show the relevance of institutions. Robinson mentions laboratory and field experiments just as case studies as a possible means to circumvent the problem of the uniqueness of institutions illustrated vividly by the example of the Trobriand Islanders. Shirley ventures a similar conjecture when she suggests that breakthroughs ‘will also come from other methodologies’ such as comparative case studies and field experiments.

I agree with both Robinson and Shirley in the sense that other, or additional, approaches are likely to enable us to get additional insights. This does, however, not affect the central point of my paper. Indeed, on page 2, I write the following:

Academics have sought to demonstrate the relevance of institutions relying on a host of approaches: from case studies via natural experiments and comparative institutional analysis to cross-country regressions. No matter what approach one relies upon, if one wants to show that institutions matter – or that they do not – one needs a reliable way to identify and measure them. It is this issue I am concerned with here.

In other words, the issue of how to measure institutions needs to be dealt with, no matter what approach we want to rely upon. Regarding approaches, it would seem to make sense to think about how cross-country studies on the one hand and case studies on the other can meaningfully be made to complement each other. (Brady et al. Reference Brady, David and Jason2006 is one of the few papers that discusses options for “mixed methods”). Both approaches have pros and cons and since these are not identical, it would seem natural to combine both to complement each other. Over the past years, huge data sets referring to the nation state level have become available. If similar data become available for the state, the regional or even the local level, their many additional insights might be used; e.g., by drawing on hierarchical models. In that sense, we should also be careful not to write the chronicle of a death foretold.

References

Brady, Henry, David, Collier and Jason, Seawright (2006), Toward a Pluralistic Vision of Methodology, Political Analysis, 14: 353–68.CrossRefGoogle Scholar
Hayo, B. and Voigt, S. (2007), ‘Explaining De Facto Judicial Independence’, International Review of Law and Economics, 27 (3): 269290.Google Scholar
Robinson, J. A. (2013), ‘Measuring Institutions in the Tobriand Islands: A Comment of Voigt's Paper’, Journal of Institutional Economics, 9 (1): March.CrossRefGoogle Scholar
Roland, G. (2004), ‘Understanding Institutional Change: Fast-Moving and Slow-Moving Institutions’, Studies in Comparative International Development, 38 (4): 109131.CrossRefGoogle Scholar
Shirley, M. M. (2013), ‘Measuring Institutions: How to be Precise though Vague’, Journal of Institutional Economics, 9 (1): March.CrossRefGoogle Scholar