Skarbek (Reference Skarbek2020) argues that qualitative research methods can exploit evidence not accessible through statistical methods, evidence that is especially important for institutional analysis. Skarbek asserts that investigations relying solely on quantitative methods often: (1) exclude important questions because we lack the necessary numbers; (2) fail to distinguish between multiple causal mechanisms; and (3) lead researchers to prefer simpler concepts and theories that can be more easily measured. This is especially problematic for institutional research, where the answers to important questions are hard to quantify, the causal mechanisms are diverse, and the concepts and theories are complex, multifaceted, and multidimensional.
Skarbek suggests two qualitative methods to study institutions: comparative case studies and process tracing. (Process tracing methodically traces causality from changes in independent variables through intervening variables to the dependent variable.) Surprisingly, Skarbek defends qualitative methods in terms of causality, which is usually purported to be the great weakness of qualitative methods. Instead, he suggests that qualitative analyses can meet some of the same tests of inference as quantitative analysis. Drawing on a literature well known outside of economics, he defines general principles of good research design and criteria, such as validity and unbiasedness, that can be used to judge the quality of both quantitative and qualitative evidence. Skarbek's criteria are demanding. To meet his standards, researchers doing case studies would have to assemble large samples, carefully select representative cases that all have similar non-causal variables, and create a realistic counterfactual. Process tracing would have to provide useful information, an unbiased sampling procedure, and tests of whether the evidence is necessary and/or sufficient. And researchers using either method would need conscientiously to probe alternative explanations.
Skarbek is treading on well trampled ground. Consider what Ronald Coase said over 20 years ago:
Economics, over the years, has become more and more abstract and divorced from events in the real world. Economists, by and large, do not study the workings of the actual economic system. They theorize about it… If we are to understand the effect of different institutional arrangements on the working of the economic system, the obvious way to do this is to compare what happens in different countries with differing arrangements (Coase, Reference Coase1999).
Many authors besides Coase have criticized economists' focus on theory and sophisticated quantitative analysis to the neglect of real world institutions.Footnote 1 Skarbek's main contribution, and it is an important one, is the brief, cogent, and instructive way he presents his arguments and his defense of qualitative methods as a complement to mainstream methods rather than a confrontation, which may be more persuasive than Coase's frontal assault.
Skarbek reveals his true Coasean colors in his examples. Drawing on his own extensive experience, Skarbek suggests that researchers investigating police and crime should consider interviewing large numbers of people living in different parts of the city, going to crime hot spots, and spending several weeks riding around in a policy cruiser to allow them ‘to see, feel, and hear what police are doing “on the ground.”’ (p. 5) This is a far cry from the applied economist crunching numbers in front of a computer. His examples make the article particularly helpful and give it a liveliness that papers on methodology usually lack.
Others have proposed similar techniques to study complex institutional problems: most notably, the group of economists and political scientists who argued for ‘analytic narratives’ (Bates et al., Reference Bates, Greif, Levi, Rosenthal and Weingast1998). These authors presented five narratives to support their view that researchers could derive empirically testable, general hypotheses from particular cases by employing rational choice and game theory. They describe their approach as ‘resembling’ process tracing, but with a ‘greater emphasis on theory’ (ibid.: 16). Indeed, the narratives include formal game theoretic models. A cursory search of Google Scholar (on January 6, 2020) turned up over 220 publications that purport to use this analytic narrative approach since Bates et al. was published in 1998. Not a bad track record over the last 21 years.Footnote 2 But, I suspect that Skarbek's article may have even wider impact, since he does not limit his advice to rational choice or game theories or to those wishing to develop formal models.
Will process tracing spawn the same enthusiasm as case studies? Applied economists are familiar with case studies (think Adam Smith's pin factory), even if many also reject them as lacking external validity by definition. Process tracing, despite its wide acceptance in political and other social sciences, is more removed from their experience. It would be a pity if economists continue largely to ignore process tracing, since it can inform theory, strengthen case studies, and enhance our understanding of institutional mechanisms.
Skarbek does not consider ‘datafication’, which is rapidly transforming qualitative information into quantitative data (as described by Prüfer and Prüfer in Menard and Shirley, Reference Prüfer, Prüfer, Menard and Shirley1998). It would have been interesting to know his views. The proliferation of information from the Internet and connected devices, text mining, machine learning, and other computation advances is turning judicial decisions, media searches and postings, political policies and speeches, and the like into data, which are then analyzed by computers using pattern recognition software. This revolution in data science is both promising and dangerous. As interviews with relevant actors and on the ground observation are replaced by large numbers of data points gathered, categorized, and prioritized by machine learning, we may move even further from Coase's call to understand how activities are actually carried out within different institutional frameworks. Skarbek's excellent advice could not be timelier.
Skarbek (Reference Skarbek2020) argues that qualitative research methods can exploit evidence not accessible through statistical methods, evidence that is especially important for institutional analysis. Skarbek asserts that investigations relying solely on quantitative methods often: (1) exclude important questions because we lack the necessary numbers; (2) fail to distinguish between multiple causal mechanisms; and (3) lead researchers to prefer simpler concepts and theories that can be more easily measured. This is especially problematic for institutional research, where the answers to important questions are hard to quantify, the causal mechanisms are diverse, and the concepts and theories are complex, multifaceted, and multidimensional.
Skarbek suggests two qualitative methods to study institutions: comparative case studies and process tracing. (Process tracing methodically traces causality from changes in independent variables through intervening variables to the dependent variable.) Surprisingly, Skarbek defends qualitative methods in terms of causality, which is usually purported to be the great weakness of qualitative methods. Instead, he suggests that qualitative analyses can meet some of the same tests of inference as quantitative analysis. Drawing on a literature well known outside of economics, he defines general principles of good research design and criteria, such as validity and unbiasedness, that can be used to judge the quality of both quantitative and qualitative evidence. Skarbek's criteria are demanding. To meet his standards, researchers doing case studies would have to assemble large samples, carefully select representative cases that all have similar non-causal variables, and create a realistic counterfactual. Process tracing would have to provide useful information, an unbiased sampling procedure, and tests of whether the evidence is necessary and/or sufficient. And researchers using either method would need conscientiously to probe alternative explanations.
Skarbek is treading on well trampled ground. Consider what Ronald Coase said over 20 years ago:
Economics, over the years, has become more and more abstract and divorced from events in the real world. Economists, by and large, do not study the workings of the actual economic system. They theorize about it… If we are to understand the effect of different institutional arrangements on the working of the economic system, the obvious way to do this is to compare what happens in different countries with differing arrangements (Coase, Reference Coase1999).
Many authors besides Coase have criticized economists' focus on theory and sophisticated quantitative analysis to the neglect of real world institutions.Footnote 1 Skarbek's main contribution, and it is an important one, is the brief, cogent, and instructive way he presents his arguments and his defense of qualitative methods as a complement to mainstream methods rather than a confrontation, which may be more persuasive than Coase's frontal assault.
Skarbek reveals his true Coasean colors in his examples. Drawing on his own extensive experience, Skarbek suggests that researchers investigating police and crime should consider interviewing large numbers of people living in different parts of the city, going to crime hot spots, and spending several weeks riding around in a policy cruiser to allow them ‘to see, feel, and hear what police are doing “on the ground.”’ (p. 5) This is a far cry from the applied economist crunching numbers in front of a computer. His examples make the article particularly helpful and give it a liveliness that papers on methodology usually lack.
Others have proposed similar techniques to study complex institutional problems: most notably, the group of economists and political scientists who argued for ‘analytic narratives’ (Bates et al., Reference Bates, Greif, Levi, Rosenthal and Weingast1998). These authors presented five narratives to support their view that researchers could derive empirically testable, general hypotheses from particular cases by employing rational choice and game theory. They describe their approach as ‘resembling’ process tracing, but with a ‘greater emphasis on theory’ (ibid.: 16). Indeed, the narratives include formal game theoretic models. A cursory search of Google Scholar (on January 6, 2020) turned up over 220 publications that purport to use this analytic narrative approach since Bates et al. was published in 1998. Not a bad track record over the last 21 years.Footnote 2 But, I suspect that Skarbek's article may have even wider impact, since he does not limit his advice to rational choice or game theories or to those wishing to develop formal models.
Will process tracing spawn the same enthusiasm as case studies? Applied economists are familiar with case studies (think Adam Smith's pin factory), even if many also reject them as lacking external validity by definition. Process tracing, despite its wide acceptance in political and other social sciences, is more removed from their experience. It would be a pity if economists continue largely to ignore process tracing, since it can inform theory, strengthen case studies, and enhance our understanding of institutional mechanisms.
Skarbek does not consider ‘datafication’, which is rapidly transforming qualitative information into quantitative data (as described by Prüfer and Prüfer in Menard and Shirley, Reference Prüfer, Prüfer, Menard and Shirley1998). It would have been interesting to know his views. The proliferation of information from the Internet and connected devices, text mining, machine learning, and other computation advances is turning judicial decisions, media searches and postings, political policies and speeches, and the like into data, which are then analyzed by computers using pattern recognition software. This revolution in data science is both promising and dangerous. As interviews with relevant actors and on the ground observation are replaced by large numbers of data points gathered, categorized, and prioritized by machine learning, we may move even further from Coase's call to understand how activities are actually carried out within different institutional frameworks. Skarbek's excellent advice could not be timelier.