To what extent can non-governmental organizations (NGOs) enhance responsiveness and facilitate equal access to citizens?Footnote 1 While there is a large number of NGOs whose mission is to further these goals in many countries, how successful they are is difficult to gauge. At the same time, their efforts are much needed, especially in situations where governments themselves lack the capability or the motivation to enforce equal access. Beyond ethnic discrimination in the labor market, in education, and in housing, unequal access to public services and public data seems to be a serious concern in today's Hungary according to a leading civil rights NGO. Access to public information is considered to be a crucial civil right that not only enhances the effective functioning of democratic systems, but also boosts citizens’ participation in public life.Footnote 2
While there exists a large and growing literature on discrimination by local governments (Hemker and Rink, Reference Hemker and Rink2017; Distelhorst and Hou, Reference Distelhorst and Hou2014), little work has been done to explore ways in which such biases may be ameliorated. To our knowledge the only such effort is Butler and Crabtree (Reference Butler and Crabtree2017) who find no effect of an information treatment delivered by researchers. One crucial issue is that interventions implemented by researchers might not be taken seriously by local governments (Kalla and Porter, Reference Kalla and Porter2019; Butler and Crabtree, Reference Butler and Crabtree2017), and governments themselves might not have sufficient incentives to intervene when they observe discriminatory behavior.
In this paper, we study the responsiveness of Hungarian local governments to request for information, and their possible discrimination of the Roma minority. We focus our attention on the Roma minority in Hungary, an ethnic group that faces widespread prejudice (Simonovits et al., Reference Simonovits, Kezdi and Kardos2018) and discrimination in Hungary (Miller et al., Reference Miller, Gounev, Pap, Wagman, Balogi, Bezlov, Simonovits and Vargha2008). We report the results of an intervention we implemented in collaboration with a major Hungarian NGO that sought to inform local governments of their duties to provide citizens with equal access to information. We evaluated this intervention with an audit study (Distelhorst and Hou, Reference Distelhorst and Hou2014) that was comprised of requests sent to local governments through email, in which we randomly varied the purported ethnicity of the sender as well as the nature of the request itself. We estimated discrimination toward the Roma minority by comparing responses to inquiries sent by ostensibly Roma versusnon-Roma Hungarians, and tested the impact of an intervention by comparing discrimination across treated and control municipalities.
Through our audit experiment, we found that ostensibly Roma individuals were discriminated against by local governments in terms of their getting a response at all, in the quality of the information in the responses, as well as in the tone of the responses. The extent of discrimination was substantively large: Roma individuals were more than 20 percent less likely to receive a response, and the quality of the information they received was over 30 percent lower based on our qualitative coding. These results are similar in magnitude to those found by (Distelhorst and Hou, Reference Distelhorst and Hou2014) as well as by Einstein and Glick (Reference Einstein and Glick2017) both in terms of baseline responsiveness and discrimination, and support the findings of Hemker and Rink (Reference Hemker and Rink2017) in establishing more subtle forms of discrimination.
Regarding the effect of our collaborative intervention, we found that informing local governments of their duties to provide information to citizens reduced their discriminatory behavior against the Roma. Our results suggest that the reduction in discrimination was driven by the local governments’ efforts to reallocate resources to responding to Roma citizens (with less effort put into improving toward non-Roma citizens), but without meaningful improvements in the quality of the responses. Assessing the long-run impact of the intervention, we find that these short-term changes by and large disappeared in a month. These findings point to the difficulty that civil rights NGOs face in fighting unequal treatment faced by minorities. While on the one hand litigation in a few key high-profile cases can change policy outcomes, cheaper and more scalable interventions, like the one reported here, are unlikely to lead to long-lasting effects.
Research design
Our study took place in Hungary, and we targeted local governments.Footnote 3 Among other things, one purpose that these local governments serve is to provide information to their constituents as well as to anyone inquiring about local issues. In particular, the Hungarian Information Law of 2011 mandates that any request of information pertaining to the operation of the local government, or local issues, qualifies as a request for public data, and as such local governments are mandated to disclose the requested information within 15 days.
Audit experiment
Our audit study employed a within-subjects design (Fang et al., Reference Fang, Guess and Humphreys2019) in which local governments were contacted from fake email accounts signaling the sender's Roma or non-Roma ethnicity. The emails were sent out in two waves: from July 9 to July 17, and then from August 9 to August 17. Altogether we attempted to send 2520 different emails from nine different accounts.Footnote 4
We included one of four different requests in these emails. The first was a question about a biking trip that the requester was planning to make near the targeted town. The second one inquired about nurseries in the area, and hinted at the sender's family's thinking about moving to the settlement. The third asked about the opening hours of the local cemetery, and whether it was accessible to people with disabilities. The fourth one inquired about possible venues for a wedding in the area.Footnote 5 We sought to balance a variety of considerations when designing these requests. First, we wanted to maximize the appropriateness of the four inquiries from the viewpoint of the targeted municipalities in order to reduce the possibility that they would find out about our research objective. Second, we sought to homogenize the length of the requests and ask specific questions so that we could evaluate the quality of the responses.
The key manipulation in our email request was the purported ethnicity of the requester, which we randomly assigned as either Roma or non-Roma. We cued Roma ethnicity using stereotypically Roma sounding names that formed the basis of the fake email addresses, and they were also included in the emails themselves. We also varied the gender of the requester through the email address and the signature. We held the style of the email constant and we used a relatively educated language in order to increase baseline response rates.
In our within-subject design each municipality received two emails. The order of the gender and ethnicity was independently randomized so that each municipality received a Roma and a non-Roma request. Finally, a last treatment arm randomly assigned whether municipalities were followed-up if a response was not received within a week. As we explain below the purpose of this treatment arm was to introduce exogenous variation into the response probability in order to mitigate non-random missingness in our measure of response quality.Footnote 6
Intervention
The intervention was carried out by the Hungarian Civil Liberties Union (HCLU, Hungarian: Társaság a Szabadságjogokért, abbreviated TASZ), one of the major Hungarian civil rights NGOs. HCLU was founded in 1994, it has focused its efforts on monitoring legislation, pursuing high profile litigation, and providing free legal aid assistance. Given HCLU's interest both in the transparency of local governments and in discrimination, it seemed the most credible source for our information campaign.
Our intervention was based on the “monitoring condition” in Fang et al. (Reference Fang, Guess and Humphreys2019), and was designed to raise the salience of anti-discrimination norms as mandated by relevant Hungarian laws. The wording of the message was designed in a collaborative effort between the research team and legal specialists of HCLU. The email first introduced HCLU and mentioned that one of the NGO's main goals was to facilitate equal access to public data. Then, it explained the legal status of requests sent to local governments (see above). It also emphasized that municipal governments were legally bound to respond to all requests. Finally, it reinforced the norm of the equal treatment of citizens requesting information. The email included the official stamp of the organization and was signed by a manager of the organization. The full wording of the message is included in online Appendix A1.
We used block randomization to obtain a treated sample of municipalities that the letters were sent to. In particular, we created blocks crossing county and population quintiles. The emails were sent out by HCLU on June 26, 2020, two weeks before the first round of the audit. With the use of mass mailing software we were able to identify local governments that opened the email.Footnote 7
Dependent measures and analysis
Our primary outcome was the receipt of a response to a given request. We coded a request as having received a response if (1) it was sent within the 15 days mandated by law, and (2) it was not an automatic reply. To obtain more subtle measures of discrimination, research assistants scored responses on two dimensions. First, we defined helpfulness as the degree to which the response provided information requested by the sender. The advantage of this measure is that we can score it even for requests that did not receive a response as zero.Footnote 8 Second, we defined politeness as the degree to which the response was respectful.Footnote 9
We explored two main sets of quantities of interest. First, we estimated baseline levels of discrimination as the differential likelihood and quality of responses received by minority and majority senders. To measure baseline discrimination in receiving a response we estimated linear models that compare the probability of response across requests sent by Roma (minority) and non-Roma (majority) senders to the same municipality. Technically, this corresponds to linear probability models with unit fixed effects. One complication that arises here is that we only observe response quality for requests that were actually responded to. Naive comparisons of response quality conditionals on observing a response would lead to post-treatment bias Coppock (Reference Coppock2019). To mitigate this problem we utilized a Heckman selection procedure where the excluded variable in the selection equation was an indicator for municipalities that were planned to be followed up in the case of no response.Footnote 10 Given that this indicator is randomly assigned, this procedure leads to consistent estimates even in the face of missing outcomes due to non-random selection.Footnote 11
Second, we explored the impact of the intervention on discrimination. To explore the effect of the intervention on discrimination, we estimated the same models but included indicators for Treatment status, Roma sender, and their interactions. For this analysis, we also assessed the persistence of a possibly occurring treatment effect by comparing the effect of the intervention across the two waves. In our main text, we deviate from our PAP in that we report results separately for the two waves of the experiment given that we uncovered very large differences in the treatment effects in post-hoc analysis.
Results
We begin our analysis by reporting discrimination against the Roma. Specifically, we focus our attention on municipalities in the control group (i.e. those who were not sent an email by HCLU) and compare our outcome measures between requests sent by minority and majority senders. For outcome measures observed for each request, we report estimates of discrimination estimated via OLS regressions with unit fixed effects and controls for request type and gender of the sender. For outcomes only observed for requests receiving a response, we estimate discrimination utilizing Heckman selection models as well as fixed-effect models for the subset of municipalities that responded to both requests.Footnote 12
We report baseline rates of anti-Roma discrimination in the control group in Table 1. We find strong evidence that requests sent by individuals perceived to be Roma were much less likely to receive a response compared to identical requests made by non-Roma individuals. The estimated difference is over 13 percent-points, or over 20 percent of the baseline response rates amounting to 56 percent (column 1). These estimates are quite precise and allow us to rule out discrimination rates below 8 percent at 95 percent significance. They are also broadly similar to those found by Distelhorst and Hou (Reference Distelhorst and Hou2014) as well as by Einstein and Glick (Reference Einstein and Glick2017) both in terms of baseline responsiveness and discrimination.Footnote 13
Table 1. Baseline rates of discrimination
Notes: Estimates in (1) and (2) are from linear models with municipality fixed effects. Estimates in (3) and (4) are from Heckman selection models with randomly assigned follow-up as an excluded predictor of the selection equation. 95 percent confidence intervals are computed based on standard errors clustered by municipality. All regressions include indicators for the type of request and gender of the sender.
We find similar patterns analyzing the content of the emails. First, when we use our comprehensive measure of response quality that imputes 0 for requests that did not receive a response, we find even more striking levels of discrimination: a 13 points difference, or an almost 30 percent difference compared to the baseline experienced by non-Roma senders (column 2). Using our sample selection models we find smaller, but still substantively important differences across the responses received by the non-Roma and the Roma. These differences in terms of information content and tone were on the magnitude of 5 points on a 100-point scale, or about 10 percent of the quality of emails received by non-Romas (columns 3 and 4). We obtain slightly smaller estimates when using fixed-effects models on the subset of municipalities that responded to both requests (columns 5 and 6)
To assess the impact of our intervention of discrimination, we first visualize differential response rates to Roma and non-Roma emails by treatment group, separately for two waves that took place 2.5 and 7 weeks after the intervention. Figure 1 reveals three striking patterns. First, both in contrast to the control group and in absolute terms, discrimination disappeared in municipalities assigned to the control group. Second, we find no evidence that the intervention increased the likelihood of response or their quality for non-Roma senders, instead it appears that response rates to requests sent by Roma requesters increased at the expense of responsiveness to non-Roma.Footnote 14 Finally, there appears to be no difference between treated and control municipalities by the second wave (i.e. 7 weeks post intervention).
Figure 1. Treatment effects decay. Bars are average response rates by experimental conditions and waves. Error bars are 95 percent confidence intervals.
Table 2 reports our estimates of the causal effect of the intervention. We report two sets of estimates: discrimination rates in the treatment and control groups (estimated separately in the two groups of municipalities) and the interaction between the Treatment and Roma sender capturing the reduction in anti-Roma discrimination due to the intervention (estimated on the full set of municipalities). We again report estimates separately for the two waves that took place 2.5 and 7 weeks after the intervention and again for four different outcome measures. We report linear regressions in the case of outcomes observed for all requests and Heckman selection models for the helpfulness and the politeness of the responses.
Table 2. Treatment effects on response rates and quality
Notes: The first two rows present estimates on the control and treatment groups, respectively and report discrimination rates (the difference between Roma and non-Roma). The third row presents estimates on the full sample and report differential discrimination rates (the difference between Roma and non-Roma compared across treated and control municipalities) based on the interaction of Roma and Treated. Estimates in (1), (2), (5), and (6) are from linear models. Estimates in (3),(4), (7), and (8) are from Heckman selection models with randomly assigned follow-up as an excluded predictor of the selection equation. Each regression includes blocked fixed effects and controls for request type and requester gender. Robust standard errors in brackets. * denotes statistical significance at 5 percent.
Our point estimates suggest that in the short run discrimination against the Roma shrank in treated municipalities. In the first wave of our experiment, Roma in the control group were about 10 percent less likely to receive a response, while in the treatment group ostensibly Roma citizens were almost 1 percent more likely to receive a response (column 1). In the specification declared in the PAP, we cannot reject the null-hypothesis that the interaction term is 0 (p=0.095, 95 percent CI [−1.7 to 21.3]).Footnote 15
We see a similar pattern when we take the amount of information into account: large and significant differences in response quality in the control group but not in the treatment group (column 2). Shifting our focus on the helpfulness and politeness of the responses received (columns 3 and 4) we find relatively similar patterns of discrimination in the treatment and control groups: the responses Roma senders received were of lower quality in both treated and control municipalities. Finally, our estimates assessing the long run impact of our intervention are clear: the effects of the intervention appear to have dissipated entirely by the second wave. We find very similar patterns of discrimination against the Roma in both the treatment and the control group—and substantively small and statistically insignificant differences across these patterns both in terms of response rates and response quality (columns 5-8).
While HCLU contacted 626 municipalities, only in 147 (23 percent of them) did administrators open the email as recorded by the CRM system used by HCLU. Such low compliance could be the result of scarce resources in local administrations. Alternatively, it could have been driven by the association of our partner-organization with liberal and anti-government political views. In other words, the weak intent-to-treat effects of the intervention are in part explained by low compliance raising the question of whether delivering the same message from a different source and through different means could have changed our results.
To estimate the treatment effect of the intervention on the treated, we estimated instrumental variables regressions with the take-up of the treatment (and its interaction with Roma sender) as an endogenous predictor and assignment to the HCLU treatment and its interaction with Roma sender as instruments. Our results—reported in online Appendix C—imply CACEs that are much larger in magnitude than our estimates of intent-to-treat effects. Of course given that compliers are likely to differ from those who failed to open the emails, it is impossible to infer from the CACE on how the overall effect of the intervention would have increased had compliance been larger.Footnote 16
Discussion
This paper reported the results of an intervention seeking to reduce discrimination against Roma citizens requesting information from Hungarian municipalities. To measure discrimination, we conducted an audit experiment in which we targeted Hungarian local government officials with various requests. We found a substantively large degree of discrimination against Roma citizens both in terms of response rates and in terms of response quality. Our intervention was an online campaign in which a major civil rights NGO reminded municipalities of their legal obligation to respond to requests for information. While we found some evidence that the intervention reduced discrimination in the short run, its effects had dissipated in about a month.
What are the implications of these findings? On the one hand, our results on anti-Roma discrimination support a growing body of evidence on how government agencies can themselves contribute to the discrimination of vulnerable minorities (Distelhorst and Hou, Reference Distelhorst and Hou2014; Hemker and Rink, Reference Hemker and Rink2017). On the other hand, our estimates of the impact of our information-campaign suggest that even interventions that are seemingly powerful in the short term can prove ineffective over the long haul. It is likely that successful campaigns aiming to reduce prejudice or discrimination against minorities will take much more effort—e.g. online or face-to-face awareness raising training for civil servants—and require the cooperation of targets. To the extent that such cooperation can be achieved, the design and implementation of our intervention could serve as a promising way for NGOs and collaborative efforts with researchers to test the effects of such programs.
Of course the generalizability of our findings is limited given the context in which our study was implemented. In the last decade, Hungary has taken an illiberal turn (Krekó and Enyedi, Reference Krekó and Enyedi2018) which manifested in a dismantling of institutional guarantees for governmental accountability (Pap, Reference Pap2017).Footnote 17 Crucially, civil rights NGOs have become targets of the Hungarian populist regime: their work was severely hampered by the introduction of a new NGO law in 2017 and the stigmatization of NGOs relying on foreign financial support (they were labeled as “Soros-funded NGOs” in the 2017 law) placed these organizations into a highly politicized role. These contextual factors might have undermined the effectiveness of our intervention by reducing the credibility of its source. At the same time our study also demonstrates that societies where such organizations are the most needed are also those where their voice is the hardest to be heard.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/psrm.2021.55.
Acknowledgements
This research was supported by the Young Researcher's Grant by the National Research, Development and Innovation Fund (FK 127978).
