Myths of Official Measurement: Auditing and Improving Education Data in Developing Countries
This paper focuses on the challenge of creating accurate census data on student achievement in settings of weak governance, using evidence from two large Indian states. I use direct audit evidence to show that levels of student achievement in Madhya Pradesh, from a large census covering ~7 million students annually, are severely inflated in official data due to cheating by students and teachers. I further document, using a sharp regression-discontinuity design with multiple thresholds, that there is little evidence of any accountability-led change as a result of the test scores. In a follow-up randomized experiment, covering over 2400 schools in a different state (Andhra Pradesh), I evaluate whether tablet-based testing, which makes cheating harder for both students and teachers, could reduce distortion. I find paper-based assessments proctored by teachers severely exaggerate achievement, in both private and government schools, in comparison to a retest. In contrast, I find no evidence of such distortion in tablet-based assessments. These results suggest that business-as-usual learning assessments may be compromised even without high-powered incentives. They are unlikely to be useful for policy action or for research. However, technology-based testing may provide a scalable solution, implemented within government systems in developing countries, to address the challenge of building accurate official data systems.