Tomoko Harigaya, PxD’s Chief Economist and Director of Research, reflects on the work of the 2021 Nobel Prize winners in Economic Sciences, and how their contributions to the field have informed PxD’s research practice.
Last week, Jonathan Faull, our Director of Communications, sent me a WhatsApp message: “I had a dream that Owen [PxD’s CEO] woke me up with a phone call to tell me Michael [Kremer] won the Nobel again…”. I laughed and responded saying that he was working too much.
Jonathan’s dream didn’t become a reality, but it wasn’t too far off in spirit. I woke up on Monday, thrilled to learn that the three empirical economists, David Card, Josh Angrist, and Guido Imbens, had won the [deep breath] 2021 Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel. As per the Royal Swedish Academy of Science’s press statement, Card won the prize “for his empirical contributions to labour economics”; Angrist and Imbens won “for their methodological contributions to the analysis of causal relationships”.
The laureates’ work on natural experiments and empirical methodologies for causal analysis has shifted the field of economics’ focus towards empirical rigor. Continuous, incremental advances in causal inference methodologies, following Card, and Angrist and Imbens’ groundbreaking work in the early 1990s, expanded the economists’ toolbox, improving our ability to generate credible answers to important policy questions through both experiments and quasi-experiments.
Here is how one specific contribution of the laureates, acknowledged by the Swedish Academy of Sciences, improves our ability to understand program impact in the context of PxD’s service:
Randomized experiments are considered a “gold standard” because of the counterfactual that researchers can establish through random assignment of an intervention. However, in real-world experiments, whether implemented through a government or a non-government partner, enforcing random assignment is often not an option. Angrist and Imbens introduced an application of a statistical method to estimate a relevant average treatment effect in these cases.
Suppose that the average proportion of farmers who listen to PxD’s advisory messages fluctuates around 60% in India. When we evaluate the impact of our service by randomizing the offer to access the service, we expect that approximately 40% of the farmers in the treatment group would not listen to advisory messages. Our benchmark analysis typically compares all farmers assigned to the treatment group to all farmers assigned to the control group, regardless of whether they actually “take up” the service and listen to advisory messages or not. But this comparison only tells us the average effect of the offer to access the advisory service, not the average effect of the advisory service itself. It is a fairly common practice to scale the estimated impact from this benchmark comparison (the “intent-to-treat” or ITT effect) by the difference in the take-up rate between the treatment and control groups to obtain the average impact of the service for an engaged farmer (the “treatment-on-the-treated” or TOT effect). A policymaker may be interested in the cost-effectiveness implications from the ITT estimates, but the TOT estimates are useful for implementers trying to iterate product design and improve the impact per farmer.
The ITT is a weighted average of the treatment effect among those who can be induced to listen to advisory messages under the intervention and the treatment effect among those who would not listen to the messages regardless of the intervention. Since the latter effect is likely to be zero – if a farmer doesn’t listen to messages when offered the service, the advisory service is unlikely to change their behavior – we could scale the ITT by the take-up rate of the advisory service to get the average impact among those who choose to listen to messages when offered the service.
Angrist and Imbens brought clarity to when – and for whom – this TOT estimate could capture a valid average treatment effect. When uncertain, they showed that if simple conditions are met, the instrumental variable (IV) approach can be applied to recover the average treatment effect but only for a specific subgroup of the study population – often called “compliers” – who are induced to receive the treatment under the intervention but otherwise wouldn’t. In the example of PxD’s service cited above, this means that using the average treatment effect of PxD’s advisory service, we can estimate the average impact among those who choose to listen to messages when offered the service.
This may seem like a minor refinement in the interpretation of impact estimates from experiments. But these details make a meaningful difference in how we understand what we know and what we don’t and how we apply that understanding to the next phase of testing and scaling.
As Angrist and Jörn-Steffen Pischke put it crisply in a 2010 review of the “credibility revolution” in economics:
This year’s Nobel prize is a recognition that the process of building credible empirical knowledge holds the rigor of research design and empirical methods at its core.