Precision Development’s goal is to generate large aggregate impacts for the poor. Research plays a central role in advancing that objective. In a collaborative post, our Research Team surveys their work in 2021 and outlines an exciting portfolio of activities planned for 2022.
As a recent video produced by our colleagues at J-PAL recounts, Precision Development would not exist were it not for the practical application of research in the world. By intentionally deploying experiments and other methods, we continue to use research to inform the design of our services (innovation research) and to evaluate our impact (evaluation research), and these two types of research often overlap. PxD’s research practice aims to integrate with service design and delivery, and we leverage research to iterate our theory of change, identify learning objectives, and gather insights in every program and initiative.
Needless to say, in 2021 we were busy! Our research and operations teams have collaborated on 26 experiments, of which 14 are completed and 12 are ongoing.
Evaluation Research
At the end of 2021, PxD has three ongoing randomized controlled trials (RCTs) to rigorously evaluate the impact of our services on downstream user outcomes, such as the adoption of improved farming practices, improved agricultural yields, or impacts on learning. In addition to these three evaluations of our own services, we are collaborating with One Acre Fund (OAF) to evaluate aspects of their work with Farmer Promoters.
Excitingly, PxD is running a large-scale RCT with nearly 15,000 rice farmers in India to estimate the impact of the Ama Krushi service on farmers’ practices and yields, and how the magnitude of these impacts vary by farmer characteristics. Data on production costs and revenues will be collected from a subset of farmers for a back-of-the-envelope calculation on the service’s benefit-cost ratio. This RCT also aims to test the effectiveness of using satellite-based imagery to detect yield changes in GPS-measured smallholder rice plots and to compare these to self-reported farmer yields. Early in 2022, we will conduct the midline survey and process satellite images to assess the impact in the first season. We are eager to share our initial findings!
In western Uganda, PxD is collaborating with the International Food Policy Research Institute (IFPRI), the Hanns R, Neumann Stiftung (HRNS), and Technoserve to measure the stand-alone impact of PxD’s two-way voice-based advisory service and the complementary impact of digital reminders implemented by PxD as part of the Uganda Coffee Agronomy Training (UCAT) program. This RCT plans to integrate the use of machine learning to identify agricultural practices that have the highest impact on improving coffee yields and to identify farmer characteristics that are most closely associated with the increased adoption of such practices. The COVID-19 pandemic pushed back the initial implementation schedule for UCAT and we now plan to collect endline data for approximately 4,000 farmers in 2022.
In the education sector, Innovations for Poverty Action (IPA) and PxD have concluded data collection for a randomized evaluation to assess the impact of our two-way SMS math education service, ElimuLeo, on learning and knowledge. Implementation of the RCT has now concluded and we have extended access to the platform to children in the control group. We have recently surpassed the milestone of over 500,000 problems solved on the platform!
Since 2018, PxD has partnered with One Acre Fund (OAF) to test low-cost incentives to enhance the performance of approximately 11,000 Rwandan volunteer extension agents, commonly known as Farmer Promoters (FPs). With one FP in each of Rwanda’s 14,500 villages, PxD’s collaborative intervention targets a majority of FPs in the country. Importantly, FPs do not receive monetary compensation for their service, but rather receive in-kind benefits such as free inputs. Throughout 2021, PxD and OAF have been conducting an RCT to examine whether motivational SMS messages improve FPs’ performance, and to test whether motivational messages designed to appeal to the FP’s specific personality traits lead to improved FP performance and farmer outcomes. This evaluation includes three main interventions: a subsidized input promotion campaign, a demonstration plot and farmer training campaign, and an agroforestry tree planting campaign. We aim to launch endline surveys in early 2022.
Service Design and Innovation Research
As part of our efforts to leverage research to improve service design and delivery, we often conduct rapid hypothesis testing. We are also exploring ways of increasing engagement with our Ama Krushi push call service. In 2021 we deployed an A/B test that experimented with sending “heads up” SMS messages at different times to alert farmer users of an impending IVR call from the Ama Krushi service. Results are very promising: sending an SMS reminder 10 minutes before a service call increases pick-up rates by 5.9 pp (a 17% increase from the control mean pick-up rate of 34%). Conditional on picking up, the 10-minute SMS reminder also increases average listening rates by 4 pp (a 25% increase over a control mean of 15.4%). Considering both the success of the 10-minute reminders and the relatively high costs of sending SMS messages via the Ama Krushi service, one implication of this experiment would be to send short SMS reminders to farmers who are at risk of becoming inactive on the service to nudge engagement for a limited period and to promote recommendations that have particularly high-impact or are time-sensitive.
Reminders also play an important role in nudging farmers to take action on optimal planting dates. Planting at the optimal time is positively correlated with yields, yet misconceptions, lack of knowledge, or limited resources often result in farmers planting at a later date. In the 2021 Long Rains season in Kenya, we tested whether sending SMS messages recommending optimal planting times would nudge adoption. This experiment targeted a sample of approximately 15,000 OAF farmers and included additional treatments such as messages addressing misconceptions on planting times and direct endorsements from Field Officers. We found that SMS recommendations for optimal planting times are effective in nudging average self-reported planting dates closer to the optimal date. This effect is larger for late planters. Late planters in treatment groups reported planting, on average, four days closer to the optimal date relative to the late planters in the control groups. These results are valuable for improving service delivery in Kenya and for nudging farmers toward more timely planting. We are keen to conduct further research on the mechanisms underlying planting time decisions.
A/B tests deployed by PxD have shown that a narrator’s voice can impact user engagement in various contexts. In 2021 we completed a first A/B test on our nascent program in Nigeria to measure the effect of the narrator’s gender on pick-up and listening rates for PxD’s push call platform. Over the course of a seven-week period in the Wet Season, a sample of over 45,000 farmers was randomly allocated into two groups, receiving weekly audio content narrated either by a female or male voice, respectively. We found that female farmers have systematically lower pick-up rates than male farmers, yet this gap is slightly mitigated when the narrator is female. In addition, we found that all farmers (both male and female) when listening to content voiced by a female narrator were 1.4 pp more likely to listen to the full push call, given a mean completion rate of 51.8% of the audio content. Going forward, our service in Nigeria will primarily use female narrators when recording audio content for push calls. Nevertheless, we will benefit from further research to generate evidence on the long-term effect of this action on user engagement.
Our nascent research agenda in our newest program in Colombia is leveraging past experimentation on human contact and direct endorsements, particularly when it comes to increasing user engagement in the early stages of service delivery when name recognition and trust levels are limited. We have recently concluded profiling surveys and are analyzing how user engagement with PxD’s Un Mensaje por El Campo platform varies before and after farmers receive the profiling survey. SMS messages are not broadly used by smallholders in the Meta region of Colombia and many farmers need to be guided on how to access and navigate their SMS inbox to realize that they have been receiving messages from our service. Surveys and direct contact with surveyors from our team provide an opportunity to mitigate low digital literacy and improve trust in our service.
Moreover, we continue to explore ways to support farmer access to recommended agricultural inputs. In Kenya, we are building a proof-of-concept digital tool to improve linkages between farmers and agro-dealers by providing local agro-dealer contact information on our MoA-INFO platform. This digital Agro-dealer Contact Directory (ACD tool) will help us gather insights on the perceived value of reducing search costs among farmers and agro-dealers, as well as potential effects on farmers’ input search behavior and input access. Using these insights, we can then determine if there are other low-touch, scalable engagements with agro-dealers that will help us to improve farmers’ welfare. Drawing on the evidence from the literature on digital phone books, we have launched a two-way agro-dealer pilot with over 33,000 MoA-INFO farmers and 142 agro-dealers. This pilot aims to elicit feedback from farmers and convey information about their input demand to agro-dealers, mitigating informational barriers that prevent agro-dealers from consistently stocking inventory. Farmers are often uncertain about which inputs are in stock and agro-dealers are uncertain about quantities of inputs demanded by farmers, with many reporting being out-of-stock during peak times of the year. Stock-outs raise uncertainty and search costs for farmers and can lower adoption rates for improved inputs, especially if farmers have to travel far to other markets. Experimentation with the ACD tool aims to reduce search costs and improve access to recommended inputs.
Crowdsourcing local information from farmers and other actors along the agricultural value chain holds many potential advantages for our services. The key is to identify scalable ways to crowdsource sufficiently granular information to improve the quality of our advisory content. In Gujarat, we are generating insights on the operational feasibility and scalability of collecting accurate and timely information on crop health issues from locally recruited gig workers and tracking the volume and quality of data submitted by these workers via a tailored WhatsApp chatbot.
Finally, sourcing and providing information on pest prevention and management continues to be one of our research priorities. We are particularly excited about an ongoing pilot in Karnataka, India, which is reinforcing good management practices for white stem borer, a major coffee pest. In a recent round of qualitative surveys, we found that a majority of respondents had experienced white stem borer infestation in the previous coffee season, and many farmers reported using antiquated corrective measures that are no longer recommended by the Indian Coffee Board, or even practices that have been banned. Research undertaken in 2021 on Fall Armyworm infestation in Kenya and Zambia, provides some indicative evidence that our advisory increases knowledge and adoption of recommended pest prevention and management practices, and that sending reminders for priority practices may result in higher knowledge and adoption. Building on this evidence, we are partnering with J-PAL to test whether repeating key advisory messages on white stem borer and using different types of reminders and nudges, lead to higher rates of adoption for recommended practices among a sample of 2,400 coffee farmers. This experiment is ongoing and we aim to share results by mid-2022.
Looking forward to exciting research opportunities in 2022
Throughout 2021, PxD expanded dairy and livestock advisory services in Kenya, Ethiopia, Pakistan, and India. In Kenya, we are partnering with three dairy cooperatives to iterate and build a two-way SMS advisory service for dairy farmers. Building on the results of a gender survey to assess the gendered division of labor within dairy-producing smallholder households, we are excited to commence an RCT of the service in early 2022, ahead of the Long Rains seasons. We aim to rigorously evaluate the impact of the service on milk production (both in terms of quality and quantity) and sales for a random sample of 5,000 dairy farmers using administrative data made available from our dairy cooperative partners.
Finally, we are excited about our nascent climate research agenda, which includes both adaptation and mitigation efforts, and aim to continue scoping opportunities within our services and geographies to strengthen smallholders’ climate resilience, generate scalable low-cost climate mitigation innovations, and, above all, to improve smallholders’ agricultural productivity and income. We are excited to launch new products integrating granular weather forecasting with actionable advisory in Punjab, Pakistan, and Karnataka, India. Our weather pilots aim to incorporate innovation research to understand how to communicate uncertainty and probabilistic events in low-literacy settings, as well as the value of forecasting versus tracking advisory. Moreover, after recently securing new climate mitigation funding, we are eager to conceptualize research pilots to measure low-cost scalable innovations and rigorously measure changes in both farmer and climate outcomes. We intend to establish a proof of concept for our environmental theory of change and seek further validation of the applicability of PxD’s communication for behavior change model on new research areas, such as the potential for cash payments to farmers for activities that sequester soil carbon and contribute to climate change mitigation efforts.
At PxD our decision-making draws on existing evidence and scientific knowledge; understanding about how humans, institutions, firms, and markets behave and function; contextual information and knowledge about targeted users and local conditions; and our prior experience (as well as resource envelopes and other pesky exogenous factors). Research and inquiry are critical to generating insights and uncovering new information across all of these categories of knowledge. We are privileged to work with colleagues who rigorously generate and leverage evidence to guide our work and decision-making as we develop, iterate, and scale innovations that impact poverty at scale.
We look forward to sharing new insights with you throughout 2022!