Sam Carter, Revati Vaidya, Simon Rubangakene, Georges Poquillon and Habtamu Yesiget recount how we improved our content and services by learning about our end-users response to messages recorded with men’s voices and women’s voices.

Do farmers act differently when they hear a woman’s voice giving them agricultural advice? How much does this matter in the nine different countries where we operate? These are questions we often ask ourselves since interactive Voice Response (IVR) and push calls are the bread and butter of our outreach to smallholder farmers. 

Rigorous research is in our DNA, since PAD was born out of a study proving that mobile-phones can cheaply deliver actionable agricultural advice to farmers. We routinely test and evaluate different versions of our service to identify the best way to engage farmers with our services and, ultimately, turn our advice into action. We’ve tested the length of voice calls, the order in which menu options are presented, and use of urgent language in our messages among many others.

In Uganda, Ethiopia, and India, PAD has conducted A/B tests to understand whether the gender of the person narrating the advisory content has an impact on farmers’ pick-up and listening rates. In Uganda, we found that switching from a male to female narrator increased farmers’ engagement with PAD content, and as a result, we will use a female narrator for all new content developed for that program. In Ethiopia, on the other hand, we found that both pick-up and listening rates were higher when content was voiced by a male narrator than a female narrator. Meanwhile, in the Indian state of Gujarat, we found no statistically significant difference in engagement rates based on the gender of the narrator. Taken together, these results suggest the need for a nuanced, contextually-grounded approach to understanding the role that the narrator’s voice plays in determining farmers’ engagement with our service.

This blog post presents a summary of each experiment design and its results, and teases out the implications of these results in time and place:

Uganda 

Simon Rubangakene, Project Associate and Georges Poquillon, Senior Research and Operations Manager

PAD’s service in Uganda

In Uganda PAD delivers digital training to smallholder farmers as part of an initiative run by Uganda Coffee Agronomy Training (UCAT) – a program implemented by the non-profit organizations Hanns R. Neumann Stiftung and TechnoServe – which delivers training to approximately 60,000 farmers in western Uganda via Farmer Field Schools. UCAT aims to train farmers to implement Good Agronomy Practices (GAPs) to increase yields and – by association – livelihoods.

PAD provides digital training to 4,000 UCAT farmers who are participating in a randomized evaluation in which farmers receive standalone and reinforcement training. The evaluation intends to understand the impact of varied approaches to agronomy training and/or extension services. 

PAD’s digital advisory is pushed to farmers’ mobile phones in four different languages (English, Luganda, Runyankole and Rutooro). Standalone farmers who are not part of the UCAT program are also able to call in and record questions, which are answered by an agronomist within 48 hours. Standalone farmers and those attending training run by Hanns R. Neumann Stiftung receive IVR calls on a weekly basis, whereas farmers attending training run by TechnoServe receive advisory messages once a month. 

How the A/B test varied the gender of the narrator, and why we decided to test this question

Prior to this A/B test, PAD’s advisory service in Uganda included content delivered by both male and female narrators. Due to the way in which language-use was distributed, few farmers receiving advisory from PAD were receiving advice voiced by a female narrator which made it difficult to identify listening patterns linked to voice. The Uganda team decided to implement an A/B test to better understand how farmers’ listening rates varied based on the voice of the narrator.

The study ran for eight weeks, from October 14, 2020 to December 9, 2020, a time coinciding with the runup to, and active period of the coffee harvest in Uganda. For this A/B test, we randomly assigned the 4,009 UCAT farmers receiving advisory from PAD to one of two groups. Group 1 (2,004 farmers) received content voiced by a narrator whose gender did not match their own. Group 2 (2,005 farmers) received content voiced by a narrator of the same gender. We stratified the sample to ensure that both groups were balanced in terms of language and the organization implementing the in-person training. 

Results of this A/B test

Switching the message content from being voiced by a man to being voiced by a woman positively and sustainably increased the probability that female farmers picked up IVR calls, while there was no effect on the likelihood of a male farmer taking a call. With regard to the propensity to listen to most of the IVR call content, switching from a female to a male narrator reduced the listening rate of male farmers, while the opposite held true when switching from a male to a female narrator. However, the effect on the male completion rate was found to be lower over time. In the case of female farmers, switching from a male to a female narrator increased the likelihood that they listened to most of the message, with more long-lasting effects. 

The findings suggest that both male and female UCAT farmers are more sensitive to messages voiced by a female narrator. However, the short-term nature of some effects may prevent such a change from generating long-run changes in farmers’ engagement. Moreover, only two narrators of each gender were involved in the recording, so we cannot rule out the possibility that we are capturing a “narrator” effect rather than a gender effect. 

Implications for programming and future research

A direct consequence of the findings of this study is that the new UCAT content will be entirely voiced by female narrators. Farmer engagement following the introduction of this new content will be monitored to assess whether any change arises, leading to improved listening and completion rates.

Ethiopia 

Habtamu Yesigat, Director of Programs, Ethiopia

PAD’s service in Ethiopia 

PAD has collaborated with the Ethiopian Agricultural Transformation Agency (ATA) since 2017 to measure and improve the effectiveness of ATA’s 8028 farmers’ hotline service. The 8028 farmers’ hotline, introduced in 2014, is an IVR service available in five languages which has reached more than 5.4 million farmers. A farmer can call into 8028 – free of charge – and navigate through different menus to reach advisory content on 21 crops and four livestock commodities. 

How the A/B test varied the gender of the narrator, and why we decided to test this question

Historically, the 8028 farmers’ hotline content has been narrated by a female journalist. In an international benchmarking study that PAD carried out in 2018, we realized that inbound systems in other settings had tested a variety of techniques to attract more farmers to call in and to encourage users to listen longer. Varying the narrator of the agricultural content (for example, varying the gender of the narrator, varying whether the narrator is a journalist or a subject matter expert, or using the voices of celebrities, agriculture leads, and scientists) is among these strategies.

The evaluation covered 462,000 farmers enrolled on the 8028 farmers’ hotline service and ran during Ethiopia’s primary rainy season in 2019 (June through August). Farmers who participated in the study were growing a variety of crops including wheat, maize, malt barley, sesame, and teff. At the beginning of the study, each farmer was randomly allocated to one of the three treatment arms, and received a series of push calls throughout the season to provide relevant to the farmer’s crop and location.

The three treatment arms were: female journalist narrator, male journalist narrator, and a male agronomist narrator. The study assessed whether there was a difference in the average time a caller spent listening to the service when we changed the gender and profile of the narrator. However, because there was no treatment arm with a female agronomist narrator, we were not able to fully understand the interactions between the perceived gender and expertise of the narrator.

Results of this A/B test

Both the pick-up rate (proportion of farmers who picked up the phone) and listening rate (proportion of the call duration to the total length of the content) were higher for the male journalist narrator and the male agronomist. We saw no significant difference in pick-up or listening rates when comparing between the male journalist and the male agronomist.

With this evidence, and acknowledging the fact that we didn’t have a female agronomist voice as a treatment arm, we can only speculate as to the mechanisms driving the higher engagement rates for male voices. There is a socio-cultural association between agriculture and men which may contribute to the difference. In addition, we did not include a treatment arm featuring a female agronomist; it may be the case that farmers were more interested in listening to an agricultural expert rather than a female journalist, but that they would have been equally receptive to a female agronomist. 

Implications for programming and future research

This evaluation has helped us to think about the multiple factors that must be considered when evaluating a change to the narrator’s voice. When we next implement an outbound call service in Ethiopia (which we will likely do in the next few months), we plan to repeat this experiment with the additional confounding factors in mind.

Gujarat, India 

Revati Vaidya, Process and Product Innovation Associate

PAD’s service in Gujarat, India

The Indian state of Gujarat is home to PAD’s oldest program, which began operating in 2016. The service, called Krishi Tarang ( “agriculture wave” or “vibe” in Gujarati and Hindi) currently serves over 36,000 farmers. Krishi Tarang provides farmers with free, customized information in two ways: i) weekly voice messages are pushed to enrolled farmer’s mobile phones (“outbound” service), and ii) farmers have the ability to call into an IVR hotline to record a question and access other information (the “inbound” service). Messages recorded by farmers via the hotline are answered by agronomists via push call within 48 hours. Traditionally, calls from Krishi Tarang have been narrated by a male voice.

How the A/B test varied the gender of the narrator, and why we decided to test this question

At the beginning of the 2016 Kharif season, PAD implemented an evaluation to test four different narrator voices to see how they impacted farmers’ listening rates. Approximately 3,000 farmers, who were registered as Krishi Tarang users at the time, were each assigned to receive calls from one of four different narrators: a male agronomist (who identified himself as an agronomist during the recording), a farmer (generally male, who identified himself as a farmer during the recording), male voice (no occupational emphasis), or a female voice (no occupational emphasis).

This A/B test took place over four weeks during Kharif, from the end of May to the 21st of June, 2016. 

Results of this A/B test

No statistically significant differences were observed among the listening rates across the four groups (i.e four voices). No one voice emerged as the best choice. 

Implications for programming and future research

Given that there were no statistically significant differences in listening rates observable across the four voices, no changes were made to the service.

Quo Vadis?

To recap: in Uganda, switching from a male narrator to a female narrator led to higher engagement, while in Ethiopia, the opposite was true. In Gujarat, there were no significant differences based on either the gender or the stated expertise of the narrator.

From a social science perspective, these results underline the extent to which gender is a social construct bound by time and place. Gender dynamics are highly context-specific, and findings cannot easily be generalized from one setting to another. The association between a narrator’s gender and their perceived expertise is just one of many important questions that we, as researchers, must try to understand before drawing firm conclusions about the role of the narrator’s gender on listening rates within a given population. At a more general level, these diverging results reinforce the importance of local learning and local knowledge, and the importance of customizing our services and information in situationally-appropriate ways (see for example our recent blog post on more intentionally understanding the needs of female farmers in the Kenyan dairy sector).

From an implementation perspective, these results demonstrate the value of collecting data on the gender of PAD users to improve both individual customization and overall program design. However, collecting detailed information on user characteristics involves important tradeoffs: We need to ask more questions of farmers to deliver personally-tailored services, yet asking too many questions will lead some farmers to drop off. Other farmers may balk at perceived privacy implications. Moreover – as detailed in another recent blog post – in smallholder households, mobile handsets are often shared among adults, so we are not always sure whether the person answering the phone is the person whom we have profiled.

As PAD continues to emphasize gender in our research and program implementation, we anticipate wrestling with difficult questions. For example, what if we find that behavior change is more likely to occur when our content is delivered in a way that reinforces patriarchy or other existing social prejudices (e.g., a narrator whose accent is associated with a particular social class or privileged group)? On the other hand, we have also begun to consider whether our service might be able to specifically target social biases based on gender and other characteristics. For instance, we might test two versions of a message delivered by a female narrator: one that includes an introduction highlighting the narrator’s technical skills, and one that doesn’t, to understand whether emphasizing the narrator’s qualifications might generate additional behavior change.