By Catherine Lalonde and Kathleen Schaffer
Catherine Lalonde is the senior program officer for the Francophone Africa program, and Kathleen Schaffer is the senior program officer for the Anglophone Africa program.
Evaluating advocacy is far from simple. Advocacy is not straightforward, as advocates often need to readjust strategies to influence decision-makers when government leaders and policies change. So it’s often difficult to attribute a policy success to a specific advocacy effort. We are grappling with these challenges firsthand as we evaluate our advocacy project Mobilizing Advocates from Civil Society (MACS).
MACS begins with the formation of alliances of civil society organizations—groups of concerned citizens, community organizations, health professionals, religious leaders, and activists—and equips them with the skills to monitor governments’ commitments and their implementation. In Burkina Faso and Kenya, these MACS alliances work across the continuum of reproductive, maternal, newborn, and child health (RMNCH) to push for more progressive, effective health policies and programs.
Alliance members have been working side by side for the past three years to harmonize their messages, hone in on their concrete advocacy ask, and diligently navigate the political environment within their countries. Because alliance members know best what project success looks like, it was critical to include them in the design of the evaluation through a participatory approach.
The evaluation of the MACS project will reveal planned and unplanned outcomes, FCI’s contributions to these outcomes, and key enabling and hindering factors. Specifically, the evaluation will answer the following:
- Effectiveness of alliances: How well did the national alliances function? How effective have these alliances been in holding governments accountable to their promises to RMNCH?
- Capacity to advocate: To what extent has FCI prepared alliance members with the skills and knowledge they need to be powerful advocates? How much has the capacity of alliance members increased over the course of the MACS project?
- Achievements: What are the gains we achieved through our advocacy efforts?
Through workshops in early June in both Burkina Faso and Kenya, we asked alliance members to tell us how the project should be evaluated by defining criteria of success and identifying specific ways we could measure evaluation questions.
To understand how they define positive change within their respective contexts, we asked alliance members to identify project achievements. Some notable achievements included:
- Advocacy targets have now become allies.
- The government listens to us.
- We gained credibility with decision-makers by completing a budget analysis; we are now recognized as both RMNCH and budget experts.
- We have pride in the work we’re doing.
- We can mobilize and move quickly.
- We are learning by doing, and we are learning from each other.
- We are creating evidence for advocacy.
In Burkina Faso, alliance members said that the MACS project is helping to improve democracy. Civil society organizations are asking government officials for budget and spending transparency and increases in RMNCH budgets, and people are watching them make these asks. Burkina Faso only recently emerged from a 27-year dictatorship, yet advocates are working with the government to create positive change! We never imagined our project would have had such a broad impact, but now the Burkina Faso alliance sees democracy as a criterion of success! This will be even more important as the government of Burkina Faso recovers from its September 2015 political crisis.
Apart from discussing criteria of success, alliance members defined concepts in each evaluation question and devised a four-part scale to measure them. For example, alliance members defined advocacy capacity (under question 2) as how well participants were able to recall and describe the new knowledge they obtained from technical trainings, or how well they could adapt the skills they learned to their context.
Using information from the pre-evaluation workshops, we drafted interview questions, and we are working with our teams in Burkina Faso, Kenya, and New York to conduct a desk review of relevant documents, interviews and potentially focus groups and surveys.
Through this participatory exercise, we culled rich information on project successes and areas to strengthen by consulting the very people with whom we’re working and whose lives we seek to improve. Alliance members were just as interested in grading themselves and their work as they were in grading how well FCI supports and leads the alliances. They feel a real sense of ownership and responsibility, and they are keen to use the evaluation as a learning tool for improving future work. In Kenya, one alliance member commented, “Because of our participation in this evaluation exercise, we’ve learned how to evaluate our own programs.”
When we started this project, we had very black-and-white, fixed criteria for success, some of which were unreasonable. But then we realized that these alliances have achieved different results than we or they originally imagined, and that they see these achievements as meaningful and important gains for improving women’s and children’s health. Advocates in Burkina Faso and Kenya are proud about what has been achieved so far, and excited about what they have the potential to achieve in the future. The real magic of using a participatory approach is discovering results you could have never imagined—and you just might learn that the project contributed to democracy.