Do NGOs make a difference, and how would you know?

Everyone in development wants to know what difference their work is making.  International NGOs such as CARE are asked the question in different ways. Our supporters want to know their money is helping those who need it most.  Our staff and partners want to know how we can do things better.  Our institutional donors want assurance of effectiveness, impact and value for money.  Our detractors wonder whether we really make any difference.  Against this background, CARE recently launched its Asia Impact Report [pdf]. This provides analysis of CARE’s programs and projects undertaken with partners over five years across 16 countries of Asia and the Pacific, ranging from Afghanistan to Vanuatu.  Our intention was to be more accountable and transparent; to get a better understanding of the impact of CARE’s work in the region; to improve our evidence base for CARE programs and advocacy; and to inform improvements in program monitoring and knowledge management.

The report compiles project information, drawing primarily on project evaluations and completion reports.  Data on project impact and outcomes from different countries was compiled against indicators aligned with particular MDGs, so we could get a sense of how CARE’s work has contributed to progress against each goal.  Country level impact examples gave a good sense of how CARE works in a given sector, and what has been achieved.  Where possible, impacts were also aggregated across two or more countries to get a bigger picture.  Independent technical review of CARE’s statistical analysis of aggregated impact data was undertaken by the firm TANGO International.  The impact analysis was supplemented by a survey of external stakeholders to obtain their views on the contribution and impact of CARE’s work; and a value for money analysis, using case studies to look at the investment and social return in four selected projects.

So what did we learn?  From a comprehensive report, here’s a few observations:

The report gives us strong evidence that our work is making a real difference to people’s lives.  This report provides specific evidence of where CARE and our partners have made a positive impact on the lives of millions of poor and vulnerable people in Asia. At the aggregate level, examples include reaching 9 million people through initiatives addressing income poverty, including increasing the average annual income of almost 2.7 million poor and marginalised people on average by 117% in Bangladesh, India, Sri Lanka and Vietnam.

NGO programs are often characterised as innovative but with sadly limited reach. However, the report shows NGOs are also able to achieve larger scale impacts – either directly with our partners, or through our advocacy with partners and governments.  For instance, in Bangladesh our work with 2 million of the country’s poorest people in areas of maternal and child health, nutrition and income generation produced substantial results.  Evidence showed a 4.5% annual reduction in stunting among children in CARE’s target population, compared with a 0.1% annual decline in Bangladesh as a whole over the same period.  Analysis also showed that women’s empowerment was the single biggest contributor to reducing malnutrition and child stunting compared to other project interventions, even the direct provision of food to mothers. In Cambodia, CARE’s development of an effective model of bilingual education for ethnic minority populations in one province, coupled with research and advocacy, has led to changes to the National Government’s education policy and replication in five other provinces – multiplying the impact of CARE’s effort for the benefit of even more marginalised ethnic minority people.

In terms of value for money (VFM), case studies show our program can provide significant positive returns for each dollar invested.  When considering how to address VFM in the report, we quickly concluded that a whole-of-portfolio analysis would be methodologically problematic and unlikely to give meaningful results.  Instead, we took the case study approach noted above to see what we could learn, piloting a modified form of the Social Return On Investment (SROI) approach.  In themselves, the results were encouraging – a social return of between $7 and $42 for each dollar invested.  However, the analysis has also stimulated useful discussion within CARE Australia on methodologies for VFM assessment, and the extent to which our systems and processes already incorporate VFM considerations.  We are feeding this learning into broader discussions within the sector.

Finally, there are things we need to do a lot better.  The experience of the report highlighted a number of lessons for us, particularly in monitoring and evaluation (M&E), impact measurement and knowledge management.  Aggregating impacts across diverse operating environments is tricky enough, but made more difficult by a wide variation of approaches taken to M&E at the project level. Synthesising evaluations becomes more challenging when gaps in our knowledge management systems mean we have to spend time tracking down a mislaid evaluation report.  We need to keep improving our impact measurement processes so we are measuring what really counts, rather than just what we can count.  Feedback from our partners included much that was positive about working with CARE, but also comments on making sure that we really learn from our field experience, and improving our impact at the policy level.

The CARE team who produced the report is proud of the end product, but we’re also aware that if we’d had better systems and processes in place, we would have been able to tell an even more complete and compelling story about our work.  Several issues identified are already being addressed within CARE, including through the development of an improved Program Information and Impact Reporting System; a strong focus on long term programs with a 10-15 year time horizon, with broad scale impact at the core; and moves within CARE International towards greater coherence and quality of our programming efforts.

We hope the review is a useful contribution to the broader dialogue on aid effectiveness.  We are aware of the strong interest and commitment around the impact agenda from many donors and gratefully acknowledge the valuable support of AusAID and DFID towards elements of the report.  We believe the regional five-year impact approach may be unique among NGO reporting, and would welcome feedback.

Andrew Rowell is Manager of Quality and Impact for CARE Australia.