Comments

At least one paper in that ACIAR series uses the term impact evaluation in a way that does not seem consistent with 3ie's criteria; other papers in the series are about "impact assessments", which also do not seem consistent 3ie's criteria. From limited sampling, it is likely there is an issue with the methodology from a 3ie perspective, although I'm not saying there is any wrong with the methodology based on the objectives of the ACIAR papers. On the other hand, the ACIAR series simply may not meet 3ie's criteria on publication.

it's great this conversation is opened up. i come from background of 25 years of communicating about aid (as a fundraiser).lessons learned: 1. don't talk about aid, talk about people 2. 'hero' is not Australian Government or DFAT or formerly AusAID, it is 'Australia' or 'Australian people' 3. go to where audience is 'at', not where we'd wish they would be 4. simple, emotional, first person 5. show problems 'fixed' 6.. and be patient 🙂

Hi, thank you very much for the information.

The table doesn't seem to capture any of ACIAR's impact evaluations - is there a reason for that? Are they not considered to be methodologically sound? There are quite a few for the Pacific - see <a href="http://aciar.gov.au/publication/term/25" rel="nofollow">here</a>

Ashlee - Your arguments rest on the assumption that DFAT management cares about aid.

It is not for me to tell 3ie how to categorize things, but as an author of several of those 18 papers I wouldn't consider many of them as "impact evaluations". By my definition, perhaps only the financial literacy study, which had explicit investigator-driven randomization. We would hope that all econometric studies that hope to uncover causal effects would be using good research designs that are not (too) prone to omitted variable bias (aka selection effects). Thus, I would think that the work we did on transport access and poverty in PNG using an IV strategy would qualify despite 3ie omitting it. When there is randomization available, as with the migration lotteries, it is good to use it. But it is hardly the silver bullet that too many development people think it is ("what works" should be more properly called "what worked, in that particular context when implemented by those particular people" because absent a theory as to why it worked there is no guarantee that it will work elsewhere - e.g. Rozelle et al Stanford REAP have new results on deworming that show no effect on school performance in rural Western China, contrary to the very publicized effects in Kenya).

For those interested, <a href="https://dfat.gov.au/aid/how-we-measure-performance/ode/Documents/impact-evaluation-discussion-paper.pdf" rel="nofollow">here is a discussion paper</a> on impact evaluations from AusAID / ODE. The paper is from 2012 and shows that greater use of impact evaluations was on the radar. Perhaps the embers of the Impact Evaluation Working Group are still warm?

Thanks, Ash, for this great policy brief and blog series. As far as I can tell, DFAT (and other aid agencies) have almost nothing to lose and everything to gain from communicating more, and more openly, about what Australian aid does and the difference that it makes in people's lives. I think your recommendation on increasing engagement mainstream media is a particularly important one; I imagine many of the people who go to the aid pages of the DFAT website or follow it on Twitter are already well-informed about Australian aid -- the challenge is reaching the broader public.

Here in Oxfam's water governance programme in the Mekong region we are putting the Augusto Boal 'Theatre of the Oppressed' into action to achieve development goals. We are working on some of the most conflict ridden and politically sensitive issues of natural resource grabbing and have found theatre is a valuable way for communities to examine and identify their own solutions. Skills are required to safely facilitate these processes and we are training 8 young people from the region in these methodologies. They will go on to work with hundreds of community members. government officials and private developers. You can learn about them <a href="https://drive.google.com/folderview?id=0BzneKwk1NKwmSFJCVDVIcGtaZE0&usp=sharing" rel="nofollow">here</a>. If you would like more information please contact me.

Thanks Patrick, I agree good panel data would be better than the cross section we currently have. Still the cross section is, I think, better than individual data points, and makes for a good complement to qualitative work. I agree with you that the nature of the issues of the day could well be important (we say as much in the blog and the paper). Avoiding the cases of any individual NGOs, in a broad sense I completely agree understanding issues such as values and how they might be associated with the variables we've studied makes a lot of sense and is the stuff of interesting future work (I think we discuss this possibility in the paper). Terence

Terence I think it may have something to do with values c.f Save the Children of recent times. NGOs have alway spoken out on values issues and the evidence seems to support the funding effect (in aggregate) seems to me at least, to be relatively weak. In the 1980s when they were more outspoken they even got more funds. Perverse incentives maybe

I just watched 'they only drink it in the Congo' in London, written by an aid industry outsider, Adam Brace. But it is excellent - touches on the white saviour complex, the realities of conflict and violence in diaspora communities and at home in the DRC, and the terrible disappointment of trying and failing to make things a bit better. Shame I can't see your work - sounds great!