Comments

Thanks Shaun, I agree we live in a world of limited resources and that efficacy matters. It naturally follows that I think evidence is important too. My main difference with MacAskill when it comes to aid is simply that we shouldn't be (overly) guided in choosing the work we do by the **type** of evidence we can obtain about the efficacy of our work. Terence

Thanks for elaborating Terrence. But I would still suggest in a world of limited resources (the only kind we are likely to know) evidence that a problem is important gets us part of the way there, but we should still be wanting some evidence of (positive) results if we devote our resources to that problem. Sadly there may be some insoluble but important problems, or at least those we don't have the wherewithal to solve now, that we must logically pass over to ensure our resources go where they can do good.

I wonder if how many MDBs a country is a member of is a very good measure of anything other than how unfocussed their aid program is. Would there not be something to be said for a country deciding on which MDB delivered the most effective and relevant outcomes and putting all their resources behind that.

Yes, join AfDB, and perhaps leave EBRD?

Thanks Sue, it is very interesting to get an NGO's perspective. And also to hear of events such as the one at the university of California. For what it's worth (and conceding you have greater practical experience than I) I do still think aid organisations (NGOs, governments, and multilaterals) should be doing more and better impact evaluations (not necessarily RCTs; there are 'silver standard' alternatives, which might suffice). However, the point about cost that you make is a very good one (I think RCTs can be done for <500k, but good ones will naturally cost more than most other evaluation alternatives). High costs mean there's a trade off, ever dollar spent on an RCT is a dollar that could be spent on aid work. And choosing how to spend that dollar isn't clear cut. It would be great I think if government aid donors could come to the party here more often in their NGO work and allocate special NGO funding to cover the cost of best-possible evaluations in instances where the learning is important. There's still a trade-off, but nice if it wasn't born by a cash-strapped NGO. Thanks again. Terence

Thanks Terence for this review. I am the social accountability adviser for World Vision (citizen engagement in service delivery). World Vision and other NGOs, including CARE, Oxfam, Plan and Save the Children do undertake rigorous evaluations including randomised control trials when circumstances permit. See <a href="http://www.iig.ox.ac.uk/output/briefingpapers/pdfs/iiG-briefingpaper-18-community-monitoring.pdf" rel="nofollow">here</a> an Oxford led study of our social accountability work in Uganda. However RCTs are rare because of the lag times and the costs of such studies (between $500K and $1m) make these a prohibitive evaluation option except in very large projects. Even then, RCTs are such a specialised field that finding a researcher interested in the particular intervention can be difficult. I was recently at a 'match making' workshop run by the University of California and advocating a very interesting evaluation proposal around testing whether giving politicians information - as we do in our social accountability work - actually stimulates them to successfully advocate for better services to District government on behalf of their constituents as we are seeing happening across very different contexts including Indonesia, PNG and Uganda. We would have gone ahead but for the giant sample needed which only a government could provide. Unfortunately, donors also appear to be less willing to fund high quality researchers to do rigorous qualitative or mixed method evaluations that can be more suited to international development interventions. Beyond this, M&E consultants in international development are paid an absolute pittance, so it is no wonder we struggle so hard to get good evaluations.

I spend the last three years working with NGOs in all areas that include management, education, training, and or advocacy on current issues affecting the local Papua New Guineans, thus I am very much interested in some of the position s advertised. I like working with such an innovative organization who has the vision to educate and bring about services to the vulnerable population of PNG. I need direction on how to apply for Women's Empowerment and Protection Manager.

Thanks Shaun, An interesting comment and a useful chance to elaborate. I'm a utilitarian (or, at least, I think that's the least-worst political philosophy). MacAskill is not (see endnote page 215), but he is very close, meaning our views are almost identical in principal. The issue is to do with practice. Here, the question, when concerned with consequences, is how does one make choices given uncertainty both about probability (of success, or risk of occurrence etc) and magnitude of effect. As I state in the review, MacAskill is aware how hard this can make decision making (i.e "this is not exactly physics" p. 180). However, he does not seem to understand that this issue is as true for aid-NGO work as it is for many of the other issues he covers in the book. This, in turn, has him coming up with lists of recommended NGOs, whilst effectively dismissing the work of Oxfam, UNICEF and World Vision (presumably the UK versions of these NGOs). The certainty in pages 120 and 121 is striking and my read of these pages is that MacAskill is very confident he knows how priorities should be set in aid. I differ. As a Leap of Faith Utilitarian (I like your term, thanks) I believe that we should gather as much evidence as possible ("we should always try to gather evidence";"Randomised control trials are great"), and should not make decisions on blind faith (unless perhaps in crisis circumstances). However, there are many types of work an aid NGO could do, which clearly have value if they work, but which cannot be RCT'd. In the case of my examples. I think they are important, not because I had a whim after my second coffee last Friday, but on the basis of evidence: 1. Climate change (why worry: because the best available scientific models suggest there is a chance it will be utterly catastrophic). 2. Economic inequality (why worry: because it is high (good evidence) and it is growing (good evidence); there is also tonnes of evidence of diminishing marginal utility, meaning unnecessary inequality is welfare inefficient; meanwhile there is little evidence that quite significant increases in equality would harm growth in many countries). 3. Poor government aid (why worry: because the history of aid giving is festooned with examples of poor government aid and government aid is much larger than private aid in most countries.) To do aid work in the way MacAskill wants would mean aid-NGOs avoiding these areas. To do aid work in the way I want would mean tolerating some uncertainty of efficacy in exchange for allowing ourselves to tackle very important issues. Thanks again. Terence

Thanks CDH, that is a good point. They do RCTs where they can, and at times couple them with interesting methodologies such as process-tracing and where they can't they still evaluate their work. For a large NGO working across a range of types of work, this seems pretty much gold star.

Since you mention OxfamGB, worth noting that they openly publish impact evaluations of mature projects as well as other effectiveness reports <a href="http://policy-practice.oxfam.org.uk/our-approach/monitoring-evaluation/effectiveness-reviews" rel="nofollow">here</a>. They're not necessarily RCTs but as you say, not everything is going amenable to that

You accuse the author of failing to think carefully, yet the nub of your argument seems to be that his requirement for objective evidence of good fails to support some of the activities you personally like based on your "leap of faith" thesis. You might genuinely want to consider what principled approach you are advocating here.

While I appreciated the somewhat retro colours that were used to organise documents in the old AusAID page above, my main reason for suggesting a return to this is that it makes it easy for both DFAT staff and external readers to identify the state of documentation for an activity and also easier to find a document than in the somewhat quirky ordering on the DFID site.