Comments

Thanks Chris for your detailed comments. I agree that ODE forms only part of the evaluation team and indeed the article is directed to AusAID and not specifically ODE. I also agree that evaluation has to be tailored to methodologies that are suitable to those needs (and RCTs are not suitable in many cases). That said, where practical and cost-effective, an RCT is generally considered the most rigorous quantitative technique. The key way any other quantitative approach would be superior to an RCT is if available observable data is very highly correlated with unobservable characteristics (of units in the sample) or if unobservable characteristics are unlikely to influence outcomes or participation – both which don’t seem to happen widely in practice. Apart from the above point, the debate focuses largely on the disadvantages of RCTs (which certainly exist), and less on the disadvantages of RCTs relative to other quantitative techniques. In the large majority of cases where practical and where the benefits outweigh the costs, RCTs represent the most appropriate quantitative approach. Combine that with the most appropriate qualitative component, and it would provide a strong basis for causal inference – hence the term platinum standard. Unfortunately only a small percentage of projects will lend them to this type of scrutiny and hence other evaluation approaches to provide rigour, quality and validity are more practical. I am encouraged that AusAID are looking at an array of ex-ante evaluation approaches which can only improve on the good steps AusAID has taken to make performance standards stronger. As to your last point, I can only hope that development effectiveness is measuring value for money and feeding into new programs and projects.

I think this kind of research (produced by male economists, I notice) treads a potentially counter-productive path, it really worries me. I don’t think we should be trying to argue for women’s representation in economic terms. It paves the way for people to then produce research showing the opposite (of which there is some) and then argue over the numbers, losing the point that women are half the population therefore they ought to be half the decision-makers. It also essentialises and limits women who do gain positions of power. They end up being disproportionately punished for not living up to the superior standards expected of them.

Some of these issues were considered about 20 years ago, under the heading of the "peace dividend" after the end of the Cold War. During the 1990s, there seemed a strong movement to support the UN peacekeeping efforts through the "Blue Berets" - note "Berets", not "Helmets". If we are to spend money on the Australian Defence Forces, then their use in supporting the UN peacekeeping missions would be welcome. As we did after the first Gulf War in northern Iraq ("Operation Habitat"), in Cambodia, Somalia, partially in Rwanda and could do elsewhere. Still. Recognising that the ADF is on a war-footing at the moment, then there are military skills in medicine and building clean water wells that can still be utilised to protect civilians in times of civil unrest and armed conflicts. Especially, if you happen to believe in the following obligations of the Convention on the Rights of the Child (signed by Australia) and CROC's Article 38(4): "4. In accordance with their obligations under international humanitarian law to protect the civilian population in armed conflicts, States Parties shall take all feasible measures to ensure protection and care of children who are affected by an armed conflict".

Thank you Jonathan. It's interesting: in terms of welfare implications Dinuk and Paul's post on the growth impacts of women in parliament is much more significant than this post here. And yet it's my post that has attracted more attention. The reason for this I think is probably, as you suggest, a mixture of guilt and the fact that the story is a personal one that most development folks can related to. While it is the big things and big trends that really matter for development, at the end of the day we are all individuals caught up for the most part in our own lives and dilemmas (or, at least, I know I am), and I guess that's why it has resonated.

A fantastic post Terence. It's interesting that the topic of aid worker entitlements consistently gets a unique level of development community engagement. Duncan Green's <a href="http://www.oxfamblogs.org/fp2p/?p=8258" rel="nofollow">piece</a> on the Nairobi pool furore has certainly gotten it's fair share of comments. One of Chris Blattman's most commented on <a href="http://chrisblattman.com/2009/05/09/should-development-agencies-fly-business-class/" rel="nofollow">posts</a> was also of the same ilk, instead asking the question of whether development agencies should fly business class. I must say your post did provide a much more balanced critique than his! I wonder, is it an underlying sense of guilt (or entitlement) that people so willingly throw themselves into public discussion about this issue, or is it rather that it is one of the only topics in this sector that development workers (traditionally from the developed world) can most easily relate to?

Thanks for the reply and explanation. Indeed an interesting and important area for future research.

Dinuk, I am coming to this conversation late having just returned from the field, but you raise some important points that deserve exploration. First, I am not sure how helfpul it is to put 'platinum' or even 'gold' labels on approaches in evaluation. As a professional evaluator, I have always felt the best approach is to think carefully through what you are trying to achieve with your evaluation and then choose and explore the methodologies that are going to suit those needs. The whole issue of labels tends to open debates around standards rather than focusing on rigour, quality and validity. Some nice responses on this recently in the Journal of Economic Literature from Ravallion and Rosenzweig in their reviews of Banerjee and Duflo's book "Poor Economics". Second, the current performance and evaluation policy in AusAID allows programs significant freedom in choosing their own approaches to performance tracking. Ex-ante evaluation is encouraged and is increasingly being adopted by the more performance oriented parts of the agency (admittedly it is often the better funded country programs - Indonesia, PNG and Philippines). There are also evaluators working with a number of AusAID teams and they are regularly having conversations about pipeline evaluative activities. It is often overlooked that ODE only forms one part of the quality system in the agency and there is a whole team in the Program Effectiveness and Performance Division tasked with driving the evaluation policy. The real challenge for this section is driving policy change alongside adequate and supported cultural reform of an agency filled with Development Generalists. Which brings me to my third point. The main constraint in taking on this approach has always been expertise rather than cost or willingness. AusAID has few professional evaluators in its ranks and even fewer individuals capable of getting their heads around an RCT or quasi-experimental evaluation design. The field of evaluation in Australia is heavily qualitatively focused and the ability to draw on a pool of quantitative evaluation experts that understand the development context is not straight forward. As I understand it, AusAID is having conversations with J-PAL and others about running this type of evaluation, but there are all sorts of constraints and challenges that need thinking through before launching into the misguided policy (see recent USAID proclamations) of believing independent evaluations will be the answer. You suggest universities as a possible partner and there is value in better utilising these institutions. However, universtities have often had their own issues with being involved in these operational undertakings. Contrary to popular belief, my experience has been that AusAID is very open to starting a conversation with groups that could undertake this type of work and the political barriers are often overcooked. Thanks for raising the issue as I think it is an important component of the development effectiveness debate. What I think is even more important though is raising the whole issue of where the issue of effectiveness and value for money is going. There are some important aspects of the recent AusAID agency results framework that have huge ramifications for how the sector works and this deserves further discussion. <em>Christopher Nelson is a Monitoring and Evaluation specialist with the World Bank Group and a former M&E advisor at AusAID.</em>

Thanks Scott for your comment. We agree that our hypothesis is not the only possibility; we were thinking of the average country and control for the possibility a fairer society is driving the result. There could be an ideal limit (i.e. is there a critical mass of females in parliament where collectively they can influence legislation that increases growth?) and as you point out diminishing marginal returns. And there are likely to be several channels via which this macro effect could work; corruption and public goods provision are two potential channels. Chattopadhyay and Duflo (2004, Econometrica) find that women invest in different types of public goods (e.g. more drinking water provision, but less of some other things). Other interesting areas to look at are whether there is a difference between developed and developed countries. Clearly, more research is needed to see how such factors might affect aggregate economic growth.

Thank you Seini - excellent comment (and sorry for the delay moderating it, it came through after I'd gone to bed last night.) The point you make is important. It stands to reason that if locally engaged staff feel inequitably treated their performance may suffer (never mind the inherent unfairness of the situation). One solution would seem to be to pay local staff higher rates. Here's the complicating fact: they already, usually, live comfortable lives by local standards, their pay typically putting them amongst the middle classes of their home countries. And there is a risk that if they are paid a lot more, aid might end up distorting local job markets, meaning that the most talented local staff strive to be aid agency employees rather than working for their own governments or private sectors, which do desperately need their skills. At least, that is the counter argument to what you are suggesting. Although I confess it rings somewhat hollow in my own ears when I make it. And I'm sure there may well be, in many cases, room for improvement in local pay. Thanks again. Terence Oh - and while I remember. The salaries of the staff of multilaterals is currently causing a scandal of sorts: http://www.guardian.co.uk/business/2012/may/29/christine-lagarde-pays-no-tax

Thanks Terrence for starting this flow of reflections. The aid worker-aid recipient living standard divide is an issue I've thought about a lot, and for which I've concluded that there are no easy solutions. People seem to draw the "too much" line at different levels: for some, a jeep may be too much in the way of luxury, while for others such a vehicle may be justifiable, but a luxury hotel with a pool is definitely too much. But perhaps there are some objective indicators of where 'too much' might lie. For instance, some psychology researchers have looked into what makes work environments in developing country contexts most effective, and have concluded that pay differentials, when they become sufficiently large, certainly don't help. Across a six-country, 202 organisation sample, they found that expatriate workers’ salaries were, on average, four times the level of salaries for local workers in the same organisation. In the two Pacific countries they looked at, the expat : local salary ratio was 9 : 1 (!). Being psychologists, they weren’t just interested in these numbers, though – they also asked people how they felt about them. Here’s what they found: “Local participants reported that the pay disparities they experienced lead to significant feelings of injustice at work. They told us that the disparities left them feeling significantly de-motivated in their everyday work. At the same time however, and equally worryingly, international participants were largely unaware of the feelings of injustice caused by their significantly higher pay (and benefits), and did not recognise that these disparities were leading to reduced motivation in their local colleagues.” (see: http://www.humworkpsy.org/resources/journals/Aid_salary_disparities_Just_%20Change_June2010.pdf ) For me this finding raises a lot of questions about how we could do things better. And perhaps it adds an additional clause to your article title…”Should(n’t) aid workers lead comfortable lives – even if they’re not expats?”

Thank you Rhianon. You wrote: "So the question is, for me, not so much “should” skilled international aid workers live comfortable lives, but will we be able to get them to come if they don’t?? It would take a massive cultural shift." It would also take an economic shift of sorts, I think. Right now I think it's reasonable enough for aid workers to worry about their mortgages and kids' schooling and the like, which means they aren't going to work low paid jobs forever. At least not while our economies back home leaves us worrying about these sorts of things.

I think that arguments about what "should" be fade a bit in the face of what "can" be. I have done stints as a volunteer (AVI) and am now in Fiji as a paid worker on a good local wage in an aid funded NGO. It was annoying that it was more lucrative for me to be a volunteer... but what is more annoying is that we currently have vacant positions for roles that would have a fantastic impact on local quality of community services, but we can't fill them because we pay a middle-upper class local wage - and we simply can't get people from developed countries to consider it. Great useful jobs, enough money for a very comfortable life here - but when converted to Australian or New Zealand dollars, people don't want to take the money. So the question is, for me, not so much "should" skilled international aid workers live comfortable lives, but will we be able to get them to come if they don't?? It would take a massive cultural shift.