Comments

It is important to note that in some aid agencies, the fervor for RCTs has led to an expansion of projects that can be evaluated in this way at the expense of those for which RCTs are not appropriate but which might have a longer lasting impact. For example, Andrew Natsios, a former Administrator of USAid describes in "The Clash of the Counter Bureaucracy and Development" how the demand for the so-called "gold standard" of RCTs (although I see that it is now "platinum" - is this evaluation inflation?) has biased the choice of projects that USAid supports towards those that are suitable for RCT evaluation. He points out that initiatives that have the most transformative impact are those that are difficult to measure and also carry the greatest risk of failure. No good bureaucrat wants a failed project, so risk taking in this way is eschewed. I have long maintained that we can learn as much from failed projects as we can from successful ones as long as the reasons for failure are carefully documented and become part of institutional memory and future project design. Alas, one of consequences of the rapid turnover of staff in many aid agencies means that institutional memory is rare or non-existent.

Dinuk, I agree with your observations regarding the need for mixed methods approaches, and an ex ante evaluation perspective. I am the research director of a large evaluation of the AusAID-funded, Australian Sports Commission-implemented Australian Sports Outreach Program. This evaluation is employing a mixed methods approach, however the quantitative component is restricted due to the ex post nature of the design (and therefore no baseline, no possibility of a control etc), this is a pity but there is not much we can do about it (so is this a 'bronze' or maybe 'silver' standard design?). Due to the fervor surrounding RCTs there is increasing pressure on evaluation consultants to employ the most rigorous methods possible, however for the reasons you mention this is almost always not possible; more needs to be done to ensure that donors understand when such methods are possible and when they are not. I agree with your call for AusAID to review upcoming projects to assess which of them may be suited to platinum standard evaluations. David <em>Dr David Carpenter is the Principal Consultant, International Development at Sustineo in Canberra</em>

Many thanks to Jonathan Pickering for his thoughtful assessment of our paper and blog. Jonathan sets out three reasons for optimism about a demonstration project approach. Firstly, he mentions the $100 billion climate finance target by 2020 agreed at Copenhagen as a potentially major source of REDD funds for Indonesia. Here we simply repeat the fundamental point we make in the paper and blog: “The prospect of large-scale REDD funds arriving some time in the future is too distant and uncertain to provide sufficiently strong incentives to tackle the deep-seated drivers of deforestation and peat conversion in Indonesia.” The second reason given for optimism is that not only is the 2011 moratorium in place, but that a new regulation on spatial planning is in place for the Kalimantan provinces. As our paper points out, abuses of the moratorium are rife. Sadly, such abuses are ongoing, the latest case in point being the Tripa peat swamp in Aceh. This area should have been safe from conversion for oil palm, being covered both by the moratorium and by Indonesian law which protected deep peat like Tripa from conversion. But Tripa was excised from the moratorium, and in recent weeks canals were dug and fires set to destroy the forest. A legal appeal by an Indonesian NGO was rejected. The Presidential Regulation on spatial planning in Kalimantan also needs close examination for its actual impacts on the ground. The CIFOR response was cautious enough; another commentator argued that the decree’s removal of the `Limited Production Forest’ category could result in the loss of a further 3 million hectares of forests in Central Kalimantan. We agree with Jonathan’s third point that demonstration activities have an important role to play. But then it is up to KFCP to actually demonstrate something. While its community efforts may have been useful, after nearly four years not a single hectare of drained peatland has been re-flooded, nor a single tonne of peatland emissions avoided. Are the foundations of a peatland REDD mechanism in place? As far as we are aware, long-identified gaps in scientific knowledge of peatland emissions remain unfilled. Until this happens, it will not be possible to establish a sound accounting methodology for peatland emissions, not can any emission reductions achieved through KFCP project be measured. Not to our knowledge have baseline reference emission levels been defined. Thus little progress has been made in demonstrating how a peatland REDD mechanism could work. Over the last four years perhaps another 5 million hectares of forest have been lost in Indonesia. How many more will be lost by the time KFCP has successfully demonstrated a peatland REDD methodology? Perhaps, as Jonathan argues, KFCP is raising the profile of peatland emissions in UNFCCC negotiations. Though what, if anything, these negotiations are leading to is unclear. And the use of KFCP in the negotiations is a double-edged sword, as it adds to the pressure to put a positive spin of KFCP progress. Thus, Australia’s December 2011 submission to UNFCCC’s SBSTA states that `The Kalimantan Forests and Climate Partnership (KFCP) is one of the most advanced large-scale REDD+ demonstration activities in Indonesia’. It is a sentence often seen in official reporting on KFCP progress. In what sense KFCP is one of the most advanced REDD activities escapes us. It is this slow progress, and the assertion that KFCP may be performing as well if not better than other REDD projects, against the backdrop of continued deforestation, that draws us to the conclusion that there has been too much focus on REDD demonstration activities, and that it is time for a new approach. Erik Olbrei and Stephen Howes

Thanks Steve for a useful account on the 'rise and fall' of poverty (as an issue of development policy focus, not poverty itself) in the Pacific. Poverty is a prickly and often politicized issue in many Pacific countries, including my own, the Marshalls. Some, but not all, leaders are still convinced that we in the Pacific, with our extended families and access to land and basic resources like fish and traditional crops, are somehow magically immune from hardship. But for many people this is far from reality. In the Marshalls I would say fully one-third of all families are really struggling to survive - and things are not improving. This is a major reason for the massive outmigration we are witnessing. Traditional safety nets for many islanders are deteriorating, but modern safety nets are not there to replace them, so the most vulnerable among us are simply falling through the cracks, unless they can migrate or get some remittances. This is especially a problem in no-growth economies and regions, of which there are quite a number in the Pacific. But even in high growth situations, poverty persists, which has led us to the new focus on 'inclusive growth' and some resurgent interest in social protection. In short, I agree!

hi ... interesting blogs and perspectives .. just wanted to add re importance of context, particulary when aid donors are funding reform programs - need to be realistic about how long it takes to achieve expected outcomes and also the signifcant challenges and issues that needs to be addressed .. .. also ongoing costs and maintanence of various initiatives introduced through aid programs and its implications on Goverment's recurrent budget needs to be considered particularly before procurement of expensive equipment and other big ticket items .. as well as maintanence costs of infrastructure, etc funded through aid funds ..perhaps 'affordability' of aid interventions particularly for the recipient country also needs to be part of the discussions on aid efficiency and effectiveness ... welcome other's views on the issue thanks

Good article. A big issue is measuring the effectiveness of scholarships. It becomes more difficults with the short term awards. How can one determine impact of the short term awards that Australia is increasingly providing….

In reply to Joel's comment about saving lives, World Vision estimates that the delay could result in at least 250,000 lives not being saved over 4 years. This assumes an average cost of $2,000 per life saved (based on Global Fund figures - http://bit.ly/JeiHqL) and 18-20% of the aid budget going to health. The cut in planned ODA is $2.9 billion over four years. It is often possible to save lives for much less, for instance the most cited figure to prevent people dying of TB is $250 and even less for malaria and some vaccine preventable children’s diseases (see http://www.dcp2.org/pubs/PIH) . There are established effective health strategies to prevent deaths from AIDS, TB, malaria, maternal and infant health and early and poorly spaced births. There are also significant unmet needs for additional funding in all of these areas.

Megan, We got the figure from adding together the NPP's outlined on page 5 ('budget highlights') of the blue book (which you can find by following the link at the bottom of Stephen's post), which actually comes to $1.47 billion. Although these figures do cover different time commitments (1, 2 and 4 years) I think the point was to highlight that the NPP culture is still very alive and well even though the review recommended it be cut completely. Jonathan

Hi Stephen, Thanks for this balanced set of rapid reviews. Lots to digest. Not sure if Tim Costello's claim that this budget will lead to 200,000 deaths is sound science but the anger over the broken commitment is understandable. On the Africa increase, page 18 of the CAPF states that of the $1.5b going to Africa in 2015-16, $1b of that will be from global programs and $500m from country programs. So, as you note, still an increase from the suggested $200m cap but the emphasis certainly does seem to be on using multilateral partners. From a public health perspective, interesting to note lots of emphasis on water and sanitation - an area that is often neglected by aid programs (toilets just aren't such an easy aid image to sell!). Thanks again for your reviews and look forward to more discussion on this... Joel

Hi Stephen, I agree with your assertion about New Policy Proposals (NPPs) leading to fragmentation and confusion in the aid program, but not sure how you arrived at the figure of $1.5 billion. Could you please clarify? Many thanks

Mr. Nik Soni, It is very interesting blog Your blog is touch with the reality of foreign aid projects. I am doing a Ph.D. research about the Local knowledge Syndrome and donors-prompted public sector reforms with special attention to the case of Yemen. I think it is very important to focus on process. If we have an effective process, we will have an effective aid project. I think my Ph.D. research covers this subject in good way.

Thanks Robin for the comment, Our central point is to do with donor generosity. It is true that recipient countries might not feel the true force of these cuts, at least not immediately. However, we mostly wanted to highlight that globally donors seem to be viewing their aid programs as expendable, with a few rare exceptions. Even in countries which haven't cut dramatically aid, there are grumblings of discontent and politicians are distancing themselves from aid commitments. When was it decided that aid to the world's poorest should be the first to go when things at home get tough?