In previous posts (here and here), we’ve given our general reaction to AusAID’s first Annual Review of Aid Effectiveness or ARAE (rosy and out of step with AusAID’s own reporting on country-level effectiveness) and we’ve homed in on its treatment of AusAID’s operational and organisational effectiveness (in parts worrying and uninformative). We’ve emphasized that we think the ARAE is a good process than can be made better. This final post expands on that constructive note, with six specific suggestions for improvements next time around.
Give first-tier results for our region. It is surprising that under the first reporting tier, relating to international development progress, the ARAE confines itself to summarising UN reporting on progress against the Millennium Development Goals (MDGs) at the global level. It contains no analysis of recent development progress and challenges in the Asia-Pacific region, which receives the lion’s share of Australia’s aid. Still less does it paint the picture at country level, despite the fact that Australia’s aid is delivered, more so than it has been for some years, primarily through country strategies and programs. It would make sense to set the scene for a discussion of Australia’s contribution to international development (the second reporting tier) by providing a good account of the fortunes of our region.
Relate claims to review and evaluation findings. There are literally no references in the ARAE to the results of reviews and evaluations conducted at the level of country programs, sectors or activities. AusAID’s Office of Development Effectiveness is only given the briefest of mentions, and none of its evaluative work is explicitly drawn upon. The failure to relate overall claims to the findings of reviews and evaluations is in sharp contrast to the approach taken by the ADB and the World Bank in their development effectiveness reviews, which depend heavily and explicitly on findings from their independent evaluation arms. AusAID has a review and evaluation base. The ARAE should use it.
Get beyond the headlines. A reasonable person has to allow that the litany of “headline” input and output figures provided in the ARAE, which respond to the “expected results” set out in the 2012 Comprehensive Aid Policy Framework (CAPF), serves some kind of a purpose. At the very least it makes concrete what the aid program is delivering, even if the numbers quoted have a numbing effect. At the same time, it is obvious that the expected results are quantified because they can be, and don’t represent everything that aid can and should try to achieve. This should be made clear, and the data underlying the headlines made available. Every now and then the ARAE ventures beyond the quantifiable. Statements such as “our assistance improved the quality of health services by strengthening the capacity of countries to build and maintain their own health systems” (p. 14) would be welcome, if any evidence were adduced.
Separate out the contribution of Australia’s multilateral partners. For now, AusAID can point back to the 2012 Australian Multilateral Assessment as a comprehensive appraisal of the performance of the multilateral development organisations it funds. This ARAE doesn’t and didn’t need to repeat that appraisal. However, it also doesn’t make any practical distinction between results delivered through bilateral programs and those delivered through global programs—particularly through funding for multilateral organisations. Australia’s multilateral funding is imputed to CAPF result areas in a way that makes it impossible to guess how far Australian agencies, through programs under their direct control, have contributed. Surely the ARAE should be structured so that contributions by the delivery agencies of most interest—which are mainly AusAID, the World Bank, the Asian Development Bank, the World Food Programme and UNICEF—can be broadly differentiated, and agency performance considered separately.
Don’t treat processes as ends in themselves. The ARAE is understandably heavy on processes; AusAID was handed a bunch of new processes to manage courtesy of the government’s response to the 2011 independent review of aid effectiveness. However, the ARAE does not do a good job of explaining how several key processes will affect resource allocation and contribute to development effectiveness. For example, the 2012 business engagement forum is presented almost as a one-off and an end in itself. It is quite unclear what is supposed to happen next. More broadly, we note the ARAE is particularly thin on the business engagement agenda and even places text on it under an “emerging challenges” heading, without explaining why.
Consolidate the ARAE and the Annual Report. Both the ARAE and AusAID’s Annual Report purport to be annual reports on the effectiveness of Australian aid. The latter only covers AusAID but thereby covers 90 per cent of Australian aid. Our suggestion would be to drop the pretence that it is possible to report on all aid programs spread across government, and embed the new three-tier framework within the Annual Report. Better to have one good report than two partial ones. Perhaps, if thought useful, a separate report could be brought out annually reporting on results from aid delivered by departments other than AusAID. If the ARAE continues as a separate document, key performance data in the Annual Report (such as the performance of our region against the MDGs, and the proportion of projects which are effective) should be also reflected in the ARAE.
That wraps up our series of responses to the first ARAE. The second ARAE, if it’s on schedule or combined with the Annual Report, is just nine months away. That’s not far off, but it is after the federal election. Whatever happens at the election, we hope the ARAE, or at least the three three-tier framework, survives it. Results reporting is good for the aid program. It just needs to be done better.
Stephen Howes is Director of the Development Policy Centre. Robin Davies is Associate Director of the Centre.