For the second year running, the US think tanks, Brookings and the Centre for Global Development, have released their ratings of aid donors. QuODA (Quality of Official Development Assistance) assesses donors against 31 indicators divided into four groups: impact on institutions, administrative burden, efficiency, and transparency and learning. I have calculated a composite quality ranking by averaging over the four groups. Using the 2011 rankings, Australia comes 8th out of 23, consistent with the recent Independent Review of Aid Effectiveness finding that Australia’s aid program was “good but improvable.”
The ODE report “The quality of Australian aid: an international perspective” draws largely on the Brookings report. It emphasises one of the Brookings results in particular, namely its comparison with other bilateral donors who, like Australia, give at least half of their aid to countries classified as fragile (poorly governed and/or in conflict). There are six such donors and of these “Australia is second only to the UK” in terms of QuoDA quality as the AusAID ODE website notes.
This is hailed by the ODE report as an “important result” and “key finding”. It shows that Australia is “not an average donor” and that “other donors can learn from Australia’s example.” Indeed, the report argues, once context is taken into account by making a comparison with this more limited group of donors, Australia’s aid program emerges as “world standard” and “among the world’s best.” A Devpolicy blog authored by ODE head, Dereck Rooken-Smith, says that the result shows that Australia is delivering “effective aid in fragile and conflict-affected states.”
The Brookings report, “Benchmaking against progress: an assessment of Australia’s aid effectiveness”, on which ODE’s is based, puts less weight on this one result, and presents more material to balance it (in particular, a critical analysis of Australia benchmarked against Ireland). Nevertheless, the claim that the result in question is an “important” one originates with the Brookings report. Laurence Chandy, the report’s author, goes on to argue that it “indicates that Australia’s efforts to heed good practice principles, despite the challenges posed by the context of its aid program, are paying off.”
Is this “2nd out of 6” result, as we might call it, an important one, as claimed by the two reports? For it to be important, the conclusion drawn from it that Australia is a world leader among donors heavily exposed to fragile states needs to be robust. That is, it should not change if minor alterations are made to the way in which the question is explored. (I use “exposed” here in the sense that a donor which is highly exposed to fragile states is one which has a high share of its budget going to fragile states.)
One problem with comparing Australia with other donors who give 50% or more to fragile states is that Australia sits right at the edge of that group, with just a 50% share (see the graph below from the Brookings report). It would be equally if not more plausible to take as the comparator group the top half of aid agencies in terms of their exposure to fragile states, a group in which Australia comes in the middle, 6th out of 12. But, in this group, Australia also comes 6th out of 12 in terms of the composite QuODA index, which is in fact a worse relative ranking than its overall 8th out of 23.
Alternatively, rather than putting Australia in the group of the top 6 donors in terms of exposure to fragile states, since Australia comes 6th, one might just as well put it in a second group of five in terms of exposure to fragile states. In that group, Austrlia is not the second best but the second worst in terms of quality.
Finally, put Australia not in the 50%+ group of donors for exposure to fragile states, but in the 40-50% group (where it just as well fits), and it comes 3rd out of 6 for quality.
In other words, depending on how the problem is framed, Australia can emerge as a relatively good, average, and relatively bad donor compared to other donors with a similar exposure to fragile states. That shows that the conclusion drawn from the “2nd out of 6” result is not robust. The result itself therefore shouldn’t have been singled out and labelled as important. Nor should any conclusions at all have been drawn from it, let alone been repeatedly highlighted.
Why isn’t the result robust? It turns out that there is no relationship at all between aid quality, as measured by QuODA, and exposure to fragile states. That’s what the second graph (below) shows. (For technical details on the graph, see the notes at the end of the blog.) It is true that three of the five worst agencies in terms of quality – US, Belgium and Italy – also happen to be the three which are most exposed to fragile states. This is what drives the “2nd out of 6” result for Australia. But it turns out to be a fluke. It is not that in general donors which provide more aid to fragile states are worse donors. Rather the picture that emerges is that some donors are better than others, and that some focus more on fragile states than others, but that there is little if any link between the two.
The lack of a relationship between aid quality and exposure to fragile states is surprising as one would certainly expect aid to be higher quality in better-governed countries. There are broader issues with the utility of the QuODA rankings which I won’t go into here. (See earlier Devpolicy blogs – here and here – for a discussion in relation to Australia’s transparency ratings.) But probably the explanation for the lack of a link (or correlation) is that QuODA is not a full measure of aid effectiveness. Rather, it primarily measures things under donor control, such as transparency and selectivity. And there is much less reason to think that these characteristics of aid under donor control would be heavily influenced by the nature of the countries in which the aid is delivered.
It is unfortunate that so much has been made by ODE of a conclusion which lacks foundation. This episode illustrates the need for greater contestability, which would, one hopes, pick up and weed out weak analysis such as this. One way to do this, which I have written about earlier, would be through the establishment of an Independent Evaluation Committee to overseee ODE’s evaluations.
And what about Brookings, the prestigious US think tank? As I meantioned earlier, the Brookings paper is a wide-ranging one, and I am focusing on just one of its results here. Nevertheless, it is unfortunate that there was no testing for robustness in relation to the “2nd out of 6” result . The error is made more serious by the failure of first-published version of the paper to disclose that the Brookings Institution in fact receives considerable financial support from the Australian aid program ($800,000 over two years according to the AusAID website) to work on issues of aid effectiveness.
 Notes in relation to the second graph: The vertical axis ranks countries by donor agency quality, where 1 is the highest quality and 23 the lowest. This composite quality ranking which I calculate is obtained by averaging the ranks obtained on the four QuODA indicator groupings, and then ranking countries by this composite indicator (excluding multilaterals at this stage). The horizontal axis ranks countries by their exposure to fragile states, where 1 is the highest exposure and 23 the lowest. Exposure is measured by the share of bilateral aid going to fragile states. The QuODA data used comes from the 2011 QuODA brief, and the fragile state exposure data comes from the Chandy Brookings paper, both referenced in the text. The line shows the correlation between the two variables (or, actually, the lack thereof). The Spearman correlation coefficient is a very low 3%. I also looked at the correlation between exposure to fragile states and the four separate QuODA dimensions. All of the correlations are small; two are negative. I haven’t used actual shares of aid to fragile states rather than ranks, because I don’t have the data, but the shares can be estimated from the Brookings paper (Figure 11, reproduced above as the first graph) and their use of them doesn’t change the results.