Australian aid investments managed by DFAT are rated on a scale of 1 to 6 in relation to a number of criteria. Scores of 4 or higher are required for the investment to be regarded as satisfactory.

DFAT rates both ongoing and completed investments, but rightly now gives primacy to the latter in its overall assessments of aid performance. The figure below, from our recently released report, shows average efficiency and effectiveness scores for both ongoing and completed investments from 2014 to 2022. Final ratings (for completed investments) show a sharp decline in performance, whereas ongoing ratings show an improvement. In 2021 and 2022, only three in five completed investments were rated satisfactory or better on both effectiveness and efficiency, whereas nine in ten ongoing investments were thus rated.

Why the worsening performance when we look at completed investments, and why the large and growing disconnect between the assessments of ongoing and completed investments?

You can read our report for a full explanation, but the key factor identified by the report’s regression and decomposition analysis is a shift in 2019 to a more independent rating system for completed investments.  The “optimism bias” that afflicts self-awarded ratings has been reduced for final ratings because, since 2019, the rating of completed investments has been the responsibility not of the investment’s managers but of a central unit within DFAT, that undertake its review with the assistance of external consultants. Ongoing ratings, however, remain the responsibility of investment managers. As a result, the gap between the rating of ongoing and completed investments has quadrupled.

The shift to a more independent rating system and the focus on final ratings should be preserved. However, the current system in which a large share of investments is rated as unsatisfactory, but the aid program is nevertheless reported to be “on track” is unsustainable. We conclude our report with five additional recommendations.

First, DFAT needs to take its own performance assessments more seriously, and reverse some of the recent decline. Since 2020, the average completed investment rating has been less than satisfactory for both effectiveness and efficiency. Of course, not all projects will succeed, but this is worryingly low. A review is needed, followed by change.

Second, DFAT should guard against solving the problem by grade inflation. After all, whether a project is regarded as satisfactory is, in the end, often a matter of judgement. We recommend against adopting a performance target, and in favour of retaining consultant review and central responsibility for final ratings, as part of a serious validation process to ensure confidence in the accuracy of project assessments.

Third, a communication effort is needed. Reporting a high share of unsatisfactory projects without explanation is a great vulnerability. In 2021 and 2022, 36 aid investments worth about one billion dollars were rated as unsatisfactory. This includes ministerial flagships and other large and critical investments. No explanation has been provided for why these investments are regarded as failures – several still have glossy websites. Rather than hoping that no one will notice, it would be better for DFAT to explain what went wrong with these interventions, and what it is doing to improve the health of the portfolio.

Fourth, ongoing ratings are meaningless due to the large disconnect with final ratings, and the performance assessment system for ongoing investments should be overhauled.  More than half the unsatisfactory completed investments rated in 2021 and 2022 were rated as satisfactory in their final ongoing investment rating. The improvement in ongoing ratings during the pandemic further stretches credulity.

Fifth and finally, the Office of Development Effectiveness (ODE) and the Independent Evaluation Committee (IEC) should be re-established. The problems identified in our report have largely occurred since the abolition of the ODE, DFAT’s evaluation body, and its oversight body, the IEC. One of the roles of the ODE, prior to its abolition in 2020, was to assess “DFAT’s internal performance management systems”. This was a role distinct from that of the performance branch, which is responsible for managing the performance system, as against assessing it.

In short, the trends revealed in this report, and the lack of any response to them to date by DFAT, make a compelling case for the reintroduction of the ODE and IEC as DFAT’s aid effectiveness champion and watchdog.

Download the full Devpol report, ‘Why are two-in-five Australian aid investments rated unsatisfactory on completion? An investigation into recent trends in Australian aid performance assessments’.