The Australian government is about to start lending money to the Pacific as part of a new aid initiative intended to help with infrastructure in the region. With Australia poised to start lending, the question needs to be asked: are the loans likely to work? Our analysis of the effectiveness of loans in the Pacific has led us to conclude Australia needs to proceed with considerable care.
Our analysis involved a global dataset of nearly 18,000 aid projects. The dataset covers a number of donors and has its origins Dan Honig’s impressive book Navigation by Judgement. (To learn about the data, see the note at the end of this post.) Each project in the dataset comes with a score of how effective it was. These scores come from donors’ project assessments (conducted by donor staff or external evaluators).
As Honig points out in his book, there’s an obvious problem with project assessments: owing to political and organisational incentives, donor staff and evaluators probably appraise projects too kindly. This is an issue, but as long as this inflation occurs roughly equally across projects, we can still use differences between projects to learn when, why and where projects are likely to succeed.
Because we were interested in what more lending might mean for the Pacific, we conducted analysis focused on the loan-funded projects in the dataset (11,821 projects) and we compared average assessments between the Pacific and the rest of the developing world.
You can see the results of this comparison in the chart below. Assessments range from one to six, with higher being better.
Loan effectiveness: the Pacific and rest of developing world
The chart shows that, on average, the assessed outcomes of aid projects funded through loans are worse in the Pacific than they are elsewhere. The difference is statistically significant.
The average project in the Pacific still scores above the mid-point of three, which approximates a ‘satisfactory’ outcome, but too much shouldn’t be read into this. As discussed, aid appraisals are probably too kind everywhere on Earth. As a result, absolute scores are not meaningful. Rather, it is the difference between groups – in this case the Pacific and the rest of the world – that is elucidating.
The difference between the Pacific and the rest of the world is not massive (-0.39). But reality may be worse than the difference suggests. Because donors’ appraisals tend to cluster together rather than being spread along the full range of the scale, it’s likely the differences between the rest of the world and the Pacific are greater than they appear in the chart.
At this point the careful reader may have thought up benign explanations for the under-performance of the Pacific. Maybe the donors that provided the data on Pacific projects are particularly inept, or gloomy in their project appraisals? Or perhaps the difference stems from some oddity in the data such as Pacific projects coming from an earlier era of aid giving, when donors made more mistakes? Or possibly projects are smaller in the Pacific, and this leads to harsher assessments? Or maybe Pacific loans are focused on a certain, particularly difficult, sector?
We thought about this too. We also accounted for it. All the charts in this blog were produced with differences between donors already controlled for. (If we don’t do this, the Pacific effect is slightly larger). Similarly, in OLS regressions, we added controls for dates, sectors, and project size. We also re-ran the analysis focusing only on economic development and excluding projects from prior to 2000. We also tried dropping projects in Papua New Guinea. None of these changes shifted our results in a substantive way. Likewise, turning the dependent variable into a binary success or failure measure didn’t change matters much either. (Enthusiasts can read results here.)
Our findings do not mean that all loans in the Pacific are doomed to failure, but they do highlight the difficulties of lending successfully to the region, which ought to be sobering for any donor about to embark on a lending spree.
Although our primary focus was loans, we also took the chance to see whether the Pacific was as challenging for aid projects funded by grants. You can see the results in the next chart. (Once again, the chart shows the effect of the Pacific after differences between donors were controlled for).
Grant and loan success in the Pacific
As you can see, the average effect for grants is also negative, although it is not statistically significantly different from zero, and the magnitude of the impact of the Pacific on grant success is less than is the case for loans. (This was also true when we add additional controls to regression models in line with the discussion above.)
For technical reasons we are not as confident of the finding that loans perform less well than grants in the Pacific. But even if loans and grants are equally hard to deliver in the region, there would still be particular reason worry about loans. If a grant goes awry the money is simply wasted. If a loan fails, the recipient has a pile of debt and nothing to show for it.
Just because other donors have struggled to lend successfully in the Pacific does not mean Australia is destined to fail. But the track record of loans in the region provides cause for caution. It won’t be in Australia’s interest if its foray into aid lending leaves the Pacific even further indebted and still beset with development woes.
Data note: Data come from World Bank, ADB, DFID, JICA, the Global Fund, two German government aid agencies and the Australian aid program. Because most of these agencies do not provide loans to the Pacific, our Pacific loan analysis is based foremost on the World Bank and ADB. Dataset creation involved taking Honig’s original data and augmenting it with a large body of additional World Bank and ADB data. We also added publicly available DFAT data. We are very grateful to Honig for sharing his data and similarly appreciative of the World Bank, ADB and DFAT for making some of their data publicly available. We are also grateful for additional DFID data from the authors of this paper.