The curious case of sustainability

Indonesia Steel Bridges Project (image: Robert Cannon)
Written by Robert Cannon

Indonesia Steel Bridges Project (image: Robert Cannon)The federal budget has now been presented. Analyses of Australia’s foreign aid commitments are currently popular topics for discussion in the development community. Curiously, even though sustainable growth is a major purpose of Australian aid, there is comparatively little discussion about the evidence for the sustainability of benefits as a justification for further aid.

Understanding sustainability

To assist in understanding sustainability it is helpful to begin by sharpening our focus. First, we need to focus on the concept of sustainability. Many will think immediately of environmental sustainability, however, the conception of sustainability explored in this post is different. It is encapsulated in the clear definition [pdf] used by the former AusAID: “the continuation of benefits after major assistance from a donor has been completed”.

Second, making generalisations about sustainability is risky, given the kaleidoscope of countries involved, different cultures, development sectors, and the variety of assistance provided. Focusing on a specific sector, as has been done so well for education, and further narrowing our focus on one country may help our understanding of the dynamics of sustainability. The focus here is on Indonesian education.

But pause for a moment and look at the photograph above. The bridge on the right is an example of the kind of sustainability being considered here. The bridge was constructed during the Australian-funded Indonesia Steel Bridges Project. Over 2,000 bridges were built from 1980 to 1992 in 24 provinces. Overshadowed by the gleaming newer structure, the older bridge contributed to the circumstances for the creation of that new bridge by supporting a two-lane highway in Sulawesi.

Here we have strong evidence of the sustainability of benefits from the bridges project. First, the older bridge is clearly still there and the cars indicate it is doing its intended job. Second, the existence of the new bridge is an indicator that the older bridge has made a contribution to development in the region. Finally, an AusAID evaluation supports the sustainability of benefits from that project.

The sustainability of benefits from education projects

This is positive news from an infrastructure project. But is there comparable evidence of the sustainability of benefits from assistance to Indonesian education? There is actually very little and much of it is in donor reports, typically Independent Completion Reports and Project Completion Reports. However, these reports are not independent. They are commissioned by the donor, reviewed by the donor, and often further reviewed by development partners.

To assess the evidence of sustainability from these sources, I located and analysed 48 donor reports of education projects from 1973 to the present. These reports represent the work of 10 different donors over more than 40 years.

Key findings from my analysis are:

  • 64 per cent of projects were rated as likely sustainable; the remaining projects were either rated as uncertain (23 per cent) or negative (14 per cent).
  • Most evaluations were conducted at or close to project completion. In other words, there is only an estimate of potential sustainability.
  • The actual sustainability of benefits has not been assessed for most projects. Of the 13 projects that were evaluated two or more years after completion, only half were considered to be actually sustainable.

These findings of potential sustainability are similar to the findings of a larger and broader study for the Independent Review of Aid Effectiveness in 2011, but a 2010 ADB post-completion evaluation was more positive with 63 per cent successful or highly successful.

Is achieving actually sustainable benefits from half the projects reviewed good enough? How do findings compare with government programs and projects in Australia, for example? We are not really able to answer such questions. However, in its latest education project design for Indonesia, Inovasi [pdf] the Department of Foreign Affairs and Trade (DFAT) signals its own bleak assessment of sustainability. One of the Inovasi design’s chapter headings bluntly asks: ‘Challenge: Millions More Attending School But Not Learning – Why Haven’t Reforms Taken Root?’

With more than 40 years of Indonesian development experience, it is reasonable to enquire why we do not have stronger evidence of the actual sustainability of benefits from aid to education. Why have the ten donors in the analysis, who have provided over USD 4.5 billion in assistance since 1973, been so incurious and for so long about sustainability arising from this massive investment to support Indonesian education?

What might be done to assist in sustaining the benefits from current and future educational development assistance?

I suggest five strategies for consideration until such time that we have a stronger base of evidence for policy and practice, something that DFAT’s Inovasi could provide in the future.

  1. Approach educational development with a better balance of cultural, political and technical realities in both design and implementation. Recent Indonesian theses, one a study [pdf] of the Australia Indonesia Basic Education Program and the other [pdf] of the sustainability of active learning in North Maluku, call attention to the need for this approach. Failure from ignoring culture is also demonstrated in a 2005 study of schools in Java.
  1. Build strategies from a deeper understanding of local and institutional cultures, of what we know works, where it works, and why strategies work. The outstanding example of the need for this approach in education is illustrated in Gerard Guthrie’s well-researched book, The progressive education fallacy in developing countries, about the challenges of educational development in Papua New Guinea.
  1. Explore ways of democratising sustainability by inculcating the idea of shared responsibilities among all beneficiaries to move away from a dependency mentality where beneficiaries assume all responsibility rests with government and donors.
  1. Consider the implementation of culturally suitable concepts drawn from education. Some concepts offer the potential for learners, teachers and others to act as sustainable agents of change themselves:
  • Lifelong learning: The idea of moving away from a focus on teaching towards sustaining learning throughout life.
  • Sustainable assessment: The recent idea that assessment can be designed to contribute to meeting future learning needs.
  • Feedback: Learning is not sustainable if it requires continuing support and feedback from teachers and development experts. Recent work on feedback is focusing on the contribution of others to lifelong learning through sustainable assessment.
  1. Pay attention to sustainability at all stages of the project cycle and ensure it is tightly coupled with the deepening of change.

There is presently a lack of congruence between aid funding concerns, and donors’ soaring policy rhetoric about sustainability and poverty reduction on the one hand, and the neglect of rigorous inquiry about the actual sustainability of benefits on the other. This curious situation also raises concerns about the effectiveness of educational development. AusAID began to address such concerns in 2012 with its enquiry into educational development. Sadly, that positive initiative seems to have gone nowhere.

Robert Cannon is an educational evaluation adviser to USAID in Indonesia.

Robert Cannon

Robert Cannon is an educational evaluation adviser to USAID in Indonesia.

8 Comments

  • The concept of sustainability is very critical and its more than environmental though many still believe environmental sustainability is still key to development and growth.

  • As Simon reminds us above, the sustainability of behavioural and organisational change is difficult to achieve. That is true whether we are working in Australia or in Indonesia. However, I think that there are some approaches that can increase the chance of success.
    The first is for sustainability to be systematically addressed in designs and annual plans, and be a focus of monitoring and evaluation. Leaving the serious thinking on sustainability to the exit report stage (as happens) is too late.
    Some people take the view that an activity will be sustainable just because it aligns with the recipient government’s priorities, and has a senior-level steering committee. If that was the case, every planned educational change in Australia would have been sustainable and that has not been the case. Local ownership is necessary but not sufficient; and ownership at Secretary General level is not the same as ownership at district level.
    AusGuideline 6.4 Promoting Practical Sustainability may be dated (2005), but it remains a useful and relevant guide and checklist for planning, management and monitoring.
    The second factor is for a gradual handover of financing. It is common for donor financing to continue at 100% until the last day, then stop. It would not be surprising if the activity gradually disappeared into the machinations of the budget cycle. A better approach would be for a staged withdrawal that has the recipient government funding some activities, but still with ongoing support. (Bob has noted the need for this above.) This stage would occur once benefits start to become clear and the credibility of the activity is established – not less than 3 years. Dan notes the need for time for the development to occur. Another reason for time, and flexibility, is for cultural issues to emerge and be addressed. It is unlikely that they can all be identified at the design stage.
    Finally, we are concerned about sustainaining organisational change. There is a considerable body of management literature on the management of change, which people working in development could be utilising -and change models are consistent with the sustainability principles. For example, a plan to develop ‘champions of change’ is a plan to develop local ownership. Perhaps a design could be specifically structured as change model; and perhaps the most important technical assistance may be on the management of change, rather than, for example, on school management.

  • Bob, many thanks for this insightful posting – and for raising some key issues.

    The key point you make, for me, is the following: “With more than 40 years of Indonesian development experience, it is reasonable to enquire why we do not have stronger evidence of the actual sustainability of benefits from aid to education. Why have the ten donors in the analysis, who have provided over USD 4.5 billion in assistance since 1973, been so incurious and for so long about sustainability arising from this massive investment to support Indonesian education?”

    What has this money achieved? An optimistic response will point to the huge gains in access. According to Indonesian Government data, 81 per cent of children aged 6 to 18 are now enrolled in school (net enrolment rate). When children (or adults) outside the specified age ranges are included (gross enrolment rate), the percentage increases to 91 per cent. Gross enrolment in primary school is 109%, in junior-secondary, 101%. Participation in early childhood has reached about 70% (a massive increase over the last decade) and in senior secondary 76%. These are significant achievements for a young nation which at the time of independence provided schooling to less than six per cent of its citizens.

    Dropout rates have reduced dramatically. In 2011-12, just one per cent of children enrolled did not complete primary school, with over 95 per cent continuing on to junior secondary schooling. The dropout rate for junior-secondary is 1.7%. While still an issue in remote areas, attendance rates (for both teachers and children) are also improving. Gender parity has largely been achieved. Adult literacy is around 95%.

    But when we look for gains in quality and in learning outcomes, the story is rather different. While we might argue about the validity and relevance of international tests, there do tell us something. Indonesia ranked 64 out of 65 participating countries in the last PISA test (2012). The country’s ranking dropped from 57 in 2009. TIMSS and PIRLS rankings are similar – and, similarly, have declined over the years of testing.

    After nearly 25 years working in education in Indonesia, a question I often ask myself is, if we already know what works (as we sometimes claim), then why are we still here, and why are doing pretty much the same thing, after 30-40 years? Why hasn’t active learning become mainstream? Why isn’t school-based management working as effectively as it could be? Why are the management and governance of basic education still mired by corruption and inefficiency?

    The answer you offer is a good one: We need to “approach educational development with a better balance of cultural, political and technical realities in both design and implementation.”

    This is well demonstrated in the studies referred to in the post. It is the key to Guthrie’s argument. And I think it could be the key to the future in Indonesia. We know that Indonesian teachers can ‘get it’. They are just as capable as any of implementing active learning approaches in their classrooms, for example. But how many actually sustain the changes beyond the life of a project? I am not convinced that many of the changes are sustained in any real way. Sustained change requires changes in cultural perspectives (beliefs, norms, relationships, roles, expectations …). And this is unlikely to be achieved with a few cascade training events – even when followed up with coaching.

    Donors also need to work in the political context in order to get sustained change. People behave as they do (in the classroom, the office or the community) because they are motivated to do so by complex sets of incentives and disincentives. If we continue to ignore this and assume that teachers will change their practice (despite the technical and cultural hurdles) – with no real incentive to do so, we are surely kidding ourselves! I don’t just mean financial incentives, of course. But why would a teacher adopt an active learning approach if it (1) takes a lot more effort, (2) earns disfavor from colleagues, parents and supervisors who don’t understand or appreciate the approach, (3) confuses kids, sometimes making them unruly and disruptive in the short term, (4) does not help them get better exam scores (the only vaguely objective measure of their performance), and (5) does not advance their career?

    And here, perhaps is the key thing. In order to work in the political and cultural dimensions, foreign assistance programs must be far more integrated and ‘owned’ by local partners. On a slightly optimistic note, I do see that this is increasingly the case. Foreign consultants, like me, are a very small part of the picture these days. We bring something that I think is still needed – a certain broader perspective, a kind of objectivity, a proficiency in English and familiarity with international standards, rigorous thinking and so forth that is still needed – but it is only a small part of the equation.

    But the real problem, highlighted in this post, is that we really don’t know for sure. We don’t know what impact all those programs, all that funding, has had. We really don’t know what works, from a technical, cultural and political perspective.

  • Bob, I’m struck by the bridge analogy, and wonder if the original project goal was to replace the old bridge or supplement it (to provide the two-lane highway). What I’m getting at here is that the outcomes of a project might not always (or often) be the ones the funder planned for, and in some cases there may be unanticipated changes, both for better or for worse. This suggests to me that when we revisit projects to evaluate sustainability we cannot simply rely on quantitative indicators devised to measure predicted outcomes from the project proposal — we need to be much more flexible and insightful. On the whole, my impression is that revisiting projects after completion, even very expensive projects, is the exception rather than the rule, and that such ex-post-facto evaluations are not done well or over sufficient time to judge sustainability with any sophistication. I used to teach planners and architects, and there is a well-know concept in the field known as post-occupancy evaluation, which involves visiting built structures some time after completion to compare the designers’ conceptions of how the space would be used with the way that the occupants in fact use it. Such evaluations are in fact extremely rare, and most architects regard them with indifference. Hmmm.

  • Bob, thanks for a thought provoking post. It makes me wonder if we have really grasped how to even think about this issue properly.

    Just as environmental sustainability has a specific meaning other types of projects and programs may need to adopt different concepts of sustainability. The author contrasts the evident sustainability of a bridge project with the harder to demonstrate and lower reported levels of sustainability of education projects. But couldn’t this be because we are again talking about different types of sustainability?

    In one sense sustainability is intrinsic to a bridge. It fulfils its purpose by not falling down. Once built a bridge should continue to be a bridge for at least a minimum projected lifetime and we should regard it as a failure if it were not to do so long past the aid project that built it.

    On the other hand, many aid projects, including the education projects you mention, are about engendering some kind of behavioural change. Surely we cannot expect behavioural change to have the same sort of sustainability as a bridge particularly as we are dealing with human beings and the myriad of unpredictable influences that impinge on their lives. Surely this kind of sustainability, if it is legitimate to seek it, is of a different order and needs different tools to understand and measure it.

    • Thanks Simon. Yes, I agree it is an issue that has not been given enough thought. Neglect might be a more accurate description. Your question goes to the heart of the problem with sustainability. As your observations imply, it is a complex matter and I remain to be convinced that the idea of sustainability is necessarily the most helpful concept here, particularly when the word seems to be used increasingly as a ‘feel good’ slogan in so many contexts now. Maybe continuing benefit or continuing impact?

      The central idea of sustainability in projects of all kinds has been the continuation of benefits which, in the ideal situation, will be clearly and realistically defined in the project design. I wonder if they are, especially in education. The behavioural change you refer to in education should, ideally, be strong enough to survive most of the unpredictable influences and support continuing change and improvement and not be limited to the implementation of (say) one unchanging teaching method or one specific management tool. Unlike bridges, an additional complication in education is that many projects operate at several interlocking levels and, further, can also include physical infrastructure such as school buildings.

      All this leads to the difficult matter of how you measure sustainability. We certainly need to do much better than a lot of past practice which has been little more than a quick judgement resulting from a very limited sample of cases (districts, schools, teachers) that have received assistance. Another curiosity is the contrast between the attention given to evaluating sustainability, which has been very limited, with the attention given to impact evaluation.

      As you correctly observe, it seems we have not yet grasped how to think about the issues properly. Curious!

  • Bob, Thank you for raising this important issue. I think all the points you raise are worthy of consideration for future development assistance for education. In my experience one of the ways to improve sustainability is to provide direct technical assistance to those who need to produce something. This often means sitting with the persons responsible for a product over relatively long periods of time introducing more effective and efficient tools that allows them to perform a required task. The skills the counterparts receive through this technical assistance methodology become well ingrained and longer lasting and thus more sustainable compared with one off training.

    • Yes, one of the saddest phases of projects is finalisation when so many enthusiastic people ask where continuing support, of even the most limited technical kind, will come from. Donors and governments alike are generally unwilling or unable to provide access to continuing help of this kind. Short term and one-off training is one of the most ineffective ways to bring about sustainable change.

Leave a Comment

Tweet
Share
Share
+1