Comments

Robert, publication of my thesis is forthcoming (next month or two) and I would be happy to put something together for the blog on learning as that is the central tenet of the thesis. In essence, I argue that our heavy focus on the instrumental nature of indicators as the measures for success means we miss some of the good stuff (and the bad!) because we don't seek to dig further into what is really going on and reflect on what this might mean for how we deliver our programs. My big push is to look at a broader range of evaluative methodologies to learn from development work that builds on the approaches already being utilised. Chris.

Geoff, thank you for explaining the detail in a cascade approach. I note in the Independent Completion Report for the project you describe, the outstanding AusAID Indonesia-Australia Basic Education Partnership, that this approach was very positively evaluated indeed. As educators, we could analyse to the advantage of improving project impact, why it is that such good practices often seem to get lost in subsequent projects. My hunch is that during project design and implementation stages, the time pressure on donors, contractors and advisors alike lead to surface approaches in much the same way that examination pressure leads to surface approaches to learning among students.

Christopher, your final suggestion is excellent and one that would really add weight to the assertions in M&E that we are truly interested in learning (when I suspect that 90% of the real agenda is accountability). It would be great if you could find time to write a blog here about your work on learning as, being research-based, it would advance the discussion considerably. In any case, can you provide a reference or a link to your work?

Hi Owen, Great to read your reply. I particularly appreciate the observation about the forces of demand operating to encourage a surface approach. This acts as a very useful counterbalance to the forces demanding depth that I have been seeing recently and my general view (confirmed by you also) that donors seem content to supply surface development, although this would likely be hotly denied. Best wishes, Bob.

Hi Bob, Thank you for a most useful application of notions of 'surface and deep' to 'development.' I am currently coming to the end of an Australian Volunteers International assignment in a university in north west China, and soon to start another similar assignment in Vietnam. The issues presented are dear to my heart. Your blog (that I was a little surprised I could access, given certain restrictions in China) provided something of a 'refresher' in Western ways of thinking and expressing. I saw your reference to 'context' as critical, to the extent that context has the potential to significantly impede open intellectual debate. Within a 'Western development provider' constituency, I would hope that what you present receives a considered hearing as I strongly support the positions presented in the blog. Part of the challenge is the difficulty, both technically and politically, in evaluating and reporting on the effectiveness of development. Further, currently the 'surface approach to development' is so often accepted with such appreciation that the need for, and added benefit of, a 'deep approach to development' would be seen as unnecessary by many 'donors' and 'recipients.' I hope that during my year in China I have been 'walking the talk' at the coalface of Chinese university classrooms. The students are receptive!

Hi Christopher, Thank you for your comment. I am aware of the research associated with the Justice for the Poor Programme. And I agree it is very promising. And that the World Bank and AusAID deserve credit for at least beginning investigations in this area (while we're at it the World Bank deserves credit for its work with the RDP programme too). Yet you write: "Our team in the Solomons would be happy to provide additional information, but they regularly travel into these communities and offer a dfferent kind of service that is not based around delivering from the capital and embraces informal systems of governance. The service delivery program is still in its infancy, but there is reason to be optimistic we can make it happen." This is written in the present tense, which suggests that active programme work (as opposed to research and scoping) is already happening. If so can you please advise me where? And what issues it is being geared to tackle? There was no sign of any such work in progress when I visited the Weather Coast. However, if it work has started three since then, that would be excellent news.

Terence, in response to your concerns I would encourage you to visit the World Bank's Justice for the Poor program website where you will see some of the innovative work being done on precisely the issues you raise. In fact, we are currently in the process of designing a program in the Solomon Islands to provide a pilot 'advisory services' arrangement for these communities in their dealings with more powerful commercial interests. This work seeks to use analysis and research to better engage with communities where there are sites of conflict and injustice. Essentially this means actually resourcing, researching and mediating the relationships you refer to. Our team in the Solomons would be happy to provide additional information, but they regularly travel into these communities and offer a dfferent kind of service that is not based around delivering from the capital and embraces informal systems of governance. The service delivery program is still in its infancy, but there is reason to be optimistic we can make it happen. The World Bank deserves some credit for taking this work on (particularly when it falls outside its normal set of aid modalities) and AusAID deserves credit for providing funding to make it happen. Encourging these approaches is essential if we are to get things like the EITI commitments beyond the meeting rooms. <em>Christopher Nelson is an M&E specialist with the World Bank in Sydney and works on the Justice for the Poor program.</em>

Robert, thanks for the insight. I have also been intrigued by where notions of learning might sit in a broader development context. No doubt we need to think carefully abut the quality of learning that takes place within education programs, but we rarely look further to how it plays a role in the full spectrum of aid interventions. My doctoral research took up this challenge by applying a phenomenographic methodology (built around Marton's work) to an agricultural program in Mozambique. What was particularly interesting in using this approach was the surprising diversity of ways that stakeholders engaged with the project. Putting these observations into a learning hierarchy was helpful in highlighting how different the impact of a development program can be for members within the same community. Understanding these differences can be particularly powerful, but only if we allow for diversity in how programs are delivered and designed. This means using research in the area to encourage donors to allow innovative and alternative development models to be trialled as an additional and necessary part of their standard programmatic work. We set aside a designated proportion of funds for monitoring and evaluation (M&E), so why not look at having a percentage set aside for learning and innovation that is properly thought through and recorded.

Robert Cannon’s blog ‘A deep or surface approach to development – what can learning research teach us?’ points to the potential shallowness of cascade training approaches, especially when carried out as ‘one-shot’ training exercises, and emphasises the need to take a deep approach to learning. He raises important issues that affect capacity development, change management and the achievement and sustainability of good development practices. The essential pre-condition for capacity development and for effective change programs is that a supportive environment exists or can be developed. It is not enough that the proposed program be supported by senior officials, it must also be supported by those closer to the action. Clearly, people will only support change programs if they at least conditionally accept the need for change. They also need to see that there is a realistic way forward to achieve the change, that they have or will acquire the necessary knowledge and skills, and they need to be supported throughout the change process. A one-shot surface approach by program implementers is unlikely to meet these conditions. Training alone is not capacity development. True capacity development requires much more than the training of specified target groups. Supporting changes to systems, process and procedures will almost certainly be needed, as well as parallel development activities for other stakeholders than those initially identified. The development process will be iterative and based on on-the-job learning and continuous quality improvement is therefore essential. All of these issues require deep understanding and engagement by the people involved. Post-program sustainability depends on local ownership and requires deep commitment by local stakeholders. Also, simply mainstreaming new practices is not a sufficient test of sustainability. True sustainability requires that these practices can be updated and improved even when no external support is available. Again, people need to have a deep understanding be able to do this, which is unlikely to be achieved through conventional cascade-training. These are serious problem for program planners, who will likely face time and budget constraints. The issue with cascade training is that the training of a particular target group (e.g. 1,000 science teachers) is seen as being the priority. Part of the answer for planners is to conceptualise the key task as one of trainer/mentor development, with the initial teacher-training activity being part of that development program rather than being the end in itself. I was part of a team working with District Education Services to support the implementation in schools of the Indonesian Government’s policies on active learning, school-based management and community participation; as well as to support the capacity development of the Services. Rather than pursuing targets for project-support training, we emphasised the development of a group of local trainers in each District. We modified the cascade approach to include a mentoring stage. Our national consultants ran the initial trainer training, then mentored the trainers as they planned and delivered their first few cycles of cascaded training, supporting ongoing improvement in the quality of the courses. With our ultimate goal being that all potential stakeholders were trained (and not just the relatively few funded by the project), these initial courses were primarily seen as a way to train the trainers – and we did meet our training indicators. This linked training/mentoring approach was repeated by the local trainers, who followed up their formal training with in-school mentoring. This proved to be a successful way to minimise the potential losses and distortions likely in cascade training. Over time and with mentoring support our trainers developed competence and confidence as professional trainers, able to continue the project-initiated training and mentoring programs, adapt and improve them over time, and design and deliver local, low-cost programs for other stakeholder groups. Being local, they could be effective in influencing teachers, parents and the community to support the changes, providing ongoing local support and serving as ‘change champions’. In effect, this approach applied the strategies supporting deep and sustainable development summarised in the Cannon blog. It aligned with government policy and earlier initiatives; provided several cycles of training, mentoring and feedback; allowed for whole-school and whole-system development and built communities of professional practice in Districts and across the Province. From a planning and implementation point of view, this is a relatively low-cost strategy. The major expenditure is at the beginning of the process, with the selection, training and mentoring of trainers through the first cycles of training. The actual cost of training and mentoring teachers, school committee members and principals by local people, carried out locally, is low and allows for near-100% coverage over time. The time demands on project consultants are heavy at the beginning and decline over time. This decline in cost and time demands on a project provides for an easily-understood exit-strategy and allows for resources to be diverted to new Districts. Nevertheless, as the blog points out, some modest continuing technical support after the formal exit, perhaps utilising a peer district system, is likely to support sustainability and improvement.

Thanks for articulating so succinctly how we NGO's in developing countries experience the shift in NZ aid approaches. It is abundantly clear that no matter now much good we are doing addressing critical community issues in developing countries, there is no way that we can access support from NZ unless we are an NZ agency... the "lip service" to partnerships between NZ based organisations and NGO's in developing countries actually appears to be rhetoric maze that we emerge from, blinking, right back where we began.

Hi Terence. The essence of what I’m on about might well be enunciated in your statements. Whether it’s a chicken and egg situation about which comes first, the fact is that pollies in so called developed countries are able to excuse any accountability for overseas aid projects by merely stating the bottom line and hoping to bask in the reflected glory. Most people in Australia for example, have absolutely no idea what happens to our aid money in practice. Why? Well that leads me onto your point about there being not a lot of evidence. Exactly right. Where do we really go to see any real evidence of any long term results from aid being spent? Billions spent in PNG over the last decades but no real improvement in the lives of the rural population and the urban poor. Before I get howled down for saying that we shouldn’t spend it, that’s not what I’m on about. What we must see is accountability for results. How, you may well ask? Well it’s simple really. No project should be commissioned without an agreed benchmark being established prior to the project commencing. There should then be transparent reporting and auditing of the results achieved. Surely that’s only common sense? The fact that aid monies end up in the pocket of dictators and corrupt officials is well documented. PNG’s Deputy Commissioner of Police is on record for publically stating that at least half his nation’s budget is lost to corruption. Is it any wonder nothing seems to change in today’s PNG? Even the previous PNG PM Somare is on record for acknowledging that the PNG public service is notorious for demanding a six pack before actioning a simple request. It’s common knowledge that many public servants almost never turn up for work yet still collect their pay. Their appointments are reputedly not made on merit but on family or cultural ties. Am I guilty of overstating my points? All I’m doing is stating what is common knowledge in today’s PNG and consistently reported in the local media. What’s the answer? I have previously written a submission to AusAID (that is posted on the Web) making some practical suggestions on how a valid aid scheme might work. That presumably disappeared into the Black Hole that all suggestions to government seem destined to end up in. There is one more aspect that’s terribly important. Any aid worker who really wants to make a difference must be prepared to learn the local language and customs. This doesn’t happen overnight. Australia previously had a facility for providing some initial training at the International Training Institute in Mosman, Sydney. That establishment is now history and the building falling apart. If an aid project officer from Australia stays in better accommodation than the people he/she is working with, that might well be accepted if the worker is able to effectively communicate directly with those they are attempting to assist. There is no bigger issue in my view than this. Many short term aid workers are disconnected from those they are ostensibly trying to assist by an impenetrable barrier of a true lack of understanding each other’s language and culture. This barrier only breaks down when aid workers get out of their barb wire surrounded compounds and security guarded forts and sit down in the dust with the people we are really trying to help. If anyone wants an example of how this actually works, try contacting Lydia Kailap and her husband in PNG’s Port Moresby who run a self-funded school for the capital’s poor and homeless. I’m sure you’ll get some good, practical advice.

Thanks Paul, Andrew, Deg and Beth for your comments and sorry for my delayed reply. Just a couple of points to note. Paul - most survey data that I'm aware of suggests that publics in developed countries believe that their governments should and do give more aid than they actually do. So I'm not sure if our aid budgets can really be accurately viewed as a product of politicians pandering to public sentiment. Also, the claim that aid helps politicians in recipient countries enrich themselves and also that it puts off any potential day of reckoning when citizens of these countries might hold politicians to account has a grain of truth to it. Certainly, in the past (and even in the present in the case of some donors) aid that has been primarily given for geostrategic rather than humanitarian reasons has ended up lining the pockets of tyrants and dictators. But this is much less prevalent (at least for the better donors) these days as conditionality and other control mechanisms are applied to try and ensure aid spending helps the needy. Similarly, there's not a lot of evidence to suggest that modern era aid leads to more rather than less corruption in developing countries. In short, the world of aid has changed somewhat for the better in the last two decades and while your concerns are valid, I think it would be easy to overstate them. Beth and Andrew, For the most part I think Andrew is right. Aid work is hard and comes with its particular set of stresses and if it is really poorly paid people with the best skills for the job (particularly when they reach the age of kids and a mortgage) won't do it. On the other hand, while this goes some way to excusing the lifestyles of aid workers I do also think Beth has a point and that, at some point, this line of thinking becomes a convenient excuse for salaries and luxuries that can't be justified. How often this point is passed, I'm not sure. Thanks for your comments.