Comments

Hi Gerard, This is an interesting read, although not all aid funded. Also note the external validity limitations of rcts discussed below. https://www.povertyactionlab.org/policy-insight/improving-learning-increasing-motivation-targeting-instruction-and-addressing-school Terence

Thanks for the comment, Terence. I'm always interested to see any examples of successful educational outcomes - what cases did you have in mind?

I wish I could agree with Garth’s comments that it is ‘amazing what has been achieved especially in areas that are well understood such as … education’ and also ‘the main impediment to further progress is lack of funding rather than lack of knowledge’. Sadly, evidence suggests the opposite may well be the case - for both. For example, Gerard Guthrie, who has already commented here, has written two important books ‘The Progressive Education Fallacy’ and ‘Classroom Change in Developing Countries’ that show how a lack of understanding of the important place of traditional epistemologies in developing counties such as PNG and elsewhere has seriously hampered educational development. My own work on the sustainability of benefits from educational development assistance to Indonesia shows how little of the considerable effort and resources over 40 years has yielded longer term benefits. In both cases, my view is that we in the development community have not applied educational and local knowledge to the extent necessary, favouring instead imported, top-down, western economic and managerial approaches. More funding in areas of education that are not well understood will likely lead to further waste and disappointment. More positively, the adoption of analytical approaches such as Problem Driven Iterative Adaptation advocated by Harvard’s Center for International Development and currently implemented by DFAT in Indonesia, may yield a significantly stronger base of understanding to achieve more in education and in other areas of development.

Thanks Gerard, Interesting comment. Education isn't my specialty but my understanding is similar to yours. Aid - in aggregate - has been better at producing outputs than outcomes. That said there are good cases of aid-funded education work delivering education outcomes too, so it's clearly possible. Terence

I'm not sure I agree with Garth's comment about 'well-understood', at least in regard to education. The main benefit of aid to education has been to increase enrolment levels, especially at primary level (Riddell and Nino-Zarazua 2016). However, quality-focused aid trying to introduce Anglo-American (and Australian) curriculum philosophies, teacher education and classroom methods has been a widespread failure. Accumulating evidence of the lack of success in fundamentally changing formalistic classroom teaching has been ignored or downplayed. In PNG, this evidence goes back to the 1970s. I've also found evidence from over 600 research and evaluation studies in at least 31 other 'developing' countries that shows similar findings. Often projects successfully generate material and professional inputs, but I have not uncovered a single methodologically sound example of success in generating paradigm shift to 'progressive' styles, nor impact on increasing student achievement from trying to change teaching styles. Actually, the evidence of failure is there and fundamental cultural reasons are understood, at least in the broad, but the evidence just happens to be ignored. Why?

Thanks Sally

Really sensible response, thanks Terence!

Really interesting reflections on an often-overlooked part of the research process: returning the findings to those they came from. Thanks for your open and honest sharing about challenges, positives and principles of this process. It's something close to my heart as a researcher and helpful to learn from how others have approached this.

Hi Suzanne, Good to hear from you. Sorry we didn't get to chat in Wellington. With all aid projects some selection has to be made: which country to work in, which region to work in and so on. This is as true with projects that are evaluated using RCTs as any other method of evaluation. With RCTs, once that selection has been made, the actual allocation of people, or villages, or appropriate units, to the treatment is done randomly. What would be a more ethical approach? With respect to internal validity, with random allocation, so long as sample size is large enough, the group of people (or villages, or whatever) that receive the treatment will be effectively identical to the group that does not receive the treatment. (It is also possible to test for this, and control for this, at least on observed traits using regressions). That means it is very much possible to compare outcomes in this sense. With respect to external validity, it is not guaranteed that a treatment that worked in one location will work in others. This is a serious limitation. Although: (a) it does not invalidate utility for individual evaluations and (b) can be overcome *to some extent* with meta-analysis. There are many other problems with RCTs -- there is much that matters in both aid work and development that they can tell us nothing about, they don't answer how/why questions well -- and so on. But their strengths are real enough. They eliminate confounding factors, they reduce subjective assessment (albeit not entirely in all instances). As for positivism: it seems to me that the null-hypothesis significance testing, which forms an integral part of how people read the result of RCTs, isn't really positivism at all, rather it's Popperian or post-positivist. But as to whether this broader family of approaches can be applied to social science...sure it's applicability can be questioned, but so can that of post-structuralism, critical theory, interpretivism, and so on. Debates amongst philosophers will continue for a long time to come. I really enjoy listening to podcasts about this stuff in the weekends. For now though, at the very least I think it's fair to claim that broadly empirical approaches to social science produce enough useful knowledge to warrant their continued use. I'm very happy to concede that this should be done with humility about what we can and can't know. This should be the case with everything in aid work. Have a good week. Terence

Thanks Garth, I think you make a really important point with the following: "Like so many people the authors appear to have an inflated view of the scale of aid both in terms of donor and recipient income and energy. Given how little effort is put into aid it is amazing what has been achieved especially in areas that are well understood such as basic health services, water, sanitation, education and agriculture."

I take issue with randomised controls not having ethical issues - apart from who gets to select which village (just one example) has access to a particular programme that conditions are such that its not possible to compare outcomes, which is what RCTs aim to do. The notion that positivist methods can be applied to social research is questionable at best

Dear Michael: I enjoyed re-reading your article and attending your seminar at UPNG. Agreed that strengthening the judiciary may be a way to counter the excesses of the PNG Parliament. However one should also be wary of buying into the neoliberal agenda which includes " a relocation of power from political to economic processes, from the state to market and individuals, and finally from the legislature and executive authorities to the judiciary." (Thorsen and Lie 2010).