Comments

Thanks Dinuk, That's a really good comment. It's important to be aware that RCTs -- like everything in development, and all research methods -- can fail. Thanks for highlighting this. If RCTs are to be done they should be done well, and used only when appropriate. As you say, other methods may be more appropriate in particular circumstances. On point I'll make in favour of RCTs is that their problems are somewhat more transparent than those of some other methods. For example, it's easy to see if an RCT is under-powered. And some other analytical flaws are easy enough to spot. For me at least, the ways that a complex regression using panel data can be tweaked are basically opaque. Similarly, it may be impossible to tell who's voices are excluded from a qualitative piece of work and why. Or what community dynamics might skew a participatory evaluation. I'm don't want to make this a binary. I'm not claiming RCTs are perfectly transparent and all other methods complete opaque. Obviously, this isn't the case. Much can still be insufficiently illuminated in an RCT's methods section. But as I thought about your comment over the weekend, I decided it was still fair to say that RCTs have something of a methodological visibility advantage over at least some of the other methods in the tool kit. None of this changes my agreement with your main point though: they can still fail. Terence

Nice share Amanda, you have covered the problems faced by people in data collection & also various solutions for it. The other means of collecting data in such area could be possible by using mobile forms that function seamlessly on tablets & smartphones, an app like Brew Survey uses offline mobile forms feature to collect data even when the device has no internet connection.

Thanks for bringing this to our attention. I think another thing to consider is that some RCTs do fail. One reason is insufficient buy-in from stakeholders. Given that RCTs in social sciences, generally (although not always), take at least one to two years to complete, it is important to ensure buy-in from all stakeholders. One RCT we attempted failed because a new team leader didn’t believe in RCTs while another failed because we weren’t able to sufficiently incentive people in the control and treatment groups to participate in follow-up interviews. A lack of buy-in at implementation level has also led to contamination. In one RCT we were undertaking, a certain government official believed anyone who wanted the intervention should be offered it. While we wanted to know if it worked, the government official believed it did work. Afterwards, when I asked why he went against our protocol, he stated that he would be doing a disservice to his people by not providing the intervention. This clearly relates to the ethical argument against RCTs and I certainly understood his perspective. There are many other reasons for failure – eg: insufficient statistical power (possibly due to small sample sizes), insufficient resources, changing priorities – to name a few. That all being said, I am certainly not anti-RCT. I would say I’m pro-RCT, just as I’m pro-Quasi-Experiment or pro-Participatory Evaluation … it all depends on what is most appropriate under the circumstances (which can also be subjective!). Thanks again for your post.

I think it is useful to acknowledge that the AIIB is intended to have an counter-hegemonic effect (i.e. against the Bretton Woods institutions and their neo-classical paradigms of economic development). The AIIB web-site is admirably transparent as to the organisation and its aims and makes no mention of counter-hegemonic goals. For what these might be it is necessary to turn to recent Chinese literature discussing economic development and economic growth. This brief paper (One Belt, One Road: China’s Strategy for a New Global Financial Order (https://archive.monthlyreview.org/index.php/mr/article/view/MR-068-08-2017-01_4)) does explain the underlying rationale for both the establishment of the AIIB and its fixation on infrastructural development. Here is a quote: ‘Yet the discourse of “peaceful development” has its own blind spots, which reflect China’s domestic contradictions. For instance, how can the AIIB avoid the damage done by the World Bank and others to the environment and indigenous livelihoods? How can China promote infrastructure investments that drive local development through diversity and sustainability, and not simply serve its own need for export outlets? The challenge, in other words, is to ensure that the AIIB and Silk Road Fund do not simply become East Asian counterparts of the IMF and World Bank. Given that OBOR is a contest for institutional influence in East Asia, the deciding factor for success or failure may be the competitiveness of its guiding discourses. China must promote a message of social justice and equitable development to counter the soft power of institutional transition that the United States has pushed since the 1980s.’ But in my view what is of even greater interest is to understand the Chinese view of economic development, poverty alleviation and social justice – the Chinese theory of economic development. The best exemplar of these views is Wen Tiejao whose work encompasses both rural reconstruction and classical political economy and I suggest that it is his views that have been persuasive in the creation of the AIIB. Three useful papers are listed here (http://commons.ln.edu.hk/southsouthforum/ia/wen_tiejun/) ‘Deconstructing Modernization’ ‘Centenary Reflections on the ‘Three Dimensional Problem’ of Rural China’ ‘Four Stories in One: Environmental Protection and Rural Reconstruction in China’ Also useful is: ‘The Development Trap of Financial Capitalism: China’s Peasant Path Compared’ (Agrarian South: Journal of Political Economy 2(3) 247–268) Economic development discourse in China seems to make little use of the practice of referential literature citations as is the practice of western neo-classical economists. Instead concepts and ideas are referenced to development policies and historic events. From a western perspective Chinese economic development thinking shows some connections to and similarities with the classical political economy of David Ricardo and Piero Sraffa (both ‘Commodities’ and ‘re-switching’). One can also find a certain resonance with the Soviet Russian NEP debates between teleological planning (plan maximalism) and genetic planning. Other background to the thinking of Chinese economists about economic development is the realisation that the neo-classical economists, World Bank and its sister organisations have no credible and coherent theory of economic growth (this was the subject of an op-ed in ‘The Economist’, (April 14, 2018, p.67)). Should Australian aid to the Pacific be directed towards infrastructure? A key part of the AIIB initiative is that it expects governments of sovereign countries to define their problems and bring their projects to the AIIB ‘shovel ready’. For an example see the AIIB/India Gujarat rural roads project. It will be interesting to see how ‘facility shopping’ pans out over the long term. Vailala

Thank you for this article on palliative care, a topic close to my heart for professional and personal reasons. And I fully agree with the report on that the present situation regarding access to pain relief and palliative care in many countries is a "medical, public health, and moral failing and a travesty of justice". I am pleased and honored that you used one of my pictures on palliative care in Kerala to illustrate your text.

Congratulations and thank you Jonathan and Alex for achieving this major, useful milestone for aid transparency in the Pacific. This clearly maps out the politics of aid across the region. Looking forward to the additional information on other flows in the coming weeks and great to see the commitment for an update every 3 years.

Thanks for the post Matt, thought-provoking as always. As you say, there are some obvious drawbacks to the proposed approach, which, as you also state, can be managed. The better question though is: will they be managed if this shift is triggered more by geopolitical considerations than aid effectiveness? One further issue I would like to see gain more prominence in the hard infrastructure discussion in the Pacific is climate resilience. Australia has made climate finance commitments and is also moving to integrate climate change across the aid program (see here: http://dfat.gov.au/aid/topics/investment-priorities/building-resilience/climate-change/Pages/climate-change.aspx). If Australia invests more in hard infrastructure in the region it will be critical to ensure the systems and processes are in place to make sure these large-scale investments are designed, built and managed in ways that make sense both in today's climate and the less hospitable climate we are rapidly heading towards; and will, undoubtedly, arrive during the lifecycle of any new hard infrastructure investments commissioned in the next few years.

Hi Suzanne, Thanks for your comment. Good to hear from you again. For what it's worth, I think the scientific method applies both to the material and human world, for the simple reason that we're part of the material world. Having said that, the scientific method becomes more and more complex the further we move away from physics, simply because the number of variables and their potential interactions become much more numerous. And, as you say, our culture and beliefs constrain and shape the views we hold. This doesn't mean that all views are equally right though, it just makes the task of learning harder still. (I should add even physicists suffer some of these problems too.) Moreover, these problems don't mean that RCTs won't work. Indeed, one of their strengths is the robustness of their method and the clear results it provides. Having said that, I don't think RCTs can answer all the important questions. And I agree that there is an important role for qualitative research. In the specific case of the bed nets, which I should stress was a hypothetical example. (RCTs may have been used in their study but I don't know). We know for obvious reasons that bed nets stop mosquitoes. Less clear, at least less clear a decade ago, were questions such as: 1. Will people use them as required (answer: yes (to the best of my knowledge)) 2. Did charging a small price for bed nets reduce their use (answer: yes) 3. Are insecticide treated bed nets more effective (answer: yes) To the best of my limited knowledge many of these questions were answered without RCTs, but they were answered with empirical social science. Thanks for your comments. Terence

Hi Eileen, Thank you for your reply. With respect to the cost. The treatment group only involved 84 students (more on that in a second). My guess is that, compared to the cost of the education system in Mauritius, the cost of upgrading approximately three pre-schools is a small price to pay for learning that investing a lot of extra money in high quality pre-schools (at least of this type) is not likely to deliver the benefits they anticipated. Certainly, the cost of improving 3 pre-schools wouldn't be enough to improve the overall quality of pre-school education in Mauritius. As for control variables, if the sample is large enough, the treatment and control groups will be effectively the same, regardless of the complexity of human life. This is one of the big advantages of RCTs over non-experimental data. (A sample of 174 may not be large enough, I'm not here to defend the study in its entirety, although the authors do show the treatment and control groups are balanced across an impressive array of variables in Table 1. A sample of 174 may also be under-powered more generally, but the fact that some people run poor quality RCTs doesn't mean the method itself is flawed.) As for schools not being laboratories, I think this critique is perhaps more persuasive with respect to the external validity of the study. (See the blog, for my views on the issue of external validity.) Thanks for your comment. Terence

Thanks Terence. Imagine if the money spent on creating 'high quality preschools for the experiment' was spent on making the low quality schools maybe even average quality? So maybe instead of 50% children getting high quality, then maybe 80% get medium quality? Regarding the 'control of variables' - unfortunately any teacher in the world can tell you that controlling variables (in education that means children, parents, teachers) is extremely difficult. This is what non-education researchers just don't get. The closest I saw was a researcher from a science background I was working with who suggested the only theory to explain what went on in an early years classroom was chaos theory. RCTs are designed for use in laboratories. Schools are not laboratories.

Thanks Terence, there is certainly some heat in this debate! I suggest that one of the issues underlying it is that of the unity of science – that scientific method applies equally to the study of the material and social world. I am persuaded by the debate that says we can only ever argue about this because neither side can position itself outside of discourse and proclaim ‘truth’ from a position of ‘nowhere’. We are all shaped by history, society, and culture, which in turn influences the questions we chose to ask as well as the answers we hear. That does not make me anti-science but rather concerned that methodology is selected according to the questions we are trying and answer. Testing new drugs may demand RCTs but will never provide what Clifford Geertz calls the ‘thick description’ critical to understanding our social and cultural worlds. Whatever the methodology selected, ethical research or evaluation demands informed consent. If anyone needs reminding about the importance of informed consent go back to the Cartwright Report. The gains from that were hard won and need to be protected and extended. As the NYT article shows, random selection is critical to RCTs and the wellness programme was fortunate in getting 500 volunteers from which participants and a control group could be selected. What’s the equivalent process in an international development context, in which donors have the cash and communities want and need support? Let’s take the bed-net evaluation as an example – it is possible to do an RCT but the question is why would you go to the trouble and expense? It is well known that mosquitos spread malaria so what questions would you need an RCT to answer? How would the participants be selected – would it be open to all community members in the first instance? Would the control group be offered treatment should they contract malaria during the trial? They are after all part of an experiment. An alternative is to identify participants using the ‘snow-ball’ method and, having gained informed consent, interview until you’ve reached ‘saturation’ point about the factors that limit access and use and those that facilitate it? A well-designed study couples this with other data collection methods. The approach does not make claims about cause and effect but rather seeks a deeper understanding of what people say they do and why (users and non-users) and makes recommendations on that understanding. It seems to me that participation and partnership demands approaches that involve participants in research and evaluation and that the decision to use RCTs, given how resource intensive they are, should be reserved to answer those questions that can best be answered by experiment and not because of an ideological position about the unity of science.

Hi Australian Foreign Minister Julie Bishop’s comments on Monday that Australia will compete with China in the Pacific over funding infrastructure projects, to ensure that small island countries “retain their sovereignty”, sounds rather ironic given Australia’s highly interventionist stance towards the region in recent decades. However, irrespective of the merit of Chinese infrastructure projects and the sustainability of the debt accumulated by Pacific states to pay for them, the Australian Government’s decision to use the aid program to fund infrastructure is welcome and long overdue. Thank you !!