Comments

Thanks for the post Matt, thought-provoking as always. As you say, there are some obvious drawbacks to the proposed approach, which, as you also state, can be managed. The better question though is: will they be managed if this shift is triggered more by geopolitical considerations than aid effectiveness? One further issue I would like to see gain more prominence in the hard infrastructure discussion in the Pacific is climate resilience. Australia has made climate finance commitments and is also moving to integrate climate change across the aid program (see here: http://dfat.gov.au/aid/topics/investment-priorities/building-resilience/climate-change/Pages/climate-change.aspx). If Australia invests more in hard infrastructure in the region it will be critical to ensure the systems and processes are in place to make sure these large-scale investments are designed, built and managed in ways that make sense both in today's climate and the less hospitable climate we are rapidly heading towards; and will, undoubtedly, arrive during the lifecycle of any new hard infrastructure investments commissioned in the next few years.

Hi Suzanne, Thanks for your comment. Good to hear from you again. For what it's worth, I think the scientific method applies both to the material and human world, for the simple reason that we're part of the material world. Having said that, the scientific method becomes more and more complex the further we move away from physics, simply because the number of variables and their potential interactions become much more numerous. And, as you say, our culture and beliefs constrain and shape the views we hold. This doesn't mean that all views are equally right though, it just makes the task of learning harder still. (I should add even physicists suffer some of these problems too.) Moreover, these problems don't mean that RCTs won't work. Indeed, one of their strengths is the robustness of their method and the clear results it provides. Having said that, I don't think RCTs can answer all the important questions. And I agree that there is an important role for qualitative research. In the specific case of the bed nets, which I should stress was a hypothetical example. (RCTs may have been used in their study but I don't know). We know for obvious reasons that bed nets stop mosquitoes. Less clear, at least less clear a decade ago, were questions such as: 1. Will people use them as required (answer: yes (to the best of my knowledge)) 2. Did charging a small price for bed nets reduce their use (answer: yes) 3. Are insecticide treated bed nets more effective (answer: yes) To the best of my limited knowledge many of these questions were answered without RCTs, but they were answered with empirical social science. Thanks for your comments. Terence

Hi Eileen, Thank you for your reply. With respect to the cost. The treatment group only involved 84 students (more on that in a second). My guess is that, compared to the cost of the education system in Mauritius, the cost of upgrading approximately three pre-schools is a small price to pay for learning that investing a lot of extra money in high quality pre-schools (at least of this type) is not likely to deliver the benefits they anticipated. Certainly, the cost of improving 3 pre-schools wouldn't be enough to improve the overall quality of pre-school education in Mauritius. As for control variables, if the sample is large enough, the treatment and control groups will be effectively the same, regardless of the complexity of human life. This is one of the big advantages of RCTs over non-experimental data. (A sample of 174 may not be large enough, I'm not here to defend the study in its entirety, although the authors do show the treatment and control groups are balanced across an impressive array of variables in Table 1. A sample of 174 may also be under-powered more generally, but the fact that some people run poor quality RCTs doesn't mean the method itself is flawed.) As for schools not being laboratories, I think this critique is perhaps more persuasive with respect to the external validity of the study. (See the blog, for my views on the issue of external validity.) Thanks for your comment. Terence

Thanks Terence. Imagine if the money spent on creating 'high quality preschools for the experiment' was spent on making the low quality schools maybe even average quality? So maybe instead of 50% children getting high quality, then maybe 80% get medium quality? Regarding the 'control of variables' - unfortunately any teacher in the world can tell you that controlling variables (in education that means children, parents, teachers) is extremely difficult. This is what non-education researchers just don't get. The closest I saw was a researcher from a science background I was working with who suggested the only theory to explain what went on in an early years classroom was chaos theory. RCTs are designed for use in laboratories. Schools are not laboratories.

Thanks Terence, there is certainly some heat in this debate! I suggest that one of the issues underlying it is that of the unity of science – that scientific method applies equally to the study of the material and social world. I am persuaded by the debate that says we can only ever argue about this because neither side can position itself outside of discourse and proclaim ‘truth’ from a position of ‘nowhere’. We are all shaped by history, society, and culture, which in turn influences the questions we chose to ask as well as the answers we hear. That does not make me anti-science but rather concerned that methodology is selected according to the questions we are trying and answer. Testing new drugs may demand RCTs but will never provide what Clifford Geertz calls the ‘thick description’ critical to understanding our social and cultural worlds. Whatever the methodology selected, ethical research or evaluation demands informed consent. If anyone needs reminding about the importance of informed consent go back to the Cartwright Report. The gains from that were hard won and need to be protected and extended. As the NYT article shows, random selection is critical to RCTs and the wellness programme was fortunate in getting 500 volunteers from which participants and a control group could be selected. What’s the equivalent process in an international development context, in which donors have the cash and communities want and need support? Let’s take the bed-net evaluation as an example – it is possible to do an RCT but the question is why would you go to the trouble and expense? It is well known that mosquitos spread malaria so what questions would you need an RCT to answer? How would the participants be selected – would it be open to all community members in the first instance? Would the control group be offered treatment should they contract malaria during the trial? They are after all part of an experiment. An alternative is to identify participants using the ‘snow-ball’ method and, having gained informed consent, interview until you’ve reached ‘saturation’ point about the factors that limit access and use and those that facilitate it? A well-designed study couples this with other data collection methods. The approach does not make claims about cause and effect but rather seeks a deeper understanding of what people say they do and why (users and non-users) and makes recommendations on that understanding. It seems to me that participation and partnership demands approaches that involve participants in research and evaluation and that the decision to use RCTs, given how resource intensive they are, should be reserved to answer those questions that can best be answered by experiment and not because of an ideological position about the unity of science.

Hi Australian Foreign Minister Julie Bishop’s comments on Monday that Australia will compete with China in the Pacific over funding infrastructure projects, to ensure that small island countries “retain their sovereignty”, sounds rather ironic given Australia’s highly interventionist stance towards the region in recent decades. However, irrespective of the merit of Chinese infrastructure projects and the sustainability of the debt accumulated by Pacific states to pay for them, the Australian Government’s decision to use the aid program to fund infrastructure is welcome and long overdue. Thank you !!

Due to the presumption of innocence, it is the other way around. Why did the accusers not produce a shred of evidence since 2012 that I had falsified anything? The judge in the National Court was very clear about this: "In spite of this overwhelming evidence (presented by Dr. Schram) Mr. Saulep continues to dispute the authenticity of the applicant's doctorate degree. I find this ridiculous and difficult to fathom especially when neither he or the police are in receipt of evidence from the European University Institute in Florence Italy, confirming their allegations and suspicions". http://albertschram.blogspot.com/2018/05/wrongful-dismissal-and-malicious.html#more

Thank you Mathew for your comment. The relevant World Bank document is the ‘Papua New Guinea - Road Maintenance and Rehabilitation Project’ (http://documents.worldbank.org/curated/en/262651468775145760/Papua-New-Guinea-Road-Maintenance-and-Rehabilitation-Project). This report includes some details of the Social Impact Monitoring Plan and makes brief mention (in a somewhat incoherent way) of the proposed research on pp 97 and 99. ‘The effect of the sub-projects on the population served by the roads involved will be assessed periodically through social studies to be undertaken under the project. The assessments will be designed to detect changes in: household incomes; cash crop/commodity production and sale; prices of commodities imported from other regions; changes in school attendance/reduced drop-out rates; health benefits from improved access to aid posts, clinics and hospitals; etc. The Project Implementation Plan (PIP) contains the terms of reference of the study for monitoring the socio-economic impacts of road and bridge improvements under the RMRP. The PNGVs will collect basic social mapping data and the SIA component will supplement this with surveys designed to villager satisfaction with the implementation of the RMRP sub-projects and gather information about the poverty alleviation effects and any adverse impacts. Baseline studies will be conducted before implementation, at mid-point and at the end of the RMRP.’ The relevant TOR for the PIP contained precise details as to the content of the survey research and methodology and the use of PNG Volunteers for data gathering. I have not been able to find these documents in the World Bank archive. Very briefly the research design was prepared as a response to a casual remark by a World Bank staffer that while the WB spent hundred of millions of dollars on road improvement schemes it had no idea as to any measurable effects this expenditure might have on poverty reduction – especially in countries where the informal economy and subsistence agriculture were demographically dominant. The research design drew on John Gibson’s valuable data on PNG household consumption patterns and the Agogo Mawuli and Ogis Sanida NRI paper ‘Landowners' Mineral Rent-quest and Use in Papua New Guinea’ which provides a useful guide as to how rural PNG households spend cash income. Also of relevance were studies made of artisanal fisheries and small business development along the Morobe coast. The questionnaire (about 26 items) was designed to elicit data not only on domestic household consumption but also data relevant to household welfare and household capital formation and the linkage of these to the provision of low quality rural roads. The WB RMRP project although initiated in the 1990s was not implemented until around 2004. The WB appeared to have farmed out the SIA components to Ausaid for review following which the SIA parts were assigned to SMEC/Finnroad. The involvement of PNG Volunteers appears to have ended and the focus of the research/questionnaire greatly altered. Much of the emphasis on non-monetised subsistence production (the consumption basket) and village household capital formation appears to have been lost in translation. The final analysis, done by Finnroad in 2007/8 paid little attention to these matters, concentrating instead on conventional measures of cash income and transport accessibility. The WB project assessment document describes the SIA findings here (http://documents.worldbank.org/curated/en/845161468286308542/pdf/NonAsciiFileName0.pdf) on pp 40 – 42. A brief Finnroad account can be found here – https://trid.trb.org/view/877540. So, yes I do think that the research is yet to be done, at least in PNG. Why am I making a point of this? Because I saw the research as an attempt to step beyond the usual framework of road infrastructure investment evaluation in terms of the EIRR. This is an issue of concern to Chinese development economists, especially Wen Tiejun, and, in my opinion underlies the anti-hegemonic orientation of the AIIB. Vailala

Thanks Garth, that's an excellent link. I hadn't seen it previously. Very useful. Thanks again. Terence

Thanks Chris, That's a great comment. I agree with all of your points, but would note: At times you will have to collect your own data. This is unavoidable, particularly if you want quality data. That said, I think the cost is a small price to pay for learning. I agree, RCTs can help with learning how a project works too. I'd just add that there may be limits and that a great alternative would be to combine RCTs with process tracing as Oxfam Great Britain does. Quant and qual methods can be friends 🙂 Thanks again. Terence

Hi Ben, Thanks for your comment. I think the wasting the control group's time is about as good a ethical issue as can possibly be raised. (Absent some non-RCT-specific ethical lapse). The situation is still no worse than the normal evaluation of a project that did not work, though. What's more, people fill out census forms, HIES and DHS surveys for no benefit. Not quite as intrusive, of course, but it's not as if RCTs are new in wasting people's time for data. Moreover, if the treatment is effective, the compensation could be expanding it to the control group. Re your attention stimulation point. I agree. It will be a problem with all good evaluations though. Moreover if the attention factor proves to be critical then perhaps we've learnt something useful that should be built into aid more generally. "Socio-economic surveys are commonly used in RCTs – they are generally very crude instruments because they rely on complex, often confidential, information from recipients that very few of us could provide accurately." It would be interesting to know what % of RCTs do actually rely on poor data (I don't think surveys inevitably generate such data, but know this is a real possibility). Once again, this isn't specific to RCTs, though. No evaluation will ever be better than the quality of its data. Great points. Thank you for engaging. Terence

Nicely balanced blog Terence. I just wanted to clarify a couple of the 'limitations' you mentioned of RCTs. Cost of RCTs - RCTs can be done for free. This is can occur when data is already going to be collected. For example, governments regularly collect data about things like who has paid their taxes. Here is example from Latin America of an RCT that uses government administrative data: https://www.povertyactionlab.org/evaluation/role-vat-tax-enforcement Programs RCTs can be used on - RCTs can be used to evaluate both micro and macro ('universal') level policy changes (even large scale infrastructure). This can be achieved using a nudge to encourage people to take up a new service or product. Here is an example from Africa of an RCT that evaluates the impact of a large infrastructure project: https://www.povertyactionlab.org/evaluation/household-water-connections-tangier-morocco How programs work - RCTs can be used to determine why/how a program works. This can be achieved by having multiple treatment groups whereby some people are provided with a sub-component of the program and other people receive the whole program. If there is no difference between the impact of the sub-component and the entire program this indicates that it is just the sub-component that is making a difference. Here is an example of an RCT in South Asia that used this approach: https://www.povertyactionlab.org/evaluation/improving-immunization-rates-through-regular-camps-and-incentives-india