Comments

Hi Eileen, Thank you for your reply. With respect to the cost. The treatment group only involved 84 students (more on that in a second). My guess is that, compared to the cost of the education system in Mauritius, the cost of upgrading approximately three pre-schools is a small price to pay for learning that investing a lot of extra money in high quality pre-schools (at least of this type) is not likely to deliver the benefits they anticipated. Certainly, the cost of improving 3 pre-schools wouldn't be enough to improve the overall quality of pre-school education in Mauritius. As for control variables, if the sample is large enough, the treatment and control groups will be effectively the same, regardless of the complexity of human life. This is one of the big advantages of RCTs over non-experimental data. (A sample of 174 may not be large enough, I'm not here to defend the study in its entirety, although the authors do show the treatment and control groups are balanced across an impressive array of variables in Table 1. A sample of 174 may also be under-powered more generally, but the fact that some people run poor quality RCTs doesn't mean the method itself is flawed.) As for schools not being laboratories, I think this critique is perhaps more persuasive with respect to the external validity of the study. (See the blog, for my views on the issue of external validity.) Thanks for your comment. Terence

Thanks Terence. Imagine if the money spent on creating 'high quality preschools for the experiment' was spent on making the low quality schools maybe even average quality? So maybe instead of 50% children getting high quality, then maybe 80% get medium quality? Regarding the 'control of variables' - unfortunately any teacher in the world can tell you that controlling variables (in education that means children, parents, teachers) is extremely difficult. This is what non-education researchers just don't get. The closest I saw was a researcher from a science background I was working with who suggested the only theory to explain what went on in an early years classroom was chaos theory. RCTs are designed for use in laboratories. Schools are not laboratories.

Thanks Terence, there is certainly some heat in this debate! I suggest that one of the issues underlying it is that of the unity of science – that scientific method applies equally to the study of the material and social world. I am persuaded by the debate that says we can only ever argue about this because neither side can position itself outside of discourse and proclaim ‘truth’ from a position of ‘nowhere’. We are all shaped by history, society, and culture, which in turn influences the questions we chose to ask as well as the answers we hear. That does not make me anti-science but rather concerned that methodology is selected according to the questions we are trying and answer. Testing new drugs may demand RCTs but will never provide what Clifford Geertz calls the ‘thick description’ critical to understanding our social and cultural worlds. Whatever the methodology selected, ethical research or evaluation demands informed consent. If anyone needs reminding about the importance of informed consent go back to the Cartwright Report. The gains from that were hard won and need to be protected and extended. As the NYT article shows, random selection is critical to RCTs and the wellness programme was fortunate in getting 500 volunteers from which participants and a control group could be selected. What’s the equivalent process in an international development context, in which donors have the cash and communities want and need support? Let’s take the bed-net evaluation as an example – it is possible to do an RCT but the question is why would you go to the trouble and expense? It is well known that mosquitos spread malaria so what questions would you need an RCT to answer? How would the participants be selected – would it be open to all community members in the first instance? Would the control group be offered treatment should they contract malaria during the trial? They are after all part of an experiment. An alternative is to identify participants using the ‘snow-ball’ method and, having gained informed consent, interview until you’ve reached ‘saturation’ point about the factors that limit access and use and those that facilitate it? A well-designed study couples this with other data collection methods. The approach does not make claims about cause and effect but rather seeks a deeper understanding of what people say they do and why (users and non-users) and makes recommendations on that understanding. It seems to me that participation and partnership demands approaches that involve participants in research and evaluation and that the decision to use RCTs, given how resource intensive they are, should be reserved to answer those questions that can best be answered by experiment and not because of an ideological position about the unity of science.

Hi Australian Foreign Minister Julie Bishop’s comments on Monday that Australia will compete with China in the Pacific over funding infrastructure projects, to ensure that small island countries “retain their sovereignty”, sounds rather ironic given Australia’s highly interventionist stance towards the region in recent decades. However, irrespective of the merit of Chinese infrastructure projects and the sustainability of the debt accumulated by Pacific states to pay for them, the Australian Government’s decision to use the aid program to fund infrastructure is welcome and long overdue. Thank you !!

Due to the presumption of innocence, it is the other way around. Why did the accusers not produce a shred of evidence since 2012 that I had falsified anything? The judge in the National Court was very clear about this: "In spite of this overwhelming evidence (presented by Dr. Schram) Mr. Saulep continues to dispute the authenticity of the applicant's doctorate degree. I find this ridiculous and difficult to fathom especially when neither he or the police are in receipt of evidence from the European University Institute in Florence Italy, confirming their allegations and suspicions". http://albertschram.blogspot.com/2018/05/wrongful-dismissal-and-malicious.html#more

Thank you Mathew for your comment. The relevant World Bank document is the ‘Papua New Guinea - Road Maintenance and Rehabilitation Project’ (http://documents.worldbank.org/curated/en/262651468775145760/Papua-New-Guinea-Road-Maintenance-and-Rehabilitation-Project). This report includes some details of the Social Impact Monitoring Plan and makes brief mention (in a somewhat incoherent way) of the proposed research on pp 97 and 99. ‘The effect of the sub-projects on the population served by the roads involved will be assessed periodically through social studies to be undertaken under the project. The assessments will be designed to detect changes in: household incomes; cash crop/commodity production and sale; prices of commodities imported from other regions; changes in school attendance/reduced drop-out rates; health benefits from improved access to aid posts, clinics and hospitals; etc. The Project Implementation Plan (PIP) contains the terms of reference of the study for monitoring the socio-economic impacts of road and bridge improvements under the RMRP. The PNGVs will collect basic social mapping data and the SIA component will supplement this with surveys designed to villager satisfaction with the implementation of the RMRP sub-projects and gather information about the poverty alleviation effects and any adverse impacts. Baseline studies will be conducted before implementation, at mid-point and at the end of the RMRP.’ The relevant TOR for the PIP contained precise details as to the content of the survey research and methodology and the use of PNG Volunteers for data gathering. I have not been able to find these documents in the World Bank archive. Very briefly the research design was prepared as a response to a casual remark by a World Bank staffer that while the WB spent hundred of millions of dollars on road improvement schemes it had no idea as to any measurable effects this expenditure might have on poverty reduction – especially in countries where the informal economy and subsistence agriculture were demographically dominant. The research design drew on John Gibson’s valuable data on PNG household consumption patterns and the Agogo Mawuli and Ogis Sanida NRI paper ‘Landowners' Mineral Rent-quest and Use in Papua New Guinea’ which provides a useful guide as to how rural PNG households spend cash income. Also of relevance were studies made of artisanal fisheries and small business development along the Morobe coast. The questionnaire (about 26 items) was designed to elicit data not only on domestic household consumption but also data relevant to household welfare and household capital formation and the linkage of these to the provision of low quality rural roads. The WB RMRP project although initiated in the 1990s was not implemented until around 2004. The WB appeared to have farmed out the SIA components to Ausaid for review following which the SIA parts were assigned to SMEC/Finnroad. The involvement of PNG Volunteers appears to have ended and the focus of the research/questionnaire greatly altered. Much of the emphasis on non-monetised subsistence production (the consumption basket) and village household capital formation appears to have been lost in translation. The final analysis, done by Finnroad in 2007/8 paid little attention to these matters, concentrating instead on conventional measures of cash income and transport accessibility. The WB project assessment document describes the SIA findings here (http://documents.worldbank.org/curated/en/845161468286308542/pdf/NonAsciiFileName0.pdf) on pp 40 – 42. A brief Finnroad account can be found here – https://trid.trb.org/view/877540. So, yes I do think that the research is yet to be done, at least in PNG. Why am I making a point of this? Because I saw the research as an attempt to step beyond the usual framework of road infrastructure investment evaluation in terms of the EIRR. This is an issue of concern to Chinese development economists, especially Wen Tiejun, and, in my opinion underlies the anti-hegemonic orientation of the AIIB. Vailala

Thanks Garth, that's an excellent link. I hadn't seen it previously. Very useful. Thanks again. Terence

Thanks Chris, That's a great comment. I agree with all of your points, but would note: At times you will have to collect your own data. This is unavoidable, particularly if you want quality data. That said, I think the cost is a small price to pay for learning. I agree, RCTs can help with learning how a project works too. I'd just add that there may be limits and that a great alternative would be to combine RCTs with process tracing as Oxfam Great Britain does. Quant and qual methods can be friends 🙂 Thanks again. Terence

Hi Ben, Thanks for your comment. I think the wasting the control group's time is about as good a ethical issue as can possibly be raised. (Absent some non-RCT-specific ethical lapse). The situation is still no worse than the normal evaluation of a project that did not work, though. What's more, people fill out census forms, HIES and DHS surveys for no benefit. Not quite as intrusive, of course, but it's not as if RCTs are new in wasting people's time for data. Moreover, if the treatment is effective, the compensation could be expanding it to the control group. Re your attention stimulation point. I agree. It will be a problem with all good evaluations though. Moreover if the attention factor proves to be critical then perhaps we've learnt something useful that should be built into aid more generally. "Socio-economic surveys are commonly used in RCTs – they are generally very crude instruments because they rely on complex, often confidential, information from recipients that very few of us could provide accurately." It would be interesting to know what % of RCTs do actually rely on poor data (I don't think surveys inevitably generate such data, but know this is a real possibility). Once again, this isn't specific to RCTs, though. No evaluation will ever be better than the quality of its data. Great points. Thank you for engaging. Terence

Nicely balanced blog Terence. I just wanted to clarify a couple of the 'limitations' you mentioned of RCTs. Cost of RCTs - RCTs can be done for free. This is can occur when data is already going to be collected. For example, governments regularly collect data about things like who has paid their taxes. Here is example from Latin America of an RCT that uses government administrative data: https://www.povertyactionlab.org/evaluation/role-vat-tax-enforcement Programs RCTs can be used on - RCTs can be used to evaluate both micro and macro ('universal') level policy changes (even large scale infrastructure). This can be achieved using a nudge to encourage people to take up a new service or product. Here is an example from Africa of an RCT that evaluates the impact of a large infrastructure project: https://www.povertyactionlab.org/evaluation/household-water-connections-tangier-morocco How programs work - RCTs can be used to determine why/how a program works. This can be achieved by having multiple treatment groups whereby some people are provided with a sub-component of the program and other people receive the whole program. If there is no difference between the impact of the sub-component and the entire program this indicates that it is just the sub-component that is making a difference. Here is an example of an RCT in South Asia that used this approach: https://www.povertyactionlab.org/evaluation/improving-immunization-rates-through-regular-camps-and-incentives-india

Hi Terence, I'm in agreement for the most part - RCTs have an important place. I'm also of the view that RCTs can be highly unethical - Elieen has given a good example, but there are many others from rural development that could be used. Data collection in RCTs also commonly involves taking the time of control groups that have received nothing from the development activity. Leaving that aside, RCTs generally do not control for at least one important factor - the attention and stimulation to recipients that goes with a development activity. This factor may be as important or more important than the technology/support on offer. You mentioned that RCTs can't always pinpoint what caused the impact and this is an example of that. You also mentioned the need for good data. Socio-economic surveys are commonly used in RCTs - they are generally very crude instruments because they rely on complex, often confidential, information from recipients that very few of us could provide accurately. So still worth discussing maybe...

Hi Eileen, You make it sound like the experiment in question denied preschoolers in question access to 'high quality' preschools. As best I can tell from skimming the methods what actually happened is that 50% of the students in the sample were given 'high quality' preschooling that they would have otherwise failed to receive. Then, from the abstract, the main finding was: "The findings show that quality of preschool education had no significant effect on children's overall educational attainment." (Although there may be heterogeneous effects.) No one was denied high quality preschooling in this experiment. What's more the intervention was discovered not to help. (Plus or minus a few caveats). I fail to see how this was unethical. "I was at an aid and development seminar a couple of years ago where many people undertaking evaluations mistakenly believed they were doing RCTs when they were obviously not. It’s a term that is being used inappropriately because bureaucrats believe it sounds good." This is disappointing to hear, I appreciate you raising the point, and I share your concern. "Doing a systematic evaluation and collecting rigorous data does not require use of this method." I agree. I thought I said as much in the blog post, sorry if I was unclear. "This is especially true in education when it is almost impossible to distinguish between the many variables that can impact on student learning and performance." But, within constraints, that is what RCTs do: they control effectively for the influence of other variables. I agree there are issues (listed in my post), but it's worth giving RCTs credit for what they can do well. Thank you for your comment. Terence