Comments

Thanks Chris, That's a great comment. I agree with all of your points, but would note: At times you will have to collect your own data. This is unavoidable, particularly if you want quality data. That said, I think the cost is a small price to pay for learning. I agree, RCTs can help with learning how a project works too. I'd just add that there may be limits and that a great alternative would be to combine RCTs with process tracing as Oxfam Great Britain does. Quant and qual methods can be friends 🙂 Thanks again. Terence

Hi Ben, Thanks for your comment. I think the wasting the control group's time is about as good a ethical issue as can possibly be raised. (Absent some non-RCT-specific ethical lapse). The situation is still no worse than the normal evaluation of a project that did not work, though. What's more, people fill out census forms, HIES and DHS surveys for no benefit. Not quite as intrusive, of course, but it's not as if RCTs are new in wasting people's time for data. Moreover, if the treatment is effective, the compensation could be expanding it to the control group. Re your attention stimulation point. I agree. It will be a problem with all good evaluations though. Moreover if the attention factor proves to be critical then perhaps we've learnt something useful that should be built into aid more generally. "Socio-economic surveys are commonly used in RCTs – they are generally very crude instruments because they rely on complex, often confidential, information from recipients that very few of us could provide accurately." It would be interesting to know what % of RCTs do actually rely on poor data (I don't think surveys inevitably generate such data, but know this is a real possibility). Once again, this isn't specific to RCTs, though. No evaluation will ever be better than the quality of its data. Great points. Thank you for engaging. Terence

Nicely balanced blog Terence. I just wanted to clarify a couple of the 'limitations' you mentioned of RCTs. Cost of RCTs - RCTs can be done for free. This is can occur when data is already going to be collected. For example, governments regularly collect data about things like who has paid their taxes. Here is example from Latin America of an RCT that uses government administrative data: https://www.povertyactionlab.org/evaluation/role-vat-tax-enforcement Programs RCTs can be used on - RCTs can be used to evaluate both micro and macro ('universal') level policy changes (even large scale infrastructure). This can be achieved using a nudge to encourage people to take up a new service or product. Here is an example from Africa of an RCT that evaluates the impact of a large infrastructure project: https://www.povertyactionlab.org/evaluation/household-water-connections-tangier-morocco How programs work - RCTs can be used to determine why/how a program works. This can be achieved by having multiple treatment groups whereby some people are provided with a sub-component of the program and other people receive the whole program. If there is no difference between the impact of the sub-component and the entire program this indicates that it is just the sub-component that is making a difference. Here is an example of an RCT in South Asia that used this approach: https://www.povertyactionlab.org/evaluation/improving-immunization-rates-through-regular-camps-and-incentives-india

Hi Terence, I'm in agreement for the most part - RCTs have an important place. I'm also of the view that RCTs can be highly unethical - Elieen has given a good example, but there are many others from rural development that could be used. Data collection in RCTs also commonly involves taking the time of control groups that have received nothing from the development activity. Leaving that aside, RCTs generally do not control for at least one important factor - the attention and stimulation to recipients that goes with a development activity. This factor may be as important or more important than the technology/support on offer. You mentioned that RCTs can't always pinpoint what caused the impact and this is an example of that. You also mentioned the need for good data. Socio-economic surveys are commonly used in RCTs - they are generally very crude instruments because they rely on complex, often confidential, information from recipients that very few of us could provide accurately. So still worth discussing maybe...

Hi Eileen, You make it sound like the experiment in question denied preschoolers in question access to 'high quality' preschools. As best I can tell from skimming the methods what actually happened is that 50% of the students in the sample were given 'high quality' preschooling that they would have otherwise failed to receive. Then, from the abstract, the main finding was: "The findings show that quality of preschool education had no significant effect on children's overall educational attainment." (Although there may be heterogeneous effects.) No one was denied high quality preschooling in this experiment. What's more the intervention was discovered not to help. (Plus or minus a few caveats). I fail to see how this was unethical. "I was at an aid and development seminar a couple of years ago where many people undertaking evaluations mistakenly believed they were doing RCTs when they were obviously not. It’s a term that is being used inappropriately because bureaucrats believe it sounds good." This is disappointing to hear, I appreciate you raising the point, and I share your concern. "Doing a systematic evaluation and collecting rigorous data does not require use of this method." I agree. I thought I said as much in the blog post, sorry if I was unclear. "This is especially true in education when it is almost impossible to distinguish between the many variables that can impact on student learning and performance." But, within constraints, that is what RCTs do: they control effectively for the influence of other variables. I agree there are issues (listed in my post), but it's worth giving RCTs credit for what they can do well. Thank you for your comment. Terence

Thanks Terence for looking at this topical issue. Here is an article and book I found useful on the important and related area of monitoring: http://blogs.worldbank.org/impactevaluations/if-i-can-t-do-impact-evaluation-what-should-i-do-review-gugerty-and-karlan-s-goldilocks-challenge

I have two comments to make about your post. 1. Your argument that RCTs are not unethical denies the reality of projects like the one reported on in this article. Deliberately providing "high quality" education to young children and denying others that same education purely for experimental purposes is unethical. "Children aged three to four years were assigned randomly to high-quality preschools that were created for the experiment or to existing petites écoles (that is, low-quality preschools)". https://www.sciencedirect.com/science/article/abs/pii/S0272775717302637 2. I was at an aid and development seminar a couple of years ago where many people undertaking evaluations mistakenly believed they were doing RCTs when they were obviously not. It's a term that is being used inappropriately because bureaucrats believe it sounds good. Doing a systematic evaluation and collecting rigorous data does not require use of this method. This is especially true in education when it is almost impossible to distinguish between the many variables that can impact on student learning and performance.

If the blog interested you, this article in the NY Times will also be of interest: https://www.nytimes.com/2018/08/06/upshot/employer-wellness-programs-randomized-trials.html

This is an interesting analysis of the past year's SWP figures, Stephen. One caution though -- your comment about Tonga's participation in the SWP and RSE reaching "an impressive 13% of the eligible sending population (20-45 years)" needs some qualification. Robert Ingram and I have shown in research completed for LMAP that the "eligible sending population" needs to be defined much more precisely than just the total in a particular age group in the sending country. Given that recruitment for both the RSE and SWP is strongly biased towards younger adult men it is much better to express the share of the "eligible sending population" in terms of males aged 20-39 years. The Australian Department of Home Affairs (ADHA) does not publish data on SWP participants by age, but Immigration New Zealand (INZ) has made such data available up until quite recently for RSE workers in broad age groups. If we assume the SWP workers heading for Australia from Tonga are from the same basic demographic groups as those heading for work under the RSE scheme, and there is no reason to assume they would be very different given they are doing similar types of seasonal work, then we can establish a little more precisely just what the the impact of seasonal work in the two schemes is having on the main age group affected. Data from the ADHA and INZ show that in the 2016/17 financial year 89 percent of the Tongans participating in the SWP and the RSE were males (4,025 out of 4,512 who were employed in the two schemes). If we assume the 2,394 Tongan male SWP workers had the same age distribution as the 1,621 Tongan male RSE workers then 3,276 of the 4,025 males would have been aged 20-39 years -- 81.4 percent. When we put these 3,276 males aged 20-39 years alongside the 13,000 Tongan males aged 20-39 years at the time of Tonga's census in 2016 we find that the equivalent of 25.2 percent were absent in seasonal work in Australia and New Zealand during that year. I would suggest that having around a quarter of all males aged 20-39 years absent from a country in seasonal work is reaching some sort of level that requires a more considered impact analysis. If we took Australia's equivalent male population in 2016 (3,232,461 aged 20-39) and removed a quarter of it for six or more months each year, there would be over 800,000 younger working age makes overseas -- bigger than the entire Queensland male population in that age group and almost as big as the equivalent population in Victoria. In New Zealand, the estimated male population aged 20-39 years in June 2016 was 630,640, and 25 percent of this -- 157,660 – is almost double the number of New Zealand-born males aged 20-39 years who were living in Australia in 2016 (82,464). These shares are not trivial and I would suggest that when we are analysing the impact of seasonal migration on source country populations we need to make sure we are using appropriate demographic denominators. Tonga is exceptional in terms of its level of participation in the SWP and the RSE, but Vanuatu is beginning to feature significantly in terms of shares of its males aged 20-39 overseas as seasonal workers in Australia and New Zealand (just under 10 percent in 2016/17). Most of the other countries have very small shares of their eligible populations involved in the schemes. Let's not generalise too much when it comes to acceptable saturation points -- people of both sexes aged 20-39 years are the mainstay of the younger workforces in the domestic economies. They are also the people who are engaged in international education and other forms of migration. And, of course, they are the primary group that is carrying the burden of reproduction of the populations in the islands. There needs to be a much closer examination of the impact on the sending country's society and economy of having the equivalent of 25 percent of young adult men absent every year in seasonal work. Hopefully this is research that will get some priority in the recently launched Pacific Labour Scheme.

Nic, PRIF is not really a mechanism for funding infrastructure projects. It's a coordination mechanism for the major donors operating in the infrastructure space in the Pacific (though China is not a member) . And it provides technical assistance/knowledge products. So quite different to AIIB. As far as I'm aware, there's next to no information available about the Asia-Pacific trilateral initiative

Very interesting - thank you for the comment. You don't happen to know the name of the World bank report from the 1990s to which you refer?

Thank you very much Mathew Dorman for this report.