Comments

Due to the presumption of innocence, it is the other way around. Why did the accusers not produce a shred of evidence since 2012 that I had falsified anything? The judge in the National Court was very clear about this: "In spite of this overwhelming evidence (presented by Dr. Schram) Mr. Saulep continues to dispute the authenticity of the applicant's doctorate degree. I find this ridiculous and difficult to fathom especially when neither he or the police are in receipt of evidence from the European University Institute in Florence Italy, confirming their allegations and suspicions". http://albertschram.blogspot.com/2018/05/wrongful-dismissal-and-malicious.html#more

Thank you Mathew for your comment. The relevant World Bank document is the ‘Papua New Guinea - Road Maintenance and Rehabilitation Project’ (http://documents.worldbank.org/curated/en/262651468775145760/Papua-New-Guinea-Road-Maintenance-and-Rehabilitation-Project). This report includes some details of the Social Impact Monitoring Plan and makes brief mention (in a somewhat incoherent way) of the proposed research on pp 97 and 99. ‘The effect of the sub-projects on the population served by the roads involved will be assessed periodically through social studies to be undertaken under the project. The assessments will be designed to detect changes in: household incomes; cash crop/commodity production and sale; prices of commodities imported from other regions; changes in school attendance/reduced drop-out rates; health benefits from improved access to aid posts, clinics and hospitals; etc. The Project Implementation Plan (PIP) contains the terms of reference of the study for monitoring the socio-economic impacts of road and bridge improvements under the RMRP. The PNGVs will collect basic social mapping data and the SIA component will supplement this with surveys designed to villager satisfaction with the implementation of the RMRP sub-projects and gather information about the poverty alleviation effects and any adverse impacts. Baseline studies will be conducted before implementation, at mid-point and at the end of the RMRP.’ The relevant TOR for the PIP contained precise details as to the content of the survey research and methodology and the use of PNG Volunteers for data gathering. I have not been able to find these documents in the World Bank archive. Very briefly the research design was prepared as a response to a casual remark by a World Bank staffer that while the WB spent hundred of millions of dollars on road improvement schemes it had no idea as to any measurable effects this expenditure might have on poverty reduction – especially in countries where the informal economy and subsistence agriculture were demographically dominant. The research design drew on John Gibson’s valuable data on PNG household consumption patterns and the Agogo Mawuli and Ogis Sanida NRI paper ‘Landowners' Mineral Rent-quest and Use in Papua New Guinea’ which provides a useful guide as to how rural PNG households spend cash income. Also of relevance were studies made of artisanal fisheries and small business development along the Morobe coast. The questionnaire (about 26 items) was designed to elicit data not only on domestic household consumption but also data relevant to household welfare and household capital formation and the linkage of these to the provision of low quality rural roads. The WB RMRP project although initiated in the 1990s was not implemented until around 2004. The WB appeared to have farmed out the SIA components to Ausaid for review following which the SIA parts were assigned to SMEC/Finnroad. The involvement of PNG Volunteers appears to have ended and the focus of the research/questionnaire greatly altered. Much of the emphasis on non-monetised subsistence production (the consumption basket) and village household capital formation appears to have been lost in translation. The final analysis, done by Finnroad in 2007/8 paid little attention to these matters, concentrating instead on conventional measures of cash income and transport accessibility. The WB project assessment document describes the SIA findings here (http://documents.worldbank.org/curated/en/845161468286308542/pdf/NonAsciiFileName0.pdf) on pp 40 – 42. A brief Finnroad account can be found here – https://trid.trb.org/view/877540. So, yes I do think that the research is yet to be done, at least in PNG. Why am I making a point of this? Because I saw the research as an attempt to step beyond the usual framework of road infrastructure investment evaluation in terms of the EIRR. This is an issue of concern to Chinese development economists, especially Wen Tiejun, and, in my opinion underlies the anti-hegemonic orientation of the AIIB. Vailala

Thanks Garth, that's an excellent link. I hadn't seen it previously. Very useful. Thanks again. Terence

Thanks Chris, That's a great comment. I agree with all of your points, but would note: At times you will have to collect your own data. This is unavoidable, particularly if you want quality data. That said, I think the cost is a small price to pay for learning. I agree, RCTs can help with learning how a project works too. I'd just add that there may be limits and that a great alternative would be to combine RCTs with process tracing as Oxfam Great Britain does. Quant and qual methods can be friends 🙂 Thanks again. Terence

Hi Ben, Thanks for your comment. I think the wasting the control group's time is about as good a ethical issue as can possibly be raised. (Absent some non-RCT-specific ethical lapse). The situation is still no worse than the normal evaluation of a project that did not work, though. What's more, people fill out census forms, HIES and DHS surveys for no benefit. Not quite as intrusive, of course, but it's not as if RCTs are new in wasting people's time for data. Moreover, if the treatment is effective, the compensation could be expanding it to the control group. Re your attention stimulation point. I agree. It will be a problem with all good evaluations though. Moreover if the attention factor proves to be critical then perhaps we've learnt something useful that should be built into aid more generally. "Socio-economic surveys are commonly used in RCTs – they are generally very crude instruments because they rely on complex, often confidential, information from recipients that very few of us could provide accurately." It would be interesting to know what % of RCTs do actually rely on poor data (I don't think surveys inevitably generate such data, but know this is a real possibility). Once again, this isn't specific to RCTs, though. No evaluation will ever be better than the quality of its data. Great points. Thank you for engaging. Terence

Nicely balanced blog Terence. I just wanted to clarify a couple of the 'limitations' you mentioned of RCTs. Cost of RCTs - RCTs can be done for free. This is can occur when data is already going to be collected. For example, governments regularly collect data about things like who has paid their taxes. Here is example from Latin America of an RCT that uses government administrative data: https://www.povertyactionlab.org/evaluation/role-vat-tax-enforcement Programs RCTs can be used on - RCTs can be used to evaluate both micro and macro ('universal') level policy changes (even large scale infrastructure). This can be achieved using a nudge to encourage people to take up a new service or product. Here is an example from Africa of an RCT that evaluates the impact of a large infrastructure project: https://www.povertyactionlab.org/evaluation/household-water-connections-tangier-morocco How programs work - RCTs can be used to determine why/how a program works. This can be achieved by having multiple treatment groups whereby some people are provided with a sub-component of the program and other people receive the whole program. If there is no difference between the impact of the sub-component and the entire program this indicates that it is just the sub-component that is making a difference. Here is an example of an RCT in South Asia that used this approach: https://www.povertyactionlab.org/evaluation/improving-immunization-rates-through-regular-camps-and-incentives-india

Hi Terence, I'm in agreement for the most part - RCTs have an important place. I'm also of the view that RCTs can be highly unethical - Elieen has given a good example, but there are many others from rural development that could be used. Data collection in RCTs also commonly involves taking the time of control groups that have received nothing from the development activity. Leaving that aside, RCTs generally do not control for at least one important factor - the attention and stimulation to recipients that goes with a development activity. This factor may be as important or more important than the technology/support on offer. You mentioned that RCTs can't always pinpoint what caused the impact and this is an example of that. You also mentioned the need for good data. Socio-economic surveys are commonly used in RCTs - they are generally very crude instruments because they rely on complex, often confidential, information from recipients that very few of us could provide accurately. So still worth discussing maybe...

Hi Eileen, You make it sound like the experiment in question denied preschoolers in question access to 'high quality' preschools. As best I can tell from skimming the methods what actually happened is that 50% of the students in the sample were given 'high quality' preschooling that they would have otherwise failed to receive. Then, from the abstract, the main finding was: "The findings show that quality of preschool education had no significant effect on children's overall educational attainment." (Although there may be heterogeneous effects.) No one was denied high quality preschooling in this experiment. What's more the intervention was discovered not to help. (Plus or minus a few caveats). I fail to see how this was unethical. "I was at an aid and development seminar a couple of years ago where many people undertaking evaluations mistakenly believed they were doing RCTs when they were obviously not. It’s a term that is being used inappropriately because bureaucrats believe it sounds good." This is disappointing to hear, I appreciate you raising the point, and I share your concern. "Doing a systematic evaluation and collecting rigorous data does not require use of this method." I agree. I thought I said as much in the blog post, sorry if I was unclear. "This is especially true in education when it is almost impossible to distinguish between the many variables that can impact on student learning and performance." But, within constraints, that is what RCTs do: they control effectively for the influence of other variables. I agree there are issues (listed in my post), but it's worth giving RCTs credit for what they can do well. Thank you for your comment. Terence

Thanks Terence for looking at this topical issue. Here is an article and book I found useful on the important and related area of monitoring: http://blogs.worldbank.org/impactevaluations/if-i-can-t-do-impact-evaluation-what-should-i-do-review-gugerty-and-karlan-s-goldilocks-challenge

I have two comments to make about your post. 1. Your argument that RCTs are not unethical denies the reality of projects like the one reported on in this article. Deliberately providing "high quality" education to young children and denying others that same education purely for experimental purposes is unethical. "Children aged three to four years were assigned randomly to high-quality preschools that were created for the experiment or to existing petites écoles (that is, low-quality preschools)". https://www.sciencedirect.com/science/article/abs/pii/S0272775717302637 2. I was at an aid and development seminar a couple of years ago where many people undertaking evaluations mistakenly believed they were doing RCTs when they were obviously not. It's a term that is being used inappropriately because bureaucrats believe it sounds good. Doing a systematic evaluation and collecting rigorous data does not require use of this method. This is especially true in education when it is almost impossible to distinguish between the many variables that can impact on student learning and performance.

If the blog interested you, this article in the NY Times will also be of interest: https://www.nytimes.com/2018/08/06/upshot/employer-wellness-programs-randomized-trials.html

This is an interesting analysis of the past year's SWP figures, Stephen. One caution though -- your comment about Tonga's participation in the SWP and RSE reaching "an impressive 13% of the eligible sending population (20-45 years)" needs some qualification. Robert Ingram and I have shown in research completed for LMAP that the "eligible sending population" needs to be defined much more precisely than just the total in a particular age group in the sending country. Given that recruitment for both the RSE and SWP is strongly biased towards younger adult men it is much better to express the share of the "eligible sending population" in terms of males aged 20-39 years. The Australian Department of Home Affairs (ADHA) does not publish data on SWP participants by age, but Immigration New Zealand (INZ) has made such data available up until quite recently for RSE workers in broad age groups. If we assume the SWP workers heading for Australia from Tonga are from the same basic demographic groups as those heading for work under the RSE scheme, and there is no reason to assume they would be very different given they are doing similar types of seasonal work, then we can establish a little more precisely just what the the impact of seasonal work in the two schemes is having on the main age group affected. Data from the ADHA and INZ show that in the 2016/17 financial year 89 percent of the Tongans participating in the SWP and the RSE were males (4,025 out of 4,512 who were employed in the two schemes). If we assume the 2,394 Tongan male SWP workers had the same age distribution as the 1,621 Tongan male RSE workers then 3,276 of the 4,025 males would have been aged 20-39 years -- 81.4 percent. When we put these 3,276 males aged 20-39 years alongside the 13,000 Tongan males aged 20-39 years at the time of Tonga's census in 2016 we find that the equivalent of 25.2 percent were absent in seasonal work in Australia and New Zealand during that year. I would suggest that having around a quarter of all males aged 20-39 years absent from a country in seasonal work is reaching some sort of level that requires a more considered impact analysis. If we took Australia's equivalent male population in 2016 (3,232,461 aged 20-39) and removed a quarter of it for six or more months each year, there would be over 800,000 younger working age makes overseas -- bigger than the entire Queensland male population in that age group and almost as big as the equivalent population in Victoria. In New Zealand, the estimated male population aged 20-39 years in June 2016 was 630,640, and 25 percent of this -- 157,660 – is almost double the number of New Zealand-born males aged 20-39 years who were living in Australia in 2016 (82,464). These shares are not trivial and I would suggest that when we are analysing the impact of seasonal migration on source country populations we need to make sure we are using appropriate demographic denominators. Tonga is exceptional in terms of its level of participation in the SWP and the RSE, but Vanuatu is beginning to feature significantly in terms of shares of its males aged 20-39 overseas as seasonal workers in Australia and New Zealand (just under 10 percent in 2016/17). Most of the other countries have very small shares of their eligible populations involved in the schemes. Let's not generalise too much when it comes to acceptable saturation points -- people of both sexes aged 20-39 years are the mainstay of the younger workforces in the domestic economies. They are also the people who are engaged in international education and other forms of migration. And, of course, they are the primary group that is carrying the burden of reproduction of the populations in the islands. There needs to be a much closer examination of the impact on the sending country's society and economy of having the equivalent of 25 percent of young adult men absent every year in seasonal work. Hopefully this is research that will get some priority in the recently launched Pacific Labour Scheme.