Comments

Hi Terence, I'm in agreement for the most part - RCTs have an important place. I'm also of the view that RCTs can be highly unethical - Elieen has given a good example, but there are many others from rural development that could be used. Data collection in RCTs also commonly involves taking the time of control groups that have received nothing from the development activity. Leaving that aside, RCTs generally do not control for at least one important factor - the attention and stimulation to recipients that goes with a development activity. This factor may be as important or more important than the technology/support on offer. You mentioned that RCTs can't always pinpoint what caused the impact and this is an example of that. You also mentioned the need for good data. Socio-economic surveys are commonly used in RCTs - they are generally very crude instruments because they rely on complex, often confidential, information from recipients that very few of us could provide accurately. So still worth discussing maybe...

Hi Eileen, You make it sound like the experiment in question denied preschoolers in question access to 'high quality' preschools. As best I can tell from skimming the methods what actually happened is that 50% of the students in the sample were given 'high quality' preschooling that they would have otherwise failed to receive. Then, from the abstract, the main finding was: "The findings show that quality of preschool education had no significant effect on children's overall educational attainment." (Although there may be heterogeneous effects.) No one was denied high quality preschooling in this experiment. What's more the intervention was discovered not to help. (Plus or minus a few caveats). I fail to see how this was unethical. "I was at an aid and development seminar a couple of years ago where many people undertaking evaluations mistakenly believed they were doing RCTs when they were obviously not. It’s a term that is being used inappropriately because bureaucrats believe it sounds good." This is disappointing to hear, I appreciate you raising the point, and I share your concern. "Doing a systematic evaluation and collecting rigorous data does not require use of this method." I agree. I thought I said as much in the blog post, sorry if I was unclear. "This is especially true in education when it is almost impossible to distinguish between the many variables that can impact on student learning and performance." But, within constraints, that is what RCTs do: they control effectively for the influence of other variables. I agree there are issues (listed in my post), but it's worth giving RCTs credit for what they can do well. Thank you for your comment. Terence

Thanks Terence for looking at this topical issue. Here is an article and book I found useful on the important and related area of monitoring: http://blogs.worldbank.org/impactevaluations/if-i-can-t-do-impact-evaluation-what-should-i-do-review-gugerty-and-karlan-s-goldilocks-challenge

I have two comments to make about your post. 1. Your argument that RCTs are not unethical denies the reality of projects like the one reported on in this article. Deliberately providing "high quality" education to young children and denying others that same education purely for experimental purposes is unethical. "Children aged three to four years were assigned randomly to high-quality preschools that were created for the experiment or to existing petites écoles (that is, low-quality preschools)". https://www.sciencedirect.com/science/article/abs/pii/S0272775717302637 2. I was at an aid and development seminar a couple of years ago where many people undertaking evaluations mistakenly believed they were doing RCTs when they were obviously not. It's a term that is being used inappropriately because bureaucrats believe it sounds good. Doing a systematic evaluation and collecting rigorous data does not require use of this method. This is especially true in education when it is almost impossible to distinguish between the many variables that can impact on student learning and performance.

If the blog interested you, this article in the NY Times will also be of interest: https://www.nytimes.com/2018/08/06/upshot/employer-wellness-programs-randomized-trials.html

This is an interesting analysis of the past year's SWP figures, Stephen. One caution though -- your comment about Tonga's participation in the SWP and RSE reaching "an impressive 13% of the eligible sending population (20-45 years)" needs some qualification. Robert Ingram and I have shown in research completed for LMAP that the "eligible sending population" needs to be defined much more precisely than just the total in a particular age group in the sending country. Given that recruitment for both the RSE and SWP is strongly biased towards younger adult men it is much better to express the share of the "eligible sending population" in terms of males aged 20-39 years. The Australian Department of Home Affairs (ADHA) does not publish data on SWP participants by age, but Immigration New Zealand (INZ) has made such data available up until quite recently for RSE workers in broad age groups. If we assume the SWP workers heading for Australia from Tonga are from the same basic demographic groups as those heading for work under the RSE scheme, and there is no reason to assume they would be very different given they are doing similar types of seasonal work, then we can establish a little more precisely just what the the impact of seasonal work in the two schemes is having on the main age group affected. Data from the ADHA and INZ show that in the 2016/17 financial year 89 percent of the Tongans participating in the SWP and the RSE were males (4,025 out of 4,512 who were employed in the two schemes). If we assume the 2,394 Tongan male SWP workers had the same age distribution as the 1,621 Tongan male RSE workers then 3,276 of the 4,025 males would have been aged 20-39 years -- 81.4 percent. When we put these 3,276 males aged 20-39 years alongside the 13,000 Tongan males aged 20-39 years at the time of Tonga's census in 2016 we find that the equivalent of 25.2 percent were absent in seasonal work in Australia and New Zealand during that year. I would suggest that having around a quarter of all males aged 20-39 years absent from a country in seasonal work is reaching some sort of level that requires a more considered impact analysis. If we took Australia's equivalent male population in 2016 (3,232,461 aged 20-39) and removed a quarter of it for six or more months each year, there would be over 800,000 younger working age makes overseas -- bigger than the entire Queensland male population in that age group and almost as big as the equivalent population in Victoria. In New Zealand, the estimated male population aged 20-39 years in June 2016 was 630,640, and 25 percent of this -- 157,660 – is almost double the number of New Zealand-born males aged 20-39 years who were living in Australia in 2016 (82,464). These shares are not trivial and I would suggest that when we are analysing the impact of seasonal migration on source country populations we need to make sure we are using appropriate demographic denominators. Tonga is exceptional in terms of its level of participation in the SWP and the RSE, but Vanuatu is beginning to feature significantly in terms of shares of its males aged 20-39 overseas as seasonal workers in Australia and New Zealand (just under 10 percent in 2016/17). Most of the other countries have very small shares of their eligible populations involved in the schemes. Let's not generalise too much when it comes to acceptable saturation points -- people of both sexes aged 20-39 years are the mainstay of the younger workforces in the domestic economies. They are also the people who are engaged in international education and other forms of migration. And, of course, they are the primary group that is carrying the burden of reproduction of the populations in the islands. There needs to be a much closer examination of the impact on the sending country's society and economy of having the equivalent of 25 percent of young adult men absent every year in seasonal work. Hopefully this is research that will get some priority in the recently launched Pacific Labour Scheme.

Nic, PRIF is not really a mechanism for funding infrastructure projects. It's a coordination mechanism for the major donors operating in the infrastructure space in the Pacific (though China is not a member) . And it provides technical assistance/knowledge products. So quite different to AIIB. As far as I'm aware, there's next to no information available about the Asia-Pacific trilateral initiative

Very interesting - thank you for the comment. You don't happen to know the name of the World bank report from the 1990s to which you refer?

Thank you very much Mathew Dorman for this report.

I get wary of debate about 'infrastructure' that suggests everybody knows what is meant when they hear the word. I am continuously attending meetings in Dili where people speak about infrastructure as if it is something that can be installed once, and then forgotten. Many people fail to recognize that infrastructure covers a number of sectors - transport and communications, health, education, small business and commerce etc. etc. and that it includes both hardware and software. The term 'soft infrastructure' I find ridiculous as it seems to ignore the fact that all infrastructure, even road building, requires software, i.e. the personnel network of knowledge and skills to make it work, keep it maintained and not fall into disrepair. I find the most neglected component of infrastructure is knowledge infrastructure, i.e. the cables and wires, plus the personnel with knowledge and skills which will enable a country to have a functioning, post office, library and internet system to get books and learning materials around to all educational institutions for teaching and learning of the knowledge attitudes and skills needed to develop the country. Such a system, if it existed, would inevitable help the small and medium enterprise sector as well (as happens in Fiji). The UN's World Summit on the Information Society had a great deal of sensible recommendations to make on this but they are rarely discussed. Timor-Leste came to independence at a time when it was widely believed the private sector (in reality foreign telcos) could take care of all that, it didn't happen. Timor-Leste is at the bottom of the WEF's computer readiness index and, despite offers by Australia's AARNET to assist universities, little has been done to ensure students and academics have access to good library and computing facilities.

Thank you Mathew for this post. Unfortunately, to date, much of the commentary focussed on the China Belt & Road Initiative and the China AIIB concessional loans has tended to emphasise a geopolitical perspective. Instead I would prefer that the China concessional loans initiative be evaluated in development terms because I think that the initiative to fund infrastructure has a basis in the Chinese experience of development and reflects evolving Chinese theories about poverty alleviation and how sustainable economic growth and development is achieved. The Chinese emphasis on infrastructure investment concentrating on roads seems to have crystallised following the ‘Global Learning Process and Conference on Scaling Up Poverty Reduction’ held in Shanghai in 2004. The Roads Improvement for Poverty Alleviation (RIPA) program was launched with World Bank support in selected provinces in the mid-1990s. RIPA focuses on linking those rural villages and townships which do not currently have basic all-weather access to the existing road networks. In the case of Henan Province, quantitative analysis resulting from an ex-post evaluation of the RIPA components showed that remote settlements that had been engaged in subsistence farming prior to the program had made markedly better progress in economic development, social development, and poverty alleviation than comparable populations in control areas. http://web.worldbank.org/archive/website00819C/WEB/PDF/CASE_S-4.PDF Your reference to the paper by John Gibson and Scott Rozelle (‘Poverty and Access to Roads in Papua New Guinea’) is very appropriate. A scheme for measuring the poverty alleviation effect of roads in PNG was included in a report to the World Bank in the 1990s. This scheme was based on using Gibson’s earlier work on household income and consumption, modified and applied by a questionnaire to be administered by the PNG National Volunteer Service in selected areas where there had been significant road development. In my view this is still research waiting to be done in PNG. Doing so would be especially appropriate in the context of the PNG LNG Project and the Papua LNG Project where significant road infrastructure is being created under the tax credit scheme. Vailala

Interesting read Jesse. In 2011 when Brisbane was flooded on 11-13 January, I lost most of my possessions in my apartment in East Brisbane. I managed to drive my car out and save important documents and personal items. The Queensland Government provided a quick response by making grants of $900 per person to assist in short term recovery. It still took 3 months to get things back to normal and I look back on this event as an example of the resilience of Queenslanders. It was the same in Tonga, during TC Gita. I was evacuated to the Scenic Hotel near the airport. The day after I was clearing downed electricity lines to allow vehicles taking AusAid supplies delivered by Australian C17 to reach Nuku'alofa. Residents of nearby houses destroyed by the cyclone were scavenging materials to rebuild while I was clearing the road. They didn't wait for handouts.