What does it take for data to inform policy? Without effective communication and political support, reports often gather dust, and information-based policy remains in the realm of fantasy for researchers and development programs. This case study, of education development in Indonesia, suggests that data can inform policy – when the right conditions apply.
North Kalimantan is Indonesia’s newest province. Located in the north-east of the island of Borneo, it shares a lengthy border with the Malaysian states of Sabah and Sarawak. The province is around three times the size of the United Kingdom, and is home to 525,000 people, a mix of indigenous Dayaks and coastal Malays – mainly Javanese and Bugis.
The Innovation for Indonesia’s School Children (INOVASI) program is an eight-year partnership between the governments of Australia and Indonesia. INOVASI, which commenced in 2016, is managed by Palladium on behalf of the Australian Department of Foreign Affairs and Trade (DFAT).
INOVASI began its work in the district of Bulungan, North Kalimantan, by collaborating with the government to collect and analyse data on children’s learning outcomes. The results were startling. A government survey found that 47% of grade 4 students were not yet able to read independently. INOVASI’s baseline study found that only 57% of children in grades 1–3 could pass a basic letter and word recognition test; 24% of grade 3 children were not yet able to read words – let alone make sense of a textbook.
Perhaps most telling, 85% of students surveyed by INOVASI said that they “love to read”. The problem appeared not to be a lack of motivation to read, as previously thought. Rather, children didn’t have access to appropriate books: 68% of available books were dry and boring textbooks, typically too difficult for beginning and emerging readers. There were few reading books in schools and classrooms. Teaching was didactic and teacher centred. There was no space or guidance in the national curriculum for the explicit teaching of reading, beyond learning the alphabet in grade 1.
INOVASI is an adaptive program. It has adopted the problem-driven iterative adaptation (PDIA) approach developed by Harvard University’s Center for International Development. Key elements of this approach include local ownership, problem exploration, and iterative piloting of solutions. Formal and informal relationships are important.
The team leveraged the literacy data to get buy-in from local government – and to co-design a solution with stakeholders. The authority of local officials was needed to prioritise books and to change teaching practices. That meant using the shock value of the literacy data to ‘make the problem matter’ to district leaders.
The result was a literacy pilot in 2018. As the pilot was designed with local partners in Bulungan District, the following principles were agreed. The approach should be simple, affordable, and scalable – it should be feasible for local teachers and administrators to manage and implement. It should be small scale – only seven schools were involved in the initial pilot. It should be focused – on literacy only. And it should produce quick results – to generate support and to inform scale-out.
The team worked with the local government and NGOs to introduce a methodology for teaching reading, through training and coaching of early-grade teachers, and to provide quality reading books in schools and communities. It soon became clear that, while the new approach worked for most children, some still found it difficult to learn to read. A special program was developed by local teachers for these struggling readers.
The next step was to evaluate the pilot. The results were encouraging. After one and a half years, the proportion of grade 1–3 children passing the basic literacy test in Bulungan improved from 57% to 94%. This is higher than in East Java, a more developed region, where a similar pilot was conducted and the percentage of children passing the literacy test improved from 79% to 91%. In addition, the time to achieve basic literacy was reduced from three to two years. Prior to the activity, it took three years of schooling for 84% of children to pass the basic literacy test; 87% of children in the pilot program passed the same test in grade 2 – after two years of schooling. And struggling learners also improved their literacy skills, as a result of the differentiated learning approach.
These results were packaged and communicated to decision-makers in a simple format. Opportunities were created for district policy-makers, including the Bupati (district head), to share this success in the media and with national stakeholders. This increased the understanding, sense of ownership, and confidence in the actions taken by the district.
The findings and outcomes of this small pilot were leveraged to get buy-in and to inform plans for scale-out and scale-up. In 2019, Bulungan District funded a scale-out of the program to all early-grade teachers (146 schools, 506 teachers). Funding was also provided for children’s reading books for the first time. In 2020, the district funded an expansion of cluster-based training to all upper-grade primary teachers (146 schools, 488 teachers). The experience of teacher training and of coaching struggling readers informed district programs to support learning during the COVID-19 pandemic. The success in Bulungan also prompted scale-out in neighbouring districts, and adoption of the literacy training modules in the local university and provincial quality assurance body.
Drawing on the results of the pilot and scale-out in North Kalimantan, INOVASI is working with partners to inform national policy on COVID-19 response and recovery, and to leverage findings to highlight the need to address learning loss at sub-national and national levels. The experience in this small and remote district is now informing national government policy, contributing to reform of the centralised book approval process, diagnostic assessment, and, potentially, the new prototype national curriculum.
Disclosure
Both authors are part of the INOVASI program, funded by Australia’s Department of Foreign Affairs and Trade.
Super interesting work and findings, thanks for sharing, Mark and Handoko.
I found particularly interesting the comments in the report around not evaluating the pilot using RCTs, rather saving them for a post-pilot study with RISE (I expected this blog to be about pilot RCTs, so went looking through the report). I very much hope they still go ahead, indeed as pilots, as they are crucial to understand whether the impacts are real and whether it makes sense to scale.
More generally, it will be great—and much more efficient and better practice—if DFAT and in particular their contractors build these in much more at the actual pilot and policy experimentation stage than currently is standard.
Anyhow, thanks again for this super interesting piece and good luck with next steps.