The program
The Basic Education Assistance for Muslim Mindanao – Autonomous Region of Muslim Mindanao (BEAM-ARMM) was a significant education and peace initiative jointly implemented by the Government of the Philippines through the Department of Education (DepEd) and the Australian Government through the Department of Foreign Affairs and Trade (DFAT). It commenced in September 2012 and concluded in June 2017.
The program initially comprised four independent yet interrelated components in partnership with UNICEF, GIZ, BRAC and Cardno. Following an independent review in 2014, BEAM-ARMM shifted towards a programmatic approach with shared outcomes and outputs. The strategic shift to shared end outcomes triggered more effective implementation and management with DFAT, DepEd-ARMM and implementing partners agreeing on four end outcomes:
- improve access by increasing completion rates;
- improve quality of education and learning environment;
- improve employability of Out of School Youth (OSY) completers, and
- improve education governance to support Early Childhood Education, basic education and OSY.
The context
The operating context within Mindanao significantly influenced the program. The ARMM region is a fragile, conflict environment, which lags the rest of the country in terms of educational access and overall learning outcomes. Additionally, the installation of a new national and regional government heightened tensions in the final year of program implementation.
The evaluation
End of program reviews for DFAT programs are usually conducted by independent external evaluators. Given the complexity of BEAM-ARMM, the challenging environment and broad scope of work, DFAT agreed to a quasi-independent approach with the program partners self-evaluating, under the guidance of the international M&E Adviser (who is the author of this post). We employed a mixed-method methodology utilising a range of approaches to demonstrate progress towards end of program outcomes and practices of interest and uptake. The evaluation considered three major questions:
- To what extent has the program achieved stated end of program outcomes?
- How appropriate were BEAM-ARMM’s institutional and governance approaches with DepEd-ARMM and other partners?
- To what extent has the program demonstrated relevance, efficiency and effectiveness through a unified approach to implementation and management? What lessons can be learned?
All evaluations have limitations and BEAM-ARMM was no different. Major limitations were the large number of interventions and scopes of work implemented by partners across the five provinces of ARMM. There was insufficient time to cover all aspects of the program and some partners had limited resources for evaluation. It was also difficult to attribute all changes to interventions implemented by BEAM-ARMM. Finally there were issues with access to and overall quality and availability of data.
Findings
In relation to access and participation, the BEAM–ARMM target for increasing the elementary school completion rate was 13%. The elementary completion rate increased throughout ARMM by an absolute 15.3%, from 24.6% in the 2012–2013 school year to 39.9% in 2015–2016. However, completion rates remain very low with eight of nine divisions being below 55%.
Overall Gross Enrolment Rates increased from 66.8% in 2012–2013 to 88.9% in 2015–2016 and the Net Enrolment Rate increased from 55.5% to 69.6% during the same period, double the anticipated target of 7%.
A significant strategy to enhance learning outcomes was to focus on teacher training and capacity development. BEAM-ARMM trained 12,250 teachers over a three year period. The target for the program was to increase the National Achievement Test scores of year 3 students by 5%. BEAM-ARMM was able to achieve a 22.83% increase in science scores, 6.85% increase in maths scores and 3.8% increase in English scores, for an overall average improvement of 10.5% as illustrated in Figure 1.
Figure 1: End-of-Program Evaluation Results – NAT Grade 3 students
Our findings indicated that the longer students are taught by trained teachers, the better they performed overall. There was also evidence of improvements in female students’ scores as a result of supporting better trained teachers.
BEAM-ARMM also worked to improve outcomes for Out of School Youth (OSY) through providing technical and vocational education and training. The target was to train 11,000 OSY and achieve a 50% employment rate amongst the cohorts. Tracer studies demonstrated that 11,044 students were trained with an overall employment rate of 56%. Importantly there were significant improvements in income for both young men and women (Figure 2).
Figure 2: Changes in income for Out of School Youth (OSY)
In terms of educational governance, the program struggled to establish adequate baselines given the broad nature of governance and the differing interventions being implemented. Nonetheless BEAM-ARMM did manage to implement a number of interventions that strengthened governance and institutional arrangements. Case studies and discussion papers were used as a means to document information-rich approaches that have led to DepEd-ARMM taking up a number of new initiatives.
To assist the program to work in a cohesive and shared manner, a unified approach was applied to M&E, gender and social inclusion, communications, governance and overall planning and reporting. A unified approach is theoretically important but problematic in practice. Partnerships require high degrees of understanding, trust and commitment to shared outcomes to operate effectively. Whilst headway was made in key implementation and management models, the program tended to operate as separate components functioning under a broad programmatic banner.
From a governance perspective, the unified approach enabled partners to come together to share progress, experiences and lessons and to participate in key implementation and management decisions. The project coordination committee meetings were integral for decision-making and promoting a sense of shared understanding and approach. The project coordination committee process was supported by project management committees which met and confirmed component progress and discussed management concerns prior to meeting at the broader forum.
Key lessons
Overall, our evaluation found that BEAM–ARMM has made a direct contribution towards the achievement of its defined outcomes, and was able to achieve most of its intended deliverables despite the difficult operating environment and complicated program management structure. BEAM–ARMM has played a significant role in the development and contribution of a range of initiatives in partnership with DepEd–ARMM and other key stakeholders within ARMM. The key lesson learned revolved around the extended time required to implement various development models given the complex operating context in ARMM. The evaluation also highlighted the need for closer coordination, cooperation and consultation with key government stakeholders and non-state actors within ARMM.
Ty Morrissey was the international M&E Adviser on the BEAM-ARMM program from 2013–2017.
Ty recently presented, along with the BEAM-ARMM Deputy Team Leader Ina Aquino, at the Australian Aid Evaluation Forum held at Crawford School of Public Policy, ANU in August 2017. View their presentation here, or listen to an audio recording of the session here. The full End of Program Review can be accessed here.
