Teacher development evaluations

27 September 2016

In 2011 I critiqued in The Australian AusAID’s plans to focus aid on building schools and management training for principals in Indonesia. I concluded my critique by arguing for more rigorous analysis of research and local evidence of what works in improving education.

Investing in Teachers [pdf], one of three teacher development reports produced recently by ODE/DFAT, is a constructive and positive report that reflects my call for both the rigorous analysis of research and local evidence of what works by using examples from developing countries.

The first two reports in the set of three are literature reviews: Teacher Quality Evidence Review [pdf], published March 2014, and Supporting Teacher Development Literature Review [pdf], published in March 2015.

The third and major report, Investing in Teachers, was published in December 2015. This examines 27 Australian aid programs from 18 countries and considers teacher pre-service and in-service qualifications and professional development. Drawing on field evidence as well as the earlier literature reviews, Investing in Teachers presents recommendations and management’s responses to these.

Five matters arising from these reports warrant further review by ODE/DFAT.

1. Language

Language used in development documents must be unambiguous and simple. English will not be the first language for many readers, and this is something report writers must keep in mind. In an otherwise well written report, there are some words in Investing in Teachers that risk conveying the wrong messages.

‘Deliver’ is one such word. We do not deliver education like a package that is simply handed over. Education and change depend on a two-way relationship, one that continues over an extended time period.

Continuity is something Investing in Teachers does address. Recommendation 2 states, in part: ‘…DFAT education program managers should ensure…sufficient timeframes to realise expected changes—for example, five to 10 years minimum for a major national teacher development program…’ (p.7). The conceptual conflict between deliver and this important recommendation should be rectified.

Another specious word is ‘levers’, as in ‘levers of change’. This cliché suggests that educational change is a mechanical cause-and-effect process. If only it were so simple!

2. The conceptual framework

The conceptual framework was developed in the first report, Teacher Quality Evidence Review. My concern is with the complexity revealed in the framework diagram presented, inconsistently for some reason, in the three reports.

A conceptual framework should be comprehensive and simple. It should be something busy professionals can carry around in heads to act as a guide to their professional practice. This framework cannot do that. It is far too complex and the concepts shown are poorly integrated. Its focus on ‘student outcomes’ is one of several problematic matters as explained below.

3. Students

These reports perpetuate a deficiency in education writing, noted by leading educational policy practitioner, Michael Fullan, that so little progress has been made in the past 30 years in considering students as serious participants in educational change.

John Hattie shows that 50% of the variance of student achievement is from what students bring to the table. Teachers account for 30%, and home, school and peer effects for the remainder.

So why do these reports perpetuate the neglect of the student’s world and focus only on outcomes? This neglect is not acceptable. Students in developing countries are far more at risk [pdf] from factors that affect their outcomes: access, enrolment, transition from one grade to the next, early school leaving, abuse, trafficking, and health. Where external support systems for students are so weak, teachers have a critical role in supporting students or, to borrow Hattie’s metaphor, getting them to the table – and keeping them there.

4. Culture

Given that such a range of cultures, from Afghanistan to Vanuatu, is reflected in Investing in Teachers it is curious that culture has been neglected. Three regional examples of cultural issues demonstrate the need for meticulous attention to culture in educational project design and implementation:

From Papua New Guinea and China there is evidence reviewed here of the need to treat formal teaching as a deeply rooted cultural behaviour capable of improvement rather than unproductively focusing on teachers adopting progressive teaching methods.

From Indonesia, research finds that a decentralization policy had almost no impact on schools because of a failure to understand culture and to focus only on the technical aspects of development.

From Laos, a study [pdf] demonstrates clear conflicts between Lao cultural values and Western values embedded in an AusAID competency-based English language curriculum.

5. The recommendations and management’s response

The otherwise sound recommendations would benefit from some editing:

Recommendation 1: For reasons implied above, I would advocate adding culture to the ‘contextual constraints’ references.

Recommendation 2.i: Clarity demands a simple fix of the muddled use of two ideas with different meanings – ‘student’s learning performance’ and ‘student outcomes’. Do they mean the same thing? There should not be any doubt about the answer in a recommendation.

Recommendation 2.ii: A major reason to provide sufficient timeframes to realise expected changes, as noted in this recommendation, should be stated to allow for unknown cultural and contextual issues to emerge and to be addressed as projects evolve. These complexities are never fully understood at design. A corollary is that designs must permit flexibility.

Recommendation 3: The common call for better monitoring and evaluation here needs to be supplemented by clear strategies for better use of M&E outputs. DFAT demonstrated the need for this at the Forum in their presentation Review of Operational Evaluations completed in 2014.

Finally, brief comments on the responses of management.

The strong commitment expressed by management’s responses is commendable. So too is management’s recognition of the complexity of teaching and its interrelationship with other key factors in teacher quality improvement.

Of concern, however, is this comment:

‘DFAT strongly supports monitoring student learning and furthering the international evidence base…’ (p. 9).

Why not ‘…furthering student learning?’ ‘Furthering the international evidence base’ through testing introduces serious risks. I have canvased these risks in two Devpolicy blogs. Further, recent research shows negative impacts of testing on Indonesian teachers’ work and morale.

Conclusion

A simple Lao proverb is relevant here: One piece of wood doesn’t make a fence.

Why so? A focus on teachers alone would be unproductive. DFAT has recognised teachings’ interrelationships with other key factors. Two factors requiring more attention in teacher development strategies are students and the cultural context in which that development occurs.

Robert Cannon is an educational evaluation adviser to USAID in Indonesia and was an Associate of the Development Policy Centre. This post presents his comments made at the recent ODE aid evaluation forum which discussed the recent evaluation of teacher development produced by the Office of Development Effectiveness (ODE) within the Department of Foreign Affairs and Trade (DFAT). You can access the podcast here and all presentations from the forum here.

Author/s

Robert Cannon

Robert Cannon is a research associate with the Development Policy Centre. He has worked in educational development in university, technical and school education, most recently in Indonesia and Palestine.

Comments

  1. Thanks Bob, for highlighting the ODE/DFAT ‘Investing in Teachers’ report. Readers might be interested in current research within remote Aboriginal and Torres Strait Islander communities, on the CRC’s Remote Education Systems project. Many of the findings of ‘Investing in Teachers’, and your blog comments resonate with our experiences in remote Australia, where the key findings from this five-year research project are:
    1. Teacher quality is not the main issue for remote schools, but ensuring teachers have the right qualities is fundamentally important.
    2. Education should support local aspirations for culture, land, language and identity.
    3. ‘Success’ is defined as parent and community involvement in school.
    4. Imposed quick fix solutions (such as attendance strategies) are most effective where there is community engagement
    5. Community engagement is important, but engagement must have benefit for parents and students in order to be effective.
    6. Schools with more resources tend to get better outcomes.
    7. The premise of ‘remote Indigenous disadvantage’ is not supported by remote community members.
    8. Contrary to popular belief, there are plenty of jobs for school graduates to take in remote communities, but the pathway from school to work is unclear.
    9. Schools with higher proportions of non-teaching staff get better results.
    10. Too little is known about the impact of boarding schools – policies developed in an evidence-base vacuum can be potentially harmful for students and communities, unless there are opportunities for students on return to the community.

    Reply Comment

Leave a comment