The recent Office of Development Effectiveness (ODE) evaluation of the contribution of Australia Awards scholarships to women’s leadership is in many ways a good news story.
The evaluation claims (despite the difficulties of attribution) that scholarships contribute strongly to women’s professional advancement. It argues this by demonstrating that alumnae supervise more staff than before their scholarships, mentor others, and pass on skills.
Yet, “[t]he proportions of women who reported positive outcomes for a range of indicators of professional advancement and leadership were consistently slightly lower than those for men” (p. 2). In other words, while beneficial for both women and men participants, scholarships are no magic bullet for shattering the glass ceiling, and barriers like family commitments, discrimination and culture persist.
Unfortunately, this evaluation, while overall providing a positive picture of the benefits of scholarships for both women and men, faces some barriers of its own.
The main thing that holds it back is the use of data from weak tracer studies of scholarship recipients to measure self-reported changes in ‘leadership’. The evaluation itself notes the weaknesses of these tracer studies, and recommends that they be improved. It is a shame that a more comprehensive survey of graduates was not undertaken to compensate for this. The qualitative interviews suggest there is significantly more scope for rich data collection on this topic.
The tracer studies, as interpreted by this evaluation, allow measurement of leadership in three main ways: whether the individual was promoted on return from their scholarship; whether they had increased responsibilities; and whether they were transferring skills to others through formal or informal channels.
This is leadership narrowly defined—it does little to tell us about women’s true influence or power (recent ODI research shows this does not necessarily go hand and hand with increasing access to higher positions), and it doesn’t venture beyond career development markers to look at women’s leadership in a broader, contextual sense.
The limits of the tracer study data also lead to incongruities in some areas of the report. For example, in summarising the qualitative interviews in the executive summary, the report states:
“The experience of studying overseas is particularly transformational for early-career women, or women from countries or provincial settings where women are under-represented in leadership, have very limited opportunities for tertiary education, and experience restrictive social and cultural norms.” (p. 1)
Yet, when discussing how to target scholarships, the report goes on to argue that “[l]eadership exhibited to date should be a heavily weighted selection criterion” (p.30), something that could exclude those women who are early-career, or who have had opportunities for leadership restricted by context. Better, richer data would have provided a more convincing argument about where scholarships are best targeted.
This leads on to the prickly question of privilege. One frequent criticism of aid-funded scholarships is that they go to those who have already had opportunities. On recent research on Australia Awards alumni in Africa, Joel Negin wrote:
“…hearing their stories, it was clear that many of the individuals were already quite accomplished before the scholarship and it is perhaps not surprising that they succeeded subsequent to their time in Australia. When asked, it was difficult for many of the individuals to attribute their achievements to their time in Australia.”
This is something that the evaluation does not delve into, largely it seems due to the scarcity of data, or perhaps because it didn’t want to go there. In making recommendations on how the Australia Awards scholarships could better improve women’s leadership, surely having some deeper understanding of which women benefit the most, and go on to be most influential, is important. As Negin points out, the lack of a counterfactual for evaluating scholarships makes this difficult, yet more questions could have provided more answers, particularly considering the evaluation’s frequent nods to the current resource-constrained aid environment.
The tracer studies did serve some purpose though. For example, they provided sectoral data, which led the evaluation to recommend that fewer scholarships be targeted at the public sector. This seems a fair recommendation given that women in the private sector and civil society appear to have far greater opportunities for advancement after their Australian studies. Yet DFAT only ‘in part’ agrees with this recommendation to untether more of the scholarships from the public sector.
But the tracer data in this report tells us nothing about whether particular areas of study do better at improving women’s leadership outcomes, whether women feel more empowered to participate in political or public life after completing scholarships, whether they take on more leadership roles outside of the workplace (such as in community life), whether they mentor other women, or so on. Or, in the case of the public sector, whether they lead in ways that might not be captured by data that only looks at short- to medium-term steps up a career ladder that may be heavily bureaucratized.
Despite being hamstrung by limited data, the report makes some strong recommendations– for example, that “in all countries where there are 10 or more scholarships awarded annually, at least 50 per cent should be awarded to women. If it is not possible to identify sufficient suitable women to meet this requirement, the number of men awarded scholarships should be reduced until a gender balance is achieved.” DFAT only partly agrees, arguing that it will only set a target rather than a quota.
A particular highlight in the evaluation was the interviews with male alumni of the Australia Awards, and on how their attitudes on women in leadership, and on gender roles in general, had changed from their participation in the program.
Hopefully the ODE evaluation will provide the impetus for better tracer studies of scholarship graduates, both female and male. It is a shame that the evaluation didn’t make more explicit recommendations on how these studies should be strengthened and broadened.
Ashlee Betteridge is a Research Officer at the Development Policy Centre. The scholarship evaluation she discusses in this post will be one of two Australian aid performance and evaluation reports discussed at the Australian Aid Evaluation Forum held at ANU on 20 April.
Thanks for your critique of the recent ODE evaluation on the Australia Awards’ contribution to women’s leadership. As one of the authors of that evaluation, I read what you had to say with great interest, and acknowledge and agree with some of the limitations you note with this evaluation. That stated, I did want to point out that one of the biggest criticisms you raise– the issue of scholarships going to the elite, rather that fostering new leaders–is a bigger issue about the purpose of the Australia Awards within the Australian government’s aid architecture.
Specifically, scholarships have, and continue to be as much (if not more) about public diplomacy as they are about strengthening development outcomes within partner countries. The goal of the awards, according to a 2014 policy statement is to ensure, “Partner countries achieve their development goals, and have positive relationships with Australia, enhanced by the contribution of men and women with Australian qualifications, experience and networks, thereby contributing to security and economic growth in our region.” In essence the awards are as much, if not more about Australia’s interests, as they are about the interest of the partner countries these awards target. This presents some challenges from a women’s leadership perspective, and particularly in terms of whom receives such awards, and why awards are targeted the way they are. As we found in the evaluation, awards go in the main to those within the public service, with prior overseas study experience, high levels of English and a prerequisite tertiary qualification. If the aim was to truly support development outcomes then the award programme would be structured entirely different, and might look more like what the Ford Foundation’s award programme looked like. Unfortunately, given the leverage these awards often offer Australia with partner governments, that is unlikely to happen.
As we note in the report, Australia Awards are one of the few forms of development assistance that goes to individuals, under the premise that investing in these key individuals has “trickle down” effects on society. This a questionable, and possibly flawed hypothesis. Our case studies do provide examples of women (and men) returning to their countries and becoming champions for change on a broader level, but often this is impeded by structural and institutional barriers that can demoralise and challenge such agency in the medium to long term. For that reason our evaluation does make a strong argument for post-award support to strengthening leadership outcomes that benefit broader constituencies, whether it be within or outside the formal workplace (noting that often it is often outside the formal workplace that these women are doing incredible stuff as I observed from my small case study into the Women in Leadership group in Africa). I believe Joel Negin and his team have come to a similar conclusion from their own study.
Undoubtedly more can and should be done with the tracer studies in terms of the questions they ask. Most tracer studies do now ask about leadership outside the workplace setting. Unfortunately, our evaluation team faced significant challenges in obtaining raw tracer study data from various DFAT posts, and working to consolidate and analyse it under one master set. This precluded looking at this dimension of leadership from the tracer study data itself, but hopefully with the move to a global tracing approach, this can be done in the future. There are significant issues though with looking tracer study alone as they are based on self-report, often have low response rates, and do seem to capture the voices of metropolitan elites more than anything else.
Fundamentally though, questions might and should be asked about whether Australia Awards are the most appropriate way to support and strengthen women’s leadership outcomes. As you note, they do have an important role to play in shaping the “hearts and minds” of the recipients (men and women) and influencing gender norms. But this is on a very small scale, and ultimately is more about soft power and diplomacy than strengthening development outcomes for women and men in partner countries. From a value for money perspective, I might argue that if the Australian government is serious about strengthening women’s leadership, it should go about doing so through vehicles other than the Australia Awards.