The recent Office of Development Effectiveness (ODE) evaluation of the contribution of Australia Awards scholarships to women’s leadership is in many ways a good news story.

The evaluation claims (despite the difficulties of attribution) that scholarships contribute strongly to women’s professional advancement. It argues this by demonstrating that alumnae supervise more staff than before their scholarships, mentor others, and pass on skills.

Yet, “[t]he proportions of women who reported positive outcomes for a range of indicators of professional advancement and leadership were consistently slightly lower than those for men” (p. 2). In other words, while beneficial for both women and men participants, scholarships are no magic bullet for shattering the glass ceiling, and barriers like family commitments, discrimination and culture persist.

Unfortunately, this evaluation, while overall providing a positive picture of the benefits of scholarships for both women and men, faces some barriers of its own.

The main thing that holds it back is the use of data from weak tracer studies of scholarship recipients to measure self-reported changes in ‘leadership’. The evaluation itself notes the weaknesses of these tracer studies, and recommends that they be improved. It is a shame that a more comprehensive survey of graduates was not undertaken to compensate for this. The qualitative interviews suggest there is significantly more scope for rich data collection on this topic.

The tracer studies, as interpreted by this evaluation, allow measurement of leadership in three main ways: whether the individual was promoted on return from their scholarship; whether they had increased responsibilities; and whether they were transferring skills to others through formal or informal channels.

This is leadership narrowly defined—it does little to tell us about women’s true influence or power (recent ODI research shows this does not necessarily go hand and hand with increasing access to higher positions), and it doesn’t venture beyond career development markers to look at women’s leadership in a broader, contextual sense.

The limits of the tracer study data also lead to incongruities in some areas of the report. For example, in summarising the qualitative interviews in the executive summary, the report states:

“The experience of studying overseas is particularly transformational for early-career women, or women from countries or provincial settings where women are under-represented in leadership, have very limited opportunities for tertiary education, and experience restrictive social and cultural norms.” (p. 1)

Yet, when discussing how to target scholarships, the report goes on to argue that “[l]eadership exhibited to date should be a heavily weighted selection criterion” (p.30), something that could exclude those women who are early-career, or who have had opportunities for leadership restricted by context. Better, richer data would have provided a more convincing argument about where scholarships are best targeted.

This leads on to the prickly question of privilege. One frequent criticism of aid-funded scholarships is that they go to those who have already had opportunities. On recent research on Australia Awards alumni in Africa, Joel Negin wrote:

“…hearing their stories, it was clear that many of the individuals were already quite accomplished before the scholarship and it is perhaps not surprising that they succeeded subsequent to their time in Australia. When asked, it was difficult for many of the individuals to attribute their achievements to their time in Australia.”

This is something that the evaluation does not delve into, largely it seems due to the scarcity of data, or perhaps because it didn’t want to go there. In making recommendations on how the Australia Awards scholarships could better improve women’s leadership, surely having some deeper understanding of which women benefit the most, and go on to be most influential, is important. As Negin points out, the lack of a counterfactual for evaluating scholarships makes this difficult, yet more questions could have provided more answers, particularly considering the evaluation’s frequent nods to the current resource-constrained aid environment.

The tracer studies did serve some purpose though. For example, they provided sectoral data, which led the evaluation to recommend that fewer scholarships be targeted at the public sector. This seems a fair recommendation given that women in the private sector and civil society appear to have far greater opportunities for advancement after their Australian studies. Yet DFAT only ‘in part’ agrees with this recommendation to untether more of the scholarships from the public sector.

But the tracer data in this report tells us nothing about whether particular areas of study do better at improving women’s leadership outcomes, whether women feel more empowered to participate in political or public life after completing scholarships, whether they take on more leadership roles outside of the workplace (such as in community life), whether they mentor other women, or so on. Or, in the case of the public sector, whether they lead in ways that might not be captured by data that only looks at short- to medium-term steps up a career ladder that may be heavily bureaucratized.

Despite being hamstrung by limited data, the report makes some strong recommendations– for example, that “in all countries where there are 10 or more scholarships awarded annually, at least 50 per cent should be awarded to women. If it is not possible to identify sufficient suitable women to meet this requirement, the number of men awarded scholarships should be reduced until a gender balance is achieved.” DFAT only partly agrees, arguing that it will only set a target rather than a quota.

A particular highlight in the evaluation was the interviews with male alumni of the Australia Awards, and on how their attitudes on women in leadership, and on gender roles in general, had changed from their participation in the program.

Hopefully the ODE evaluation will provide the impetus for better tracer studies of scholarship graduates, both female and male. It is a shame that the evaluation didn’t make more explicit recommendations on how these studies should be strengthened and broadened.

Ashlee Betteridge is a Research Officer at the Development Policy Centre. The scholarship evaluation she discusses in this post will be one of two Australian aid performance and evaluation reports discussed at the Australian Aid Evaluation Forum held at ANU on 20 April.