Evaluating Australia’s Syrian response: will DFAT act?

#WhatDoesItTake to end the suffering in Syria? In 2011 it was estimated that one million people needed humanitarian aid. In four long years, that number has escalated to 12 million [pdf], many of whom are children [pdf].The third International Humanitarian Pledging Conference for Syria on 31 March 2015 saw donors pledge $3.8 billion in response to a request for $8.4 billion [pdf] (which is less than what the London Olympic Games cost). What is Australia doing to assist? The Office of Development Effectiveness’ 2014 Australia’s Humanitarian Response to the Syria Crisis Evaluation [pdf] has some answers.

Three people conducted the evaluation, spending in total 80 days across eight working weeks, with only eight days in Lebanon and Jordan. The terms of reference, like those for most evaluations, were ambitious for the resources available. Yet the team robustly responded to what was asked of them, particularly given the crisis’ complexity, access issues and resource constraints. They produced a clear, concise and readable report.

Australia has shown the kind of leadership this crisis needs. Through its seat on the Security Council and involvement in the High Level Group on Syria, Australia contributed to increased humanitarian assistance to people in Syria. In particular, UN Security Council Resolution 2165, co-sponsored by Australia, Luxembourg and Jordan, allowed humanitarian access across four borders and was a significant, precedent-setting intervention.

While Australia’s funding volume has been significant – a total of $130.81 million – it is modest in relation to other donors and in decline. (In fact, Australia only gave 27% of its fair share in 2014.) Graphs reinforcing this point (p.27) illustrate a drastic decline in the number of people Australia has helped, both in absolute terms and relative to those who need assistance. As the report puts it all too politely, funding levels have not “kept pace with needs resulting from the escalation in the crisis” (p.14).

In terms of aid quality, UN partners praised Australian aid for its flexibility, being generally non-earmarked. Yet as the report tells us, Australia’s aid to Syria cannot be described as “predictable or dependable” (p.12). Australia’s $131 million to Syria over the last three years has been distributed through 54 one-off allocations to 14 separate partners. Further, there are inconsistent DFAT monitoring and evaluation requirements across partners. And accountability to refugees – the ultimate beneficiaries – is low.

Analytically, the evaluation overlooks two areas. In its assessment of vulnerability targeting (p.15), responses are examined against gendered need, yet only in relation to women and girls. While this may relate to a weakness in DFAT’s approach to gender (such as in its core Gender Strategy – Promoting Opportunities for All [pdf]), the Humanitarian Action Policy [pdf] does refer to men and boys. Surely, history has taught us by now that young men experiencing insecurity and neglect in refugee camps may head back to Syria to take-up arms, or potentially become radicalised, joining fundamentalist militias. Agencies working with refugees and host communities need to include masculinities in their gender analysis, and shape programmes that meet the needs of boys and men. The evaluation missed the opportunity to raise this important issue.

The evaluation also neglected Australia’s approach to refugee resettlement. It is odd that the evaluation extends to diplomatic efforts but not immigration policies. Australia is actually doing comparatively well in accepting Syrian refugees, with a pledge to take 4,064 Syrian refugees by the end of 2015. Oxfam calculates [pdf] this is 103% of Australia’s fair share (p.10).

Six recommendations are put forward: 1) develop a multi-year Syria response strategy; 2) consolidate the partners DFAT engages with, but maintain a balance across UN, NGO and international humanitarian partners; 3) upgrade the region’s humanitarian officer position; 4) explore the creation of a predictable multi-year program fund, as well as the ability to provide supplementary funding; 5) make changes to the Humaniatarian Partnerships Agreement to enable longer term engagement in slow onset or protracted emergencies, limit the number of agencies selected in each funding round, and emphasise established, local capacity as partner selection criteria; and 6) request twice-yearly reports from UN agencies for unearmarked funding.

The evaluation team resisted the temptation to create a long list of recommendations. However, they could have offered one or two creative suggestions to stimulate thinking about thorny issues the evaluation highlighted. The report highlights the tensions Jordon and Lebanon are facing, coping with massive influxes of dispossessed people while still trying to cater for their own poor. The assistance refugees obtain creates resentment amongst citizens in need and instability. While the evaluation finding that these countries fall outside of Australia’s aid priorities is fair, there remains the potential to explore creative ways of assisting these countries, given the need associated with the crisis, and its unprecedented nature. Aid to these countries is DAC-able [pdf], and being outside of Australia’s geographic priorities didn’t stop ODA to Iraq or Afghanistan in the past, so why not Lebanon or Jordan now?

Also, the evaluation made no recommendation regarding its finding that accountability to Syrian refugees is weak. Of course, a single evaluation cannot solve a problem that troubles all humanitarian and development agencies. But again, ideas could have been offered to encourage DFAT to think about how it can leverage improvements in this important area.

Finally, what can this evaluation teach us about evaluations? Like most evaluations, this one contained a comment about resource constraints in the limitations section. Narrowing the TOR would have helped, such as focusing only on UN funded activities and enabling deeper engagement, particularly given the NGOs have heavier monitoring and evaluation requirements under the Humanitarian Partnerships Agreement. A second lesson is to build space for flexibility into evaluation design. The method was developed and agreed upon before entering the field, and the report commented (p.37) that this meant there was scant room for flexibility. Allowing space for the evaluation team to respond to field conditions, such as exploring emerging issues, or altering data collection methods, ensures the team can better understand what is occurring on the ground, leading to more profound learning. A third lesson: process evaluations provide useful information to shape an ongoing response.

This evaluation augurs well for the future of the Office of Development Effectiveness. It is ODE’s first evaluation produced without an external consultant. Yet its overall quality is high: it doesn’t seem to suffer from some of the fundamental flaws we’ve documented in other ODE evaluations. This must also be the fastest evaluation ODE has ever produced: from conception to publication in just a year.

All-in-all, there is a lot to like about this evaluation. The same cannot be said for DFAT’s response to it. All the management response says is that DFAT will “consider” the evaluation’s “suggestions”. While there is a commitment to develop a strategy, there is none to increase predictability, only a promise to “explore” solutions. That’s pretty weak. The last sentence of the DFAT response is ominous: “any decisions regarding future funding will need to be made in line with the Government’s aid policy and budget allocations.” On one level, of course it will. On another, the imminent 20% aid cut makes it more likely that Australia will withdraw from rather than intensify our Syrian engagement.

As the evaluation illustrates, Australia has shown leadership and made an important contribution to what is an unrelenting humanitarian catastrophe. Further diminution of our contribution will undermine any claim Australia has to being a good global citizen and reinforce the painful belief too many hold: the world no longer cares.

Joanna Spratt is a PhD candidate at the Crawford School of Public Policy. Joanna will be speaking on this evaluation at our third annual forum discussing Australian aid evaluations. You can register for the event here

image_pdfDownload PDF

Jo Spratt

Dr Joanna Spratt is a Visiting Fellow at the Development Policy Centre, and a Registered Nurse. She is currently Oxfam New Zealand’s Advocacy and Campaigns Director.

3 Comments

  • Glad you liked this evaluation Joanna. I should clarify that although this evaluation was led by an ODE staff member, the team did also include a consultant, Kate Sutton, whose contribution was invaluable.

    • Thanks for the clarification, Simon. Yes, I did notice that in the summary of the report but had been given contrary information. Great to have Kate’s work acknowledged again.

  • Well said, Joanna. Good evaluations (as you note, undertaken within often difficult resourcing constraints) offer critical opportunities for policy and programming improvement. There is a real opportunity here: yet the urgency in many evaluations – particularly of the summative kind – and the slow nature of bureaucratic adjustment don’t sit easily together. I’m not sure what the answer is. Perhaps engaging evaluators to help drive the implementation process is one option. Having a really senior DFAT person engage courageously with it could also move things along. It is hard to know whether the fairly standard promise to consider and explore is weak or there is more afoot here. Let’s hope it’s the latter.

Leave a Comment