Writing of the ‘diabolically difficult mid-term review’ recently on this blog, Gordon Peake asked if law and justice programs are uniquely difficult to evaluate. His blog foreshadowed an event held earlier this month by the Australian Law and Justice Development Community of Practice (COP), discussing learning from reviews of DFAT law and justice programs in Indonesia and Vanuatu. The discussion spoke to why law and justice reviews are so difficult and where their value might reside in spite of this.

There is something intrinsically difficult in determining whether, and in what ways, the often technical, short-term inputs of aid programs – such as technical assistance, workshops, trainings and so on – are contributing to meaningful improvements in people’s lives (in this case, a more just and peaceful society). Pathways to change and processes of attribution are complex and hard to trace. This is a difficult but not unfamiliar challenge to monitoring and evaluation experts.

In addition to this, justice programs are hard to review because they are political at their very core. They are not only surrounded by the kinds of politics that Gordon’s blog mentions – between individuals, institutions and at the level of diplomacy. They are political because justice is fundamentally about power: who has it (and who does not), and how it is regulated and exercised. What is ‘better justice’ that development programs are working towards? A more robust legal framework – or does that impinge on liberties? A better trained and equipped police service – or is this a coercive tool of state control? Greater rights protection for vulnerable groups – or is that foreign disruption of local norms? What constitutes ‘better justice’ is more contested and likely to spark greater debate than say, better health, or better education – although those are not entirely straightforward either.

Determining whether aid programs are contributing to ‘better justice’ is, in part, so challenging because of the diversity of views of what this means, where the priorities lie, how change is best achieved and who the appropriate allies are. Differing views on these questions are rooted in different individual assessments of what ‘better justice’ in fact looks like. And these individual assessments will vary amongst DFAT staff, managing contractors and the consultants they hire, local partners, as well as review teams. That’s a lot of competing ideas of justice that virtually never get talked about, let alone resolved.

The differing views within review teams matter because what ends up in the final report is itself political. Review teams do not sit outside of the politics that permeate the rest of justice programming. So, we see women and national experts often underrepresented in review teams and structures of power can make it more difficult for their views to make it into final reports. A review of reviews would no doubt itself produce a trove of insightful material!

In part stemming from the devilish politics that surround them, mid-term reviews are often seen to produce largely familiar findings that seem to remain unlearned, or at least unimplemented. Hard hitting findings are often toned down to the point of being banal, prompting the question of whether they’re useful at all. The COP discussion suggested that despite these shortcomings – which were viewed as pragmatic necessities – reviews can play important roles that we overlook, although one is left with the question of whether these are sufficient reasons to undertake them.

Reviews serve different interests for different constituencies. Program staff suggest it is the process, more than the final product, that is useful – providing the opportunity to reflect, learn and adapt. While many programs are meant to be adaptive and learning-oriented, in reality few meet these ambitions (as both the Indonesia and Vanuatu reviews note) and staff are rarely given the space for the kinds of reflection required for serious experimentation or adaptation. Reviews were pointed to as a welcome moment for pause. If reviews are to play this function, then it is important that there is in fact space and appetite for the kinds of adaptation that such reflection might prompt.

For DFAT, even if reviews are not widely read outside (or even inside) the department, it was suggested they can be useful for staff at Post to better understand the programs they oversee, their achievements and weaknesses and how to represent them internally. While an important function, this seems a costly way to get DFAT staff up to speed with their portfolios. A better understanding of how reviews might be useful within DFAT, beyond meeting simply an accountability function, is needed.

Finally, participants noted that local partners and stakeholders can see reviews as an opportunity to make their voices heard (to the program, the donor and their own governments). This underlines the importance of inclusive consultation and participatory methods, as well as capturing and giving voice to the views and concerns raised by partners. Yet the audience of local partners is often an afterthought in many reviews – evidenced by how few are translated into local languages. If this is indeed where their value lies, then they likely need to be reimagined in ways that better serve local audiences. It is striking that we struggled to get any representatives of the governments or justice sectors in partner countries to join the COP event.

As a researcher I am predisposed to like reviews because I’m interested in capturing learning about the politically fraught issue of how more just societies can be supported. But that’s probably too ambitious an ask for reviews, especially in a political climate where the importance of learning appears so undervalued (witness the end of the Office of Development Effectiveness and the Federal Government’s underfunding of the National Audit Office). If we can push beyond the idea of reviews as simply accountability exercises that check whether an implementer is meeting contractual obligations and adhering to the design, then we might find that their value lies in serving other functions. It is these functions that might make such diabolically difficult undertakings worthwhile.