Paul Theroux once said that the short story is the most diabolically difficult literary form to master. I daresay he never tried writing a mid-term review of a justice or policing program.
In the early part of the last decade, I worked as a jobbing consultant and a few of these assignments involved conducting such reviews. Some of my oeuvre is available here, here, and here.
I felt reasonably well qualified to be involved in these efforts. I’d published academic articles and co-edited a book on the subject and worked for three years in a policing program in Timor-Leste.
And it was certainly a line of work that generated many unique experiences. I met astonishing people doing memorable things. I acquired many a good anecdote.
But no matter whether it was in Papua New Guinea, Timor-Leste, Marshall Islands or anywhere else, a number of factors contributed to making these assignments fiendishly challenging to write up.
The first was politics. No matter where we turned, we ran up stubbornly against it, expressed in all its various forms.
Some of this politics was personality-based, the time-honoured story of people from different walks of life just not being able to get along. I recall two co-chairs of a program dedicated to fostering partnership within the law and justice agencies in a Pacific nation unable to be in same room as each other.
Some resided within the ‘political economy’ of the country, between and within institutions and political groupings. Friendships, alliances, institutional restraints, and subterranean court politics — such who played guitar or tennis with friends in high places — seemed important too, as it is everywhere. I read Hilary Mantel’s Bring up the Bodies on one assignment that involved copious turf politics and remember thinking what I was experiencing was nothing new.
Other times, such politics was the reality of bilateral relationships between states. The diplomatic meta-narrative within which these programs take place is of #partnership and #reaffirmingcommitments; hard for the reviewer to square with partners that were, betimes, lukewarm or uninterested.
The second was complexity and how to render a messy situation in a tidy manner. To borrow the words of Harold Macmillan, the late British Prime Minister, it was ‘events, dear boy, events’ that determined the jagged and mercurial course of implementation more than the design document. Such ‘known unknowns’ include changes in governments and, with them, policies, as well as political leaders, heads of agencies and bureaucrats who change their minds or simply do not grasp the issue, setting plans often entirely off course. An extreme example, but once, in Vanuatu, the government in Port Vila wrote a bold new chapter in the story of the bilateral relationship by expelling all the Australian Federal Police (AFP) staff working on a police development program halfway through our review. (The AFP returned a year later.)
The writer faces the challenge of trying to write a story where much cannot be said and where even the most soft-gloved of observation and constructive criticism in the first draft can generate consternation, whether from the staff being reviewed, the institution they are working for, their partners or the funder.
And such sensitivity around what a report may say is completely rational. Trepidation within a donor or managing contractor about reputational damage to their brand is palpable. Success in promotion rounds and winning new contracts depended upon hawking achievements, not tarrying on missteps. No Ambassador or High Commissioner wants the contents of a page-turning review jolting the happy narrative on their social media feed. Contractual insecurity snaps at the heels of the consultants preparing the reports. I didn’t get paid for these assignments until the report was accepted, an imbursement arrangement which always concentrated the mind.
Removing all sources of consternation means that the end product always feels curtailed, akin to reading a novel with some chapters taken out. Is it surprising that, if and when eventually published, these documents garner such few readers? A few years ago, the World Bank was candid enough to disclose nearly half their reports did not have a single reader.
In 2015, after one assignment, the more bracing and surrealistic details of which might get included in a mystery novel sometime, I hopped off the treadmill.
I salute the hardy souls that have perfected this task for which one gets little praise, and often wondered whether there is something sui generis about law and justice evaluations that makes them tricky. Does the ambiguity of the topic make it difficult to assess? Do those evaluating, say, a market development facility run into similar challenges?
I find writing difficult and admire the talents of the many skilled authors working within the narrow confines of this genre. I take joy in trying to discern the intent behind their intentionally ambiguous word choices.
But I often read them and think ‘I’m sure I’ve read this story before’, and wonder how much learning these painstakingly put-together efforts generate. An admirably brisk evaluation of a New Zealand-funded Community Policing program in Bougainville conducted in 2019 resounded points made in a review I was involved in back in 2013. Which begs the question of just how governments that commission these reviews actually use these documents? Who reads the final draft? Is all the effort and angst of producing them actually worthwhile?
I’ve often wondered whether there is an argument for not publishing these documents or, at the very least, having some parts public-facing (and diluted as needs be) and some parts not. There are good reasons why tweets are public and diplomatic cables embargoed for 25 years and, in a similar vein, why politicians’ memoirs are more readable than their talking points.
On 1 October, I’ll be chairing an online session at the Law and Justice Development Community of Practice about the art of reviewing law and justice programs. All are welcome to join — register here.
We’ll be looking at three reviews undertaken in 2019: Indonesia, Solomon Islands (not uploaded yet; always intriguing when that happens) and Vanuatu. We are going to explore the headline findings of these three reviews and what they tell us about what works (and what does not) in law and justice assistance. We’ll ask how much these reviews are contributing to learning, program practice and the direction of law and justice assistance.
I’m looking forward to the conversation and learning from people with greater literary gifts than me. I’d love you to join us. Got a question that you’d like me to ask? Email me.
Thanks for an interesting post. One thing that has struck me over the years is that while most people agree that governance support is a long-term undertaking, people too often seek to meet bureaucratic requirements by specifying short-term, often quantitative, performance indicators. And too often, what’s important isn’t measurable, and what’s measurable isn’t important. For example, in the field in which I used to work, electoral assistance, by far the most crucial (though usually unstated) objective was to build a culture of honesty and independence, of the type which would cause an administrator faced with political pressure to stand up for proper practice. This is the sort of thing which can only be tested when it actually happens, and one lived in hope that it never would.
Couldn’t agree with you more, Michael. So much time spent on trying to measure the immeasurable.
One figure that sticks in my mind is 60% – the amount of time Martha McIntyre estimated she spent in her job in a police program writing reports and describing achievements. It doesn’t seem a good use of anyone’s time.
Thanks for the positive feedback
Actually, towards the end of my career I sought whenever possible to configure projects cheaply, and with the smallest possible foreign footprint. Partly that was to try to keep the cost under the AusAID threshold which took one from relatively un-bureaucratic work into the world of grandiose designs, reviews, reports, M&E matrices etc. But I also wanted to avoid the situation in which a large international team becomes a structure within a structure, talking among themselves rather than working with those they are supposed to be helping.
Bula Vinaka Michael.
And the AEC and your PIANZEA/BRIDGE legacy lives on across the Pacific with many a Pacific Islander taking charge of election administration and electioneering in their home country’s.
Unfortunately, not so in Fiji. The dictatorship, as we locals perceive and experience it, still rules, despite the successive post coup election technical assistance received from well meaning NZ and Australian Advisors.
Your poor name sake experienced the wrath of electorate/candidate/Party frustration at the Elections he tried to help administer. And my, he was oh so diplomatic about it!
Your combined experience’s will sure be worth a blog or book or two on shared learnings, not to mention the laughs.
I sure would love to read more.
Warmest of regards
Thanks for your kind comments and good to hear from you. But it’s sad that Fiji’s situation so differs from the positive things we are seeing elsewhere in the region. Stay safe and well!
What a wonderful post Gordon. You’ve raised so many of the things we think about as evaluators – not least of which is how do we cut through the ‘narrative’ that gets created around a program in a way that can bring meaningful change. You’ve raised a really interesting question about whether there is something unique about law and justice programs that make this more difficult. I’m going to give that a bit more thought, and come back to you with some reflections soon. In the meantime, I thought I’d point in you in the direction of the mid-term review of for the Tonga Police Development Programme. The report is less brisk than the one I did for the Bougainville Community Policing Programme, but it resulted in some tangible changes to the programme (woohoo!). And, full kudos to MFAT for publishing these reports on their website. I think that kind of transparency is important, and doesn’t negate the confidential conversations and information that can be provided to donors/programme partners as part of an evaluation process. Also, I think of evaluations as a process, where how we do them matters as much as (in some cases more than) what we write in the report. I find this thought also helps with the drudgery of report writing 🙂
Thanks so much, Stacey. I appreciate your kind feedback and will check out the Tonga report. And you are right: all credit to MFAT for putting the report on line. (By contrast, few reports of Australian funded policing programs make their way online)
From having received lots of private feedback on the blog, my sense is that this is not a problem restricted to law and justice programs alone! Some of the tales I heard were very much in the ‘not sure whether to laugh or cry’ category. All best, and thanks again