Six years ago we wrote a paper called ‘The evaluation of politics and the politics of evaluation’, which argued that programs seeking to work in a politically smart manner faced particular challenges in monitoring and evaluating their performance. This included the need to take into account the long-term, non-linear nature of these initiatives, as well as the recognition of the complex, dynamic nature of the contexts in which they were working. These are many of the same challenges faced by programs seeking to work adaptively.
We argued that there were a number of evaluation and research methods which, when judiciously combined, were better able to answer the kinds of evaluative questions these programs asked. Yet the inherently political nature of evaluation and the ‘results agenda’ tended to favour certain methods – those that captured and quantified short-term tangible outputs, and downplayed those that potentially are better able to assess long-term, subtle changes in relationships, power and social norms (something others, such as Eyben, Natsios, Vogel, Honig, Yanguas and Mueller have also suggested).
Since we wrote that piece, and a follow up paper on what some programs were doing at the time, a lot more programs have sought to work adaptively and in politically smarter ways. Lisa Denney’s first blog in this three-part series captures some of the lessons this has generated. In this blog we set out six take-aways focused on what is being tried and learnt in setting up monitoring and evaluation (M&E) frameworks for adaptive programs, including about working in ways that recognise the political nature of evidence and evaluation.
- Be clear about the purposes of M&E
Designing these processes depends on what you want to do. Too often M&E frameworks are designed by technical specialists in ways that either exclude key stakeholders and decision makers, and/or which result in technically robust products that are too complex, expensive or time consuming to be implemented. Furthermore they can also conflate monitoring, evaluation and learning in unhelpful ways. All of which often results in lots of data being collected but rarely used. There is a growing call for ‘right-fit’ systems which are clear about the different purposes of monitoring, evaluation and learning, and more strategic about the collection and use of data to meet these different purposes.
- More complex programs require different but accessible M&E
For programs seeking to address complex and political environments, the various stakeholders, interactions and unplanned elements mean that traditional M&E – based essentially on tracking cause and effect – does not work. Practitioners who persist in applying traditional approaches end up either with long lists of indicators and questions (trying desperately to cover the whole story) or very high-level or generic measures, which provide no basis for informed decisions about why and how to adapt.
Programs seeking to work adaptively should utilise multiple data collection and analysis methods to illuminate the different aspects of the work. The aim is to provide useful information about how a program is operating, what contributions are being made to change, and the impact and relevance of this contribution. Often that means different systems for assessing program activities and program contributions to change and additional assessment about context – how that might also be changing, and the implications for the program.
This more comprehensive approach is not fully captured by a logframe or performance framework, nor can it be contained to the M&E team alone. It should be part of the fabric of the program itself. It requires thought and considerable engagement with the program implementers and others who can identify the critical factors to be tracked. The M&E therefore has to be comprehensible and accessible. As was clear at a recent Asia Foundation workshop in Manila involving M&E staff from several adaptive programs, simplifying processes, language and data collection was a common strategy. This included using telephone apps or Facebook to collect real time data from front line staff, and avoiding the often alienating language of M&E, instead using local or plain English language.
- Focus on processes and relationships as much as outputs and outcomes
It is recognised that human systems function on the basis of relationships, and it is these interactions that lie at the heart of social change. Furthermore, it is changes in relational practice, i.e. how people, organisations and states relate to each other, that can lead to new ‘rules of the game’, or institutions. However what those institutions will look like is not predictable. This raises challenges for M&E in adaptive programs about how and when to measure changes in relationships and what this leads to.
Adaptive programs are adopting a range of ways to try to address this challenge, including:
- partnering with researchers to undertake ongoing action research with a particular focus on assessing, for example, changes in power relations;
- using social network analysis, changes in social norms and outcome harvesting to assess the evolution of relationships over time and the ongoing collection of data to be able to tell a more complete story of change;
- building in social accountability processes or feedback loops to provide more real time data, particularly from those that programs are seeking to benefit; and
- more use of realist evaluation and qualitative comparative analysis methodologies to be clearer about the underlying generative mechanisms of change in particular contexts, including through using participatory methods.
- Resource the M&E and L
M&E to support dynamic programming does not start and finish with a framework and set of systems. It involves ongoing data collection and analysis processes, ideally engaging multiple stakeholders. This requires resources, planning and management attention. M&E has to be part of the work plans for program implementation. It requires scheduling and planning. Significantly, M&E for adaptive programs requires the whole team to take some responsibility for the analysis and review of the data collected and what it means for ongoing work.
- Be politically smart
Once it is recognised that monitoring, and particularly evaluation, is as political as anything else, and often needs to adapt to shifting demands, then many of the tools of adaptive programming and everyday political analysis become equally applicable. These might include:
- using everyday political economy analysis to better understand the interests, incentives and power relations of the stakeholders involved in an evaluation;
- being clearer about the multi-level theory of change and need for stakeholder engagement, which makes it more likely that M&E processes actually lead to better learning and adaptation;
- exploring whether there are coalitions, allies or networks which can combine their experience, skills and data to provide a more powerful and complete picture of change, as well as allowing multiple voices to be heard; and
- exploring framing and narratives which help communicate evidence but are also politically savvy. As many have argued, populism is not going to be beaten with spreadsheets.
- And finally
M&E for adaptive programs ought to also adapt and improve. There are few examples of a perfect M&E system for politically informed adaptive programs. The M&E has to build in its own processes of review, analysis and improvement in order to both grow with the program and stay relevant to its inquiry and assessment needs.
This is the second in a three-part series on adaptive aid programming. Part one looked at adaptive implementation, and part three at the role of research and learning. A pdf of all three posts can be found here.
Interesting! My take aways: collect the data that is necessary for the program; ensure a role for stakeholders and implementers in the data collection; and assess the context and its implications for the program.
Thank you
An interesting read and timely as we are in the process of revising input, outputs outcomes in a logframe of a political and technical project following a Mid Term Review. Linda Kelly can related to this. Will share this article with partners
Very interesting article. M&E and L is an integral part of programming. Adaptive programming just took it further, and try to make M&E and L more realistic and relevant, I believe. With multiple sources of data collection and analysis, and at various points and levels, should provide an holistic picture of the change story, and value for money- something donor programs are pressed to account for in an adaptive and complex environment. Also agree that Logframe approach is limiting in evaluating human interactions and social change, in which ‘context’ plays an important role, and use and application of, ‘thinking and working politically’, and political economy analysis, among others are necessary.
Thanks John I am glad this resonated with your experience!
Thanks very much Chris and Linda for a great read. I was interested to see that ‘M&E’ was discussed in a separate blog to ‘L’. Along this vein, and in your experience/s, what can incentivise donors and providers to better link the evaluative techniques you outline to learning processes and (ideally) real-time adjustments in programming (i.e. activities, budgets, choice of partners etc etc)?
Hi Lavinia really important question. I think part of the issue is thinking hard about what some have called the ‘decision space’ https://www.developmentgateway.org/blog/dg-white-paper-release-understanding-data-use i.e. the political and institutional incentives which enable use of data, learning and subsequent adaptation. I think Dan Honig’s new book Navigation by Judgement https://oxfamblogs.org/fp2p/book-review-navigation-by-judgment-by-dan-honig/ provides a compelling argument for not only why this the case, but why providing space for front line judgement, learning and adaptation is critical
An interesting blog. This could end up very very complex which most countries would be unable to sustain without extensive consultancy support – which is dangerous
Totally agree with you Ian, that is why we point to attempts to, and the need to, simplify language, processes and data collection. At the same we also need to recognise that there needs to be adequate support for, and resourcing of, local actors, researchers and agencies in helping to answer some difficult questions.