Comments

I believe that proposed strategies for development and accounting for development progress is something that should be subject to analysis, debate and evidence when this is available. By the tone of your response, Lant, I sense that you may not agree with this. I think this is unfortunate. It inhibits considered progress towards good development strategies for good outcomes. This is why I noted that your Paper is a welcome contribution because it foregrounds the move away from schooling goals towards learning goals, which has the potential for good outcomes. Nevertheless, there are risks when we begin to consider ways of measuring these outcomes. And this is why I argued for caution. You identify three key points in your comment. As I understand your first point, you say that I that "cannot coherently applaud the shift of focus from “schooling” … to “learning” and then claim we can do that without ever measuring learning." I did not assert that we should never measure learning. I acknowledged, for example, that you encouraged the use of country-specific goals and tests, a strategy that I believe is appropriate, if done well. Rather, I said that we should exercise great caution in the wider use of tests such as TIMSS and PISA for comprehensive international comparisons of learning. Further, we need to be alert to the potentially negative consequences of such types of tests for developing countries. This caution is not based entirely on them becoming high stakes for the systems but also because of concerns about their cross-cultural validity. Secondly, you point out that "the debate about ‘high stakes’ in countries like India ... is absurd because there are massively high stakes for the student tests at grades 10 and 12 and university entrance examinations already." If you really mean debating high stakes testing is absurd, I find this troubling; with more reason to consider the stakes and figure out how to manage them. Another of your points is "that [the Indian] system itself (and schools and teachers) avoids any measurement of its performance at all ... So introducing learning goals would not increase the stakes for students–that is there already–it would just create some tracking of overall performance on goals the society claims to care about." I agree with you that tracking is necessary, but this is where the caution I argue for is also necessary so that the stakes from tracking are understood beforehand and carefully managed so that their consequences do not filter down to schools and children with even more negative consequences. I fail to see why exercising caution is problematic, particularly in the kind of environment you describe. To do otherwise would be irresponsible. In your third point you denigrate the work of large numbers of dedicated teachers and I wonder why you do this. Yes, I am sure there are some teachers in Africa, and elsewhere in the developed world as well, who are absent too often or only work 29 minutes a day. But there are very many teachers who are, as you sardonically describe, “wonderful, intrinsically motivated teachers and doing their best at the complex task called teaching”. Where is the evidence to support your assertion about the typical Indian teacher’s commitment and that the Indian education “system itself (and schools and teachers) avoids any measurement of its performance at all”? I will go on “assuming” (to use your term, Lant) that the risks of high stakes testing will apply in developing countries just as they do in the developed, until proven wrong. I repeat that I believe the move to learning goals you argue for in your Paper is a positive direction to take. But I also repeat that we must be cautious about the risks that can arise when testing attainment of these goals becomes high stakes within developing countries with all the known and potentially negative consequences that will be counterproductive to good development outcomes.

Paul, You've misunderstood what the article is about. I think what you've put forward is a proposal for assessing aid effectiveness at the recipient country level. I wouldn't say its a particularly good proposal. Sometimes a really good consultancy has massive resturns, whereas I get the impressions you would say: the more being spent on consultancies, the worse the program. But in any case, our article is about something quite different, and much less ambitious. It's about one small piece of the aid puzzle, the production and publication of country strategies. I assume you would agree that aid agencies should produce and publish country strategies. If we agree on that, then we can also agree that we should check on whether AusAID is actually following through on its promise to produce and publish. That's what we try to do. More generally, I think assessing aid effectiveness is really hard. Assessing compliance with good practice in aid is easier. Compliance doesn't necessarily imply good aid, but its an important first step.

There is some question about when to cry wolf. Clearly if there is a wolf. But what if it is dusky and I cannot really tell if it is a wolf or an Alaskan Husky? Probably better to play it safe and cry wolf. On the other hand, crying wolf at everything that is canine leads to lots of hub-bub and confusion. I own a Bichon-Frise who weighs about 12 pounds and is white and cuddly and looks alot more like a sheep than a wolf (for reasons I don't understand she has her own Facebook page at Jaya Dog, go see). Crying wolf at a Bichon-Frise just makes you seem hysterical and a little silly. Our paper never proposes high stakes testing, never uses the words "high stakes" and cannot, in my view, be reasonably construed as proposing test for high stakes purposes for students, teachers or schools. Debates about "high stakes" should be reserved for when that is actually on the table and not when it is never mentioned. The authors current attitude seems very Victorian, the current paper isn't proposing sex but it does say things that could lead people to think about other things that might eventually lead them to think about sex and so the paper is prurient. On a less facetious note, three points. First, in developing countries there is "high stakes" measurement--it is just that measurement is about enrollments and inputs. So the debate is not about "high stakes" or not, it is about what the "high stakes" measures that drive organizations should be. I don't think you can coherently applaud the shift of focus from "schooling" (where there are scads of measures that drive policy making) to "learning" and then claim we can do that without ever measuring learning. This is like saying I am going to drive from New York to Kansas but never look at where I am. Odds of doing that seem pretty slim. Second, the debate about "high stakes" is countries like India (where I now live) is absurd because there are massively high stakes for the student tests at grades 10 and 12 and university entrance examinations already. So right now there are examinations that are crushingly high stakes for students but the system itself (and schools and teachers) avoids any measurement of its performance at all and so at times produces tragically awful schooling--especially for the poorest--with consequences at all. So introducing learning goals would not increase the stakes for students--that is there already--it would just create some tracking of overall performance on goals the society claims to care about. Third, I do not believe--for many of the reasons the blog author cites--in "high stakes" or "thin accountability" metrics. That said, many of the arguments from rich country denizens about "high stakes" is that it detracts teachers from doing more worthwhile things. But in poorly performing countries the issue is not that these wonderful, intrinsically motivated teachers and doing their best at the complex task called teaching and evil green-shade guys would sully their cherished occupation with mind numbing, task-narrowing, soul-shrinking, numbers. In India it is well documented that a typical teacher is in the classroom engaged in instructional activity less than half the hours they are paid to do so. I just heard today that a recent survey in Africa (rural Uganda I believe) found teacher instructional time was 29 minutes a day. So instructional activity of any type would be a gain. In countries at high levels of performance, like Australia or the USA or Finland I can see being very worried about high stakes testing because the system is already functioning pretty well and some combination of intrinsic motivation and thick accountability (internal and external) is working reasonably well. But this is not the problem in many countries and about them we should not assume that the same risks are present.

Michael and Bill thank you for your comments. There is a lot there and I take Bill's point on large NGOs crowding out local initiatives. Of course though my point is that it depends on the NGO, their reach, and to some extent the country. Countries with a strong local NGO sector like India Bangladesh or the Philippines see little crowding out; except in the case of Bangladesh where is the big local NGOs like Proshika or BRAC who do the crowding out. On Michaels’ point I sort of answer that in the following section and that is I don't to think it is the closeness to government or otherwise that limits NGOP effectiveness in what they set out to do but rather their world view or what Lissner calls their Weltanschauung or world view. Most NGOs run agendas based on their values, and for the majority these are altruistic or religious values, and for a minority it would be structural social change. My point is that historically it has always been thus. Lissner's Politics of Altruism was written long before the big government donor dollars were there and it reads as relevant today as it was in the 1970s. In Australia the number of NGOs accredited to receive AusAID find has halved in the last 20 years. The accreditation criteria are largely about the foreign aid system and its approaches, but with a strong emphasis on transparent partnership and local control. The Codes of Conduct are trying to move agencies towards more local control and participation in their processes but it is a slow process. I think the issue of about NGOs and what they are on about rather than their closeness or otherwise to government which fluctuates according to the national political cycle (i.e whio is in power). The one big change over the last forty years has been the internationalising of donor NGOs, where most are now part of larger global networks, but this does not mean they are more or less in the thrall of government funding, but perhaps that their world views are reinforced, which may or may not be a good thing.

Thanks Enrique for your post. It's a hard balancing act, especially when there are a small pool of organisations with the skills and resources to both implement and evaluate projects. In the private sector, there are four major audit firms that audit most if not all the large multinational companies. However there are regulations in place to ensure that the same audit firm cannot provide advice on internal controls while also undertaking the external audit. As I recall there are also regulations in place to prevent one audit firm from auditing the same company for more than 5 consecutive years. Perhaps aid donors could consider similar regulations (if they haven't done so already).

Bob, thanks for providing such an excellent synopsis of this important book and also for generating thoughtful and insightful comments by those who have responded with opinions. One of Indonesia's education goals is to make its citizens more competative in a global economy. A useful measure of progress in meeting the goal is PISA results, where Indonesia performs poorly. It might be an interesting study, if not already done, to investigate the methods used among the top performers on a continum ranging from "formalism" to "progressive". S. Korea and Finland were tops in 2009--two countries globally competative but, I assume, with very different cultural traditions. At the same time it might be interesting for an on-going education project to look at PISA results among Indonesian schools. perhaps some correlations between performance and instructional models might emerge. [Puspendik has the data.]

The overview of any huge scheme like this is fairly useless unless it is translated into previously set benchmark achievements. E.g.: 1. What percentage by country of this expenditure is spent in Australia and never reaches the country AusAID is intending to help? 2. Of the amount that is actually spent in the developing nation, how much is spent on consultancies and expatriate salaries? 3. Of each country’s total share, is there a feedback loop to establish the effects and ongoing benefits accruing? If so, who signs off on this loop? Those actually benefiting at the coal face or those in the capital of the local national government whose budget these AusAID funds augment? 4. Against what percentage of each local national budgetary item is the AusAID allocation? 5. What percentage of each country’s AusAID allocation is actually expended in rural (non metropolitan) areas on rural programs benefiting those in rural areas? 6. Finally, did the auditors actually go into the field and check first hand the benefits accrued in each program or did they only check the written reports submitted by AusAID and expatriate consultancies?

Interesting post, Dinuk. I agree on the need to plan evaluations more carefully to find the right approach for each intervention. Also, that more care needs to go into planning to ensure that existing knowledge is incorporated into the interventions. But also important is the development of a system that ensure real independence. Aid is no longer an 'a-political' issue that voters are unaware and interested in. Efforts to assess the quality of Australian aid will be undermined if this is not done properly. Australia should therefore be careful of avoiding the situation we find in the UK where DFID is the main client of the very same consultancies, NGOs and think tanks that are called to evaluate UK aid. KPMG, for example, manages the aid watchdog but also implements hundreds of millions of Pounds-worth of projects. This clientelistic approach means that think tanks like ODI are also now involved in projects as implementing agents for iNGOs and consultancies (such as PWC and KPMG) which makes their oversight roles impossible. And the same is true with smaller consultancies from communications to social development which often find themselves working with organisations that they are also evaluating. 3ie itself, supposedly the enforcers of absolute certainty, is not free from this. True independent voices are few and unpopular. This situation is not helped by the roles played by new foundations like Gates or iNGOs in using researchers to advocate for their own interests (see, for instance, Gates' development progress work) which goes as far as funding influential media outlets like the Guardian for the same purpose. The consequence is a system with few (if any) lines of accountability; one in which all participants are clearly benefiting from the status quo and the conclusion that 'more aid is good'. The public is beginning to react to this and, unless, important changes are made (and many will be big half-baked PR jobs, unfortunately), the baby will be thrown away along with the bathwater. I think the fault here lies mainly with some large bilateral funders such as DFID that have failed to recognise that different organisations play different roles in the aid sector and that their contributions demand certain degree of specialisation and even protection. A system in which consultancies, research centres, think tanks and NGOs are all expected to compete and collaborate with each other can only lead to uncomfortable and dangerous conflicts of interest. Conflicts that are incompatible with the demand for rigour and transparency in project evaluations. Australia would do well to avoid this muddling of roles. It should attempt to strengthen independent research communities with evaluation expertise separate from those tasked with implementing aid policy. Only this will allow Australian to hold its Aid industry to account.

The overview of any huge scheme like this is fairly useless unless it is translated into previously set benchmark achievements. E.g.: 1. What percentage by country of this expenditure is spent in Australia and never reaches the country AusAID is intending to help? 2. Of the amount that is actually spent in the developing nation, how much is spent on consultancies and expatriate salaries? 3. Of each country's total share, is there a feedback loop to establish the effects and ongoing benefits accruing? If so, who signs off on this loop? Those actually benefiting at the coal face or those in the capital of the local national government whose budget these AusAID funds augment? 4. Against what percentage of each local national budgetary item is the AusAID allocation? 5. What percentage of each country's AusAID allocation is actually expended in rural (non metropolitan) areas on rural programs benefiting those in rural areas? 6. Finally, did the auditors actually go into the field and check first hand the benefits accrued in each program or did they only check the written reports submitted by AusAID and expatriate consultancies?

Thanks Darian. Yours is certainly a story of academic perseverance paying off. All the best in your future career in the sector! Jonathan

Tom, I share many of your sympathies but I shied away from writing about them as I was either afraid of spreading too much hypocrisy throughout the paper (as I said, this is built off of my own experiences and not meant to be a prescriptive exercise) or I am simply too naive about the issue (particularly when we talk about image, settling in a developing country, what people are looking for in a development career etc). On the point of volunteering in rural communities, I did make the point that any work done in Indigenous communities is valued far greatly than any other volunteering one does overseas. I stopped myself from going too far into the subject because I also have never volunteered in an indigenous communities. I do, however, know many people that have done it and found it very rewarding, and plan on doing it myself at some point in the years to come. Overall, however, I don't think I share your cynicism about what drives people to work in the development sector. From the people I have met in my (limited) career to date they are all passionate and driven to see positive changes for those less fortunate. Or maybe I am just too well sheltered in my ivory tower think tank environment. If you want to flesh out your ideas further (and you are not prevented by your employer to do so) I would encourage you to write a blog for us to keep the conversation alive. Thanks a lot for your comments! Jonathan

Dear Benjamin, Thanks for your comments. I have heard about the oversight mechanisms you have mentioned, and my understanding is that they have started to become effective in improving the coordination of evaluations across government. However, in a way, this is an example of the overlapping M&E mandates I refer to in my article: as you point out, there are now several coordination mechanisms for M&E, each of which has a similar mandate but has different meeting schedules, reporting lines and composition. The risk is that the coordination of M&E might itself in the future become as complex as the M&E system itself, if it is not properly managed. The Baraza program does indeed look very promising, even though it is still in a pilot phase. As I mention in my article, it will be most useful if it becomes a forum for generating change: if the feedback that community members provide to their representatives is properly responded to, and their concerns are acted upon. Time will tell if the program will achieve this goal, but the early indications are positive. Thanks again for your contributions.