Comments

Bravo! The post 2015 agenda is fast becoming a very noisy debate dominated by the usual suspects. I don't think the ideas that CGD, ODI, etc. are bad, but I agree with Morton that once the debate heads in the direction they suggest it will become harder and harder for voices from the South to be heard and more importantly, to influence the outcome. I am hopeful that given the more varied international development community today (compared with 2000) that there are many factors and forces that will not allow this to happen. Nevertheless, we in the developed countries should be more generous, stepping back to allow others to take the lead; that, unfortunately, is not likely to happen.

Hi Stephen, I agree with you about the need to publically publish information about what we are claiming we are spending on foreign aid. The essence about published information however is whether it is meaningful or valueless? While the article you refer to may well have limited horizons, I suggest the real issue is not whether consultancies are either good or bad but whether they areeffective in producing results at the grass roots? The experience of many would suggest that after decades of huge amounts of aid monies being spent on wonderful sounding projects, the effects are minimal at best at the level where something is desperately needed. Take for example the recent media report on the provision of AusAID support in the fight against drug resistant TB in Western Provence of PNG directly opposite the Torres Strait. What is really happening as reported by a journo who actually visited the region and saw with his own eyes is I’ll bet not what is being reported to those who are signing off on the program. Sorry mate. I don’t agree with you about it being hard to address compliance issues concerning foreign aid. Having worked at the kunai roots level I can assure you it is very easy. All you have to do is accept that a published assessment is only as good as the methodology used to prepare the report. What use is there to get those wonderful, glowing reports of achievement if they are in fact, based on totally meaningless statistics and misinformnation? Total expenditure figures are the beloved addiction of politicians who quote them ad nausium but don't have any idea about what was actually achieved in reality. They are also the meat of those who feed off the public purse without achieving any long term results for those who we are supposed to be assisting. If a consultancy is worth its salt, it should provide an effective reporting mechanism that assesses achievements based on a prior set practical benchmark or target. If you can’t measure practical results, you can’t manage a program. Could it be (perish the thought), that those who are currently involved in consultancies might be slightly partial in this area of concern?

The constructive and thoughtful comments here tend to reflect a number of key matters that seem important to me. They all highlight the great importance of considering culture and context whether it is Uzbekistan or Indonesia. In the case of Indonesia, Sopantini reminds us of the need to be more local in our thinking, such as considering culture in Flores, rather than the broader concept of Indonesian culture, and Mark draws attention to different concepts of knowledge and learning that have implications for teaching. Dan asks if there is research on methods used by different countries that link to success in PISA tests. Yes, there is a literature on this of which the "Mathematics Teaching and Learning Strategies in PISA" by the OECD, 2010, is one example. In relation to your comment, Dan, these general conclusions are pertinent: “Teaching and learning strategies are complex processes that interact with one another, suggesting that in-depth, context-specific analyses are necessary to fully understand each strategy’s role in enhancing student performance. With a few interesting exceptions, most teaching and learning strategies do not have a direct, robust and consistent relationship with student performance across countries. The relationship between the strategies and performance tends to be moderated by other factors such as student attitudes and background, suggesting that these issues cannot be analysed separately (page 9).” (Both this study and the one below are discoverable by Googling the titles.) To emphasise the importance of culture and society in these tests, have a look at this publication by the Finnish National Board of Education, "Main factors behind the good PISA reading results in Finland". This publication points out the importance of reading in Finland – “one of the best library systems in the world, the library is the most beloved cultural institution” and “the number of books borrowed annually from public libraries and number of new books for children and young people is high”. Other indicators of reading are cited. Compare these values and habits with Indonesia! The above points also relate to the comments made by Chimi Thonden. Chimi’s comments about “incremental change” are raised in the discussion of the book Poor Economics here. It is a pity that there are so few examples of continuing support from national governments or donors to continue with incremental changes over time. The new USAID PRIORITAS project in Indonesia is one welcome exception to this, however. The other pity, and this is implied in Dan’s comments, is that there is very little quality research upon which to base policies and practices in relation to teaching in Indonesia. Even the data we do have from shorter commissioned studies and from routine monitoring and evaluation is not always well used in drawing lessons for practice and policy development. Like others, I have often wondered whether formal or progressive approaches are best. But while we await better evidence to help make our decisions for specific contexts, I think I will still be guided by conclusions reached by R.B. Spence in the 1928 edition of the Journal of Educational Psychology, 84 years ago: “The real problem is not "Is method A better than method B or method C?", but rather "Under what conditions does each method produce the most effective results?" And in relation to the selection and use of one approach to teaching, in this case in universities, he observed: “The decrying of the wholesale use of lectures is probably justified. The wholesale decrying of the use of lecturing is just as certainly not justified.”

I believe that proposed strategies for development and accounting for development progress is something that should be subject to analysis, debate and evidence when this is available. By the tone of your response, Lant, I sense that you may not agree with this. I think this is unfortunate. It inhibits considered progress towards good development strategies for good outcomes. This is why I noted that your Paper is a welcome contribution because it foregrounds the move away from schooling goals towards learning goals, which has the potential for good outcomes. Nevertheless, there are risks when we begin to consider ways of measuring these outcomes. And this is why I argued for caution. You identify three key points in your comment. As I understand your first point, you say that I that "cannot coherently applaud the shift of focus from “schooling” … to “learning” and then claim we can do that without ever measuring learning." I did not assert that we should never measure learning. I acknowledged, for example, that you encouraged the use of country-specific goals and tests, a strategy that I believe is appropriate, if done well. Rather, I said that we should exercise great caution in the wider use of tests such as TIMSS and PISA for comprehensive international comparisons of learning. Further, we need to be alert to the potentially negative consequences of such types of tests for developing countries. This caution is not based entirely on them becoming high stakes for the systems but also because of concerns about their cross-cultural validity. Secondly, you point out that "the debate about ‘high stakes’ in countries like India ... is absurd because there are massively high stakes for the student tests at grades 10 and 12 and university entrance examinations already." If you really mean debating high stakes testing is absurd, I find this troubling; with more reason to consider the stakes and figure out how to manage them. Another of your points is "that [the Indian] system itself (and schools and teachers) avoids any measurement of its performance at all ... So introducing learning goals would not increase the stakes for students–that is there already–it would just create some tracking of overall performance on goals the society claims to care about." I agree with you that tracking is necessary, but this is where the caution I argue for is also necessary so that the stakes from tracking are understood beforehand and carefully managed so that their consequences do not filter down to schools and children with even more negative consequences. I fail to see why exercising caution is problematic, particularly in the kind of environment you describe. To do otherwise would be irresponsible. In your third point you denigrate the work of large numbers of dedicated teachers and I wonder why you do this. Yes, I am sure there are some teachers in Africa, and elsewhere in the developed world as well, who are absent too often or only work 29 minutes a day. But there are very many teachers who are, as you sardonically describe, “wonderful, intrinsically motivated teachers and doing their best at the complex task called teaching”. Where is the evidence to support your assertion about the typical Indian teacher’s commitment and that the Indian education “system itself (and schools and teachers) avoids any measurement of its performance at all”? I will go on “assuming” (to use your term, Lant) that the risks of high stakes testing will apply in developing countries just as they do in the developed, until proven wrong. I repeat that I believe the move to learning goals you argue for in your Paper is a positive direction to take. But I also repeat that we must be cautious about the risks that can arise when testing attainment of these goals becomes high stakes within developing countries with all the known and potentially negative consequences that will be counterproductive to good development outcomes.

Paul, You've misunderstood what the article is about. I think what you've put forward is a proposal for assessing aid effectiveness at the recipient country level. I wouldn't say its a particularly good proposal. Sometimes a really good consultancy has massive resturns, whereas I get the impressions you would say: the more being spent on consultancies, the worse the program. But in any case, our article is about something quite different, and much less ambitious. It's about one small piece of the aid puzzle, the production and publication of country strategies. I assume you would agree that aid agencies should produce and publish country strategies. If we agree on that, then we can also agree that we should check on whether AusAID is actually following through on its promise to produce and publish. That's what we try to do. More generally, I think assessing aid effectiveness is really hard. Assessing compliance with good practice in aid is easier. Compliance doesn't necessarily imply good aid, but its an important first step.

There is some question about when to cry wolf. Clearly if there is a wolf. But what if it is dusky and I cannot really tell if it is a wolf or an Alaskan Husky? Probably better to play it safe and cry wolf. On the other hand, crying wolf at everything that is canine leads to lots of hub-bub and confusion. I own a Bichon-Frise who weighs about 12 pounds and is white and cuddly and looks alot more like a sheep than a wolf (for reasons I don't understand she has her own Facebook page at Jaya Dog, go see). Crying wolf at a Bichon-Frise just makes you seem hysterical and a little silly. Our paper never proposes high stakes testing, never uses the words "high stakes" and cannot, in my view, be reasonably construed as proposing test for high stakes purposes for students, teachers or schools. Debates about "high stakes" should be reserved for when that is actually on the table and not when it is never mentioned. The authors current attitude seems very Victorian, the current paper isn't proposing sex but it does say things that could lead people to think about other things that might eventually lead them to think about sex and so the paper is prurient. On a less facetious note, three points. First, in developing countries there is "high stakes" measurement--it is just that measurement is about enrollments and inputs. So the debate is not about "high stakes" or not, it is about what the "high stakes" measures that drive organizations should be. I don't think you can coherently applaud the shift of focus from "schooling" (where there are scads of measures that drive policy making) to "learning" and then claim we can do that without ever measuring learning. This is like saying I am going to drive from New York to Kansas but never look at where I am. Odds of doing that seem pretty slim. Second, the debate about "high stakes" is countries like India (where I now live) is absurd because there are massively high stakes for the student tests at grades 10 and 12 and university entrance examinations already. So right now there are examinations that are crushingly high stakes for students but the system itself (and schools and teachers) avoids any measurement of its performance at all and so at times produces tragically awful schooling--especially for the poorest--with consequences at all. So introducing learning goals would not increase the stakes for students--that is there already--it would just create some tracking of overall performance on goals the society claims to care about. Third, I do not believe--for many of the reasons the blog author cites--in "high stakes" or "thin accountability" metrics. That said, many of the arguments from rich country denizens about "high stakes" is that it detracts teachers from doing more worthwhile things. But in poorly performing countries the issue is not that these wonderful, intrinsically motivated teachers and doing their best at the complex task called teaching and evil green-shade guys would sully their cherished occupation with mind numbing, task-narrowing, soul-shrinking, numbers. In India it is well documented that a typical teacher is in the classroom engaged in instructional activity less than half the hours they are paid to do so. I just heard today that a recent survey in Africa (rural Uganda I believe) found teacher instructional time was 29 minutes a day. So instructional activity of any type would be a gain. In countries at high levels of performance, like Australia or the USA or Finland I can see being very worried about high stakes testing because the system is already functioning pretty well and some combination of intrinsic motivation and thick accountability (internal and external) is working reasonably well. But this is not the problem in many countries and about them we should not assume that the same risks are present.

Michael and Bill thank you for your comments. There is a lot there and I take Bill's point on large NGOs crowding out local initiatives. Of course though my point is that it depends on the NGO, their reach, and to some extent the country. Countries with a strong local NGO sector like India Bangladesh or the Philippines see little crowding out; except in the case of Bangladesh where is the big local NGOs like Proshika or BRAC who do the crowding out. On Michaels’ point I sort of answer that in the following section and that is I don't to think it is the closeness to government or otherwise that limits NGOP effectiveness in what they set out to do but rather their world view or what Lissner calls their Weltanschauung or world view. Most NGOs run agendas based on their values, and for the majority these are altruistic or religious values, and for a minority it would be structural social change. My point is that historically it has always been thus. Lissner's Politics of Altruism was written long before the big government donor dollars were there and it reads as relevant today as it was in the 1970s. In Australia the number of NGOs accredited to receive AusAID find has halved in the last 20 years. The accreditation criteria are largely about the foreign aid system and its approaches, but with a strong emphasis on transparent partnership and local control. The Codes of Conduct are trying to move agencies towards more local control and participation in their processes but it is a slow process. I think the issue of about NGOs and what they are on about rather than their closeness or otherwise to government which fluctuates according to the national political cycle (i.e whio is in power). The one big change over the last forty years has been the internationalising of donor NGOs, where most are now part of larger global networks, but this does not mean they are more or less in the thrall of government funding, but perhaps that their world views are reinforced, which may or may not be a good thing.

Thanks Enrique for your post. It's a hard balancing act, especially when there are a small pool of organisations with the skills and resources to both implement and evaluate projects. In the private sector, there are four major audit firms that audit most if not all the large multinational companies. However there are regulations in place to ensure that the same audit firm cannot provide advice on internal controls while also undertaking the external audit. As I recall there are also regulations in place to prevent one audit firm from auditing the same company for more than 5 consecutive years. Perhaps aid donors could consider similar regulations (if they haven't done so already).

Bob, thanks for providing such an excellent synopsis of this important book and also for generating thoughtful and insightful comments by those who have responded with opinions. One of Indonesia's education goals is to make its citizens more competative in a global economy. A useful measure of progress in meeting the goal is PISA results, where Indonesia performs poorly. It might be an interesting study, if not already done, to investigate the methods used among the top performers on a continum ranging from "formalism" to "progressive". S. Korea and Finland were tops in 2009--two countries globally competative but, I assume, with very different cultural traditions. At the same time it might be interesting for an on-going education project to look at PISA results among Indonesian schools. perhaps some correlations between performance and instructional models might emerge. [Puspendik has the data.]

The overview of any huge scheme like this is fairly useless unless it is translated into previously set benchmark achievements. E.g.: 1. What percentage by country of this expenditure is spent in Australia and never reaches the country AusAID is intending to help? 2. Of the amount that is actually spent in the developing nation, how much is spent on consultancies and expatriate salaries? 3. Of each country’s total share, is there a feedback loop to establish the effects and ongoing benefits accruing? If so, who signs off on this loop? Those actually benefiting at the coal face or those in the capital of the local national government whose budget these AusAID funds augment? 4. Against what percentage of each local national budgetary item is the AusAID allocation? 5. What percentage of each country’s AusAID allocation is actually expended in rural (non metropolitan) areas on rural programs benefiting those in rural areas? 6. Finally, did the auditors actually go into the field and check first hand the benefits accrued in each program or did they only check the written reports submitted by AusAID and expatriate consultancies?

Interesting post, Dinuk. I agree on the need to plan evaluations more carefully to find the right approach for each intervention. Also, that more care needs to go into planning to ensure that existing knowledge is incorporated into the interventions. But also important is the development of a system that ensure real independence. Aid is no longer an 'a-political' issue that voters are unaware and interested in. Efforts to assess the quality of Australian aid will be undermined if this is not done properly. Australia should therefore be careful of avoiding the situation we find in the UK where DFID is the main client of the very same consultancies, NGOs and think tanks that are called to evaluate UK aid. KPMG, for example, manages the aid watchdog but also implements hundreds of millions of Pounds-worth of projects. This clientelistic approach means that think tanks like ODI are also now involved in projects as implementing agents for iNGOs and consultancies (such as PWC and KPMG) which makes their oversight roles impossible. And the same is true with smaller consultancies from communications to social development which often find themselves working with organisations that they are also evaluating. 3ie itself, supposedly the enforcers of absolute certainty, is not free from this. True independent voices are few and unpopular. This situation is not helped by the roles played by new foundations like Gates or iNGOs in using researchers to advocate for their own interests (see, for instance, Gates' development progress work) which goes as far as funding influential media outlets like the Guardian for the same purpose. The consequence is a system with few (if any) lines of accountability; one in which all participants are clearly benefiting from the status quo and the conclusion that 'more aid is good'. The public is beginning to react to this and, unless, important changes are made (and many will be big half-baked PR jobs, unfortunately), the baby will be thrown away along with the bathwater. I think the fault here lies mainly with some large bilateral funders such as DFID that have failed to recognise that different organisations play different roles in the aid sector and that their contributions demand certain degree of specialisation and even protection. A system in which consultancies, research centres, think tanks and NGOs are all expected to compete and collaborate with each other can only lead to uncomfortable and dangerous conflicts of interest. Conflicts that are incompatible with the demand for rigour and transparency in project evaluations. Australia would do well to avoid this muddling of roles. It should attempt to strengthen independent research communities with evaluation expertise separate from those tasked with implementing aid policy. Only this will allow Australian to hold its Aid industry to account.

The overview of any huge scheme like this is fairly useless unless it is translated into previously set benchmark achievements. E.g.: 1. What percentage by country of this expenditure is spent in Australia and never reaches the country AusAID is intending to help? 2. Of the amount that is actually spent in the developing nation, how much is spent on consultancies and expatriate salaries? 3. Of each country's total share, is there a feedback loop to establish the effects and ongoing benefits accruing? If so, who signs off on this loop? Those actually benefiting at the coal face or those in the capital of the local national government whose budget these AusAID funds augment? 4. Against what percentage of each local national budgetary item is the AusAID allocation? 5. What percentage of each country's AusAID allocation is actually expended in rural (non metropolitan) areas on rural programs benefiting those in rural areas? 6. Finally, did the auditors actually go into the field and check first hand the benefits accrued in each program or did they only check the written reports submitted by AusAID and expatriate consultancies?