What kind of data revolution do we need?
Morten Jerven, Assistant Professor at Simon Fraser University and author of Poor Numbers, has five fundamental propositions for a successful agenda for data for development.
“In my book Poor Numbers I have described a basic knowledge problem in economic development statistics. Poor Numbers focused on the provision of data needed for the economic governance of African countries, and concluded that the numbers that we base economic decisions on are currently poor and seriously misleading. To explain this outcome I point to two dominant factors. First, many poor countries currently have weak statistical systems. Recording and data collection is particularly constrained in poor economies. Second, demand for data has been largely external to these poor economies and this demand has been largely uncoordinated and therefore often disruptive to the regular supply of data. As a result statistical offices have become de facto data collection agencies for hire, and the supply of data has been irregular and of very uneven quality. The state of affairs has been called a ‘Statistical Tragedy’ by the World Bank chief economist for Africa.
I suggest that we do not want to repeat past mistakes. On the basis of the diagnosis provided in Poor Numbers I put forward five fundamentals propositions that need to be taken into account before we embark on a data revolution. If we don’t, then this demand for evidence is going to be counterproductive and do more harm than good.
1. We need to focus on data supply, not only on global data demand.
Consistently the development community is repeating the mistake of naively demanding more and better data. The monitoring of specific projects should be tempered by a realistic assessment of the capacity of the statistical office to deliver information. The Millennium Development Goals agenda identified targets, but gave less thought to where the information should come from. We need to turn this important development question upside down. Rather than asking: ‘what kind of development should we target’, the question should be – ‘What kind of development are we able to monitor’?
2. A data revolution has to focus on costs as well as benefits.
More data is only better data if it is meaningful information and there are no opportunity costs to its supply. A useful exercise is to sift through the current 8 goals, 18 indicators and 48 targets and think about the cost of supplying all those data on an annual basis for all poor countries. In practice there are more gaps than real data observations in the global datasheet. Moreover, the supply of one indicator may come at the cost of another. When one member of staff is pulled away from balance of payment statistics in order to provide health statistics, we get more of some data and less of others. If there is a reliable national sampling frame, the costs of a Living Standard Measurement Survey could be up to 2-3 million dollars for an average country and take about 18 months to two years to complete. The supply of data has a cost, and we should rethink the list of indicators with that in mind.
3. Many statistics are public goods, and therefore we need to worry about market failures in their provision.
Not all statistics are public goods. Market research and opinion polls are supplied privately for private demand. But statistics on health, the economy, expenditure and population are public goods. There are well established theorems on why the supply of public goods is a public matter – and therefore mostly, but not always, pertains to the state. There is also an inherent risk of the tragedy of the commons. This occurs when demand overshoots supply, and particularly where private demand only has to cover the private costs (like when per diem is pulling staff out of their office), there is a negative social externality (no staff to complete a task a with lower financial reward) and the market demand for data does not cover the general cost of maintaining a national statistical office.
4. Paying for results is not only naïve but dangerous.
Hand in hand with the demand for a data revolution comes the idea that the development community should not only embrace evidence based policy, but also move towards paying for results. This would be a great idea if poor countries were closely observed labs where you could make objective observations, and the act of observation did not affect the facts. It may be a terrible idea if this is not the case. The data basis is so meager, and the data collection and aggregation process so vulnerable, that if you put forward strong incentives for reaching a particular number the data will be manipulated. This happened to immunization data, poverty rates in Tanzania, primary school rates in Kenya, and maize production in Malawi. What you get is policy driven evidence, the opposite of what we need. If we think that a child is vaccinated, has escaped the poverty line, goes to school and has enough food, when this is not the case, it is not a statistical mistake but a real tragedy
5. We need to worry more about local demand and applicability to have a sustained improvement in data for development.
Local demand for data needs to come into focus. A statistical office is only sustainable if it serves local needs for information. This is the time for the development community to remind ourselves that while we demand evidence for policy, we must not forego the opportunity to invest in accountability. Obtaining data is not a technocratic exercise but rather one of building institutions. For all this talk of ‘institutions matter’ and ‘governance’ in development circles, there has been a surprising gap in analyzing the statistical office. Statistical offices are first and foremost institutions that provide information to promote a discourse between citizens and states. We should spend more resources to find out which data matters for the citizens of these countries.
The statistical offices and their ability to provide timely high quality data on their countries have long been neglected. This is a chance to set these past mistakes right, and to move beyond a statistical tragedy, to stake a path towards statistical capacity building. It might not be a revolution, but considering these fundamentals of provision of data in poor countries, is a step towards better data for development.”
Morten’s post is part of a new blog series on post2015.org on the data revolution. Find out more here.
Post2015.org - what comes after the MDGs? 
Saying “Not all statistics are public goods” raises the question about whether we are discussing here Official statistics and the statistical system that poduces them or more generally any quantitative data produced by any organisation using statistical technics.
It is because Official statistics were Poor numbers up to being an Offical statistics tragedy that since 2000 PARIS21 advocates for the adoption at country level of what is now known as National strategy for the developement of statistics (official statistics) (NSDS). As was said during a recent meeting in Africa, a true data revolution would start with the implementation of agreed upon NSDSs with full support of all revolutionnaries.
your five recommandations naturally fit into any NSDS.
Regards
Posted by Gérard Chenais (@GChenais) | November 6, 2013, 12:08 pmDear Gerard,
I guess you feel that the NSDS and PARIS21 is getting the attention it deserves, and that is why you advocate it here and in other blog comments. Sure, most of these propositions may fit within an NSDS. I have no axe to grind with NSDS, but think that perhaps that they are a bit vague sometimes. Maybe that is why you and other find it hard to advocate them? It has been since 2000, but despite the attention to development statistics lately, NSDS is not at the tip of everyone’s tongue.
And: If I read the PARIS21 and NSDS mission statements correctly, it is more about enabling ownership of MDG data. That is not the same as discussing how to negotiate global demand. And here, and elsewhere, I talk of the need to really map statistical capacity and inadequacies of data.
“During a meeting hosted by the OECD Development Assistance Committee on the 18th and 19th of November 1999, one hundred high-level statistical and policy officials from developing countries, key international organisations, regional banks and bilateral donors came together to discuss the problems faced by policymakers when statistics are outdated or inadequate, and by statisticians with limited resources and low status in government.”
It is good to discuss the problem, but I urge a more proactive role in diagnosing the problem.
Regards,
Morten
Posted by Morten Jerven | November 8, 2013, 3:41 pmDear Morten,
To me there is no more need to advocate NSDS with countries as shown in the PARIS21 periodic NSDS Status Report. Also most of developped countries have a multiannual programme for statistics. NSDS are first of all national decisions that are not intended to suit every expectation because they can not; there is also a learning process at work and second or third NSDSs are of a much better quality than the earlier ones.
The intention has been very early on to have an holistics (inclusive) approach to statistics developpement. But, for obvious reasons, PARIS21 did not prescribe a best strategy or any preferred strategy; so this might make it looking vague sometimes ; rarely at country level.
There is much more concern with outsiders, be they statisticians or users, as many have a more restricted center of interest and too often a lack of indepth understanding of how difficult producing statistics in a developping country could be and really is; some can not even figure out why it is so important to abide by the national statistical legislation, much less to make great effort and take time to improve it.
As you know the UN working group on monitoring and indicators was established in September 2012 and released its report ‘Statistics and indicators for the post-2015 development agenda’ on july 2013. Some elements of a diagnose can be found in it.
On can read:
-Agency work with countries is still stove-piped to agency counterparts in countries which are often outside national statistics offices, such as labour, education, health and environment ministries, leaving little opportunity for strong coordination at the national level in MDG indicators compilation.
-Donor coordination is inadequate both to prevent overlap and duplication and to promote effective harmonization of collection programmes. Strengthened participation of donors in whatever mechanism might be set up to compile statistics in the post-2015 agenda should also be considered.
– It is understandable that in most developing countries the costs of integration and harmonization across disparate data sources and responsibilities are high and the immediate benefits in meeting current demands are less compelling.
– A broader, more inclusive global work programme for technical cooperation in statistics post-2015 should take account of this concern, drawing on the work of PARIS21 in developing individual strategies with countries for the development of national statistics ……. and greater support for implementation of these strategies.
A global strategy and a global work programme would be natural steps following a diagnose; I trust we are heading to that !
Gérard
Posted by Gérard Chenais (@GChenais) | November 11, 2013, 4:43 pmGerard,
I could not agree more with you. We are dealing with a situation where, perhaps, we have not been able to explain well what the National Strategy for the Development of Statistics (NSDS) is and its potency to transform National Statistical Systems. I see nothing new in the talk about the “data revolution”. There is also nothing new or fundamental in the five propositions made by Morten.
I contend that a well designed NSDS properly mainstreamed in national policy and planning processes and properly implemented, has the potential to engender a “data revolution” in our countries. I, therefore, wish that instead of conjecture on how to deliver the “data revolution”, we focus on how the NSDSs are designed and implemented in the countries.
Africa is experiencing a statistical renaissance and the NSDS is fueling this renaissance. Therefore, talk about a “statistical tragedy” or even “poor numbers…” is not well-informed.
Posted by Prof. Ben Kiregyera | November 6, 2013, 5:19 pmDear Ben,
I was hoping to discuss this with you in Paris at OECD and PARIS21 last spring, but it never came through. http://mortenjerven.com/poor-numbers-paris21-and-the-oecd-development-centre-on-friday-17-may-11-13h/
Hopefully we can discuss this in full soon.
Just two notes.
First. I do not think you mean that any talk of a ‘statistical tragedy’ or ‘poor numbers’ is per definition not informed. I think that we all agree that there is a knowledge problem in development statistics. The size of it, and the timing of it is a matter of discussion and analysis.
Second. How is your book – “A Statistical Renaissance in Africa” coming along? I will be curious to see how you define your terms, because usually the term ‘renaissance’ is used to mean a rebirth – after dark ages, or neglect – and signifies a return to a golden age. So. The way I read it your use of the term ‘Statistical Renaissance’ implicitly presupposes a period that was not so golden… you don’t want to call it a statistical tragedy, and you do not want to call it poor numbers. What do you suggest we call it Ben?
Posted by Morten Jerven | November 8, 2013, 3:54 pmDear Morten,
Are you trying to belittling those responding to your blog entry. Should we follow your “five fundamental extraordinary awesome propositions” and forget about NSDS. NSDS is a strategic plan. It is sad that some are misled by your shallow book. I did not get anything new in your book. Renaissance mean a rebirth… it is not ‘used to’ it has still the same meaning. I look forward to reading Ben’s book.
Posted by Jack Agunta | November 12, 2013, 12:26 pmMoreten Jerven shares so obvious and so ignored points about data! Most of the problems he tackles permeate the (mis)conceptualization of M&E “systems” and subsequent Knowledge Management ideas. Ah, by the way. I have seen in many instances such focused “strategies” (papers?) become narrow as they run the risk of not addressing (it cannot) and integrating other elements of the managerial system. In Portuguese, for instance, when one translates”MIS”, it’s to say a “system to manage information”, and not, as it is supposed to be, an “information system to support management”. As a practitioner, I’d love to see everyone supporting the concept of “optimum ignorance”. As Jerven puts it: ‘What kind of development are we able to monitor’?
Posted by Valeria Salles | November 7, 2013, 12:17 pmMorten Jerven’s call not to repeat past mistakes is very welcome indeed, but I would question whether the solution can be entirely supply-driven. The debate does not just centre on what is technically feasible for those generating and using data, but also what is meaningful for the lives of people who will be most affected by the data – or indeed the lack of data. A perfect illustration is the data collection and analysis on the situation of older people.
No one can deny the huge demographic transformations taking place across the globe [http://tinyurl.com/acqxlgo] – the world is ageing – yet data collection and analysis on the situation of older people remains a statistical blind spot. UNAIDS, to its credit, recently recognised this and has produced for the first time regional data on the incidence of HIV in people aged 50 and over [http://tinyurl.com/olf3z4f]. While this data isn’t exactly ‘granular’ it is a huge step forward in recognising that ageing must be taken into account in international development.
To learn from the mistakes of the MDGs, we cannot be satisfied with implementing what we have better, because what we have does not include people in later life. Gathering and using data on older people is possible as has been illustrated by the Global AgeWatch Index [http://tinyurl.com/bo29j2a]. The evolution of our data collection and use must therefore be informed sometimes by the non-technical aspirations and realities of the people we are trying to help. This is why the coupling of the data revolution with the transformative shift “leave no one behind” is so very powerful and compelling and has the potential to change the landscape of development.
Posted by Ken Bluestone | November 8, 2013, 5:26 pmMorten generalizes to sub-Saharan Africa by just citing a handful of countries. Is that a good sampling frame :), I found it very shallow and behavior of Wistle blower. I would be happy if Morten concentrates in a couple of countries and have a depth view of what is going on. Call it poor number: a case of x, y, z countries. I am worried when you say “I suggest that we do not want to repeat past mistakes. On the basis of the diagnosis provided in Poor Numbers I put forward five fundamentals propositions that need to be taken into account before we embark on a data revolution. If we don’t, then this demand for evidence is going to be counterproductive and do more harm than good.” is it my way or the highway. There are a number of constructive documents done on how to bring about the statistical landscape in Africa. You did not dare to look at them… You are I think in self promotion state-of-affair.
Jack
Posted by Jack Agunta | November 12, 2013, 12:14 pmMy reflections on Morten’s book: Is it poor numbers or poor knowledge of the African Statistical system?
Although, the writer is mainly focusing on data quality; particularly the lack of reliability , accuracy and transparency (and sometimes validity, vaguely introduced and defined by the author), however , unfortunately the author himself has been trapped in this quality problem, since he was occasionally, generalized from his study sample of eight Anglophone countries to the continent as a whole without sharing with the readers the methodology that permits him to generalize, and without providing the tolerance limits of error due to such generalization. Indeed even the choice of this sample is not statistically justified; as such selection was merely done due to a sort of trade-off between breadth and depth as author has announced. Indeed if the choice of Ghana and Nigeria (Western Africa three neighboring countries in Eastern Africa sharing common history (Kenya, Tanzania and Uganda) plus other three neighboring countries in Southern Africa (Botswana, Zambia and Malawi) reduce the diversification within this subjectively non- random sample, hence such deliberate choice has weakened its ability to generalize at least at the Sub-Saharan Africa level, consequently the sample itself is biased. The only merit of this document is that it is suggesting within an interdisciplinary approach of knowledge to introduce the concept of validity within the package of data quality dimensions.
However, the term was loosely defined at a sense to indicate the ability of a state to collect taxes and information, in another sense (the concept of validity is related to whether the measure is accurate, in another context the concept indicates lacking or absent of creditability or integrity of a national statistical office, though integrity/creditability is a data quality dimension, then there is a need for a precise and concrete statistical conceptualization of the definition for the term to be qualified to fit in the statistical literature. Admitting that the author reflected and documented his perceptions, as user of statistics rather than professional statistician, his conclusion is rather pessimistic, since he has attempted to generalize malpractice rather than best practice, this is in turn due to the writer lacking all sided and current stock of information as well as not well informed with literatures devoted to capacity building in Africa ( Immense Work in statistical capacity building including contributions of AUC; AfDB, ECA, UNSD, and country contributions). Due to the above reasoning, most of the titles and subtitles of this manuscript can be renamed in a more precise manner to document case studies and to be organized at best as an article dealing with some aspects of challenges in data quality in selected African countries.
Posted by Ali | November 19, 2013, 1:16 pm