In the accompanying post to this one, I showed how the volume of health, and particularly malaria data, collected in Papua New Guinea (PNG) through the country’s National Health Information System (NHIS) has been steadily increasing over the past decade or so, but that it was not being effectively used. In this post, I conclude the story with an examination of four questions: Why isn’t the data used? What are the costs of excessive data collection? Why has the data system been designed to collect too much data? And, what can be done about it?

Why isn’t the data used?

There are many reasons why NHIS (and other health-related) data are little used. First, there may be a lack of capacity to prepare, analyse and interpret the data, particularly at the provincial level and below. Second, there is limited guidance on how to act on certain observations and, in most instances, there would not even be any direct operational relevance in the detailed disaggregation provided. Third, financial resources at subnational level may be too constrained to readily act on new findings and in the specific case of malaria, the control efforts may be seen as a vertical national program.

What is more, key stakeholders may even have difficulty accessing routine NHIS data either for technical reasons (e.g. internet connectivity at the subnational level, or because the set-up of the user interface does not easily allow the extraction of primary data for analysis), or due to difficulty in obtaining permission to access the data. Yet, if data is meant to inform decision-making (and ideally improve accountability), it should be easily accessible for anyone who has a valid reason.

What are the costs of excessive data collection?

Further, there are valid concerns over the impact of the increased reporting burden on data quality. A review of ‘Monthly Report’ data shows, for example, that in the electronic NHIS database malaria indicators on page 1 often do not add up with the indicators on page 2 from which they were supposedly derived. As all indicators are digitised anyway, it is questionable why health workers have to add them up manually. Anecdotal evidence and personal experience suggest that health workers have in the past had quite different interpretations of how forms should be filled. Besides, the brunt of the work of collecting and reporting the data is borne by a health workforce that is understaffed, ageing, busy with managing patients in very challenging environments, and who – alongside the communities they serve – rarely benefit from all the data they collect. How likely is it that sufficient attention is then paid to reporting requirements when the forms become more and more complicated to fill? Of course, the concerns about quality reduce the likelihood of the data being used, creating a vicious cycle.

Merely satisfying a thirst for data – without prospect for tangible local benefit – does not justify burdening hundreds of health workers with additional data collection tasks. It is important to realise that changing reporting forms, and increasing the amount of information collected in the forms is far from costless. It requires resources to train and continuously supervise those expected to fill the forms, and it poses a growing burden on health workers, who are a very limited and precious resource (in PNG and elsewhere). The opportunity cost of collecting data that remains largely unused may include reduced time spent on clinical duties, outreach activities, or other essential administrative tasks such as drug stock management.

Why has the data system been designed to collect too much data?

So, if the ever-increasing amount of data collection by the PNG health information system may be doing more harm than good, why then is so much data collected?

The basic answer is that development partners and international organisations are fond of data. Data is generally seen as paramount to accountability. But the focus is too often on quantity, not quality.

Initiatives by the World Health Organization (WHO) have contributed to health data across the world becoming more standardised and hence more comparable. But WHO guidance on core indicators and their required disaggregation suggests a level of detail in routine data that is difficult to reconcile with the capacity of health information systems in many low- and middle-income countries.

The Global Fund to Fight AIDS, Tuberculosis and Malaria, the major funder of malaria control in PNG and globally, uses specific malaria indicators to track progress, such as the “proportion of confirmed malaria cases that received first-line antimalarial treatment according to national policy at public sector health facilities”. This indicator, if assessed through the NHIS, requires recording treatment administration stratified by diagnostic method (i.e. diagnosed based on clinical signs and symptoms, or using a laboratory confirmation such as a rapid diagnostic test or microscopy). Since, in addition, cases are recorded separately by age group, sex and whether they are an outpatient or inpatient, recording the monthly number of administered treatments requires filling 28 fields in the ‘Monthly Report’ form. This is required every month at 800 health facilities. For one reporting indicator.

What can be done about it?

The way forward is tricky. It is comparably easy to add indicators and data collection mechanisms to a health information system, but more difficult to remove an indicator. To date, initiatives to strengthen the health information system in PNG have not managed to disentangle the complicated web of partly overlapping, partly complementary, data collection systems. Instead, some weaknesses are patched-up by adding new and independent data collection mechanisms, such as a mobile-phone-based system for reporting of disease outbreaks directly to the national level. While new and innovative tools may have clear advantages (such as increased timeliness and accuracy), they should be embedded in an overall coherent strategy and ideally be integrated into a single electronic platform that stakeholders at national and subnational levels can access.

The focus should be on collecting “minimal essential data”. Routine reporting systems should collect a limited set of key indicators and for each of them, there should be a solid justification. The indicator definition should be guided by the World Health Organization, and the selection must consider legal obligations under the International Health Regulations. But a country should consider carefully which information it requires for decision-making and which data has to be reported from every single health facility for every single month. Data that is not required at this frequency or resolution might rather be collected in sentinel surveillance sites (where adequate resourcing and data quality can be ensured), through surveys, or research studies. In the case of malaria data, one might, for example, consider abandoning age group stratification in the NHIS and collect age information in selected surveillance sites, such as the ones maintained by the Papua New Guinea Institute of Medical Research (PNGIMR). This way, the burden is shifted from the clinical health workforce to dedicated specialists tasked with informing the government.

The story of the collection and use of malaria and health data in PNG is far from unique. A disconnect between health data and certain public health decisions can be found in countries at all income levels. But particularly in over-stretched and underfunded health systems, donors need to step back, and governments need to rethink. Of course, data is crucially important – but with the right balance between quantity and quality. Most importantly, a health information system must be owned locally and designed in a way that is consistent with the country’s capacity to operate it and utilise the data for improved programmatic decision-making.

This blog is part of a series. You can find the first blog here.