Finding Meaningfulness in Indicators of Country Level Ranking Reports

The amount of data available in the world for inquiry is increasing exponentially. At the same time, the relevance of evidence-based decision making is also increasing. Many country-level reports use aggregate data for building metrics and indicators to rank countries. Often with the publication of these reports, many media debates occur and we draw different inferences from the same empirical evidence. It is important to understand how to interpret data in such ranking reports and then one builds an argument or advice that is aligned with the philosophy of “evidence based”.

Given the importance of these indicators in the socio-economic landscape, for better interpretations, the individual researchers, policy makers and opinion leaders need to identify and address several problems with the nature and application of these metrics and indicators. The following is an evaluation of different type of indicators. Subsequently, a user guideline is provided for more meaningful analysis and advice.

The insight evaluation on any of the popular indicators or matrices should follow a sequential list of questions like:

• what is being measured?
• why in first place there is need to measure it?
• what sort of indicators are used for measurement? 
• what measurement mechanisms have been adopted?
• are these indicators qualifying rigor and exhaustiveness?
• “what matter most is accounted for.”
• ask questions about what it does and does not mean and finally,
• how it should be and not be used.

When Indicators Are Fake

There are several unauthentic sources providing fake metrics about economic and social indicators. Social media helps in fast propagation of such fake metrics which do not represent the true state of society and economy.

Much cannot be done about social media propagation of false metrics, except verifying the source and counter checking. Do not let such data be used in your analysis.

When Indicators Are Not Really Representative

There are several metrics which are very poor indicators of the target variable(s). For example, per capita income is a very poor proxy of poverty or absolute amount of debt is often cited as a proxy of poor economic performance to name a few.

Using such indicators in the design of any policy could be catastrophic, always seek better ones or look for supplements to model a more representative matrix.

When Indicators Have Design Glitches

Several, if not all, of the indices being produced by international organizations are miss-understood. This creates a naïve commentary on their findings. For example, Pakistan stands at bottom end of the Global Innovation Index listing. By reading this statement, one would think that Pakistan has very poor scientific output. But actually innovation, as defined in Global Innovation Index (GII) report is ‘capability to adapt innovative technology’. The construction of GII involves several input and output indicators. Inputs largely relate to human capital, institutions, infrastructure and market and business sophistication. Scientific output relates to indicators like online creativity. The rank of any country in GII then is actually an average of its rank in scientific out puts and governance indicators. Meaning that poor performance on governance indicators will always outnumber unit development in scientific outputs that might be happening in a greater frequency.

In such cases, identify the glitches, normalize the data and start talking about meaningful conversations on the facts that matter.

When indicators Have Measurement Mischiefs

Pakistan ranks 2^nd in the world on Out Of School Children (OOSC) numbers and is at 142 out of 144 countries and lowest in South Asia. Interestingly this low rank is because data was based on registered institutions only and also exclude children studying in madrasas.

For evaluating such instances, dig deeper and identify the measurement mechanisms and procedures used to construct such rankings. In first place, look for the definition of the variable under consideration and you will find what is being measured. Now evaluate it on the elixir of prudence.

When Indicators Are Representative

It is very difficult to gauge that what indicator is truly representative. Even GDP and HDI being the most popular ones might fail on a conservative criterion. Sometimes we can reasonably believe that an indicator is functionally representative enough to move forward. Or there may be cases where you cannot ask questions about its legitimacy even, the results need reflection in greater detail, or one needs to identify policy implications. A classic example is of corruption perceptions index (2019) where ranking of Pakistan created a massive media debate.

Here, where perceptions represent reality, any analysis must first dig down to the anatomy of perception, how is it built and progresses over time before posing any analysis on the subject. But, to complete the picture, see the other side as well; i.e., understand what corruption is, how is it manifested in a society and what happens when any state led effort curbs it. Also, what are the initial, latent, and lagged reactions of such interventions. Now it is time to look at the results and take position and share your analysis.

Go back to the books, dig deeper in theory, seek empirical evidence, and use common sense to put forth your analysis.