Finding Substance In Popular Reports

Country-level reports and surveys are discussed in newspaper articles, social media, and talk shows on television almost every other day. These reports could be regarding the state of human wellbeing, corruption, governance, ease of doing business, and maybe about willingness to get a shot of Covid-19 vaccine. The reports spur debates and most of the time the results are either interpreted out of context or beyond reasonable inference. In many country-level reports, aggregate data is used for building matrices and indicators to rank countries. The evaluation on any of the popular indicators or matrices should follow a sequential list of questions like; what is being measured, why in the first place there is a need to measure it, what sort of indicators are used for measurement, what measurement mechanisms have been adopted, are the indicators exhaustive, and finally how; it may be used and how it cannot be used. Different checks can be used for developing a judicious opinion and subsequent policy advice. First, several unauthentic sources provide fake matrices about economic and social indicators. Social media helps in the rapid propagation of such fake metrics which do not represent the true state of society and economy. The first check that every prudent analyst must undertake is to verify the source, use counter-checks and avoid do data that does not pass the tests Second, though some indicators look representative but are still poor indicators, for example, per capita income as a measure of poverty or absolute amount of debt as a proxy of poor economic performance. Blindly using such indicators in forming an opinion or in the design of any policy could be catastrophic. It is better to always seek better indicators or look for supplementary data. The use of triangulation is not an option but a necessity in such cases. Third, several if not all, indices produced by international organizations are often misunderstood which leads to a naïve commentary on findings. For example, Pakistan stands at the bottom of the Global Innovation Index. Now looking at this index, one would think that Pakistan has a very poor scientific output. But, innovation, as defined in the Global Innovation Index (GII) is ‘capability to adapt innovative technology’. The construction of GII involves several, input and output indicators. Inputs largely relate to human capital, institutions, infrastructure, market, and business sophistication. Scientific output relates to indicators like online creativity. The rank of any country in GII is an average of its rank in scientific outputs and governance indicators. Poor performance on governance indicators may outnumber unit development in scientific outputs leading to low rank on the GII. The indices and therefore the interpretation of the indicators can be improved by identifying the design glitches, normalizing the data, and opening a meaningful conversation on the facts that matter. Fourth, check for the measurement mischiefs. Pakistan ranks 2nd in the world on the number of out-of-school children and lowest in South Asia. Interestingly this low rank is owed to the fact that data was based on registered institutions only and also excludes children studying in madrasas. It is always advisable to dig deeper and identify the measurement mechanisms and procedures used to construct such rankings. In the first place, look for the definition of the variable under consideration and you will find what is being measured. Now evaluate it on the elixir of prudence and build advice. Fifth, recheck the representativeness of an indicator. There are instances where one can reasonably believe that the indicator is functionally representative. Still, even in such cases, the results need to be reflected upon in greater detail to develop policy implications. A classic example is of corruption perceptions index (2019) where the ranking of Pakistan created a massive media debate. In this case, where perceptions seek to represent reality, any analysis must first dig down the anatomy of perceptions. How these develop and progress over time before posing any analysis on the subject. But, to complete the picture, see the other side, that is, understand what is corruption, how it is manifested in a society and what happens when any state-led effort is made to curb it, what are the initial, latent, and lagged reactions of such interventions. Now it is time to look at the results and take a position and share your analysis. The idea is to go back to the books, dig deeper in theory, seek empirical evidence, and use common sense to bang with your analysis.