Did we forget Goodhart’s law in our fight against the pandemic?
This is the 7th post of our COVID-19 Series.
An emergency tests a Government’s response to unforeseen situations and the current pandemic is definitely a litmus test of our Government’s approach towards decision making. Although the Indian Government was relatively quick in implementing travel restrictions and calibrated airport screenings followed by the lock-down, a few dots seem to be missing.
Lockdowns of varying intensity have been implemented around the world and data indicate that they have been instrumental in slowing the spread of Covid-19. In this context, the implementation of the lockdown was an imperative step adopted by the Government. However, while the lockdown is in effect, we cannot turn a blind eye to the functioning of the healthcare system during the pandemic.
As per the international ‘best in class’ standards, India’s healthcare system was not prepared for an emergency of this scale. However, with several weeks having passed by, we have developed more testing centers and are in the process of procuring additional testing kits. Ironically, we still haven’t seen an uptick in the overall levels of testing across the whole country. With the exception of the Bhilwara village in Rajasthan and the state of Kerala, we do not see a proactive approach towards aggressive testing by the Government.
Globally, as per the advisory from the World Health Organization, countries are migrating towards an aggressive testing model. However, for India, even though the testing capacity has been ramped up over the past few weeks, there seems to be a poor culture of testing. As per the Indian Council for Medical Research (ICMR), the number of tests conducted fall below the potential testing capacity of the country. Hence, it appears that the problem is not really about the capacity. Epidemiologists and medical practitioners have all pondered on the issue but to no avail. How do we then explain such a behavior of the Government? In this piece, the author attempts to explain the response of the Government through the laws of behavioral economics.
“The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
It is well established that, when the sole parameters of measuring effectiveness are quantitative indicators, the people involved have a greater tendency to demonstrate less ethical behaviour, thereby producing undesirable and ineffective results. This is also termed as Campbell’s law.
Campbell’s law is pertinent in analysing the policy measures of the Indian Government. An epidemiologist working with the global health organization mentions that it is deeply concerning that the Government is reluctant in conducting tests and amending the testing policy, because it wants to keep the numbers low. Essentially, the number of tests conducted are the quantitative factor to measure the spread of the virus, as referred to in Campbell’s theory, and the Government has determined a way to ‘game’ the metric by conducting fewer tests.
An early reference to the concept of ‘gaming the system’ was made in the Wells Fargo Cross Selling Scandal. This case in turn forms the basis of Goodhart’s Law. In fact, Campbell and Goodhart’s laws are closely interlinked. Campbell’s idea of how a metric system incentivises people involved to adopt less ethical behaviour is akin to Goodhart’s concept of ‘gaming the system’.
Goodhart stated that, “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” A simpler version of Goodhart’s law is that, “When a measure becomes a target, it ceases to be a good measure.”
According to Goodhart, the system underneath you transforms with the very use of the metric measuring the effectiveness of the system. That is, the moment we say that the quantitative number of tests act as a check against the spread of the virus, testing fails as a measure to contain the virus. This is primarily because, people predict the outcome and consequently, find a way to alter the policies determining the outcome.
Essentially, if the performance of a country in the fight against the pandemic is indicated by the extent of spread of the virus in that country and the containment of transmission by the Government, this indicator of performance ceases to function, as those handling the data in the economy begin to ‘game’ it.
Goodhart further stated that metrics reduce dimensionality. This implies that a single figure in a metric system is not representative of the entire system, and thus does not qualify to be a good measure. The outcome based on this dimensionless metric ignores several other factors which are equally integral to achieving that outcome. It is, therefore, important to understand that conflating a factor with a metric results in failure to achieve the underlying objective of the test.
For instance, we often mistake the number of tests conducted to be representative of the widespread nature of tests being performed by a country. However, data on testing suggest that figures demonstrating the number of performed tests do not coincide with the figures of the number of individuals tested for Covid-19. This is because a patient previously tested positive for Covid-19 is subsequently tested more than once. Hence, the number of tests conducted fail as a measure to determine the extent of spread of the virus since they are not a true indicator of the number of different individuals tested for it.
Challenging Goodhart’s proposition, Douglas Hubbard mentioned in his book that “no matter how ‘fuzzy’ the measurement is, it’s still a measurement if it tells you more than you knew before.” However, as per Goodhart, the first step to developing a useful metric is trusting the data used for the measurements. If it is difficult to verify this data for the purpose of our goal, any measure grounded in this data, does little to ameliorate the situation. In such a case, the measurement comes at the expense of sound judgement or merely to cover-your-rear.
Looking at the testing policy of India, even almost two months after the first case was detected in Kerala on 30th of January, the testing policy specified testing only for people who have a travel history to affected countries and who were showing symptoms. This policy was amended later to include those who may have come in contact with people having such travel history and was further amended at the end of March to include testing of all patients admitted with severe respiratory illnesses.
However, even as we are weeks into fighting the virus with death tolls increasing by the day, India’s testing policy doesn’t include mass screening despite clear evidence of the presence of several hotspots in the country. Although a few states have sealed these hotspots to prevent the spread, aggressive testing has still not been adopted as a necessary measure. This is primarily because our data show that the number of cases detected are not as high as in the third stage of community transmission. At this juncture, Goodhart emphasizes how essential it is to question the credibility of the data before you trust the system. Understanding what these published numbers on testing mean is foremost to trusting the system and when our testing policy fails to even encompass all dimensions of the metric in the first place, the metric automatically becomes biased and the system no longer remains fair.
What is worse is that we are turning a blind eye to the ground realities of the outbreak! The concern about the spread of the virus is greater than is put out there. This is not just a fight against a pandemic but also a race against the clock. And as deeply upsetting as it may sound, we have lost out on crucial time to prevent a mass outbreak by failing to implement the right measure at the optimal time. Not only did we forget Goodhart’s law but we also didn’t learn from the patterns in other countries. A number of cases detected in South Korea were asymptomatic, and many of them were unusually among the youth. Early mass testing in India, similar to South Korea, could have played a huge role in preventing community transmission where the current testing policy, which doesn’t include mass screening would fall apart. With the virus entering the slums of Mumbai, it is almost impossible to believe that where millions of poor reside in unhygienic and crammed-in conditions, we have still not entered the third stage, of community transmission. In fact, the ICMR has finally hinted that community transmission may have happened in certain areas.
However, in this bliss of ignorance, the real problem demands our immediate attention. Amidst the Government gaming the system, we may be applauded for containing the situation well as the figures on the board remain low, but it is impossible for India to fight this pandemic, blindfolded. Irrespective of the number of tests done, the spread of the virus neither halts nor slows down. Eventually, people get infected, suffer and spread, without being detected. In the end, we achieve the goal of saving our face at the expense of having a large population which is affected but remains undetected. It increases the chances of transmission and we also end up losing out on crucial time that could save a lot of lives. Goodhart’s law mentions, that while gaming the system may be the shortcut to success, in the long-term, the system collapses because of that very game. Hence, unless we take immediate action, the situation is bound to get worse.
India has the second largest population in the world and given that our population density is even higher than China, we are in troubled waters. It is, therefore, high time that we reject denial as a strategy. The need to acknowledge the real situation and stop gaming the virus is foremost to collectively fight this pandemic. Acknowledging the situation is the most important step towards making the right investments required to tackle the consequences, adequately. In the context of our country, widespread and aggressive testing for Covid-19 will not only act as a reality check but will also help India choose the right measures essential for responding appropriately and timely.
This is where we all need to remember and repeat Goodhart’s law. To reach the desired output, you must begin by implementing the right measures. India currently ranks among the countries that are least affected by the corona virus and it has been stated that the situation in India is not as severe as the rest of the world. If we sit back and trust this, thinking that while the lockdown is in effect, everything is in control, we are yet again, forgetting Goodhart’s law.
The poor culture of testing, the metric of testing lacking dimensionality and the delay in implementing the correct testing policy in India is the reason for the apparent success of the Government in containing the spread of the virus. In reality, we are far behind the curve and have a long way to go in this battle against the pandemic. In fact, this apparent success of the country’s Government is not despite the lack of testing, understating the ‘individuals’ tested and incorrect policy decisions but primarily because of it.
Shivangi is a student at the National Law School of India University, Bengaluru.