Opinion | V-Dem's Democratic Diagnosis For India - Insight or Oversight?

When March arrives, the V-Dem Institute graciously assumes the mantle of the 'democracy weatherman,' offering us its annual prognostication on the state of our democratic ethos. The 2024 dispatch from V-Dem enlightens us yet again that India, in their esteemed assessment, continues its tenure as an 'Electoral Autocracy'. The institute eloquently describes this as a milieu where, although the spectacle of multiparty executive elections is upheld, the essence-freedoms of expression and association, along with the integrity of elections-remains wanting.

Yet, one might wonder, who anoints these arbiters of democratic classification? 'A conclave of 'experts', they say. But are these arbiters swayed by particular doctrines, or do they pledge allegiance to a specific ideological banner? The clarity on this remains as elusive as ever. In any case, why consider the tangible strides in welfare - be it access to potable water, electricity, housing, health insurance, clean cooking fuel, or toilets? After all, isn't the common populace's perspective but a whisper against the thunderous decree of the 'experts'?

Double Standards

Yashwant Deshmukh and Sutanu Guru's analysis sheds light on the disconcerting parallels between Canada's seemingly draconian Online Harms Bill and India's portrayal as an "electoral autocracy" by entities like V-Dem. This juxtaposition starkly illustrates the double standards in the global discourse on democracy. The scenario reveals a broader narrative fraught with bias, where Western democracies are often given leeway while countries like India face harsh scrutiny, regardless of their internal advancements or the complexities of their democratic processes. The situation highlights a pervasive imbalance in the international dialogue on democracy, underscoring the need for a more nuanced and equitable approach to evaluating and understanding the state of democratic governance worldwide.

Regardless, it is prudent to discuss the methodological issues with how V-Dem measures democracy. The V-Dem project, with its 4,300 country experts, faces significant challenges in measuring democracy accurately. Despite the seemingly large number of experts, their distribution across numerous countries and various dimensions of democracy means that only a fraction are knowledgeable about any given area. This can lead to superficial assessments, particularly for less-studied regions or specific aspects of democracy. Furthermore, ensuring comprehensive coverage and depth of expertise is complicated by the complex nature of democracy itself, which includes a wide range of elements like political participation, rights, and governance. These challenges are exacerbated by the inherent difficulties in maintaining consistency and comparability in evaluations across different national contexts, which can affect the reliability of cross-national comparisons.

On Expert Bias

Technically, quantifying and adjusting for expert bias is a critical challenge. Employing statistical methods such as the Bayesian hierarchical models to assess individual expert biases systematically involves making strong prior assumptions and conducting robust sensitivity analyses. This is necessary to prevent the introduction of new statistical biases. Additionally, estimating unobserved democratic attributes, as V-Dem does, necessitates accurate estimation of measurement error and expert reliability. These factors are essential for the validity of the inferences drawn from the data, necessitating the implementation of methods to assess expert judgments' internal consistency and temporal stability.

The selection and diversification of the expert pool are paramount to mitigate biases and ensure balanced views. Considering geographic, political, and academic diversity, a statistically representative sampling strategy is crucial. However, creating a composite index that captures the multifaceted nature of democracy without oversimplification is challenging. Techniques like factor analysis or principal component analysis must be carefully applied to reflect the conceptual breadth and depth of democracy accurately.

A Method Reminiscent Of 12 Angry Men

Finally, the V-Dem project's iterative estimation process, which refines and re-estimates expert judgments, presents its own challenges. Ensuring the model's convergence and stability is critical and requires continuous monitoring and validation. Providing data in various formats to cater to different analytical needs necessitates a high level of statistical literacy among users, further complicating the project's implementation and application. Addressing these technical and methodological challenges is essential for the project to produce reliable and valid measures of democracy.

The V-Dem project's methodology, reminiscent of the deliberations in 12 Angry Men, illustrates the complexities of assessing democracy globally. In the movie, jurors need help to reach a consensus, reflecting the project's challenge of compiling assessments from a vast yet uneven spread of experts across different countries. Just as the film showcases the jurors' biases and changing perspectives, the V-Dem project grapples with expert bias and the difficulty of maintaining consistent and comprehensive evaluations of democracy's multifaceted nature. The iterative process of refining and re-evaluating expert judgments in the project mirrors that of 12 Angry Men.

A recently published empirical study by Andrew T. Little and Anne Meng critiques the methodologies used to measure democratic backsliding, focusing on the V-Dem index's reliance on subjective indicators and expert judgments. They point out that these subjective measures, influenced by individual perceptions, media narratives, and the political climate, are prone to systematic biases that could exaggerate the degree of democratic decline. The distinction between objective indicators, like election results and legislative turnovers, and subjective assessments, such as the perceived fairness of elections, is highlighted, with objective metrics often showing stability or improvement in contrast to the decline indicated by subjective evaluations. This discrepancy leads to questions about the accuracy of methodologies in reflecting the true democratic state.

Local vs Global

The paper also delves into coder bias and the potential for changing standards in expert evaluations, which may affect the consistency of democratic assessments. It suggests that objective measures might not fully capture political leaders' strategic actions to undermine democracy, as these leaders adopt subtler strategies that are hard to detect. Methodological challenges are underscored, alongside a call for more refined measures to identify overt and subtle democratic backsliding accurately. Furthermore, the analysis distinguishes between local and global democratic trends, emphasising the need for localised studies to uncover specific instances of democratic erosion that broad indices might overlook, thus advocating for a nuanced approach in assessing the health of democracy across different regions and countries.

V-Dem recently came up with a working paper defending its methodology and the presence of expert bias. However, its approach to evaluating expert biases in measuring democracy highlights several methodological and conceptual challenges. If one goes through the paper, one can gauge how temporal proximity to political events may influence experts' judgments, making it difficult to discern whether changes in ratings reflect real political shifts or biases due to the immediacy of events. This concern is compounded by the fact that V-Dem relies on expert assessments in areas where direct, observable measures are unavailable, complicating the assessment of potential biases.

An Unconvincing Argument

One argument made in the paper is that if a "bad vibes" bias uniformly influenced experts, one would expect a widespread decline in democracy ratings across countries. However, V-Dem data apparently do not support this, showing significant democratic erosion in a minority of countries. This is used to argue against the presence of a pervasive negative bias. However, the validity of this conclusion can be questioned because the absence of uniform decline does not necessarily negate the presence of bias. Biases could manifest in more nuanced or variable ways, depending on the context and other factors influencing expert judgments.

Moreover, the analysis attempts to differentiate between actual changes in democracy and changes due to expert bias by examining how experts update their scores over time. The finding that a small percentage of experts changed in a more pessimistic direction is presented as evidence against a systematic bias. However, this approach may need to be more balanced as it assumes that biases should lead to consistently more negative ratings, disregarding the possibility of more complex, non-linear, or varied effects of biases on expert judgments.

Bias Is Not Simple

The discussion on the synchronisation of changes in democracy ratings and the use of statistical measures to argue against the increasing systematic bias over the last decade also seems to overlook the potential for more subtle or complex manifestations of bias. While the analysis reports increased dispersion in ratings over time, suggesting a diversity of views, this could also indicate other underlying issues, such as varying levels of access to information or differing interpretations of democratic quality, rather than a straightforward absence of bias.

Finally, the attempt to explore ideological bias through expert disagreement reveals the intricate challenge of identifying bias. The fact that significant disagreement is found only for highly subjective indicators, and more so in recent times and less democratic contexts, hints at the nuanced ways in which biases might operate. These disagreements could reflect deeper differing perspectives on democracy and its components.

Casting Doubts Over Citizens' Agency

India actively engages its citizens in the political and civic spheres, supported by institutions that promote participation and safeguard rights. Despite the V-Dem Institute's classification of India as an "Electoral Autocracy", the nation's democratic credentials are robust, as seen in its adherence to the principles of polyarchy. Regular, free, and fair elections, inclusive suffrage, the right to run for office, freedom of expression, access to diverse information sources, and a dynamic civil society all point towards India's democratic vitality.

Labelling India as an "electoral autocracy" undermines the democratic engagement of its nearly 987 million registered voters, set to participate in the forthcoming general elections. It dismisses the complex democratic processes in a country with a rich history of diverse, multi-party elections. Ultimately, it overlooks the active involvement of citizens in shaping their future, thus diminishing the significance of their collective decision-making power.

(Bibek Debroy (x: @bibekdebroy) is Chairman & Aditya Sinha (X: @adityasinha004) is OSD, Research at the Economic Advisory Council to the Prime Minister.)

Disclaimer: These are the personal opinions of the author