Projektbeschreibung (auf Englisch)

Project of the Swiss National Center of Competence in Research (NCCR)
"Challenges to Democracy in the 21st Century"

In the face of accelerating globalization, increasing inequality and ongoing democratization, a re-examination of the nature of political legitimacy and the functioning of democratic processes is taking place. Within the debates on these developments, the measurement and evaluation of democracy have received special attention. However, the most widely used indices for measuring the quality of democracy in comparative political science, such as the Vanhanen-Index, the Polity-Index and the Freedom House-Index, are being called into question. Three aspects of these indices' conceptual assumptions are being challenged: first, their focus on procedural aspects; second, their exclusive focus on the nation-state; and third, their tendency toward cultural blindness. Thus, these indices do not generate variation among established democracies and encourage unjustifiably self-congratulatory analyses. In order to remedy these deficits, the NCCR democracy barometer will develop an index that will measure the quality of democracy in advanced industrialized societies (OECD member countries) in a substantive and differentiated manner. In order to achieve this goal, the project combines the advantages of qualitative and quantitative approaches to measuring democracy and develops strategies of reaggregation and disaggregation that will allow us to take processes of globalization and "glocalization"quot; into account. The findings and data of the barometer will certainly benefit political scientists, but its critical assessment of democracy as an open-ended process will also be useful for practitioners ranging from political actors to civics teachers.

Problem, objectives, and transferal function

Measuring democracy has a long tradition in political science, dating back to the works of Dahl (1956) and Lipset (1959). These early endeavors to measure democratic systems were often underpinned by a modernizing theoretical perspective. Young democracies like those in Germany and Japan were aiming to become stabilized, and in former colonies new democracies were being established. By the middle of the 1970s, instruments for measuring democracy had been developed and were being widely used - especially the quantitative democracy indices by Vanhanen, Jaggers & Gurr and Freedom House (Gaber 2000), in addition to a few qualitative measurement tools like the Democratic Audit of Beetham (1993). Several factors explain this demand for tools to evaluate democracy: while development assistance programs relied on these indices as criteria for intervention, the third wave of democratization (Huntington 1991, 1997; Shin 1994; Diamond 1993) and the corresponding dramatic increase in the sheer number of (at least nominally) democratic states generated demand among political scientists for a more sensitive instrument capable of making meaningful distinctions beyond the traditional dichotomy of dictatorship versus democracy (Lauth et al. 2000). Although these indices are widely used in comparative political science (Vanhanen 1984; Hadenius 1992, Lane & Ersson 2003), they are increasingly vulnerable to internal and external criticism. Internal criticism encompasses methodological questions of reliability, validity and dimensionality. External criticism is concerned with the cultural blindness of existing indices, as well as their strong focus on procedural aspects which tend to overlook undemocratic outcomes and lead to a lack of variation among advanced industrial societies. Moreover, these indices’ exclusive focus on nation-states seems inappropriate in the context of globalization.
 
The NCCR democracy barometer responds to the concerns of internal criticism by refining the project's theoretical foundations, by examining questions of dimensionality and weighting certain aspects, and by increasing reliability via better data and higher transparency. Furthermore, it takes into account some critics raised by external criticism. In order to be able to make an accurate assessment of democracy, we aim to develop measurement for the quality of democracy and thus the democracy barometer will:

• generate more variation among established democracies by combining the advantages of qualitative and quantitative methods and taking substantive outcomes into account;

• take the criticism of the cultural relativism of democracy seriously by focusing on advanced industrial societies and reflecting on the possibility of culturally determined varieties of democracy. 

By meeting these theoretical and methodological challenges, the NCCR democracy barometer enables transfer in three ways:  

• first, by consolidating the theoretical democracy debate across the NCCR modules, thus contributing to the consolidation of internal theoretical reflection within the NCCR;

• second, by generating knowledge and data which serve as a foundation for teaching materials and courses within the NCCR as well as for the planned teaching tool of the civic education module;

• third and most importantly, in the long run it will contribute regular publication of raw data and analysis. Through this we aim at a critical discussion of and a benchmarking for developed democracies, which will benefit both comparative political scientists and political actors.

State of the Art

There are a large number of indices of democracy in comparative political science and in democracy research. However, since many indices only include a limited number of countries (Hadenius 1992) and/or measures (Bollen 1993; Arat 1991; Coppedge & Reinicke 1991), the most popular and widely used indexes are the Vanhanen-Index, the Polity-Index by Jaggers & Gurr, and especially the Freedom House-Index. These cover over 150 countries and in part go back to the 19th century (for details see Gaber 2000 and Hardmeier 2004):

• The Vanhanen-Index (VI), produced by Tatu Vanhanen, professor at Helsinki University, in cooperation with the International Peace Research Institute, Oslo (PRIO), was developed in 1984. The data, which is available on the internet, includes 187 countries and covers the period 1810 – 1998. Because of its proximity to Dahl's theory of democracy, the Vanhanen-index is also informally known as the "polyarchy data set." Two dimensions are recorded - competition and participation - and aggregated over the following formula: C(ompetition)*P(articipation)/100, 0 to 100 (for details and adjustments see Gaber 2000 as well as Hardmeier 2004).

• The Polity-Index (PI) was developed by Ted Robert Gurr in the 1970s and is now connected to the University of Maryland and Colorado State University, where Keith Jaggers is now in charge of its maintenance. The Polity-Index includes 150 countries which have been covered at different times. For all practical purposes the index is, like the VI, two-dimensional even if its description lists three dimensions (free and competitive elections, horizontal power limitation, and liberty rights). The actual polity-index is based on a subtraction of a value on the autocracy scale from a value on the democracy scale. Thus it results in values ranging from –10 (very autocratic) to +10 (very democratic).

• The Freedom House Index (FHI) was launched by Raymond Gastil of the University of Washington in Seattle. Today it includes 192 countries and 18 independent territories. First, two independent indices are recorded, which also describe the theoretical dimension. The index of political rights consists of ten criteria which are grouped into three parts: electoral process, political pluralism and participation, and government functioning; the index of civil liberties includes 15 criteria which are divided into four groups: freedom of speech, associational and organizational rights, rule of law, and autonomy. Then two more indices are created, with values ranging from 1 (best value) to 7 (worst value). In many publications the mean of the two indices is shown on a rotated scale whereby democracies with values between 1 and 2.2 are considered "free," those between 3 – 5 as "partly free," and those between 5.5 and 7 as "not free." 

These three indices have different theoretical foundations and in particular, different assumptions concerning the core dimensions of democracy. Many comparative analyses have focused on correlation and covariance between the indices (especially Vanhanen 2000 and Gaber 2000). Basically, the indices vary in terms of how close they are to the minimum definition of democracy, which is usually defined by four criteria: the existence of regular, free, equal and general elections; real legislative representation; and separation of powers and the recognition of certain civil rights. In this regard, the Vanhanen-Index (VI) is considered especially parsimonious and strongly oriented towards Dahl's polyarchy model (1971), which defines contestation and participation as the two central characteristics of democracies (see Vanhanen 2000: 185). The Polity-Index (PI) also considers the restriction of the power of the president as well as the access of the opposition. The Freedom House-Index (FHI), with its emphasis on political rights and civil liberties, is furthest from the minimalist definition of democracy.
 
Regardless of these conceptual differences and their various operationalizations, the three indices share at least two key features: first, they do not go beyond a procedural understanding of democracy – an assumption which will be discussed later (see external criticism). Second, they assume that democracy should not be recorded dichotomously but rather in categories and even metrically. While some still question this metric approach from a theoretical point of view (Huntington 1991), methodological objections (Przeworski 1997; Przeworski & Limongi 1997) have recently been convincingly refuted (Elkins 2000). Whereas Przeworski & Limongi have argued that the error variance of measurements of polytomies is larger than that of dichotomies, Elkins has demonstrated that with measurement of polytomies it is not only the error variance which is larger, but the total variance as well. In other words, to assess measurement quality, we should review the proportion of error to total variance rather than focusing exclusively on error variance, and according to this standard, polytomies can be measured more accurately than dichotomies. Hence, we can assume that it is possible to measure democracy on an ordinal scale. In recent times, however, the indices have been facing other types of critiques. While some criticize internal aspects concerning method and process, others formulate external reservations and question the usefulness of the indices themselves.
 
As far as internal methodological criticism is concerned, the reliability of the FHI, and to a lesser extent of the PI, have been found wanting. While the VI draws on objective data from election statistics, FHI data is decentrally recorded and coded. Furthermore, the encoding rules for the FHI are vague, and the data pool is not clearly fixed. The potential for subjective bias is thus present, as Bollen & Paxton's (2000) secondary analysis clearly demonstrates: the FHI's estimated method factor consistently correlates negatively with Marxist-Leninist countries, while predominantly Christian countries and older democracies are systematically better coded. Thus, the VI seems more reliable and thus preferable. However, VI does very poorly in terms of validity. On this point, Parry & Moyser's (2000) criticism of the VI's comparison of participation rates in measuring the quality of democracy makes sense, especially in the Swiss context of low participation. Participation rates cannot simply be compared with each other but must be placed in the context of political opportunity structures for voting, and compositional effects of the voting population should be taken into account when interpreting absolute participation rates.

Between internal and external criticism there are those who question the treatment of democracy as a one-dimensional phenomenon, and who specifically examine the aggregation of scales. While theoretically and nominally all indices presume two-dimensionality, only the VI implements this concept, which Vanhanen achieves with a multiplicative index. Corresponding criticism of the indices is either empirically supported with factor analytical methods (Munck & Verkulien 2002), or there is a stronger emphasis on theory and the issue of the weighting of various aspects of democracy. For example, Elklit (1994:108) concludes that the only comparison possible is on an ordinal but multidimensional scale and that we should thus avoid aggregating different indices into a single case, and instead discuss the weight of different dimensions from an intercultural perspective.

This cultural relativism perspective (Sowell 1994) also underpins the external criticism of democracy indices' general assumption that democracy is a homogenous phenomenon. In keeping with the so-called "relativistic turn" which is currently dominant in the philosophy of science but less prevalent in the social sciences (Hagner 2001:20; Cartwright 1999), the universal criteria that indices use to measure democracy are increasingly viewed as inappropriate by those who argue that democracy can only be evaluated by locally-defined and context-sensitive criteria (Biryukov & Sergeyev 1994; Parekh 1994). The basic complaint is that existing indices generate too little variation because they focus exclusively on the procedural aspects of democracy. In other words, this procedural focus leads to inflated scores and thus encourages self-congratulatory tendencies (Berg-Schlosser 2000), which is unjustified for at least two reasons. First, it fails to conceptualize democratization as an open-ended process or to admit the fact that no country can claim that it is perfectly democratic (Whitehead 2002: 27). Second, it fails to respond to the contemporary concern that under conditions of globalization, established democracies are as much in need of critical assessment as unconsolidated ones (Beetham & Weir 2000: 76). In this regard we should also critically assess whether national indices make any sense at all and what kind of strategies of disaggregation and reaggregation might be able to capture the reality of democracy under globalization. So far, this very fundamental question has barely been raised (for an exception see Dunleavy & Margetts 1994).

Design, Organization and Milestones

The new democracy barometer envisioned in this NCCR takes these criticisms into account. The focus will be on the 30 OECD member countries.
 
The barometer combines the advantages of quantitative and qualitative methods. Conceptually, it is based on the qualitative method of the democratic audit (Beetham 2000, 1993), and also relies on what Schiller (1999) has described as the "qualification of democracy" strategy in order to account for the broad range of concrete forms and stages of democratic development. However, in contrast to some of these authors, we are convinced that it is possible to quantify these qualification criteria of democracy and to generate an ordinal, multidimensional value of democracy, especially where advanced industrialized countries are concerned. Measures to implement this approach include the following theoretical, organizational, methodological and analytical steps:

Autumn 2005 – autumn 2006: refinement of the concepts

In order to evaluate democratic quality and performance, we need criteria that make such an assessment possible. Thus, the first priority is to find a working definition of democracy, which will be no trivial task considering the debates on the theory of democracy (Schmidt 2000). Furthermore, even followers of Schumpeter's standard "minimum" or "procedural" definition only agree on its relevant dimensions, without agreeing on how to operationalize them. Then, in a second step, identification of the defining elements of democracy (i.e. rule of the people, popular control, equality or responsive rule) is on the agenda and will control for their underpinning values and the operative principles identified (Schmitter & Karl 1993; Beetham 1993). Here, in addition to establishing criteria for the minimal definition of democracy, we will also identify additional criteria, as more elaborate models of democracy have done, e.g. the participatory or associative model (Warren 2001), the cosmopolitan model (Held 1994), the reflexive or deliberative model (Steiner et al. 2005; Dryzek 2000), as well as the substantive concept of democracy (Saward 1994, Scharpf 1999). Therefore, one point will receive special attention in this period: the application of theoretical considerations regarding substantive democracy and the discussion of ideals and realization of democracy.

The milestones of this first year will consist of:

• organization of brown bag lunches in cooperation with modules 1 and 2 in order to discuss issues related to principles of democracy and integration of substantive democracy and outcomes;

• workshop with external experts in order to discuss the working definition and the defining elements of democracy as well as the criteria in autumn 2006;

• presentation of the theoretical framework at an international conference.

Autumn 2006 – autumn 2007: test phase

In the second year we will focus on aspects of data collection and processing. The following tasks will be crucial during this phase:

• First, we need to choose a small number of OECD countries in order to test the feasibility of the democracy barometer. We will start by collecting data on Switzerland, Germany (as an old EU member state), Poland (as a new EU member state), Japan (as an Asian country), USA (as a non-European democracy), and Sweden (as a specific type of a welfare state). 

• Second, practical questions regarding access to data and disaggregation and reaggregation will be considered, i.e. whether the quality of democracy is better measured in the local dimension or should include the supra- and international level. 

• Finally, during the actual processing of the data, questions concerning weighting and dimensionality as well as variation will be examined.  

The corresponding milestones are:  

• data collection for the six test countries;

• the launch of a lecture series on "Quality and Measuring of Democracy" with international scholars;

• presentation of the test country data set (including comparison with the existing indices of democracy and analysis of variance among the countries) at an international conference;

• organization of a two-day retreat with external experts.

Autumn 2007- autumn 2008: refinement of data collection and of the coding scheme

The third year will start with the evaluations of the two-day retreat and the international conference and an adjustment of the coding scheme. Furthermore, it will be dedicated to establishing cooperative relationships that will support our data collection efforts for the remaining OECD countries. We will start the data collection for these countries during this phase.
 
The milestones of this phase are:

• to establish further cooperation, and

• to undertake fundraising activities.

Autumn 2008 – autumn 2009: implementation and consolidation

In the fourth year, we will conclude the data collection for all OECD countries. The fourth year will also be dedicated to the publication and dissemination of results. The milestones are: 

• publication of data and findings in peer-reviewed journals: i.e. Journal of Democracy, European Journal of Political Research, West European Politics;

• to set up the Democracy-Barometer website, which will allow for broad dissemination of the data and ensure worldwide accessibility. The website can build on the database run by the ISN (International Relations and Security Network) - a widely recognized and respected web-based knowledge platform in the field of international relations under the auspices of the CIS. With the technical support of the ISN we will be able to launch a project-specific ISN-dossier;

• organization of a two-day retreat with external experts.

The following years will be dedicated to deepening the analysis and broadening the data set with respect to the time period and the countries covered. Political Representation in Western Democracies"

Significance, benefits, and target audience

The democracy barometer project will contribute to both the theoretical and methodological development of the current debate on measuring democracy. At the theoretical level, it will offer an instrument capable of producing a differentiated assessment of established democracies in OECD with qualitative methods for the evaluation of democracy. This instrument will integrate dimensions not yet included in existing indices (globalization, glocalization, mediatization, outcomes). Methodically, the project's importance lies in the quantification of criteria which so far have only been evaluated qualitatively. These two contributions will enable the development of a long-term comprehensive and differentiated data set for OECD countries which will be useful for researchers in a variety of areas, but particularly for research on democratization and democratic theory as well as comparative government and public policy. Accordingly, the academic target group is large, and includes philosophy, social geography and developmental sociology in addition to political science.

The democracy barometer will also be useful for political actors in governments and administration, as well as for interest organizations and NGOs. It allows for a benchmarking with other democracies and a critical observation of democratic development over time – a task which has become more pressing for established democracies since September 11th, when the fight against terrorism suddenly took precedence over basic democratic guarantees, especially civil rights. Finally, the democracy barometer should also be of interest to other regions of the world, and our ongoing critical reflections on the problem of ethno-centricity will make our findings more generalizable to non-Western countries.