Contesting commensuration: Public response tactics to performance evaluation of academia

Characteristics of commensuration

Various forms of ratings, rankings and performance evaluations constitute a prominent, almost taken-for-granted part of today’s society (Martins, 2005; Rindova et al., 2017; Wedlin, 2007). For example, it would be almost impossible to imagine a school system without the grading of pupils. Likewise, it is difficult to see how governments can control local authorities without different types of summary reports on their financial status.

As suggested in the seminal work by Espeland and Stevens (1998) (see also Espeland and Sauder, 2007; Stevens and Espeland, 2005), such ratings, rankings and performance evaluations are very powerful as they involve commensuration, i.e. they imply “the expression of measurement of characteristics normally represented by different units according to a common metric” (Espeland and Stevens, 1998, p. 315). According to this stream of research, commensuration thus has two important characteristics.

First, it involves a chain of translation (Latour, 1999) where the multiplicity and particularity of organizational reality is reduced into a standardized and single rate or rank that works as a proxy (Rindova et al., 2017), or stand-in (Dambrin and Robson, 2011), for that reality. Publically announced ratings and rankings dealt with in this study are prominent examples, but commensuration comprises all types of efforts to express value quantitatively (Stevens and Espeland, 2005). However, commensuration not only makes particular qualities of the evaluated entities visible, but also makes many, if not most, of their qualities invisible. For through describing entities using a particular format – e.g. by means of a shared metric – commensuration makes everything that does not fit that format irrelevant. Or as argued by Espeland and Stevens (2008, p. 432), commensuration “facilitates a peculiarly modern ontology, in which the real easily becomes coextensive with what is measurable.”

Second, commensuration is inherently relative in character (Espeland and Stevens, 1998; Stevens and Espeland, 2005). For commensuration not only simplifies and unites entities by describing them in a standardized way, it also constructs particular relationships between these entities, i.e. “[i]t creates relations between attributes or dimensions where value is revealed in comparison” (Espeland and Stevens, 1998, p. 317). For example, by ordering entities in terms of hierarchies on a single, universal scale, some of them are constructed as generally better than others. And this is so irrespective of whether such differences may be negligible (Espeland and Sauder, 2007), or whether lower ranked entities may outperform higher ranked ones on other criteria than the one expressed by the chosen scale (see e.g. Elsbach and Kramer, 1996). Accordingly, while commensuration transforms all types of differences into a standardized format, thereby uniting them, it also creates differences between entities. Again, however, these differences are expressed in terms of magnitude on a single scale, i.e. as matters of more/less or better/worse, rather than as matters of kind (Espeland and Stevens, 2008; Stevens and Espeland, 2005).

Commensuration and the “reactive conformity thesis”

As argued by Espeland and Stevens (1998, 2008), Espeland and Sauder (2007) and Sauder and Espeland (2009), commensuration in general, and public ratings and rankings in particular, influence the subjects of such evaluations in many different ways. Although some do not explicitly refer to the notion of commensuration, there seems to be a general consensus among most researchers that public ratings and rankings tend to lead to conformity in organizational behavior. For example, Gioia and Corley (2002, p. 110) observed how rankings pushed organizations to “conform and perform to the rankings criteria,” while Martins (2005, p. 701) noted that “rankings constitute sources of institutional isomorphic pressure on organizations.” Along these lines, Wedlin’s (2007) study suggested that while a ranking of business schools left some room for differences in organizational practices, it nevertheless codified a template for a good business school of high international standard. A template that created isomorphic pressures on organizations to adapt to this role model (see also Mingers and Willmott, 2013; Parker, 2011; Willmott, 1995).

The literature on ratings and rankings has offered a number of explanations as to why commensuration should lead to organizational conformity. For example, Gioia and Corley (2002) suggested that rankings push organizations to adapt to the rankings criteria because such compliance provides access to important resources from key constituents (see also Gendron’s, 2015 discussion about the implications of the paying-off mentality in academia). And along these lines, Espeland and Sauder (2007, p. 13) found that even small differences in rank made law schools adapt to the evaluation criteria “not only to [influence] the reactions of prospective students, but also to other constituents such as trustees, boards of visitors, and alumni, all of whom provide financial and administrative support to the schools.”

It has also been argued that commensuration is generally associated with highly held values of objectivity and rational decision-making (Espeland and Stevens, 2008). And when such depersonalized and public form of knowledge is deemed superior to personalized and local knowledge by different stakeholders (Espeland and Stevens, 1998), it is difficult to be against such “proxies” of organizational performance, at least in the long run. For as Jeacle and Carter (2011) found when exploring the role of the ranking device TripAdvisor, this type of commensuration constitutes an important generator of trust in general, and of system trust in particular (i.e. trust in the ranking devices per se, see also Giddens, 1990).

Another stream of research has stressed the explanatory value of perceived discrepancies between the position assigned to an organization in rankings and the position implied by the organization’s identity in the minds of its top managers (Martins, 2005) for understanding behavioral conformity. Along these lines, for example, Elsbach and Kramer (1996) found that business schools used a number of response tactics so as to address perceived identity threats, i.e. when public rankings called into question the merit of core, enduring and distinctive features associated with their organizations. Similarly, Wedlin (2007) argued that rankings contribute to shaping the international business school field by codifying an organizational template that is used in identity-formation processes of organizations – a template that signifies identities and is formed around perceptions of what is real, good and legitimate (see also Mingers and Willmott, 2013; Parker, 2011).

Finally, Espeland and Sauder (2007) (see also Sauder and Espeland, 2009) suggested two key mechanisms that explain what they refer to as organizational reactivity. Apart from commensuration elaborated on in this study, they show how rankings evoke self-fulfilling prophecies. The premise is, they suggest, that when “subjects change their behavior in response to prediction; prediction then, becomes a new and dynamic factor that changes the conditions under which it comes true” (p. 11, emphasis in original). And along these lines, Espeland and Sauder (2007) found that law schools used previous rankings as the basis for internal resource allocation, or even adopted improved rankings as an explicit organizational goal, thereby reinforcing the legitimacy of and conformance to the ranking criteria.

Hence, as Rindova et al. (2017, p. 14) conclude in a recent comprehensive literature review of the ranking literature, an overall key insight from this research “is that quantification removes and disincentivizes individuation and increases conformity” (see also Gendron, 2008; Mingers and Willmott, 2013; Parker, 2011; Sauder and Espeland, 2009). In fact, even critical management scholars, who are perhaps best equipped to “see through” and resist the conforming effect of commensuration such as journal rankings and citations counting, generally have ambivalent attitudes to it (see e.g. Annisette et al., 2015; Butler and Spoelstra, 2012, 2014).

Context

As suggested above, commensuration is an inherent part of modern society, and a prominent example is the widespread use of ratings and rankings so as to asses various forms of performance in the university sector (Kallio et al., 2016; Lynch, 2015; Mingers and Willmott, 2013; Wedlin, 2007). In order to investigate how organizational representatives can publically “contest” this type commensurability, i.e. try to influence how performance evaluations should be construed among important stakeholders, we investigated how 26 Swedish academic institutions publically responded to a state-initiated assessment of their educational programs in business administration. At least two characteristics of this context made it particularly useful for investigating how the subjects being rated or ranked may publically contest commensurability.

First, at the time of the study, the Swedish university sector was (and still is) quite subdivided in terms of the positional status of different actors (cf. Elsbach and Kramer, 1996; Martins, 2005; Sauder and Espeland, 2009). One group of academic institutions belonged to the six largest and oldest universities. Within the Swedish community, these business administration departments, together with their counterpart at the privately owned Stockholm School of Economics (SSE), were considered high-esteem and officially referred to themselves as members of the “U6-group/network”[1]. The second category consisted of business administration departments from four new universities (awarded university status around the turn of the millennium) and 13 smaller so-called university colleges. These departments could, in principle, give the same type of educational programs as do those in the U6-group (given that they were awarded the qualification by the Swedish National Agency for Higher Education). However, they were generally considered less esteemed and only disposed a very marginal proportion of the state-funded research budget.

Second, the Swedish Government had changed the overall system for the evaluation of educational performance in the university sector. Inspired by a regime of “new public management involving a proliferating culture of audit, accountability and performativity” (Knights and Clarke, 2014, p. 336; see also Espeland and Sauder, 2007; Kallio et al., 2016), it was decided that assessments should be done by a panel of external, independent reviewers who should focus on the achievement of learning outcomes (Govt. Bill 2009/10:139, 2009, p. 139). Unlike previous assessments that had been primarily focused on preconditions (such as the proportion of teachers having a PhD degree) and extant processes for ensuring a good education (e.g. the existence of internal routines for quality assurance), the new system was thus output oriented and even included a small pecuniary reward for those who received the highest grade. Or more precisely, the expert panel should focus on the quality of students’ bachelor/master theses, and students’ experiences of the educational program in question[2]. And on these grounds, the panel assessed all educational programs and graded them on the scale “inadequate quality,” “high quality” and “very high quality.”

This was a quite dramatic performance evaluation system change. For not only did poorly performing academic institutions risk losing their entitlement to award educational programs rated “inadequate quality” if the problems were not satisfactorily addressed within a specified time limit (Report 2011:3 R, 2011), it also resulted in a rather unorthodox grouping of business administration departments. In fact, several of the departments belonging to the highly esteemed U6-group received the rate “inadequate quality” on one or more of their educational programs, while quite a few of the less highly esteemed departments got the rates “high quality” and “very high quality.” This performance evaluation thus provided us with an empirical context where we could expect interesting dynamics as both winners and losers would take the opportunity to influence how their particular rates were publically construed so as to try to improve or maintain their positional status within the organizational field (Elsbach and Kramer, 1996; Martins, 2005; Sauder and Espeland, 2009).

Data collection

Considering our interest in public responses, our primary source of data was the press releases that all universities/university colleges published almost immediately after the results of the new performance evaluation was announced by the National Agency for Higher Education in mid-May, 2012. These press releases were deemed useful for two reasons. First, they granted us access to firsthand, written data where the subjects of the evaluation themselves formulated how they perceived, and wanted important others to perceive, the outcomes of the evaluation. Second, this performance evaluation was deemed very important by all universities/university colleges in question. This increased the likelihood that they would not only respond promptly and decisively, but would also invest considerate effort into formulating the public response (in the form of quite lengthy press releases). And indeed, 26 press releases were immediately published (one per university or university college) amounting to more than 30 pages of text. In about half of the cases, the signing author of the press releases was a press officer. In the other half of the cases, the author was not explicitly specified.

The press releases typically consisted of two parts. One part provided background information on the evaluation as such (e.g. aim, focus and overall results). The other part consisted of statements about the received grade(s) from one or more representatives of the organizations in question. In the great majority of the cases, the Vice Chancellor, a Faculty Head or the Head of the business school commented on the outcomes of the performance evaluation, which further stresses the gravity of the situation. It was also common that representatives from the business administration departments (e.g. the head of department, a professor, or the director of studies) were quoted in the press releases.

In order to get background information on how the performance evaluation and its outcome was perceived by actors in the business administration field, we also conducted semi-structured interviews with representatives from the business administration departments whose educational programs had been evaluated. These interviews included questions like: Was it an important event? Were the outcomes perceived as reliable and fair? and What were the reactions when they acquired the results regarding their own and other business administration departments?

Overall, we conducted interviews with department heads (or similar positions) from 21 of in total 26 units. The remaining five department heads were not available for an interview, typically because of time limits. On average the interviews lasted around half an hour, and were all but one tape-recorded (in that particular case, the respondent did not want the interview to be recorded).

We also collected complementary material concerning the new performance evaluation system as such. Particularly important in this respect was information issued by the National Agency for Higher Education. Such information sources included Power Point presentations showed at so-called pre-evaluation information meetings, and the Report 2011:3 R (2011), which described the background, aim, design and implementation of the newly developed performance evaluation system.

Data analysis

Drawing upon Ahrens and Chapman (2006) and others (Silverman, 2011; Bazeley, 2013), we analyzed the data through working back and forth between the empirical observations, the extant literature and an emergent taxonomy of public response tactics. However, after having reached a preliminary understanding of the material, we conducted the following more systematic coding of it.

Based on the view of ratings and rankings as “engines of status anxiety” (Sauder and Espeland, 2009; see also Elsbach and Kramer, 1996; Martins, 2005), we first looked for in vivo expressions (see e.g. Bazeley, 2013) in the press releases where actors sought to positively differentiate one’s own organization (in-group) in relation to others (out-group) so as to improve their relative positional status. Such expressions could include that “Close to 40% of all programs in the country are severely criticized. […, but] the result is positive for us” (emphasis in original), and “in comparison with other institutions that have been criticized this is ‘mild.’”

In the second phase of the coding, these in vivo expressions were systematically compared to identify commonalities and differences. A striking commonality was that essentially all actors tried to (re)define how the performance evaluation per se should be publically construed among important stakeholders. This overarching observation led us to develop the notion of “contesting commensurability.”

On a more detailed level, however, our analysis of in vivo expressions suggested a number of differences in terms of how this type of “public contesting” of the performance evaluation was manifested. For example, some actors tried to elaborate on the (new) groupings that the performance evaluation had engendered (e.g. through inclusion and exclusion of group members), while others focused on (de-)emphasizing the significance of the ascribed performance rate as such. This type of differences led us to identify three forms of public contesting, focusing on the “comparison object,” the “comparison dimension,” and the “comparison rate,” respectively.

We also found that these attempts at contesting commensurability differed between so-called winners and losers, i.e. whether educational programs given by a particular business administration department were awarded the grade “inadequate quality” or “high quality.” This led to the idea that positional status emerging from the performance evaluation in terms of winners or losers was important for understanding the different forms of public contesting observed.

Based on the three foci of public contesting and the two types of positional status emerging from the performance evaluation, the different manifestations of public contesting were compared and collapsed into a set of second-order concepts referred to as “commensuration management tactics.” In total, 12 different types of commensuration management tactics were identified (see Table I in Section 5).

In a final step of the analysis, these aggregate findings were further contextualized and validated by our other sources of data. For example, our interviews with department heads further stressed the importance of the pre-study groupings of the academic community for understanding the responses to the performance evaluation (e.g. in terms of “we were very surprised to find [the large U6-University NN] among those who failed”). Related to this, the interviews also attested the importance of improving, or at least prevent deterioration of, one’s organization’s positional status in the field (e.g. “we don’t want to be perceived as ‘failed’ […] it’s a matter of long-term survival”).

As a means of further validating the emergent findings, we conducted yet another analysis of press releases that were issued when the educational programs in sociology at Swedish universities were assessed some weeks later. Overall, we were able to identify the larger part of the commensuration management tactics found in the focal coding of the business administration field, which suggests reasonable transferability of findings to other, similar contexts.

Commensuration management tactics focused on the comparison object

As argued in Section 2, a basic quality of commensuration in the form of ratings and rankings is that it contributes to constituting entities and their relations (Espeland and Sauder, 2007; Espeland and Stevens, 1998, 2008; see also Miller and Power, 2013). Furthermore, this constituting the quality of commensuration implies that there is at least some space for “rhetoric manoeuvring” where subjects to performance evaluations can seek to publically (re)define how they are grouped together with other organizations (referred to as in-group) and the relations to other groups (out-group). Again, based on the positional status anxiety hypothesis (Sauder and Espeland, 2009), we would expect that actors seek to positively differentiate one’s own organization from others in order to enhance, or at least prevent a deterioration of, its relative positional status within the organizational field of which it is part.

Our case is interesting in this respect insofar as the state-initiated performance evaluation grouped business administration departments in a way that was largely orthogonal with the traditional dichotomization of the sector (see Section 4). And given that this new grouping was perceived as threating for some actors (losers), and as a window of opportunity for others (winners), we find that it sparked two principal forms of tactics relating to the comparison object as such. Specifically, we find that actors would typically mobilize the evaluation as a means to signal (non)membership, below referred to as inclusion (see Cells 1 and 3 in Table I) and exclusion tactics (Cells 2 and 4). As suggested by the table, such tactics would also often imply that actors sought to minimize or maximize the number of members in the new in-group or out-group, respectively, depending on the positional status of the organizations in question and what they wanted to achieve.

One expression of the inclusion tactic in Cell 1 was when traditionally less esteemed actors that came out well in the performance evaluation sought to associate themselves with traditionally high-status, even prototypical, members who were also categorized in the same group. For example, one university college associated themselves with the prestigious Stockholm School of Economics (SSE); “It is only [we] who together with SSE and [University College NN] have received the rate very high quality in the subject Business Administration on the bachelor level” (press release). Moreover, this university college also sought to minimize the new high-status group through specifying “the bachelor level” (see Minimization tactic in Cell 1). For again, several more actors received the grade very high quality on one or more of their educational programs (e.g. on the masters programs), but through delimiting the social comparison to a particular program, the new in-group became even smaller and thereby more exclusive. Another example of this type of “in-group minimization tactic” was used by another university college which solely focused their rhetoric on one-year master programs; “Of 22 one-year master programs in Sweden, only three get the highest rate […] and [we] are one of them” (press release).

However, also many losers mobilized an inclusion tactic – but in this case to mitigate a deterioration of their positional status (see Cell 3 in Table I). One way of doing this was to highlight that also traditionally high-esteem actors (typically U6-members) were part of the new inadequate quality group. For instance, one university college stressed that “Among them who do not fulfill the requirements, there are both university colleges and universities” (press release), while another even explicitly listed all three U6-members (and a number of other academic institutions) “who have also been questioned” (press release).

Parallel to the minimization tactic mobilized by some winners (i.e. where the number of members in the new successful in-group is minimized), we also find substantial evidence of losers seeking to maximize the number of members in the inadequate quality group (see Cell 3). The by far most common way of doing this was to emphasize that “almost 40% of the country’s educational programs get the rate inadequate quality” (press release from a university college). Another way was to focus the presentation of results on business administration departments rather than on educational programs (which was the actual unit of assessment). Because then the percentage of “failures” could be raised from 40 to 50 percent; “The expert group has examined students’ theses […] from 26 universities and found shortages at 13 of these” (press release). In other words, many losers sought to diminish the negative effects of being part of the new low-status group through stressing that one’s own organization was in “good company” (see also the discussion about Tactic 11 below, implying that losers sought to de-emphasize significance of the comparison rate as such).

Notably, a parallel type of rhetoric was also mobilized by many winners. However, they typically used it to enhance their positional status within the academic community through excluding and distancing themselves from the less esteemed out-group(s) (cf. the Exclusion tactic in Cell 2 in Table I). For example, one U6-university sought to reaffirm its position as a winner within the Swedish community through quite harshly commenting on both the quality and the large size of the new low-status group (cf. the Maximization of out-group tactic depicted in Cell 2):

Close to 40% of all programs in the country are severely criticized. […, but] the result is positive for us, in particular when you consider the high proportion of questioned [programs]. At the same time it is worrying that one of the largest educations in Swedish academia gets such a [weak] result. But it is very good news that our educations in Business Administration without doubt have a high quality.

(press release, emphasis in original)

As suggested by Cell 4 in Table I (i.e. when actors mobilize the performance evaluation to signal non-group membership), an exclusion tactic may also be used by losers. That is, as a means of maintaining one’s relative positional status, actors may exclude and distance themselves from other actors in the new grouping imposed on them by the performance evaluation. One common way of conducting such downward social comparison (cf. Elsbach and Kramer, 1996) was to emphasize that one’s own organization nevertheless is better than other relevant comparison objects in the new low-status group:

We have received criticism on one [evaluation] criterion per education of eleven in total. With respect to all other criteria [our] educations are granted high quality. In comparison with other institutions that have been criticized this is mild [suggesting one’s own organization positively differentiates from others in the new, imposed low-status group].

(press release, emphasis added)

All in all then, our data suggest that both winners and losers may skillfully enhance or seek to “save” their positional status through mobilizing a number of tactics focused on the performance evaluation comparison object as such. Specifically, we find that they typically mobilized the performance evaluation to signal group (non-)membership through a variety of inclusion and exclusion tactics. Depending on their positional status (such as winners or losers, traditional position, etc.) and what they wanted to achieve (e.g. to pass from a low-status group into a more esteemed and/or to keep some actors out of new in-group), they would also try to rhetorically minimize and maximize in-/out-groups.

Commensuration management tactics focused on the comparison dimension

As suggested by the mid-column in Table I, a number of commensuration management tactics were also focused on the comparison dimension as such. This meant that actors rhetorically sought to (de-)legitimize the focus on educational outcomes which dominated the newly introduced performance evaluation system (Report 2011:3 R, 2011).

If we start in Cell 5, one obvious tactic used by many winners was to applaud the government’s new grip to try to evaluate the degree of attainment of the learning objectives stipulated in the Higher Education Ordinance (as opposed to the previous system focusing primarily on conditions and processes, respectively, for ensuring high quality, see Section 4). For example, one interviewee whose department came well out of the performance evaluation said that “the focus on bachelor and master theses was essentially very smart.” Likewise, another interviewee thought that “the new performance evaluation system is a step forward,” although s/he also argued that the evaluation of students’ theses “could be complemented with other output-oriented quality criteria.”

However, our data also suggest that winners not only sought to legitimize the comparison dimension as such, but also tried to construct the overall results of the performance evaluation as fact-like (cf. Chua, 1995; Power, 2004). Specifically, we identify two interrelated ways of achieving this, namely, through ambiguity reduction and selective verification, respectively (see Cell 6 in Table I). The first-mentioned tactic sought to attest the validity of the results, e.g. through stating that “all members of the expert panel [who did the evaluation] unitedly stand behind the results published today” (press release, emphasis added). Other expressions of ambiguity reduction with respect to the results are (press releases, emphases added):

Our educational programs have unhesitatingly high quality,

We’re very happy with the results, which recognise the fact that [university college NN] is in the top when it comes to Business Administration in Sweden, or

[O]ur educations in Business Administration without doubt have a high quality.

The second commonly used way of rhetorically “boosting” the validity of the comparison dimension and the overall results was to selectively link them to other, previously conducted performance evaluations (see the selective verification tactic in Cell 6). One way of doing this was to make temporal comparisons. For example, one press release stated that “It is really nice that we also in this evaluation cycle win the gold medal,” while another one argued that “the Swedish National Agency for Higher Education has today awarded [Name of university college] yet a golden star, whose popular business program gets the highest possible mark.” Yet other actors sought to make the focal performance measurement fact-like through comparing it with quality assessments undertaken by other evaluators. As one actor put it, “we are very happy since yet another independent review shows that our priorities and strategic choices have been correct” (press release).

As suggested in Table I, the commensuration management tactics mobilized by losers largely mirror those applied by winners. Notably, however, very few losers sought to de-legitimize the comparison dimension as such. That is, almost no one openly problematized the evaluation of educational output as such (although, indeed, quite a few criticized the sole use of bachelor/master theses to make the overall judgement of educational output quality). Instead, many losers sought to deflect attention of a poor result through highlighting alternative comparison dimensions (see Cell 7, cf. Elsbach and Kramer, 1996). In effect, many actors did this through emphasizing (alternative) organizational strengths that were not explicitly valued in the performance evaluation. For example, one U6-university suggested both in the press release and in the follow-up interview that they had “research of world-class” and that this was not really reflected in the new measurement of performance. Other losers argued that they first of all looked upon themselves as developing skillful, reflexive, and not least, employable practising professionals. For example, one actor argued that “[this program] is developed to increase the number of new business ventures in the country [and therefore] it does not really match the focus of the new evaluation system” (press release). In a similar vein, other actors stressed that “our educations are focused on the professions,” and that “[a key feature of our educations] is a strong cooperation with the region’s working life [and accordingly], our students are very popular on the job market” (press release).

As suggested by Tactic 8 in Table I, however, many involuntary members of the new low-status group also tried to maintain their relative positional status through explicitly questioning the accuracy and validity of the performance measurement. That is, while they (again) hardly criticized the comparison dimension as such (i.e. the evaluation of educational outcomes), they problematized its fact-like status. And largely parallel to winners, they did so by means of two types of tactics, namely, ambiguity enhancement and selective comparisons (Cell 8). Although these two are indeed interrelated, ambiguity enhancement tactics refer to actors seeking to problematize the focal measurement as such. For example, one university college noted that “the result is not clear-cut” and continued through quoting the evaluation report issued by the Swedish National Agency for Higher Education; “there is no evidence of significant shortcomings regarding the education’s content, structure and forms for examination” (press release). Similarly, another actor argued that “a minor shortage in one area can severely affect an education that in all other respects has high quality” (press release), thereby suggesting that the evaluation created seemingly large differences between groups when in fact they are very small (cf. Espeland and Sauder, 2007).

The tactic Selective comparison (Cell 8), in turn, implies that actors problematized the focal performance evaluation through selectively comparing it with other, more successful evaluations. Such tactic could be expressed in terms of “One year ago the same authority awarded [our university] the qualification to give [name of educational program […] and] at that time, the overall judgement was that we fulfilled all quality requirements” (press release). Likewise, two U6 universities stressed in their press releases that “the program in Business Administration does have high quality which has been confirmed in other assessments [link to webpage],” and “we have our own quality assurance system which we trust [and] we score well on these, other criteria,” respectively.

In conclusion then, we identify a number of commensuration management tactics that were directed toward the comparison dimension as such, i.e. toward the very format used to “transform qualities into quantities, difference into magnitude” (Espeland and Stevens, 1998, p. 316). Winners typically strived to enhance their positional status through “establishing” the validity and reliability of their successful outcome of the performance evaluation. Specifically, they tried to legitimize the performance evaluation through explicitly supporting the comparison dimension, and/or through constructing the results as fact-like through ambiguity reduction and selective verification tactics, respectively. In contrast, many losers sought to de-emphasize the importance of the comparison dimension through highlighting alternative organizational qualities (strengths) not valued in the performance evaluation. They also tried to problematize the performance evaluation’s fact-like status through ambiguity enhancement, and selective comparisons with other, more successful evaluations.

Commensuration management tactics focused on the comparison rate

Table I suggests that commensuration management tactics may not only focus on the comparison object (i.e. the new groupings of actors) and comparison dimension (i.e. educational output quality and the way it is measured), but also on the comparison rate as such (i.e. the attained performance level). To begin with, Cell 9 in Table I shows that the winners typically sought to emphasize the significance of the rate as such through results amplification. This tactic was manifested in various ways, but a common one was to stress that the successful rate referred to a core/major activity within the department (as opposed to a marginal and peripheral one). Examples of such expressions in the press releases include:

This is one of the most important and popular programs at the university,

It is the single largest program, and

We are happy that the large amount of students at the bachelor level attains such a good result.

In contrast, many losers tried to de-emphasize the significance of a low rate through relativizing and toning down the results (see Cell 11). A very common way of doing this was to argue that the results were “not relevant any more.” Along these lines, for example, one U6-member stated that “I can conclude that the critical judgements partly concern one-year masters that are now being phased out, partly concern four master programs of which one has been phased out and two have been significantly changed since the performance evaluation” (press release). Other attempts of relativizing a weak rate was to position one’s result in a more successful context, thereby constructing it merely as a “bump in the road.” Examples of such statements from the press releases include that, “the number of students have increased steadily over the years,” “[we] have always had a high number of admissions to our programs,” or to stress that the low rate does not refer to the more prestigious “[NN] program which we know has high quality.”

We also find that actors selectively tried to justify/explain a high and low rate, respectively. As far as winners are concerned, they typically mobilized two tactics (see Cell 10). The first one was to stress proactivity. For example, one actor argued that “the high quality is the result of the long-term efforts we have undertaken,” while another stated that “this is the result of active and continuous quality enhancing work on our educations.” The second tactic in Cell 10 implies that many winners justified/explained the high rate through explicitly linking it to core organizational attributes. Examples from the press releases include that actors could ascribe the success to attributes such as “liberal education,” “multidisciplinary educations,” “a practitioner focus,” and “internationalization” (where the last-mentioned actor further stressed this core attribute through writing the press release in English).

Equally interesting is, however, that losers would typically do the opposite (cf. Cell 12 in Table I). That is, they oftentimes tried to stress that the performance evaluation did not show “who they really were.” And this was done through emphasizing exceptionality. For instance, one university college stressed that the failing programs were “not pure educations in Business Administration, but combined with other subjects and also directed towards practicing professionals [as opposed to more traditional academic programs]” (press release). In a similar vein, another actor emphasized exceptionality through arguing that “this particular program consisted of a high portion of foreign students without previous experiences of writing theses [suggesting that the ‘normal’ programs had high academic quality]” (press release).

So to conclude then, our empirical evidence suggests that many actors sought to enhance (or prevent a deterioration) of their positional status through commenting on the comparison rate as such. In so doing, winners typically tried to amplify the significance of a high rate, while many losers would do the opposite (above referred to as results relativizing). We also find that winners oftentimes sought to justify the rate through stressing proactivity and also to ascribe the perceived success to core qualities/strengths of their organization. In contrast, losers typically stressed exceptionality, and through doing this, suggested that the disappointing comparison rate did not show “who they really were.”

A taxonomy of commensuration management tactics

Overall, our taxonomy of commensuration management tactics (see Table I above) suggests that: these tactics may have at least three different foci related to the rating device as such, and that they are typically mobilized by both losers and winners, although in slightly different ways. More specifically, our findings suggest that actors may try to affect how imposed ratings and rankings are publically construed in the following principal ways. First, actors may elaborate on the comparison object, i.e. on the groupings of entities and the relations between these that commensuration engenders (see e.g. Tactics 1–4 in Table I). For example, they may both seek to expand and contract the boarders of groupings to signal who they (do not) want to be associated with (above referred to as inclusion and exclusion tactics, and maximization and minimizations tactics, respectively). Again, winners often tried to minimize the size of the in-group thereby making it even more exclusive, while many losers sought to maximize the new in-group so as to show that many others, even traditionally high-esteem actors were also part of this unwanted low-status group (see also Elsbach and Kramer, 1996).

However, actors also tried to create new groups all together. A typical example is when one group as defined by the performance evaluation is subdivided into two subgroups by the subjects of this evaluation – one higher esteemed in-group, and one lower esteemed out-group. For example, some actors who were awarded the rate inadequate quality on a particular educational program could argue that they (new in-group) only failed on a single evaluation criterion, while others (new out-group) failed on several criteria.

Second, actors may elaborate on the comparison dimension. As illustrated by Tactics 5 and 7, for example, one way of doing this is to elaborate the standardized format for comparison per se by reintroducing qualities not deemed relevant by the commensuration in the first place. For example, many winners publically ascribed their success to important qualities/strengths of their organizations, while many losers highlighted alternative and more beneficial comparison dimensions. This way of trying to affect how the performance evaluation was publically construed, thus implicitly contested commensuration all together. And actors did so through demonstrating that differences between organizations should not only be seen as a matter of magnitude on a single scale, but also as a matter of kind (cf. Espeland and Stevens, 2008; Stevens and Espeland, 2005).

Another way of contesting the comparison dimension was to elaborate on the representational validity/correctness of the groupings and the relations between these that commensuration engenders (see Tactics 6 and 8). Again, winners would typically seek to construe a successful rate as fact-like, while losers would oftentimes question its validity through selective comparisons with other, more successful ratings and rankings.

Third and finally, actors may elaborate on the significance of the comparison rate (see Tactics 9–12). For example, many winners tried to increase the significance of a high rate through results amplification tactics and through stressing proactivity. In contrast, many losers tried to relativize the significance of a poor rate by arguing that the “results are not relevant any more,” or through emphasizing exceptionality (i.e. that the program(s) that received the rate inadequate quality constituted a peripheral part of the department’s education activities).

In conclusion then, we find that actors may try to contest externally imposed rates or ranks through publically elaborating on the comparison object, the comparison dimension and comparison rate, respectively. We also find that not only are threats to positional status likely to spur decisive responses (Chatterji and Toffel, 2010; Elsbach and Kramer, 1996; Martins, 2005), but also the opportunity to exploit a good rate. In fact, most of the tactics used by actors with poor rates have a mirror-image variant used by those coming well out of the rating.

Contributions

Through analyzing empirical material on public responses to an imposed performance evaluation, and integrating insights from several literatures (Elsbach and Kramer, 1996; Gendron, 2008, 2015; Espeland and Sauder, 2007; Parker, 2011; Pollock and D’Adderio, 2012; Pollock et al., 2018; Wedlin, 2007; Mingers and Willmott, 2013), our study suggests two important contributions.

First, we respond to recent calls for studies that focus more specifically on “the strategic actions that organizations can take to influence their ranking positions” (Rindova et al., 2017, p. 12). We also take account of Pollock et al.’s (2018) call for more research that problematizes what they refer to as the reactive conformance thesis which characterizes most research not only in accounting (see e.g. Bogt and Scapens, 2012; Gendron, 2008; Parker, 2011), but also in organizational studies (Gioia and Corley, 2002; Mingers and Willmott, 2013; Wedlin, 2007) and sociology of commensuration (Espeland and Sauder, 2007; Sauder and Espeland, 2009). For as Pollock et al. (2018, p. 57) argue, “scholars have largely overlooked the possible range of alternative responses.”

Addressing these calls, our paper proposes an empirically derived taxonomy of commensuration management tactics that provides analytical resources for the study of public responses to externally imposed ratings and rankings. That is, our taxonomy identifies a number of ways through which actors subjected to such performance evaluations may try to affect how imposed rates or ranks are construed among important stakeholders. In contrast to the recent work of Pollock et al. (2018), however, our study suggests that such proactive behavior may not only emanate from organizations being exposed to multiple rankings (enabling them to “see” and “reflect” upon different and often conflicting responses asked of them), but also from single performance evaluations.

Second, and related to this finding, this study is one of the first (to our knowledge) that elaborates on how the constitutive quality of commensuration makes it somewhat “plastic” (Briers and Chua, 2001). That is, while externally imposed ratings and rankings contributes to constituting entities and their relations (Espeland and Stevens, 1998, 2008) which are difficult to resist (Espeland and Sauder, 2007; Gendron, 2008; Parker, 2011; Sauder and Espeland, 2009), our study does suggest that their plasticity creates some room for public rhetoric maneuvering on the behalf of the evaluated subjects. Rhetoric maneuvering that could focus on the comparison object, the comparison dimension, and the comparison rate, respectively, as means of improving or maintaining their organization’s relative positional status.

However, this type of research on public responses to ratings and rankings is still in its infancy and more efforts are required to explore if and how such commensuration management tactics actually have an effect on how external stakeholders perceive the relative status of evaluated organizations. We also lack detailed empirical and theoretical accounts of how such processes of (re)negotiation may emerge over time – an interesting and rewarding task for further research on the commensuration management tactics developed in this study.