Dilemmas of Engagement: Evaluation and the New Public Management: Volume 10

In our public institutions it sometimes feels as though the tectonic plates of social and political contracts are shifting. Familiar coordinates are displaced or left stranded as we survey new territorial configurations and have to work out again how to find our way in public administration. The economic revolution embraced by neo-liberals and conservatives found its counterpart in a governance revolution in the very institutions that have always been designed to protect us from sharp historical and political lurches one way or another. Our public institutions – mostly coinciding with what we can call our professional institutions – which were once the filters of social and political change have become co-opted into social reform and are now all-too-frequently the conduits of that political change.

David Marquand (2004) opens his celebrated book on the ‘decline’ of public service by arguing (pace Tawney) that there is a fundamental tension between capitalism and democracy – the former dedicated to an essential economic inequality (competition); the latter dedicated to rights-based equity derived from civic ideals. His book highlights public sector institutions as the arena in which the inevitable tussle between the two is played out. The neo-liberal movement provides the ideological vehicle for reform of public institutions, designed to ‘root out the culture of service and citizenship’ (p. 2) – i.e., those civic ideals. At the heart of the struggle is professionalism, the combination of competence, judgement and principle, and neo-liberalism takes careful aim at it.

Evaluation in general and performance indicator systems in particular play an increasing role in society. We do not have a long historical set of experiences which helps us understand what exactly happens when, say, performance data for schools are made public on the internet and in news, because the emerging rules of the game in what some observers call the “knowledge society” (Stehr, 1994, 2001) and “reflexive modernization” (Beck, 1997a, 1997b) have inaugurated new relations between evaluation and performance data on the one hand and political, organizational and practical realities on the other.

My work on issues of representation in evaluation came out of a convergence of my interest in performance and poetry, a concern for evaluation use and a desire to bring some of the insights of postmodernism regarding representation to evaluation practice.

In this paper, I offer a series of reflections on some issues involved in evaluating complex public policy programmes in a context of rapid change and against the background of an activist government setting exacting standards for the delivery of public service outcomes. I argue that the overlay of pressures on programme evaluation as a result of the engagement with such complex public policy interventions presents a series of challenges for evaluation.

This chapter describes the evaluation of a large complex social change programme in England, UK. During implementation, the programme experienced a series of changes to its form, function and governance, which in turn had impacts upon the practice of evaluation itself. These changes also raised questions about the place of evaluation within a policy context that increasingly focuses upon indicators and outcomes of effectiveness and other features and tensions that characterise the New Public Management. This chapter outlines the programme and its evaluation in this context, in order to be able to explore these impacts and raise questions about how we learn from social change programmes.

In England, systematic school self-evaluation (SSE) began as model for school improvement. However, since 2000, and in the context of increasing moves toward ‘new public management’, it has become a policy priority for the Government which is inextricably linked to the inspection regime and risks becoming more closely associated with accountability than improvement. Such policies can, paradoxically, compromise school improvement (Fuhrman, 1993). This chapter will explore ways in which teachers and school leaders have attempted to engage in school self-evaluation in ways that allow them to maintain control of the evaluation agenda and retain the focus on improvement and pupils’ learning. It illustrates attempts by schools to engage with and reduce the negative side-effects of target-driven policies and the drive towards competition between schools. In particular, it will consider the benefits and challenges of inter-school collaboration through networking. It should certainly not be read as an endorsement of current policy developments in England and does not seek, as other chapters do in this volume, to address the key question of what kind of society we want? It offers a pragmatic exploration of some schools’ creative responses to the context in which they currently operate aimed at being more conducive to public rather than private good, that is meeting the needs of the ‘knowledge society’ as opposed to the ‘knowledge economy’ (Hargreaves, 2003).

So I want to take you back to about 20 years ago, as a first year graduate student I was doing a research assistantship with a professor interested in math test performance and motivation. This was the period when the U.S. was equally awed by the Japanese automobile import/production cycle and the Japanese educational system. The systems were both efficient and effective. As part of that assistantship, I helped some staff from the Midwest State Board of Education (MSBE) do a modest longitudinal study. It was all very informal – MSBE and Midwest University (MU) faculty thought it would be interesting to see how the students in 1984 compared with 1974 – called the Decade Study. MU faculty also wanted to pilot some scales assessing student motivation and test anxiety.

Traditionally, the kinds of ethical inquiry that one encounters in program and policy evaluation involve two broad matters: Evaluator (mis)conduct – that is, concerns such as conflict of interest, contractual obligations, competence, integrity and honesty, and so forth – and protection of evaluation participants’ rights to autonomy, privacy, informed consent, and so on. Ethical inquiry in both cases is typically guided by a set of ethical theories, standards, guidelines, or principles that an evaluator must interpret and apply in the situation at hand (Newman & Brown, 1996). For example, the Joint Committee's (1994) Propriety Standards speak to matters of both evaluator conduct – with respect to formal agreements/contracting, conflict of interest, and fiscal responsibility – and to matters of participants’ rights. Similarly, the Guiding Principles for Evaluators, endorsed by the American Evaluation Association (see www.eval.org), address these two ethical concerns stressing the evaluator's duty or obligation with respect to methodological competence, integrity and honesty, and respecting people. These are important matters in professional ethics in evaluation, situated largely within the discussion of the ethics of principles or rules. However, this view of ethical inquiry in evaluation is not my concern here.

In the academic year 2000–2001, I visited the U.S. What I found out concerning the “standards reform” in public education intrigued me. I had a conception of public schooling in the U.S. as a very decentralized system, mainly governed by local school boards. But during my stay, people talked about public schooling in terms of centralization, not only referring to the standards reform but also to other issues of state control, like tests and development projects. Some years earlier I had been made aware of changes in England that also were talked about in terms of centralization. Local school authorities lost much of their power in policy and decision-making to the central power of the state, and a national curriculum was introduced.

Given the importance of trust in social life, the concept has had little direct attention from evaluators.¹ Trust is central to the seeming integrity of social processes, including, of course, the social processes we call evaluation. Evaluation depends for its success on cooperative relationships and a measure of trust. Evaluation stands in an interesting relationship to trust. The credibility and utility of evaluation rests on trust. Loss or lack of trust is a major impetus to evaluation, and evaluation sometimes takes the place of trust. The process of evaluation requires trust, and evaluation is used to underpin or provide a warrant for trust.

(Editors)