The Science of Realist Evaluation: A Realist Manifesto

The Science of Realist Evaluation is a key text for anyone interested in research in the social sciences. It is a brilliant book which will justifiably become a fixture on students’ reading lists, as it brings together in one place Pawson’s thinking and ideas in relation to Realist Evaluation. For those familiar with Realist Evaluation, it broadens and deepens ideas developed in previous publications, most notably Pawson and Tilley (1997). For those new to Realist Evaluation, it is possible to start here, but it perhaps remains most rewarding to read this after Pawson and Tilley’s classic text.

Pawson writes with his usual engaging style, mixing punchy points with wit and wisdom in a seductive cocktail that embraces philosophy and methodology, practice examples and witty asides. Early chapters cover the basic principles and philosophy of realist evaluation (including some robust critique of studies purporting to be realist), the middle of the book considers working with complexity in evaluations and final chapters explore the concept of causal mechanisms. The book concludes with a chapter calling for progress through endless disputation.

There can be no doubt that Pawson likes disputation, indeed there are points at which the punchy style made me wince slightly, particularly in the no-holds barred criticism of some studies purporting to be “Realist”. However, such an approach is entirely consistent with the book’s conclusion, namely that progress in understanding and rigour in methodology can only be achieved through “a disputatious community of truth seekers”. In practice, calling for greater specificity in claims for evaluations being “realist” seems justified if Realist evaluation is not to become diluted and turned into the new “grounded theory”, namely a method claimed by many but actually practised rigorously by few.

Overall, the book presents a powerful polemic against both relativist and positivist traditions. Instead, Pawson calls for methodological pluralism, undergirded by a realist focus on understanding the underlying generative mechanisms causing particular patterns of outcomes (often summarised as the context, mechanism and outcome relationship).

In critiquing every other approach Pawson is at his strongest. His erudition and wit make him both an entertaining and a convincing guide, and one is left with little doubt that his criticisms of other approaches are well founded. However, Pawson is attempting something more ambitious than the dismantling of some existing shibboleths. In this and previous books he is trying to create a new framework for building useful and generalisable knowledge about what works. Here the concept of generative mechanisms of causation is central – these are the underlying reasons that people do things. Thus, for instance, an example of the search for generative mechanisms is given at length from a Realist literature review undertaken to inform whether there should be a smoking ban in cars. Rather than look for methodologically “robust” studies that minimise bias through methodological processes like randomisation, Pawson presents a complex iterative process that involves generating theories around causal links, selecting key questions relating to this and then reviewing the relevant literature. In the end he demonstrates, to my mind very convincingly, that this approach is far superior to the conventional generation of lists of “what works”.

Yet on three counts I retain some scepticism. First, Pawson’s approach seems to me essentially to be what I think of as good old-fashioned scholarship. There is no escape to methods to resolve complexity: rather it is rational thought and debate, reasoned and transparent decision-making and wide-ranging reading that provides a better alternative than “systematic reviews” and mechanistic reliance on methods for discovering a better approximation to the truth. I am convinced that scholarship produces better results than any reliance on method for a review, but not that this equates to a “realist” approach. Second, Pawson sees a place for – it would appear – virtually every method imaginable, but reserves nothing but scorn for randomized controlled trials (RCTs). I do not see the logic of this position. RCTs are misused if they are claimed to result in unbiased findings. However, they seem as valid as any other method within a realist paradigm. I always feel I am missing something when I read Pawson and others on this area. Finally, and most importantly, Pawson’s ultimate claim to a substantial contribution is in the idea that cumulative knowledge can be generated by paying attention to mechanisms of generative causation. He beautifully outlines just such potential mechanisms in an illuminating chapter on research in psychological interventions.

Sadly – for me – I remain unconvinced. The mechanisms Pawson adumbrates are persuasive, but when one pauses it is easy to think of other mechanisms. With further thought one could even think of other typologies of causal mechanism. The problem here is that the concept of generative mechanisms is seductive – there are surely such “things” and they surely do explain the complex findings of studies such as RCTs better than, well, RCTs. Yet it is difficult to know which mechanisms, what a mechanism genuinely is, how one knows one is observing a mechanism, and so on. In other words, what is the ontological status of “causal mechanisms”?

No doubt Pawson has an answer to such questions. Indeed, one is left in little doubt that he has an answer to most questions about realist evaluation. It is just that I am not sure they are the same answers others would give. One senses that Pawson may feel some of the same sense of doubt, as he closes the book calling for constant disputation as the only guarantor of rigour. Such a call seems well founded, as such disputation is the best guarantor of excellence in research. Yet perhaps that is all we have: constant disputation, with partial evidence in an attempt to develop and use policies that are wise and helpful. This is not by any means all there is to Pawson’s argument – but it is, for me at least, what we are left with.