Evaluation as Practical Hermeneutics—or The Long and Stony Road from a Theory of Practice to Evaluation Practice¹⁾

Thomas A. Schwandt (2002). Evaluation Practice Reconsidered (Counterpoints—Studies in Postmodern Theory of Education, vol. 211). New York-Washington: Peter Lang Publishing Group, 224 pages, ISBN 0-8204-5705-1 (paperback) EUR 33.70

Abstract: Contrary to what one might have expected from its title, Evaluation Practice Reconsidered by Thomas A. SCHWANDT is a highly theoretical and largely programmatic volume. Its focus is on an attempt to understand evaluation as a variant of practical hermeneutics, with the result that the book is not so much concerned with the actual practice as with a metatheoretical framework of evaluation. This framework is interesting and in some respects stimulating, and leads readers on to further issues, but those who are interested in methodological issues and expect answers to their practical research problems should not be misled by the title: evaluation practice, in the sense of the expectations, methods and challenges of everyday research practice, is mentioned only in passing.

Those debates on the practice of, and research into, the evaluation of human services that have taken place in recent years in the German-speaking countries continue to be remarkably abstract. Strongly programmatic and conceptual contributions remain the dominant elements of the debate and participants have repeatedly failed to agree on a general understanding of evaluation. There have been only rare attempts to find conceptual and methodological answers in an "inductive" manner, i.e. against a background of concrete, day-to-day evaluation practice and by reflecting on its institutional prerequisites, conditions for success or failure, and ways and means to deal with results. [1]

There may be a variety of reasons why debates about evaluation begin on an extremely general level of thinking. Possibly, the very early and intense discourse on the standards of evaluation at expert level (see, e.g., Joint Committee on Standards for Educational Evaluation/SANDERS 1999), regardless of the fact that these discussions were—and are—very important, could have had a calamitous side effect in this respect. It is also possible that problems remain in pinpointing evaluation research at some place between science, practice and policies (e.g. when those studying the science perceive themselves as focused on practice and application, only to wrestle with the implications of this self-perception; cf. most recently KROMREY 2003). It is therefore necessary to first search for reasonable answers to these points. In such a situation, a volume that promises in its title to concentrate on and reconsider the practice of evaluation will certainly attract attention. [2]

Thomas A. SCHWANDT is unlikely to be known to students of the theory of evaluation research. Notwithstanding several papers in American Journal of Evaluation, Evaluation Practice and Evaluation (SCHWANDT 1997, 2000, 2001a, 2001b, 2003), he has impressed himself on a broader audience as the editor of the Dictionary of Qualitative Inquiry (SCHWANDT 2001c), published by Sage. In 2002 he received the Paul F. LAZARSFELD Award of the American Evaluation Association. According to the dedication, this award is given to those "whose written work on evaluation theory has led to fruitful debates on the assumptions, goals and methods of evaluation" (cf. http://www.eval.org/awards.htm). Former award winners include Daniel STUFFLEBEAM in 1985, Michael SCRIVEN in 1986, Robert STAKE in 1988, William SHADISH in 1994, Michael Quinn PATTON in 1997, and David FETTERMAN in 2000. The list reads like a Who's Who of American evaluation research, so it appears advisable to give due consideration to the author, his book and his lines of argument as a major voice in the current discussion. [3]

We might do so in spite of the fact that this volume was published as part of a series expressly dedicated to the post-modern education discussion—which, incidentally, covers the entire spectrum of post-colonial theory formation from gender research to the debate of school curricula. This is no book written for evaluation research in education alone: its theses are addressed to all angles of the evaluation discussion. [4]

This volume is not a monograph carefully put together from beginning to end but is essentially a collection of individual papers, some of them already published elsewhere, arranged by subjects, in parts revised and supplemented by new material. For example, Chapters 5 and 6 were published in 1999 and 1996, respectively, in Qualitative Inquiry and more or less retouched for reprinting so as to reflect the subject of evaluation. One consequence of this publication strategy, at least for those who read the book from beginning to end, is the multiple repetitions of key arguments, such as the description of the underlying concept of practice, or the criticism of "modern" understandings of science and evaluation. Less nobly expressed, we might talk of exasperating redundancies. On the other hand, this allows the reader to pick out chapters of the book without risk of missing some essential point made elsewhere. [5]

The volume consists of three parts. The first part (comprising four chapters) is dedicated to evaluation. Following what looks almost like a template, its chapters all start with an outline and critique of what the author understands to be the modern understanding of evaluation—and it should be noted that the conceptual terms change over the course of the book. Accordingly, modern understanding of evaluation later re-appears under the terms naturalistic, objectivistic, instrumental, sociological, value-free or value-neutral understanding. The critical discussion is invariably followed by the counter-concept, and here too, the author is ready with a set of terms. He describes his position as "humanistic" and "hermeneutic" (e.g. on pp.17ff), and aims for an ethically committed evaluation practice (pp.33ff), which sees itself as a moment of practical philosophy (pp.47ff and 66ff). Later, he talks of a "value-critical" understanding (pp.151ff), while Chapter 10 explicitly attempts to draw conclusions for thinking about evaluation from post-modern discourses (pp.171-186). [6]

The second part draws the reader's attention to the basics of (epistemological) theory in three chapters. In Chapter 5 the author develops the theoretical background of what he describes as practical philosophy and practical hermeneutics, taking the concept of "understanding" as his model. His chief references are to the work of Hans-Georg GADAMER and Charles TAYLOR, Joseph DUNNE and, repeatedly, Zygmunt BAUMAN. In his next step, the author furiously criticizes, left and right, all those—in his opinion—evil practitioners of criteriology swishing their tails in the social sciences (Chapter 6). In Chapter 7 he takes up the trail of, amongst others, Richard RORTY by pleading—cautiously compared to many other protagonists of postmodern qualitative social research such as those in the notorious Handbook of Qualitative Research (DENZIN & LINCOLN 2000; cf. LÜDERS 1996)—that we "'poeticize' the social practice of evaluation" (pp.124ff). [7].

In its third part, the book discusses several key problems confronting all types of evaluation. Chief among these is the problem of justifying the values that are used to evaluate practice under the conditions of postmodern relativity, contingency and an emphasis on differences. Detailed attention is given to the tasks and self-perception of evaluators during the process of evaluation (especially in Chapter 11) and, in Chapter 10, the function and role of evaluation as a social institution or as the "social practice of power" (pp.172ff). [8]

As already mentioned, the argumentative lines are remarkably schematized: after first producing (and parading) the theoretical counterposition, the author next presents his own position. The passion for a clear argumentative firing line certainly illuminates his position as being brilliantly au courant with modern times and thoughts, and constantly offers stimulating insights into, and provocative probing of, the ongoing discussion among experts. But the price to be paid is that he repeatedly simplifies the "adversary" position—this is at times quite vexatious and occasionally drifts into clichés and even conspiracy theories. An example would be his criticism of the putative "criteriology" cycle. This includes, first, all those who are convinced "that it is not only possible but absolutely necessary to develop regulative norms for the choice between methodologies, values, theories, claims, and conjectures" (p.95). Even though the author may have had chiefly positivistic positions in mind and even though he once again quotes coarse differences such as those between the humanities and the sciences, or between qualitative and quantitative research (p.96), at second glance things are never quite as easy as would be suggested by his fighting terms. What is decisive is not so much whether criteria should be used (without them it would appear to be impossible to arrive at decisions and evaluations), but rather how they are developed (cf. p.97). [9]

Interestingly (particularly in the context of Forum: Qualitative Social Research), the difference between qualitative and quantitative social research, elsewhere lovingly emphasized, here plays only a subordinate part. In this case the opponent, in a much more fundamental sense, is modern social research, which is characterized, according to the author, by its tendency to "objectivize" its subjects: to accord the researcher a position that is distanced from and value-neutral to the subject and, pre-eminently, instrumental at root. The conclusion with regard to evaluation arrived at by the author is demonstrated by the following quotation, as one of many possible examples:

"In other words, what the professional evaluator examines is an evaluand—a 'thing' such as an educational or social program, project, or policy. That thing-like entity has properties or features, all of which, at least in principle, are observable and measurable. ... The rationale here is distinctly modernist reflecting at least implicit acceptance of the aim to tame and domesticate disorder and ambiguity in society by means of molding citizens and society to reflect a rational social design. Evaluators, along with other social scientists, play a key role in modern, scientifically guided societies ... as the providers of the expert knowledge necessary to shape and control the natural and social environments" (p.13). [10]

It should be noted that essential arguments of this critique are borrowed from Charles E. LINDBLOOM (1990) who is, accordingly, quoted with great frequency. [11]

It is words like these that at times provoke as much as perplex (to put it mildly) the reader. One instance is the author's refusal, maintained with admirable tenacity, to furnish proof for his sometimes slightly outré theses, something that could be accomplished by naming protagonists of the position he criticizes. In this manner, readers are constantly tormented by the question of who could have been meant by a particular comment. To remain with the example quoted above: Who are the protagonists of such an understanding of evaluation? Are the concepts of a participatory, collaborative empowerment evaluation (cf. e.g. FETTERMANN, KAFTARIAN & WANDERSMAN 1996) included? And is the differentiation of three evaluation purposes (judgment-oriented, improvement-oriented and knowledge-oriented) introduced by Michael Quinn PATTON (1997, pp.63-85), the last of which is expressly understood as an alternative to instrumental forms of utilization, a counterargument or just a well-disguised variant of the controlling rationality of evaluation? [12]

If we take a closer look at the arguments presented, a second problem arises in the form of the accompanying melodies that shape the criticism. Again taking the above as an example, saying that evaluators evaluate evaluands provokes the counterquestion: well, what else should they do? Obviously projects, programs and policies are "evaluands", objects of evaluation: a statement replete in triviality. Thomas A. SCHWANDT, however, tends to give a wording to such statements that offers the reader a choice between rejecting an idea and feeling bad about it—as if, in our example, this were a drastic as much as classic case of unwarrantedly turning a subject into an object. This might just be acceptable if the evaluations were to refer to subjects but, as the author himself states, they are about projects, programs and policies. [13]

And, just as obviously, they must be observable (to make the point again from another angle) otherwise you would get into deep water because you cannot observe anything. It is annoying when the author briskly makes things measurable and refers to properties and features, because this procedure defrauds them of a broad discussion of what can be observed, and which of this is measurable, under which conditions—at the same time suggesting that social research itself can be expertly reduced to the measurement of properties and features. [14]

This applies analogously for the second argument: it is only right and proper to point out that evaluation has long since been a form of social power and that the expert knowledge produced by evaluation becomes politically effective in many ways. But the argument is misleading when it claims that evaluation at least implicitly aims to tame disorder and ambiguity and to mold citizens. This might be just about acceptable when expending plenty of argumentative effort and strong presumptions in terms of control theory, e.g. by turning to Michel FOUCAULT or having recourse to the theses of instrumental reason set out by Theodor W. ADORNO. But first, and provided that these prerequisites are met, more effort would have to go into developing rather than just suggesting the argument, and secondly, faced with such strong theoretical assumptions, the author would inevitably collide with his own position, because according to such logic evaluation would necessarily have to be understood as a controlling instance in the sense of practical hermeneutics. [15]

In this way, a few simple suppositions and innuendos suffice to reduce heterogeneous discussion strains and development skeins to much too simple a denominator in order to make them easier to repudiate. The lack of differentiation and the sheer rigidity of the cudgel swung make many of these passages of little use for furthering the debate. [16]

It should be added that the book also includes a number of passages that demonstrate how taking a different vein is possible. One example is the discussion of the social role of evaluation. In analyzing a number of authors (such as Thomas D. COOK, William R. SHADISH, Michael SCRIVEN and Ernest R. HOUSE), the author arrives at the thesis that evaluation must not be understood just as the practice of evaluators, but has since become an effective "sociopolitical institution" (p.174) and "an institutional practice that mediates the relationship between citizens and their social world" (p.176). Even though empirical proof is again in short supply, this is at least open to argument; and it would be worth a separate analysis and discussion whether and to what extent such a diagnosis also applies for the German-speaking countries. [17]

Notwithstanding all the dissatisfaction with the critical preliminaries, we must not forget their actual function—to supply a contrast to the author's own position. Accordingly, we might be inclined to be a little lenient. After all, the severity of the criticism causes the level against which the author's arguments are measured to be raised very high indeed. [18]

In order to do justice to the author's position it helps to recognize that his argument revolves around a specific understanding of a practical concept modeled closely on philosophical hermeneutics, or, to be more precise, leaning heavily on Hans-Georg GADAMER—which incidentally explains quite a lot of the borderlines introduced, e.g. when the author embraces GADAMER's differentiation (rooted in Aristotle) between the moral knowledge of phronesis and the theoretical knowledge of epistemics (GADAMER 1975, pp.297ff). In this sense, Thomas A. SCHWANDT specifies that "Praxis is always related to our being and becoming a particular kind of person and requires a mode of knowledge called practical wisdom (phronesis)" (p.49, italicized in the original). Practice of this kind embedded in a person's given life-world provides an uncircumventable horizon of processes of communication and interpretation. Research and thus also evaluation are accordingly understood as practical hermeneutics:

"Practical hermeneutics is concerned with the mode of activity called the practical (praxis). Its matter is how an individual conducts her or his life and affairs as member of society. The goal of practical philosophy is to raise to the level of reflective awareness the exercise of distinctly human traits or basic human capacities involved in this kind of moral-political action in the world" (pp.47-48). [19]

This type of research is characterized by four moments: "ethics", "deliberative excellence", "poetics" and "rhetoric" (pp.50ff). The ethical aspect obtains its prominent role from the fact that science as a practical philosophy is to be understood chiefly as "ethics of judgment" or as a "moral science". Wisdom and consideration ("deliberation") are constitutive moments of moral knowledge, argumentation and action. The author uses "poetics" to achieve a sharp contrast to epistemic knowledge (in the sense quoted above) in order to emphasize practical reason, creativity and imagination (p.53). Rhetoric, finally, points out that practical reason is linked to the art of persuasion (ibid). [20]

Those who have given careful attention to the work of Hans-Georg GADAMER, Charles TAYLOR or, e.g., Jürgen HABERMAS will have some familiarity with large parts of these deliberations. The essential difference is that the author enriches their arguments by a few semantics loaned from recent discussions on postmodernity—e.g. when evaluators are understood to be "strong poets", narrators of "dense" stories (Chapter 7, pp.119-133). Furthermore, the thoughts of Zygmunt BAUMAN (1987) play a role that should not be underestimated. His thesis of the modern intellectual changing from being a legislator of objective universal truth and rationality to being a postmodern interpreter of local and context-bound knowledge has marked the book in many respects—not least by the role-definition proposed of the evaluator as an interpreter and teacher (see below and Chapter 11). [21]

Positive mention should be made of the fact that the author, in spite of his assumption that there is a continuum between the social sciences and literature (p.114), recognizes that social-science "narratives" are subject to special standards. Without going into the details of his exposition on validity, generalization and use as good-practice criteria (pp.128 ff), it should nevertheless be noted that the difficulties are recognized even though no satisfactory response is found: "The evaluator as poet must, of course, explore how to achieve verisimilitude or lifelikeness in stories and portrayals, how to establish narrative fidelity, and how to achieve an invitational quality in the construction of the story or portrayal" (p.132). [22]

If we now ask about the outcome in terms of evaluation of this "new way of thinking about the activity of judging the merit, worth, or significance of some human action" (blurb), the key advantage of this position will be quickly visible: with research, and thus evaluation, understood to be a "moral science", the author can proactively turn to the subjects of ethics, morals and value decisions. While the traditional discourse on evaluation typically perceives it as a proper (i.e. value-neutral) social science and accordingly begins to writhe in pain once the question is raised of how and on what basis the inevitably pending actual evaluation is performed, practical hermeneutics offers a comparatively easy approach. [23]

Accordingly, two full Chapters (8 and 9) are explicitly devoted to this complex issue, and the subject is reintroduced in many other places and contexts. As expected, the postulate of freedom from values, and the scientific understanding linked with it, is first of all rejected. The author's own position, described as a "value-critical framework" (pp.151 ff), emerges as a balance between emancipatory, i.e. normatively highly charged, concepts on the one hand (cf. pp.148 ff), and something called radically postmodern social constructivist positions on the other hand (cf. pp.161 ff):

"Evaluation practices based in a value-critical framework decenter (the) conception of the aim, nature, and place of social inquiry in social life. They do so by redefining social inquiry as a dialogical and reflective process of democratic discussion and philosophical critique" (p.151). [24]

This is not done selflessly, but with a clear objective in sight—that of "improving praxis" (italicized in the original, p.152)—and with a surprisingly clear educational impetus: the object is for "evaluator(s)-as-teacher" (p.131) to help practitioners develop practical wisdom, i.e. "to help clients cultivate this capacity" (p.153). Evaluations in this sense are understood to be an "educative experience" (p.21), "because they help people to come to a clearer understanding of who they (and others) are, to a clearer picture of the meaning of their practices and the extent of their moral responsibility for their actions ..." (p.21). In view of this functional description of evaluation we might ask about the state of innocence or benightedness among practitioners that such a perception assumes. We might also note that this dialogue-based educational understanding of enlightenment has a surprisingly modern touch about it—even though the author does not see it thus, because for him modern understanding of education equals "managing and controlling self and society" (p.21). [25]

The claim of educational enlightenment apart (suspect not least because it is nowhere substantiated as such), the debate of the values and judgments issue is a typical example of how fruitful approaches will sooner or later end up in smoke without leaving a trace. Even if we take into account that the author was primarily concerned with describing a conceptual framework, disappointment will set in at the latest when we ask what all this means for evaluation practice, for methodology, for methods and processes. In contrast to the promise of the title, the book, with the exception of a few rather cursory references, is not about the practice of evaluation, but about a rather metatheoretical framework, about—to quote the blurb again—another way of thinking about evaluation. Yet it is a long way from there to the day-to-day practice of concrete evaluation studies, and at many points the reader will be left behind, helpless, not knowing how to struggle on. We might object that behind all this, a modern, because methodical, understanding of research and evaluation is concealed. But by this point we have come full circle. [26]

It should be noted that the work also raises, without settling, questions because it pleads, on the one hand, for a specific variant of educational-interpretative-enlightening evaluation, yet on the other hand points out (quite rightly in my opinion) that social change has turned evaluations into a powerful instance of influence. Nevertheless it should be considered that, at least when we look at the ongoing discussions in the German-language and European context, both politics and public normally have an entirely different understanding of evaluation—as a way of checking whether public money is spent effectively and efficiently. In a clear majority of cases, this understanding of evaluation is simply incompatible with the proposals submitted by Thomas A. SCHWANDT. It takes a rare confluence of happy events to reconcile evaluation as philosophical and reflective criticism, as a democratic dialogue with a claim of educational enlightenment attached to it, with the issue that is in everybody's mind: "What works and what doesn't work," the untiring search for "best practice", and interest in using available means in as "targeted" a way as possible. For Thomas A. SCHWANDT, all this means is asking about the institutional and political prerequisites for his concept. [27]

Ultimately, the upshot of all this is ambiguous. Those readers who are interested in debating general paradigms of evaluation research will find numerous suggestions, quite a few pertinent provocations, and the cornerstones of an understanding of evaluation as a variant of practical hermeneutics which has not yet been put into words elsewhere. But those who are more interested in methodical issues and expect answers to the practical problems of research and processes should not be misled by the title: evaluation practice is once again (we are inclined to say) given very short shrift. [28]

Bauman, Zygmunt (1987). Legislators and Interpreters: On Modernity, Postmodernity and Intellectuals. Cambridge: Polity Press.

Denzin, Norman & Lincoln, Yvonna S. (Eds.) (2000). Handbook of Qualitative Research (2nd revised edition). Thousand Oaks: Sage.

Dunne Joseph (1993). Back to the Rough Ground: "Phronesis" and "Techne" in Modern Philosophy and in Aristotle. Notre Dame: University of Notre Dame Press.

Dunne, Joseph (1997). Back to the Rough Ground: Practical Judgment and the Lure of Technique (preface by Alasdair MacIntyre). Notre Dame: University of Notre Dame Press.

Fettermann, David M.; Kaftarian, Shakeh J. & Wandersman, Abraham (Eds.) (1996). Empowerment Evaluation: Knowledge and Tools for Self-Assessment & Accountability. Thousand Oaks: Sage.

Gadamer, Hans-Georg (1975). Wahrheit und Methode. Grundzüge einer philosophischen Hermeneutik (4th edition). Tübingen: Mohr.

Joint Committee on Standards for Educational Evaluation & Sanders, James R. (Eds.) (1999). Handbuch der Evaluationsstandards. Opladen: Leske + Budrich.

Kromrey, Helmut (2003). Evaluation. In Wissenschaft und Gesellschaft. Zeitschrift für Evaluation, 2(1), 93-116.

Lindbloom, Charles E. (1990). Inquiry and Change: The Troubled Attempt to Understand and Shape Society. New Haven: Yale University Press.

Lüders, Christian (1996). Between Stories—Neue Horizonte der qualitativen Sozialforschung? In N. Denzin & Y. Lincoln (eds.): Handbook of Qualitative Research. Sozialwissenschaftliche Literatur Rundschau, 19(31/32), 19-29.

Patton, Michael, Q. (1997). Utilization—Focused Evaluation. The New Century Text (3rd edition). Newbury Park: Sage.

Schwandt, Thomas A. (1997). Evaluation as Practical Hermeneutics. Evaluation, 3(1), 69-83.

Schwandt, Thomas A. (2000). Further Diagnostic Thoughts on What Ails Evaluation Practice. American Journal of Evaluation, 21(2), 225-229.

Schwandt, Thomas A. (2001a). Understanding Dialogue as Practice. Evaluation, 7(2), 228-237.

Schwandt, Thomas A. (2001b). A Postscript on Thinking about Dialogue. Evaluation, 7(2), 264-276.

Schwandt, Thomas A. (Ed.) (2001c). Dictionary of Qualitative Inquiry (2nd revised edition). London: Sage.

Schwandt, Thomas A. (2003). 'Back to the Rough Ground!' Beyond Theory to Practice in Evaluation. Evaluation, 9(3), 353-364.

Dr. Christian LÜDERS is head of the Department of Youth and Youth Services at the German Youth Institute in Munich. His scientific interests include child and youth services research, qualitative research, evaluation research, theory of social work, theory of pedagogical knowledge and sociology of science.

Lüders, Christian (2004). Evaluation as Practical Hermeneutics—or The Long and Stony Road from a Theory of Practice to Evaluation Practice. Review Essay: Thomas A. Schwandt (2002). Evaluation Practice Reconsidered [28 paragraphs]. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 5(1), Art. 29, http://nbn-resolving.de/urn:nbn:de:0114-fqs0401291.

Forum Qualitative Sozialforschung / Forum: Qualitative Social Research (FQS)

ISSN 1438-5627

Creative Commons Attribution 4.0 International License