Evaluation in Distance Education and E‐Learning: The Unfolding Model

Citation

Tran, N. (2013), "Evaluation in Distance Education and E‐Learning: The Unfolding Model", Journal of Educational Administration, Vol. 51 No. 1, pp. 100-102. https://doi.org/10.1108/09578231311291486

Evaluation in Distance Education and E‐Learning: The Unfolding Model is a comprehensive book that serves as a beginner's guide to program evaluation for distance and e‐learning courses. Building on Messick's framework for establishing the validity of standardized tests, Ruhe and Zumbo (2009) developed The Unfolding Model, a road map to guide evaluation practice for distance education and e‐learning. They assert that “the real power of the unfolding model, an adaptation of Messick's (1989) framework, is that it is comprehensive, dynamic, and adaptive to diverse course designs and technologies” (p. viii). The unfolding model consists of four elements – scientific evidence, relevant/cost‐benefit, underlying values, and unintended consequences. By highlighting the underlying values and unintended consequences, the authors have identified the missing features frequently observed in other evaluation models. Recognizing the rapid technological advances of our world, the model is intended to be “dynamic and will continue to evolve in response to new technologies” (p. 244). Readers of this book will be informed of the evaluation process that is centered on distance education and e‐learning.

Ruhe and Zumbo organized the book in nine chapters, each providing information necessary to understand the rationale, theory, and practice relevant to distance education and e‐learning. In Chapter 1, “Why Do We Need a New Approach to Evaluation in Distance Education and E‐Learning,” the authors provide distinctive definitions for distance education and e‐learning. They then propose a new and comprehensive approach for evaluating distance and e‐learning, one that goes beyond providing scientific evidence to highlighting the relevance, cost‐benefit, values, and unintended consequences – all central elements of the unfolding model.

Chapter 2, “The Theory and Practice of Program Evaluation” presents an overview of the various models applied in program evaluation studies with relative emphasis on the methodology, values, and consequences. The authors argue that while numerous evaluation models exist, “little consensus on their characteristics, merits, and weaknesses” (p. 25) has been established by the discipline as a whole. Readers will enjoy Chapter 3, “Evaluation Theory and Practice in Distance Education and E‐Learning,” in which the authors provide a comprehensive review of the theory and practice relevant to distance education and e‐learning over the last two decades. Ruhe and Zumbo contend that “although there has been a recurring call in the literature for a more professional approach to evaluation, there is little articulation of what this approach might look like. We maintain that this approach depends on an analysis of scientific evidence, values, and consequences” (p. 71). Second, “evaluation practice in distance education tends to be methods oriented, even though values and unintended effects have been documented” (p. 73). Consequently, they propose the unfolding model, an adaptation of Messick's framework on test validity, as a professional approach for evaluating distance and e‐learning courses. It is noteworthy to highlight that unlike other models, the unfolding model brings values and unintended consequences to the foreground in the evaluation process. In greater detail, Chapter 4, “Messick's Framework: What Do Evaluators Need to Know?,” explains the significant overlap between Messick's conception of assessment validity, which consists of construct validity, relevance, value implications, and social consequence, and the unfolding model, which comprises of scientific evidence, relevant/cost‐benefit, underlying values, and unintended consequences.

In Chapter 5, “Getting Started,” the authors share with the readers their experience on how to get started with an evaluation study including preparation for the study, tips for getting approval from the Institutional Review Board (IRB), awareness of the political context of evaluation, and consider obtaining input and feedback from various stakeholders. The chapter ends with suggestions for integrating both quantitative and qualitative approaches, sampling, and establishing generalizability for the findings in the evaluation process. Guidelines for instrument development, identification of outcome measures, data collection, analytical techniques for both quantitative and qualitative approaches, and procedures for conducting cost‐benefit analysis are also offered in Chapter 6, “The Unfolding Model: Scientific Evidence.”

Highlighting the unique features of the book, “The Unfolding Model: Values and Consequences” in Chapter 7 provides resources to help the readers identity the underlying values and unintended consequences of distance education and e‐learning courses. For example, “distance/e‐learning courses are designed for a wide range of purposes, including geographical reach, flexibility, pedagogical richness, and depth or to increase revenues and cost cuts. Therefore, your evaluation study should include an analysis of the underlying values of distance courses” (p. 167). Instructional and social consequences are two types of unintended consequences, both of which can be identified by answering the following question: “Is there a gap between how the course was expected to work and how it actually worked?” (p. 193). Specific guidelines are also given in the following areas: developing survey and interview questions, collecting data, analyzing data, generating recommendations, and writing evaluation reports – all essential activities needed to provide insights on the values and unintended consequences of distance and e‐learning programs.

Chapter 8, “Findings from Two Authentic Case Studies,” describes how features of the unfolding model have been used to guide the evaluation process of two different post‐secondary distance and e‐learning courses. Information about the methods, procedures, scientific evidence, values, and unintended consequences are clearly articulated in the evaluation reports of these studies. More importantly, “differences in these two case reports show how you can tailor the unfolding model to collect diverse kinds of scientific evidence that meets the diverse needs of stakeholders and still provides a comprehensive assessment of merit and worth” (p. 235).

In the final chapter, “Bringing It All Together,” Ruhe and Zumbo engage the readers in an exploration of the potential evolution of the unfolding model by raising a series of questions: “Does scientific evidence belong under only one category or could it belong under more than one category? Does the category or categories in which overlapping concepts are placed matter? […] Can the framework be used to evaluate interventions as well as courses? Would the model perform equally well? Would the findings be as rich?” (p. 243). Finally, the authors call for further empirical investigation of the unfolding model and acknowledge that “future applications of our framework in diverse technological and pedagogical contexts are needed to address these kinds of questions and to validate or extend our model” (p. 243).

Evaluation in Distance Education and E‐Learning: The Unfolding Model is suited for beginning evaluators or those who are interested in understanding the evaluation process for distance and e‐learning courses. Readers will find the theoretical framework and practical guidelines provided in this book helpful.