Methods in longitudinal school improvement research: state of the art

Citation

Feldhoff, P.T., Radisch, P.F. and Klieme, P.E. (2014), "Methods in longitudinal school improvement research: state of the art", Journal of Educational Administration, Vol. 52 No. 5. https://doi.org/10.1108/JEA-05-2014-0058

Methods in longitudinal school improvement research: state of the art

Article Type: Guest editorial From: Journal of Educational Administration, Volume 52, Issue 5.

In May 2013 an international conference on "Designs and Methods in School Improvement Research" took place at the German Institute for International Educational Research in Frankfurt, Germany. The conference goals were to discuss requirements for designs and methods in quantitative school improvement research and to contribute to methodological development in school improvement research. The conference sought to examine research designs and methods used by other disciplines and assess their viability for use in quantitative school improvement research.

The conference had two thematic foci. First, schools as organizations are characterized by specific conditioning factors that need to be taken into account when selecting research designs and methods. For example, school improvement studies need to take into account both the multilevel structure of schools and the need for longitudinal measurement of school improvement effects. Second, given the paucity of experimental research in this field, we wished to understand if and how experimental designs could be productively applied to leverage knowledge accumulation in the domain of school improvement. Thus, several conference presentations discussed design requirements and implementation strategies when conducting intervention studies at the school level.

Following the successful conference, the idea of introducing the conference findings to a broader community in a special issue was discussed. Given both its centrality to the study of school improvement, and the scarcity of longitudinal studies in the extant literature, this special issue focusses on the theme of longitudinal research on school improvement. The special issue is comprised of papers presented at the conference as well as other relevant studies sourced from leading scholars working in this domain.

The role of longitudinal studies in school improvement research

School improvement targets the school as a whole. As an organizational process, school improvement is aimed at impacting collective improvement capacity, the skills of its members, and the learning conditions and outcomes of students (Hopkins, 1996; Maag Merki, 2008; Mulford and Silins, 2009; Murphy, 2013; Velzen et al., 1985). In order to achieve sustainable school improvement, school practitioners engage in a complex process comprised of diverse strategies implemented at the district, school and classroom levels (Heck and Hallinger, 2010; Mulford and Silins, 2009; Murphy, 2013).

Although school leaders aim to achieve a positive impact on school-level results, it is widely recognized that school-level changes only become evident after individual teachers recontextualize, adapt and implement them in their individual classrooms (Hall, 2013; O’Day, 2002). Therefore, we view school improvement as an intentional, planned change process that unfolds at the school level. Its success, however, crucially depends on the incorporation of change in the attitudes and actions of individual teachers (Hall, 2013). As Stoll and Fink (1996) pointed out, "Although not all change is improvement, all improvement involves change" (p. 44).

We, therefore, assert that because school improvement always implies a change in organizational conditions (e.g. attitudes, behaviors, practices, capacity, outcomes) over time, it is most appropriately studied in a longitudinal perspective. The rationale for using longitudinal designs in school improvement research is not, however, only grounded in the conceptual argument that change occurs over time (e.g. Ogawa and Bossert, 1995), but also in the methodological requirements for assigning causal attributions to school policies and practices.

Ultimately, school improvement research is concerned with understanding the nature of relationships among different factors that impact on productive change in desired student outcomes over time (Hallinger and Heck, 2011; Murphy, 2013). The assignment of causal attributions is facilitated by reliable theoretical justification as well as by measurement of independent and dependent variables at different points in time (Gustafsson, 2010; Hallinger and Heck, 2011; Zimmermann, 1972). This feature of school improvement research is obviously highly relevant in the evaluation of school improvement and education reform projects whether they are conducted at the school, district or national level.

Recently Thoonen et al. (2012) highlighted the paucity of longitudinal studies in school improvement research: "In spite of the awareness that improvement, by definition, entails change in the state of schools over time, most of the literature on school improvement and educational change relies on studies that do not explicitly describe change and improvement in schools" (p. 444). This lack of longitudinal studies was also observed by Klieme and Steinert (2008) as well as Hallinger and Heck (2011). In Spring 2013, we conducted a search for literature in Eric using the keywords "school improvement" and "longitudinal." We were able to identify less than a handful of relevant articles, thereby reaffirming the assertions of these other scholars.

In the absence of longitudinal studies, quantitative research in school improvement has instead relied on cross-sectional studies. These often compare characteristics of schools with differing developmental stages (e.g. in their improvement capacity and/or school outcomes). Comparisons are typically aimed at the quality of process factors that are assumed to have an effect on school capacity (e.g. leadership, culture or evaluation) to the efficacy and the success of change processes and outcomes. However, fundamental limitations of the cross-sectional design constrain the validity of results when seeking to understand "school improvement."

Thus, we conclude that building a solid knowledge base on school improvement requires the use of longitudinal research designs, regardless of whether researchers employ qualitative, quantitative, or mixed methods. In this special issue, we focus on the latter two methodologies. Notably, however, implementing longitudinal research designs in school improvement research faces numerous challenges. These challenges encompass a wide range of conceptual, measurement, and analytical issues (Creemers et al., 2010; Hallinger and Heck, 2011; Heck and Hallinger, 2010; Klieme and Steinert, 2008; Thoonen et al., 2012). Many of these are elaborated and critically assessed in this special issue.

Objectives of the special issue

Because longitudinal studies have not been widely used in school improvement research, the editors wished to highlight current longitudinal designs in school improvement research drawn from a variety of countries. The studies contained in this special issue deal with different contents and demands in school improvement research. Owing to different research designs and methods, the papers demonstrate diverse possibilities in the manner of conducting longitudinal studies of school improvement. The papers, therefore, highlight advantages and disadvantages of different approaches. While we believe that these papers are of high quality, the complexity of school improvement still demands further development in the conceptualization of school improvement as well as in our research designs and methods. This said, the editors invite readers to read and discuss the papers critically. Hopefully, your own research will receive inspiration from these pioneering efforts to examine school improvement from a longitudinal perspective.

Introduction of the papers

The first two papers discuss methodological demands in longitudinal studies of school improvement. In their paper, Pamela Sammons, Susila Davis, Christopher Day and Qing Gu discuss the use of mixed methods in school improvement research. They show how qualitative and quantitative approaches can be fruitfully combined to explore school improvement processes. Their particular focus is on analyzing the direct and indirect influences of school leadership practices on student outcomes. The quantitative longitudinal part of this paper identifies different groups of improving and effective schools using a composite data set drawn from different national data sources. Value-added indicators of student progress over three years and raw indicators at the school level were used to identify and select cases for further qualitative analysis. The paper demonstrates how combining the strengths of different methodological approaches expand the range of possible analytical strategies. The availability of more robust analytical strategies, in turn, enables researchers to draw more refined conclusions from the data.

In her paper, Katharina Maag Merki deals with the difficulties of assigning subjects to experimental and control groups in intervention studies on school improvement. The ability to compare experimental and control groups and their representativeness of the original population is a strength of intervention studies. However, implementing this in practice is also challenging. Maag Merki points out that in most intervention studies very general factors (e.g. type of a school or district, achievement level, student composition) are used for choosing a sample. However, crucial factors for school improvement (e.g. experience in school improvement activities, improvement capacity) are seldom considered. These characteristics may also affect the comparability of groups. Maag Merki demonstrates the issues that arise from this challenge in the context of a longitudinal intervention study of teacher cooperation in Germany.

The next three papers focus on the effects of school leadership as part of the schools’ capacity. Sleegers, Thoonen, Oort and Peetsma report on a study that examined capacity building and its impact on changes in teachers’ classroom practices over a four-year period of time. Hence, they make a significant contribution to the limited empirical knowledge base on how schools build capacity over time and the resulting influence on teaching practice. In their study, school-wide capacity for improvement is defined by four different factors: leadership practices, school organizational conditions, teacher motivation, and teacher learning. Capacity is proposed as a key factor that shapes the conditions that lead to an improvement in quality of instruction. The authors use multilevel, fixed-occasion models to investigate the impact of building school-wide capacity on change in teachers’ instructional strategies at 32 elementary schools in the Netherlands. In this type of analysis, time (measurement occasions) is modeled as a unique hierarchical level (Level 1). Time is nested within teachers (Level 2), and teachers are nested in schools (Level 3). Sleegers and colleagues found that factors measured on the teacher level (e.g. engagement in professional learning activities and teacher motivation) had the largest impact on teachers’ classroom practices. Organizational factors like leadership seem to moderate this impact.

Ronald H. Heck and Philip Hallinger used longitudinal data from 60 primary schools in the USA to bring together two important strands of school improvement research that aim at enhancing student outcomes, school leadership and effective teaching. A hallmark contribution of this paper is the investigation of collaborative leadership effects on growth in student achievement via effective teaching and school improvement activities in a single analytical model. The study is also notable in its use of "cross-classified data" on students in order to track changes in the achievement of individual students over time as they study with different teachers. To consider this change over two years, they use a four-level growth curve model. The models show that instructional environment and teacher effectiveness have a positive impact on students’ ending math achievement. Leadership also has an indirect impact on students’ ending math achievement via the instructional environment.

Peter Goff, Ellen Goldring, J. Edward Guthrie and Leonard Bickman present findings from a longitudinal experimental study of a principal coaching intervention. The study sought to understand how coaching impacted principals’ characteristics and teachers’ perceptions of the principals’ behavior. The authors found differential effects of feedback and coaching and concluded that it is fruitful to couple feedback and coaching with other initiatives to sustain effective school improvement. The study is noteworthy for its application of a sophisticated experimental design in a field where experimental research is rarely found.

Alex J. Bowers and Bradford R White present a six-year longitudinal study of all elementary and middle schools in one state in the USA. The authors examine the effects of principal and teacher characteristics on student learning growth using a state-based standardized achievement test. Bowers and White used growth mixture modeling to identify different trajectories in academic achievement at the school level. They found two different trajectory groups and numerous significant influences of different principal and teacher characteristics. Strengths of this study include the relatively long time period, the large sample of schools, and the strength of the analytical method (i.e. growth mixture modeling).

As a whole the papers demonstrate the use of a wide range of longitudinal research designs and methodological approaches in studies of school improvement. The papers reinforce our belief that sophisticated quantitative longitudinal designs and methods can be applied productively in school improvement research. Moreover, these studies further illustrate how school improvement research can draw upon new primary (e.g. Golf et al., 2014; Maag Merki, 2014), pre-existing secondary data (e.g. Bowers and White, 2014; Heck and Hallinger, 2014; Sleegers et al., 2014), or a combination of both (Sammons et al., 2014). Because of the huge effort and costs that are associated with the repeated measurement of large numbers of schools in longitudinal studies, the use of existing secondary data sets in school improvement research seems especially attractive. Such data provide numerous possibilities and, as demonstrated by Sammons and colleagues, can also be complemented by the collection of additional data. This can be a very effective technique that enables the researcher to compensate for some limitations of secondary data.

In sum, we wish to suggest that the papers, individually and as a collection, demonstrate both the challenges and the potential of longitudinal research on school improvement. In our view, the authors were unusually forthright in acknowledging the practical challenges that they faced in applying research designs (e.g. experiments, longitudinal studies), measurement model (e.g. cross-classification of students and teachers, multilevel fixed-occasion models), and analytical methods (e.g. growth curve analysis, growth mixture) that are seldom found in the school improvement literature. We conclude that studies such as these that bring together theories and more robust methods will advance school improvement research in the years ahead.

Professor Tobias Feldhoff - Department of Educational Quality and Evaluation, German Institute for International Educational Research (DIPF), Frankfurt, Germany

Professor Falk Radisch - Department of Educational Science, University of Rostock, Rostock, Germany

Professor Eckhard Klieme - Department of Educational Quality and Evaluation, German Institute for International Educational Research (DIPF), Frankfurt, Germany

References

Bowers, A.J. and White, B.R. (2014), "Do principal preparation and teacher qualifications influence different types of school growth trajectories in Illinois? A growth mixture model analysis", Journal of Educational Administration, Vol. 52 No. 5, pp. 705-736

Creemers, B.P.M., Kyriakides, L. and Sammons, P. (Eds) (2010), Methodological Advances in Educational Effectiveness Research, 1st ed., Routledge, Milton Park

Golf, P., Goldring, E., Guthrie, J.E. and Bickman, L. (2014), "Changing principals’ leadership through feedback and coaching", Journal of Educational Administration, Vol. 52 No. 5, pp. 682-704

Gustafsson, J.E. (2010), "Longitudinal designs", in Creemers, B.P.M., Kyriakides, L. and Sammons, P. (Eds), Methodological Advances in Educational Effectiveness Research, 1st ed., Routledge, Milton Park, pp. 77-101

Hall, G.E. (2013), "Evaluating change processes: assessing extent of implementation: constructs, methods and implications", Journal of Educational Administration, Vol. 51 No. 3, pp. 264-289

Hallinger, P. and Heck, R.H. (2011), "Conceptual and methodological issues in studying school leadership effects as a reciprocal process", School Effectiveness and School Improvement, Vol. 22 No. 2, pp. 149-173

Heck, R.H. and Hallinger, P. (2010), "Collaborative leadership effects on school improvement: integrating unidirectional- and reciprocal-effects models", The Elementary School Journal, Vol. 111 No. 2, pp. 226-252

Heck, R.H. and Hallinger, P. (2014), "Modeling the longitudinal effects of school leadership on teaching and learning", Journal of Educational Administration, Vol. 52 No. 5, pp. 653-681

Hopkins, D. (1996), "Towards a theory for school improvement", in Gray, J., Reynolds, D., Fitz-Gibbon, C. and Jesson, D. (Eds), Merging Traditions. The Future of Research on School Effectiveness and School Improvement, Cassell, London, pp. 40-60

Klieme, E. and Steinert, B. (2008), "Schulentwicklung im Längsschnitt. Ein Forschungsprogramm und erste explorative Analysen", in Prenzel, M. and Baumert, J. (Eds), Vertiefende Analysen zu PISA 2006, Waxmann, Münster, pp. 221-238

Maag Merki, K. (2008), "Die Architektur einer Theorie der Schulentwicklung-voraussetzungen und Strukturen", Journal für Schulentwicklung, Vol. 12 No. 2, pp. 22-30

Maag Merki, K. (2014), "Conducting intervention studies on school improvement: an analysis of possibilities and constraints based on an intervention study of teacher cooperation", Journal of Educational Administration, Vol. 52 No. 5, pp. 590-616

Mulford, B. and Silins, H. (2009), "Revised models and conceptualization of successful school principalship in Tasmania", in Mulford, B. and Edmunds, B. (Eds), Successful School Principalship in Tasmania, Faculty of Education, University of Tasmania, Launceston, pp. 157-183

Murphy, J. (2013), "The architecture of school improvement", Journal of Educational Administration, Vol. 51 No. 3, pp. 252-263

O'Day, J. (2002), "Complexity, accountability, and school improvement", Harvard Educational Review, Vol. 72 No. 3, pp. 293-329

Ogawa, R. and Bossert, S. (1995), "Leadership as an organizational quality", Educational Administration Quarterly, Vol. 31 No. 2, pp. 224-243

Sammons, P., Davis, S., Day, C. and Gu, Q. (2014), "Using mixed methods to investigate school improvement and the role of leadership: an example of a longitudinal study in England", Journal of Educational Administration, Vol. 52 No. 5, pp. 565-589

Sleegers, P.J.C., Thoonen, E.E.J., Oort, F.J. and Peetsma, T.T.D. (2014), "Changing classroom practices: the role of school-wide capacity for sustainable improvement", Journal of Educational Administration, Vol. 52 No. 5, pp. 617-652

Stoll, L. and Fink, D. (1996), Changing our Schools: Linking School Effectiveness and School Improvement, Open University Press, Buckingham

Thoonen, E.E., Sleegers, P.J., Oort, F.J. and Peetsma, T.T. (2012), "Building school-wide capacity for improvement: the role of leadership, school organizational conditions, and teacher factors", School Effectiveness and School Improvement, Vol. 23 No. 4, pp. 441-460

Velzen, W.G.V., Miles, M.B., Ekholm, M., Hameyer, U. and Robin, D. (1985), Making School Improvement Work: A Conceptual Guide to Practice, Acco, Leuven and Amersfoort

Zimmermann, E. (1972), Das Experiment in den Sozialwissenschaften, Poeschel, Stuttgart

About the Guest Editors

Tobias Feldhoff, since 2011, has been an Assistant Professor for Empirical Educational Research and School Development at the Department of Educational Quality and Evaluation at the Leibniz Institute for International Educational Research (DIPF) as well as at the Goethe University, Frankfurt. From 2008 to 2011 he was a Senior Researcher at the Institute for the Management and Economics of Education at the University of Teacher Education Central Switzerland. From 2005 to 2008 he worked as a PhD Student at the Institute for School Development Research (IFS) at the TU Dortmund University, where he finished his PhD in 2010. Assistant Professor Tobias Feldhoff is the corresponding author and can be contacted at: mailto:feldhoff@dipf.de

Falk Radisch is a Full Professor for school pedagogics and general didactics at the Department of Educational Science (Faculty of Humanities) at the University of Rostock. From 2010 to 2013 he was an Assistant Professor for Quantitative Methods in Educational Research at the School of Education at the University of Wuppertal. Prior to that he was Vice Director of the Institute for Management and Economics of Education at the University of Teacher Education Central Switzerland in Zug. He finished his PhD in 2008.

Eckhard Klieme is a Full Professor for Educational Sciences at the Johann Wolfgang Goethe-University in Frankfurt/Main and Head of the Center for Educational Quality and Evaluation at the German Institute for International Educational Research(DIPF). From 2004 to 2008 he served as the Director of the DIPF. Eckhard has a strong background in educational measurement, educational effectiveness, quantitative methods, and comparative studies. He graduated from University of Bonn with Master Degrees both in mathematics and psychology, and a PhD in Psychology. Before joining DIPF, he was a Senior Researcher at the Institute for Test Development and Talent Research in Bonn (1982-1997), and the Max Planck Institute for Human Development in Berlin (1998-2001).