Abstract
Purpose
In an era of increased autonomy and accountability in education, school principals have been given the responsibility for many tasks that used to be centralized, such as hiring and managing personnel, ensuring that the curriculum is followed and that the development of the school is on the right path. In this study an exploration is attempted into the associations among institutional context, school leadership, school climate and student outcomes.
Design/methodology/approach
The authors analyze data from the Programme of International Student Assessment study 2015 to identify variations in institutional context (autonomy and accountability) among education systems to determine whether and how institutional context is associated with leadership and whether education systems, in turn, are associated with school climate and students' achievement in reading. To account for the hierarchical structure of the education systems under investigation, a three-level structural equation modeling (SEM) approach was employed to analyze the data gathered on the institutions, schools and students.
Findings
No association was found between the degree of autonomy of an education system and leadership behavior; however, accountability and leadership seemed to go together. Achievement in reading competencies was greater in school systems in which principals had more autonomy. At the school level, a relationship was found between leadership and opportunities for parental involvement.
Originality/value
The results of this study indicate the need of strong leadership in order to have better results when there is more accountability. In this sense, the authors' findings also point to the increased importance of context-sensitive leadership skills.
Keywords
Citation
Kemethofer, D., Weber, C., Brauckmann-Sajkiewicz, S. and Pashiardis, P. (2023), "Examining the trident: how data from the PISA study can be used to identify associations among context, school leadership and student outcomes", Journal of Educational Administration, Vol. 61 No. 2, pp. 162-177. https://doi.org/10.1108/JEA-02-2022-0030
Publisher
:Emerald Publishing Limited
Copyright © 2023, David Kemethofer, Christoph Weber, Stefan Brauckmann-Sajkiewicz and Petros Pashiardis
License
Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode
Introduction
Over the past few decades, many school systems around the world have undergone extensive changes in their governance structures (Cheng et al., 2016). In many school systems, including for example Australia, Canada, Germany, Israel or Singapore, individual schools received more decision-making power and responsibility concerning curriculum and the allocation of resources, and they have been placed in charge of daily operations and strategic direction (OECD, 2016). Autonomy allows schools to be innovative in teaching and learning and to respond more directly to the needs of the communities they serve. At the same time, schools are held accountable for student outcomes and mechanisms have been put in place to determine whether national achievement targets are being met (Keddie, 2015). Thus, accountability serves to counterbalance high levels of school autonomy to ensure the quality and effectiveness of decisions and actions taken at the school level. In this scenario, accountability and autonomy act as “twin dimensions of the reform agenda” (Bush, 2013, p. 697).
In an era of increased autonomy and accountability, school leaders have become key agents facing a wide variety of new tasks. They must not only orchestrate operative management issues, but also deal with organizational and human resources development (Day et al., 2016). In fact, principals must tackle an array of complex scenarios, which place them under pressure: politically, economically, socially, globally and educationally. However, changed governance structures offer several opportunities for principals too by having more room for maneuver and act as the “driving force to make accountability frameworks fit their school” (Tulowitzki, 2016, p. 236).
Thus, the purpose of this paper is to uncover associations among contextual characteristics, school leadership and student achievement from a cross-country perspective, with aspects of school climate as a possible mediator of the relationship between leadership and student achievement. For the analysis, we use data from the Programme of International Student Assessment (PISA) 2015 (OECD, 2017a).
Theoretical framework
To describe direct or indirect effects of leadership practices on student achievement several authors developed models or specified them based on empirical research. Established leadership models structure the factors by means of their level of impact on learning outcomes (e.g. Gumus et al., 2018; Gurr et al., 2022; Hallinger, 2011). In this study we focus on the impact of school leadership in different institutional contexts. The institutional context refers to system level variables and describes the features of the education system in which a principal operates. This responds to the calls for school leadership research to broaden the lenses of context (Hallinger, 2018) and for more comparative research (Klein et al., 2022). As previously mentioned, we analyze data from the PISA study 2015. This means that the selection of variables is subject to restrictions. More specifically, an exploration is attempted into the associations among institutional context, school leadership, school climate and student outcomes.
School leadership
In educational policy and research there is consensus that school principals are decisively responsible for the quality of a school. An increasing number of findings support the impact of school leadership on student achievement (e.g. Leithwood et al., 2020a; Liebowitz and Porter, 2019). Most of the studies conducted to date have supported the belief that principals exercise a measurable, though small, indirect effect on student achievement, although quantifying the effect of leadership on student achievement is difficult from an empirical perspective (Grissom et al., 2021). One reason for this is that there are many different connotations described in the context of school leadership (Bush and Glover, 2014; Hitt and Tucker, 2016). Leadership, “understood as a process of influence based on clear values and beliefs and leading to a ‘vision’ for the school” (Bush, 2007, p. 403) has an impact on school organization, culture, norms and practices and thus on teaching and learning. In this sense, leadership is the nexus of all those behaviors and practices that school principals use in order to influence the behavior of others (Brauckmann and Pashiardis, 2011). The number of promising leadership styles seems to be growing steadily, with transformational, instructional and distributed leadership being most widespread (Gumus et al., 2018; Pietsch and Tulowitzki, 2017). The consensus is that these three leadership styles are the most promising for promoting student achievement, but with inconsistent effects depending on the study and when used in combination (e.g. Day et al., 2016; Robinson et al., 2008). Consequently, notable studies in this area show that successful school leaders draw on the same repertoire of leadership practices, that seem to be successful across countries and contexts (e.g. Day et al., 2016; Gurr, 2015; Leithwood et al., 2020a, b; Pietsch and Tulowitzki, 2017).
In this study, these leadership practices are represented by four leadership styles outlined by the Organization for Economic Co-operation and Development (OECD). Instructional leadership style places focus on the quality of teaching and learning. From an instructional leadership perspective, principals have sufficient knowledge about instruction and the curriculum in order to lead teachers. They aim to improve classroom practices by setting expectations, monitoring and evaluating students and teachers, providing feedback and stimulating instructional innovations (Hallinger, 2005). A number of studies have demonstrated positive effects of instructional leadership on teaching, learning and student achievement (e.g. Robinson et al., 2008; Urick et al., 2021). The participative style entails getting teachers involved in decision-making processes and encouraging team-building processes, as needed. As a result, the commitment and sense of organizational affiliation within the school increases as well as the individual and shared job satisfaction (Pietsch et al., 2019), leading to a more positive school climate overall. In this sense, leadership is viewed, from a distributed perspective, meaning that “leading and managing involves more than the actions of the school principal” (Spillane et al., 2007, p. 104). The personnel development style involves recruiting teachers, providing training opportunities and supporting staff. The principal is responsible for creating a coherent concept for professional development that involves recognizing the knowledge, skills, experience, strengths and weaknesses of his/her staff and taking them into account (Grissom et al., 2021). Bruggencate et al. (2012) provide evidence that leadership, based on human resources, correlates with teachers' work. Finally, the structuring leadership style encompasses elements of transformational leadership such as providing clear direction and internal stability with the firm implementation of rules and regulations (Leithwood and Sun, 2012). In short, these activities provide learning conditions under which students and teachers feel safe and can achieve results. Research has shown that creating an organizational structure is associated with the school culture and processes within a school which in turn is associated with student achievement (Brauckmann and Pashiardis, 2011; Scheerens, 2016).
Institutional context: autonomy and accountability
Education policy is seen as an important component of schooling and student outcomes (Kyriakides et al., 2018). To improve educational quality, neoliberal ideologies have made their way into education policy in many countries worldwide (Cheng et al., 2016; Keddie, 2015; OECD, 2016). In particular, autonomy and accountability have become the prominent policies in education. Autonomy allows schools to be innovative in teaching and learning and to respond more directly to the needs they serve. Responsibilities transferred to schools can include numerous tasks. Neeleman (2019) classified a set of 16 decision-making areas for school leaders (e.g. staffing policy, handling financial resources) into three main categories: education, organization and staff. Similarly, the OECD (2017b) divided aspects of school autonomy into responsibility for resources (e.g. hiring and firing teachers, formulating the school budget) and responsibility for curriculum and assessment, including determining course content and establishing student assessment policies.
The shift towards more autonomy for schools is often accompanied by the establishment of centralized accountability measures (Bush, 2013). Although there is no single definition of the term accountability, accountability approaches typically involve the definition of standards, the implementation of evaluation and assessment systems, often coupled with rewards and sanctions, the use of performance data and appraisal of school leaders and teachers (Ozga, 2020). Accountability should make education systems more transparent and identify opportunities for improvement. Accountability thus serves to counterbalance high levels of school autonomy to ensure the quality and effectiveness of decisions and actions taken at the school level. Cross-country analyses show that students perform better when schools are held accountable for their autonomous decisions (Wößmann, 2007). Hanushek et al. (2013) suggest that external accountability also leads to better decision making at the local level. In addition, there are remarkable differences between countries and even within countries with respect to autonomy for schools and accountability measures (Neeleman, 2019; OECD, 2016). For example, in some countries regional or sub-regional bodies are in charge of education policy decisions. According to the OECD (2019), the degree of centralization or decentralization in a system is neither good nor bad, but depends on contextual and specific challenges.
School leadership and the importance of context
With increased decision-making at the school level, the role of the school leader gained in importance (Neeleman, 2019). Thus, (features of an education system have an influence on leadership), making it important to pay attention to the context in which principals lead (Hallinger, 2018). For example, Harris and Jones (2018) argue that sociocultural contexts shape the actions of principals and explain differences in their behavior. Principals operate under different conditions and, therefore, need to adapt their leadership to the needs, opportunities, resources and constraints of their working context (Brauckmann et al., 2023). Surprisingly, however, there is little evidence of the impact of autonomy and accountability on principal behavior and few international comparisons have been conducted (Neeleman, 2019).
Lee and Hallinger (2012) analyzed data from the Progress in International Reading Literacy Study (PIRLS) 2006 to illustrate the impact of national context on principals' use of time. They found differences in the amount of time principals spent on instructional leadership, administrative tasks and interactions with the school environment. They also found principals in less hierarchically organized education systems spent more time on classroom-related activities and on building relationships with parents and the community. On the basis of 24 country cases from all over the world Ärlestig et al. (2016) illustrated that principals in all countries respond to the relation between responsibility and accountability from higher levels. In centralized countries, school leaders spend a lot of time in administration and bureaucracy, and therefore, their attention to other leadership activities is restricted. In education systems without autonomy for staffing and recruitment limited authority of principals is reported. Using the example of three education systems (USA, Norway and Shanghai, China), Johnson et al. (2008) demonstrated the influence of education policies on leadership. High accountability measures and the threat of sanctions led school administrators to place greater importance on student achievement, including strategies on improving test scores. In contrast, leadership in Norway is characterized by democratic values and collaboration whereas Chinese principals lead top-down. Luschei and Jeong (2021) analyzed data from the PISA study 2015 to understand the relationship among school governance structures, leadership and student achievement. While the authors found a considerable range of decision-making authority at the school level, there was no significant correlation between the responsibilities of a principal and student achievement, even after controlling for leadership behavior. In summary, only few empirical studies address the relevance of the institutional context to school leadership systematically.
School climate as a mediator of leadership effects
As outlined above, principals do not directly affect student achievement. One candidate variable that mediates the effects of leadership is school climate. School climate is positively correlated with student achievement (Wang and Degol, 2016; Scheerens, 2016) and there is strong evidence that school leaders shape the climate of a school (e.g. MacNeil et al., 2009). School climate is a multidimensional construct encompassing beliefs, values, shared norms, interpersonal relationships and organizational structures. Wang and Degol (2016) identified four broad domains covering 13 dimensions of school climate: (1) The academic domain encompasses dimensions such as teaching and learning or professional development, (2) the community domain includes amongst others the quality of interpersonal relationships, feelings of connectedness to school and community partnerships as characterized for example by parental involvement in school, (3) the safety domain comprises social and emotional security, discipline and order and (4) the institutional environment domain refers to environmental adequacy and availability of resources. Hence, school climate can be described as the “heart and soul of the school” (MacNeil et al., 2009, p. 75).
Several studies provide evidence that school climate mediates the effects of leadership. Sebastian et al. (2017) found school climate to be a key variable mediating the impact of principal leadership on student achievement. According to their findings, principals have an impact on climate, which in turn has an effect on learning outcomes through classroom instruction. Furthermore, Leithwood et al. (2020b) tested whether various variables of school climate mediate the effect of leadership on student achievement. They found a significant indirect effect of leadership via a composite variable covering disciplinary climate, use of instructional time and academic press on achievement. However, there was no evidence that other aspects of school climate (e.g. teacher commitment) acted as a mediator. Further support for the mediation of leadership on achievement via school climate is provided by Dutta and Sahney (2022).
Research question and hypotheses
Based on the theoretical assumptions and prior research, we assume associations between institutional context, school leadership, school climate and student outcomes. As Hanushek et al. (2013) argue, cross-country data offer the opportunity to analyze associations between the education system context and outcome variables by exploiting variation in institutional level factors among countries. The following research question guided our analysis:
Is there an association between the institutional context (i.e. autonomy for principals and accountability), leadership behavior, school climate and student achievement? We expect on the one hand, that autonomy for principals offers more opportunities for action and accountability and on the other hand, generates more pressure to act (H1). In line with the assumption, that leadership primarily affects student outcomes indirectly, we expect, that leadership is positively related to school climate (H2) and school climate is positively associated with student achievement (H3). These three hypotheses make up a multilevel mediation model (e.g. Preacher et al., 2010) in which the effects of system-level variables (level 3) on student outcomes (level 1) are mediated by two school-level variables (i.e. leadership and school climate, (level 2)). The assumed mediation via three paths is Hypothesis 4 (H4).
Methods
This paper draws on data from the PISA 2015. PISA is administered by the OECD in member-countries around the world and measures key competencies in reading, mathematics and science. Additionally, a questionnaire is administered to students and school principals to gather background information (OECD, 2016). We analyzed data collected on 248,620 students from 9,370 schools in all participating OECD member countries. For more details on sampling see the OECD (2017b).
Measures
Outcome variable
The outcome variable in our study is student achievement in reading (reading literacy). Reading literacy is the ability “to understand, use, reflect on and engage with written texts in order to achieve one's goals, develop one's knowledge and potential, and participate in society” (OECD, 2017a, p. 15). For more information, see supplement and the OECD technical report (OECD, 2017b).
Institutional context – autonomy and accountability
In this study, we focus on institutional context variations among 54 regions. The variable “region” is provided by the OECD and divides several countries in subnational units, provided that within these regions a sufficiently large number of students and schools were sampled. These regions are often characterized by educational system differences (e.g. Basque County or Catalonia in Spain; see INNE, 2016; for results supporting the use of regions instead of countries see supplement). For assessing variables at the system level (i.e. region level), we analyzed principals' self-reports on items reflecting the educational policies of their regions (see also Luschei and Jeong, 2021).
Autonomy
Following the OECD (2017b) we explore two aspects of school autonomy: responsibility for curriculum (4 items, e.g. “Deciding which courses are offered”) and responsibility for resources (6 items, e.g. “Deciding on budget allocations within the school”). Each item was rated 0 if the issue was not within the decision-making power of the principal and rated 1 if the principal had the power to make decisions regarding the issue. To determine whether it is reasonable to assume that these principal reports reflect autonomy at the region level we calculated the intraclass correlations (ICC) and – as a measure of reliability – ICC(2) for all autonomy items (see Stapleton et al., 2016). ICCs (for binary items) ranged from 0.286 to 0.586 (median = 0.404) and ICC(2)-values ranged from 0.984 to 0.995 (median = 0.991; see Table A1 Supplement). Thus, there is considerable variation between and homogeneity within regions and region level components of the items are highly reliable. These results support the assumption that principal reports are suitable to reliably measure characteristics of the education system. Based on the results of a two-level confirmatory factor analysis (CFA; see Supplement) we estimated a single factor score representing autonomy at the region level to be used in subsequent analysis (Composite reliability ω = 0.934).
Accountability – testing
We chose five items related to testing and monitoring practices that represent typical elements of accountability policies at the system level (Ozga, 2020). Three items relate to the use of standardized tests to (1) compare school performance at the district or national level, (2) compare schools with other schools and (3) monitor schools' progress from year to year. Items were rated 1 if principals reported the respective use of standardized tests and otherwise rated 0. We used a fourth item to determine whether mandatory standardized tests were used on a regular basis to assess students' performance (0 = never, 1 = at least 1–2 times a year) and a fifth item to determine whether achievement data were published (0 = no, 1 = yes). Again, ICC and ICC(2)-values (ICC between 0.252 and 0.472; median = 0.312; ICC(2) between 0.981 and 0.983; median = 0.986; see Table A2 Supplement) supported the assumption that the items reliably measure aspects of accountability at the system level. After obtaining the results of a two-level CFA, we computed a factor score (ω = 0.956) representing accountability at the region level.
Accountability – mandatory school improvement
To capture the mandatory quality assurance aspect of accountability (OECD, 2016), we used a single item to assess mandatory school improvement through the help of external experts. The item was rated 1 when principals reported mandatory regular consultation about school improvement matters with one or more experts over a period of at least six months and 0 if there had been no consultation or consultations on a voluntary basis. The ICC for the item was 0.296, indicating again substantial variation among regions. An ICC(2) of 0.985 suggests high reliability of the aggregated item.
Leadership behavior
On the school questionnaire for the PISA study 13 items address school leadership (OECD, 2017b). Similar to other authors (e.g. Leithwood et al., 2020b; Pietsch and Tulowitzki, 2017), the OECD leadership scale (LEAD) encompasses several leadership styles and thus representing a broad definition of school leadership: Instructional leadership (e.g. “I promote teaching practices based on recent educational research”), participative leadership (e.g. “I provide staff with opportunities to participate in school decision-making”), personnel development leadership (e.g. “When a teacher brings up a classroom problem, we solve the problem together”) and structuring leadership (e.g. “I ensure that teachers work according to the school's educational goals”). Principals were asked to rate the frequency of various leadership behaviors and activities during the previous year on a 6-point scale ranging from “did not occur” (= 1) to “more than once a week” (= 6). Scale reliability (median of OECD countries) was 0.897 (see also supplement).
School climate
We included four measures of school climate provided in the data. Two scales refer to phenomena hindering student learning, referring to the safety (discipline) and academic domain (teaching) of school climate (Wang and Degol, 2016). Student learning hindered by teacher behavior (TEACHBEHA) and student learning hindered by student behavior (STUBEHA) were each assessed with five items reported on by principals (TEACHBEHA: e.g. “Teachers being too strict with students”; STUBEHA: e.g. “Students intimidating or bullying other students”). Higher values of the scales indicated a greater learning hindrance. Median reliabilities were 0.791 for STUBEHA and 0.782 for TEACHBEHA (see also supplement).
Sense of belonging to one's school (BELONG) is a measure based on student reports on five items (e.g. “I feel like I belong at school”), referring to the community domain of school climate (Wang and Degol, 2016). Students rated the items on a 4-point scale (1 = strongly agree, 4 = strongly disagree). Items were recoded so that a high score on the BELONG measure indicated a strong sense of belonging to one's school. The median reliability was 0.845 (see also supplement).
Teacher fairness (UNFAIRTEACHER, community domain of school climate) refers to students' experiences of unfair treatment by teachers within the last 12 months. The scale comprised six items (e.g. “Teacher ridiculed me in front of others”) and students responded on a 4-point scale (1 = never or almost never, 4 = once a week or more). The OECD provides a simple sum score for this variable, where higher scores indicate that students more often experience unfair treatment by their teachers.
In addition to using these scales, we computed two more sum score variables. Opportunities for parental involvement was based on four binary (yes/no) items (e.g. “Our school provides a welcoming and accepting atmosphere for parents to get involved”) and reflected the school policy for parental involvement (community domain of school climate). Higher scores indicated that schools offer more opportunities for parents to get involved. Classroom observation (academic domain of school climate) was based on two binary items (yes/no) and referred to internal methods used to monitor teaching practices (e.g. “teacher peer review”) within the last 12 months. Higher values indicated more monitoring methods.
Control variables
We chose several school level variables that are known to influence school and classroom processes as control variables (e.g. Scheerens, 2016). For details, see supplement and OECD (2017b).
Analyses
We applied a three-level structural equation modeling (SEM) approach using Mplus 8.1 (Muthén and Muthén, 1998–2017), considering student (L1), school (L2) and system level (L3) to test our hypotheses. The simplified analysis model is shown in Figure 1. Within a multilevel SEM-approach, manifest measures were decomposed in L3, L2 and – when variables were measured at L1 (such as sense of belonging) – L1 components (e.g. Muthén and Asparouhov, 2011). As we are interested in effects of education system level variables on reading achievement that might be mediated by leadership and school climate, we focus exclusively on L3-effects. We ran an overall model including all context variables, the leadership variable (LEAD), all six climate variables – which were allowed to correlate freely – and reading achievement. However, this approach results in quite a complex model at the region level. Given the small number of clusters at L3 (n = 54), this might lead to biased estimates. For example, Meuleman and Billiet (2009) showed in a simulation study that the parameter and standard error biases increase with growing numbers of estimated parameters at the upper level and are beyond acceptable limits when the number of estimated parameters at the upper level is larger than the sample size. Actually, the overall model comprises 77 parameters at L3. Thus, to back up the results of the overall model, we further estimated separate models using only one out of the six school climate variables. These results are reported in the supplement. For all analyses we controlled for the set of control variables and used weights at the school and student levels (see supplement). Moreover, additional information on model estimation (e.g. estimation of indirect effects) is provided in the supplement.
Results
Due to the focus of the analyses at the region level, it is important to know first, how much variance of the variables can be attributed to between-region differences. ICCs indicate that between 3% (UNFAIRTEACHER) and 31% (classroom observation) of the total variance are accounted for by between-region differences. The correlations of the three system level variables were only small in size (r ≤ 0.319). Of the system level variables, mandatory school improvement (r = 0.629, p < 0.001) and testing (r = 0.495, p < 0.001) correlated strongly with leadership. Thus, in regions with higher levels of accountability, principals claimed to do more leadership activities. The correlation between autonomy and leadership was weaker and not significant. Interestingly, a positive association was found between the leadership and UNFAIRTEACHER (r = 0.375, p < 0.05) and STUBEHA (r = 0.389, p < 0.05). Thus, in regions where principals showed more leadership behavior students experienced more unfair treatment by their teachers and principals claimed more student behaviors hindered learning. Similarly, leadership correlated negatively with sense of belonging (r = −0.590, p < 0.001). In line with our expectations, leadership correlated with opportunities for parental involvement (r = 0.519, p < 0.001) and classroom observation (r = 0.288, p < 0.01). Finally, none of the school climate variables correlated with reading achievement. All correlations and descriptive statistics at the region level are presented in Table A5 in the supplement.
Hypothesis 1 (H1)
The results of the three-level analysis regarding H1 (see Figure 2) showed significant associations between accountability and leadership. Principals in regions with stricter school improvement policies and stricter testing policies reported more leadership activities. Notably, we found no association between the degree of autonomy within a region and principals' leadership. Thus, concerning accountability, H1 is supported, but concerning autonomy, there was no support for H1.
Hypothesis 2 (H2)
There is mixed support for H2 (see Figure 2), which concern system level variations in leadership behavior accounting for between-system differences in measures of school climate. First, there was an association between leadership and opportunities for parental involvement. In regions where principals claimed to exercise more leadership activities, principals also described their schools as offering many opportunities for parents to get involved in their children's education. In contrast to this result, one correlation also showed a different sign than expected. Leadership was negatively associated with students' sense of belonging to their school. Thus, in regions where principals claimed to exercise more leadership activities, students reported feeling a weaker sense of belonging to their schools. To summarize, H2 was only supported in regard to opportunities for parental involvement.
Hypothesis 3 (H3)
The results regarding H3 (see Figure 2), which concerned system level variations in measures of school climate that account for between-system differences in student achievement in reading, indicated a significant association between parental involvement and reading achievement. Thus, students demonstrated stronger reading skills in regions where schools provided parents with more opportunities to get involved in their children's education. Thus, H3 was supported regarding opportunities for parental involvement, but other school climate related constructs were not significantly associated with reading achievement at the system (region) level.
Hypothesis 4 (H4) - indirect effects
As reported above, H1, H2 and H3 were supported simultaneously for opportunities for parental involvement only, that is, in regions with higher levels of accountability principals showed more leadership behavior, which in turn was related to a school policy encouraging opportunities for parental involvement and opportunity for parental involvement was associated with better reading skills. The indirect effect of mandatory school improvement via leadership and opportunities for parental involvement on reading was marginally significant (bind = 0.177, p < 0.10), thus, providing some evidence in line with H4. All other indirect effects via the three paths did not reach statistical significance.
Further notable results of interest
In addition to the reported results concerning our hypotheses, we found several other associations worthy of noting (for details see Table A6 in the supplement). First, contrary to our expectations, we found a negative direct effect of testing on reading achievement (b = −0.056, p < 0.05). Thus, in regions with a stricter testing policy, students' reading achievement was lower than in regions where the testing policy was less strict. Second, autonomy was associated with classroom observation. Principals claimed to conduct more classroom observations in regions with greater autonomy (b = 0.056, p < 0.01). Third, there was an association between testing and opportunities for parental involvement (b = 0.032, p < 0.01). In regions with a stricter testing policy schools offered more opportunities for parents to get involved in their children's education. Fourth, leadership was negatively associated with reading achievement (b = −0.249, p < 0.01), thus, in regions where principals show more leadership activities student reading achievement is lower.
Results of single analyses
Given the relatively high complexity of the overall model, we performed a set of further analyses, each considering only a single school climate variable. Overall, the single analyses (for details see supplement) confirm the results reported above.
Discussion and implications for practice, policy and research
The fact that principals in regions with strict school improvement and testing policies claimed to conduct more leadership activities is interesting. These findings indicate that when principals are held accountable for school performance and student achievement, they exhibit greater leadership, as they probably feel more pressure from higher levels of authority to excel. This also leads to greater parental involvement (as principals probably pressure parents and families to play an active role in their children's education), which leads to greater effort from students to perform well on tests, as our findings suggest. This finding shows that accountability and leadership go together (Grissom et al., 2021). Thus, when principals are held accountable, they must exhibit strong leadership skills to get good results. However, for school leaders to function well under such performance pressure, they need to have some decision-making freedom as well as appropriate and adequate training. A material and legal safeguarding of the concept of an autonomous school, especially against the influence and demands of the school supervision and the school authorities is important (e.g. Gurr, 2015). However, when we examine the actual implementation of autonomy, there are still objections as to the extent to which the current regulations strengthen the role of the principal.
Findings related to Hypothesis 2 indicate a “perceived” conflict in goals: on the one hand, students are under pressure to do better academically, and on the other hand, they are encouraged to feel good at school and be happy members of society. Furthermore, in some places, it becomes apparent that school principals act exclusively in a school-specific and problem-related manner (no autonomy to resolve that specific problem). There are also numerous indications that only jointly perceived pressure or a commonly experienced problem allows leadership action and/or change-initiating work. It is precisely this collective awareness of a problem that sharpens the view of the school's framework/enabling conditions. Consequently, a search can begin, both within and outside the school, for the lack of autonomy that has led to this, now more recognized, problem. For instance, if the need for action is recognized in the school but support is unavailable, the delegated decision-making power seems a half-hearted “pseudo-autonomy”. Thus, school autonomy requires further qualification. Research has shown that policies to enhance school autonomy do not necessarily lead to autonomy in practice (Neeleman, 2019).
Regarding classroom observation, probably management concepts closest to teaching-learning relationship in terms of content obviously receive the most attention from principals and management concepts that emphasize the school's individual accountability to the outside world receive less. Apparently, measures of de- and recentralization often lead to uncertainty on the part of the principals involved. Structurally imposed room to maneuver can have a paralyzing effect, as these measures are only partially practiced or exhausted due to fears of originally unintended transgressions. Principals need security and information and to coordinate with school supervisors.
The findings further suggest that leadership performance can be achieved through instruments of New Governance to a limited extent only. In line with the findings reported here, a highly contextual situation-specific mix of leadership approaches would likely be most effective (Day et al., 2016). In other words, our findings emphasize the influence of the school-specific leadership situation on the leadership requirements to be met (Hallinger, 2018). Rather, it appears that some leadership instruments may be dispensable in view of the individual schools' contextual conditions, since organizational guidelines might act as leadership substitutes. In this context, it remains unclear which leadership competencies are absolutely necessary in order to meet the demands placed on them. Thus, since the fragmentation into a multitude of activities makes the systematic pursuit of a goal difficult, an effective school leader would have to consider with which activities several goals can be pursued simultaneously and controlled for quality assurance.
Then, when closely examining the findings concerning Hypothesis 3, we observe a significant association between opportunities for parental involvement and reading achievement. This calls for entrepreneurial leadership in order to delegate some pedagogical responsibilities to parents and the external community at large (Pashiardis and Brauckmann, 2018). In this situation, the question arises how many and what kind of tasks, activities and functions should be delegated and should fields of action be delegated or isolated activities only? School principals might experience their autonomy as pseudo-autonomy because system management and individual school management do not mesh. Accordingly, even autonomous management action is subject to systemic contextual conditions and can lead to frustration and increase uncertainty. It also should be noted that principals often complain that their own initiatives and responsible actions have not been rewarded by school authorities and have even been rejected or sanctioned (e.g. Pont et al., 2008). It is, therefore, not surprising if autonomy imposed from above is treated by the school management in a rather dutiful manner and is not immediately integrated into the decision-making architecture of the individual school. The findings of our study, therefore, also point to the increased importance of building coalitions with partners inside and outside the school as well as across several hierarchical levels. In this context, providing orientation regarding a common development task could become primarily a communication task for the principal in the sense of strategic management.
Strengths and limitations
Using large scale data offers several opportunities, for example, a randomized and representative sample and the opportunity to compare education systems from around the globe and address scientific research in a cost-efficient way. However, there are some limitations as well. First, the operationalization of theoretical modelling is limited. Several authors describe paths through which school leaders exhibit their impact on student achievement, but these paths cannot be relied upon in secondary analyses because the variables were pre-specified. Consequently, compromises must be made in modeling theoretical assumptions.
Second, PISA provides only self-reported data from principals on their own leadership behavior and on institutional context. In various studies principals have been found to portray a more positive picture of themselves than their teachers do (e.g. Urick and Bowers, 2019). In addition, discrepancies have been found between principals' perceived autonomy and accountability and their actual autonomy and accountability and such variance within a system indicates that principals stated their perception only. It can be argued, however, that principals' perception of their school context might have a greater impact on their behavior. Nevertheless, additional sources for assessing variables such as leadership activities and institutional context might validate principals' perception of them. Finally, the cross-sectional character of the data restricted causal interpretation of the findings. Reynolds et al. (2014) underline the importance of longitudinal data to demonstrate effects; however, cross-sectional data can provide meaningful insight and the cross-sectional data from the PISA study were appropriate for addressing the research question in this study.
Conclusion
It is evident that the management approaches and instruments based on autonomy and accountability do not automatically generate new leadership practices. Therefore, school leadership action seems to be less related to systemic characteristics than to individual school characteristics. In addition, the perception or promotion of an expanded scope for action within the school is generated more by the individual initiative of the principal or institutional opportunity structures as well as by legal requirements. It remains unclear whether the lack of consistent correlations between autonomy and accountability-oriented approaches and the leadership styles of school principals can be explained by the recent trend toward increased school autonomy in numerous countries (OECD, 2016). The leadership actions of school leaders tend to be more situation-specific. Our findings suggest flexible leadership styles are loosely tied with autonomy and/or accountability-oriented approaches. This is independent of the combination of more freedom of design and simultaneous accountability intended at the system level. Rather, there is a continuous re-evaluation of and reflection on the relevance of systemic or institutional factors against the background of individual school issues which, in turn, require complementary monitoring. School leaders, apparently remain decision-makers and sovereign in dealing with autonomy and accountability related steering instruments.
Contrary to earlier assertions, the substantive responsibility of school principals not only is reduced to the field of activity defined by education law, but also results from the specific context of action of the individual school. In this context, principals are not necessarily, as is often portrayed, exclusively a multifunctional person, but rather the producer of newly conceived multifunctional working relationships. Thus, the description of working conditions seems to be particularly worthwhile, since system, organization and person related characteristics are related to each other. However, further clarification is needed as to which reference contexts are relevant for school leadership and which can be ignored. In this context, investigation should be made into whether principals' behaviors depend on their self-regulatory capacity or ability to adhere to regulations that, for example, require them to deal with more autonomy and/or more accountability.
Figures
The supplementary material for this article can be found online.
References
Ärlestig, H., Day, C. and Johansson, O. (2016), “International school principal research”, in Ärlestig, H., Day, C. and Johansson, O. (Eds), A Decade of Research on School Principals: Cases from 24 Countries, Springer, Cham, pp. 1-9.
Brauckmann, S. and Pashiardis, P. (2011), “A validation study of the leadership styles of a holistic leadership theoretical framework”, International Journal of Educational Management, Vol. 25 No. 1, pp. 11-32.
Brauckmann, S., Pashiardis, P. and Ärlestig, H. (2023), “Bringing context and educational leadership together: fostering the professional development of school principals”, Professional Development in Education, Vol. 49 No. 1, pp. 4-15.
Bruggencate, G.T., Luyten, H., Scheerens, J. and Sleegers, P. (2012), “Modeling the influence of school leaders on student achievement”, Educational Administration Quarterly, Vol. 48 No. 4, pp. 699-732.
Bush, T. (2007), “Educational leadership and management: theory, policy and practice”, South African Journal of Education, Vol. 27 No. 3, pp. 391-406.
Bush, T. (2013), “Autonomy and accountability”, Educational Management Administration and Leadership, Vol. 41 No. 6, pp. 697-700.
Bush, T. and Glover, D. (2014), “School leadership models: what do we know?”, School Leadership and Management, Vol. 34 No. 5, pp. 553-571.
Cheng, Y.C., Ko, J. and Lee, T.T.H. (2016), “School autonomy, leadership and learning: a reconceptualisation”, International Journal of Educational Management, Vol. 30 No. 2, pp. 177-196.
Day, C., Gu, Q. and Sammons, P. (2016), “The impact of leadership on student outcomes”, Educational Administration Quarterly, Vol. 52 No. 2, pp. 221-258.
Dutta, V. and Sahney, S. (2022), “Relation of principal instructional leadership, school climate, teacher job performance and student achievement”, Journal of Educational Administration, Vol. 60 No. 2, pp. 148-166.
Grissom, J.A., Egalite, A.J. and Lindsay, C.A. (2021), “How principals affect students and schools: a systematic synthesis of two decades of research”, New York, available at: http://www.wallacefoundation.org/principalsynthesis (accessed 10 August 2021).
Gumus, S., Bellibas, M.S., Esen, M. and Gumus, E. (2018), “A systematic review of studies on leadership models in educational research from 1980 to 2014”, Educational Management Administration and Leadership, Vol. 46 No. 1, pp. 25-48.
Gurr, D. (2015), “A model of successful school leadership from the international successful school principalship project”, Societies, Vol. 5 No. 1, pp. 136-150.
Gurr, D., Drysdale, L. and Goode, H. (2022), “An open systems model of successful school leadership”, Journal of Educational Administration, Vol. 60 No. 1, pp. 21-40.
Hallinger, P. (2005), “Instructional leadership and the school principal: a passing fancy that refuses to fade away”, Leadership and Policy in Schools, Vol. 4 No. 3, pp. 221-239.
Hallinger, P. (2011), “Leadership for learning: lessons from 40 years of empirical research”, Journal of Educational Administration, Vol. 49 No. 2, pp. 125-142.
Hallinger, P. (2018), “Bringing context out of the shadows of leadership”, Educational Management Administration and Leadership, Vol. 46 No. 1, pp. 5-24.
Hanushek, E.A., Link, S. and Woessmann, L. (2013), “Does school autonomy make sense everywhere? Panel estimates from PISA”, Journal of Development Economics, Vol. 104, pp. 212-232.
Harris, A. and Jones, M. (2018), “Why context matters: a comparative perspective on education reform and policy implementation”, Educational Research for Policy and Practice, Vol. 17 No. 3, pp. 195-207.
Hitt, D.H. and Tucker, P.D. (2016), “Systematic review of key leader practices found to influence student achievement”, Review of Educational Research, Vol. 86 No. 2, pp. 531-569.
INNE (2016), “OECD review of policies to improve the effectiveness of resources use in schools. Country background report Spain”, Madrid.
Johnson, L., Møller, J., Jacobson, S.L. and Wong, K.C. (2008), “Cross‐national comparisons in the international successful school principalship project (ISSPP): the USA, Norway and China”, Scandinavian Journal of Educational Research, Vol. 52 No. 4, pp. 407-422.
Keddie, A. (2015), “School autonomy, accountability and collaboration: a critical review”, Journal of Educational Administration and History, Vol. 47 No. 1, pp. 1-17.
Klein, E.D., Bronnert-Härle, H., Boone, W.J. and Muslic, B. (2022), “Constructs of leadership and diverging institutional environments: an exploratory comparative study in the United States and Germany”, School Effectiveness and School Improvement, Vol. 33 No. 4, pp. 564-587.
Kyriakides, L., Georgiou, M.P., Creemers, B.P.M., Panayiotou, A. and Reynolds, D. (2018), “The impact of national educational policies on student achievement: a European study”, School Effectiveness and School Improvement, Vol. 29 No. 2, pp. 171-203.
Lee, M. and Hallinger, P. (2012), “National contexts influencing principals' time use and allocation: economic development, societal culture, and educational system”, School Effectiveness and School Improvement, Vol. 23 No. 4, pp. 461-482.
Leithwood, K. and Sun, J. (2012), “The nature and effects of transformational school leadership”, Educational Administration Quarterly, Vol. 48 No. 3, pp. 387-423.
Leithwood, K., Harris, A. and Hopkins, D. (2020a), “Seven strong claims about successful school leadership revisited”, School Leadership and Management, Vol. 40 No. 1, pp. 5-22.
Leithwood, K., Sun, J. and Schumacker, R. (2020b), “How school leadership influences student learning: a test of ‘the four paths model’”, Educational Administration Quarterly, Vol. 56 No. 4, pp. 570-599.
Liebowitz, D.D. and Porter, L. (2019), “The effect of principal behaviors on student, teacher, and school outcomes: a systematic review and meta-analysis of the empirical literature”, Review of Educational Research, Vol. 89 No. 5, pp. 785-827.
Luschei, T.F. and Jeong, D.W. (2021), “School governance and student achievement: cross-national evidence from the 2015 PISA”, Educational Administration Quarterly, Vol. 57 No. 3, pp. 331-371.
MacNeil, A.J., Prater, D.L. and Busch, S. (2009), “The effects of school culture and climate on student achievement”, International Journal of Leadership in Education, Vol. 12 No. 1, pp. 73-84.
Meuleman, B. and Billiet, J. (2009), “A Monte Carlo sample size study: how many countries are needed for accurate multilevel SEM?”, Survey Research Methods, Vol. 3 No. 1, pp. 45-58.
Muthén, B.O. and Asparouhov, T. (2011), “Beyond multilevel regression modeling: multilevel analysis in a general latent variable framework”, in Hox, J. and Roberts, J.K. (Eds), Handbook of Advanced Multilevel Analysis, Taylor & Francis, New York, NY, pp. 15-40.
Muthén, L.K. and Muthén, B.O. (1998–2017), Mplus User's Guide, 8th ed., Muthén & Muthén, Los Angeles.
Neeleman, A. (2019), “The scope of school autonomy in practice: an empirically based classification of school interventions”, Journal of Educational Change, Vol. 20 No. 1, pp. 31-55.
OECD (2016), PISA 2015 Results (Volume 2): Policies and Practices for Successful Schools, OECD Publishing, Paris.
OECD (2017a), PISA 2015 Assessment and Analytical Framework: Science, Reading, Mathematic, Financial Literacy and Collaborative Problem Solving, OECD Publishing, Paris.
OECD (2017b), PISA 2015 Technical Report, OECD Publishing, Paris.
OECD (2019), Education Policy Outlook 2019: Working Together to Help Students Achieve Their Potential, OECD Publishing, Paris.
Ozga, J. (2020), “The politics of accountability”, Journal of Educational Change, Vol. 21 No. 1, pp. 19-35.
Pashiardis, P. and Brauckmann, S. (2018), “New public management in education: a call for the edupreneurial leader?”, Leadership and Policy in Schools, Vol. 25 No. 2, pp. 1-15.
Pietsch, M. and Tulowitzki, P. (2017), “Disentangling school leadership and its ties to instructional practices – an empirical comparison of various leadership styles”, School Effectiveness and School Improvement, Vol. 28 No. 4, pp. 629-649.
Pietsch, M., Tulowitzki, P. and Koch, T. (2019), “On the differential and shared effects of leadership for learning on teachers' organizational commitment and job satisfaction: a multilevel perspective”, Educational Administration Quarterly, Vol. 55 No. 5, pp. 705-741.
Pont, B., Nusche, D. and Moorman, H. (2008), Improving School Leadership Improving School Leadership, Vol. 1, Policies and Practices, Paris.
Preacher, K.J., Zyphur, M.J. and Zhang, Z. (2010), “A general multilevel SEM framework for assessing multilevel mediation”, Psychological Methods, Vol. 15 No. 3, pp. 209-233.
Reynolds, D., Sammons, P., Fraine, B., de van Damme, J., Townsend, T., Teddlie, C. and Stringfield, S. (2014), “Educational effectiveness research (EER): a state-of-the-art review”, School Effectiveness and School Improvement, Vol. 25 No. 2, pp. 197-230.
Robinson, V.M.J., Lloyd, C.A. and Rowe, K.J. (2008), “The impact of leadership on student outcomes: an analysis of the differential effects of leadership types”, Educational Administration Quarterly, Vol. 44 No. 5, pp. 635-674.
Scheerens, J. (2016), Educational Effectiveness and Ineffectiveness: A Critical Review of the Knowledge Base, Springer, Dordrecht.
Sebastian, J., Huang, H. and Allensworth, E. (2017), “Examining integrated leadership systems in high schools: connecting principal and teacher leadership to organizational processes and student outcomes”, School Effectiveness and School Improvement, Vol. 28 No. 3, pp. 463-488.
Spillane, J.P., Camburn, E.M. and Stitziel Pareja, A. (2007), “Taking a distributed perspective to the school principal's workday”, Leadership and Policy in Schools, Vol. 6 No. 1, pp. 103-125.
Stapleton, L.M., Yang, J.S. and Hancock, G.R. (2016), “Construct meaning in multilevel settings”, Journal of Educational and Behavioral Statistics, Vol. 41 No. 5, pp. 481-520.
Tulowitzki, P. (2016), “Educational accountability around the globe: challenges and possibilities for school leadership”, in Easley, J. and Tulowitzki, P. (Eds), Educational Accountability: International Perspectives on Challenges and Possibilities for School Leadership, Routledge, London, New York, NY, pp. 233-238.
Urick, A. and Bowers, A.J. (2019), “Assessing international teacher and principal perceptions of instructional leadership: a multilevel factor analysis of TALIS 2008”, Leadership and Policy in Schools, Vol. 18 No. 3, pp. 249-269.
Urick, A., Liu, Y., Ford, T.G. and Wilson, A.S.P. (2021), “Does instructional leadership mediate effects of student home resources on opportunity to learn and math reasoning skills? A cross-national comparison of 4 th grade students”, International Journal of Leadership in Education, Vol. 24 No. 6, pp. 876-914.
Wang, M.-T. and Degol, J.L. (2016), “School climate: a review of the construct, measurement, and impact on student outcomes”, Educational Psychology Review, Vol. 28 No. 2, pp. 315-352.
Wößmann, L. (2007), “International evidence on school competition, autonomy, and accountability: a review”, Peabody Journal of Education, Vol. 82 Nos 2-3, pp. 473-497.
Corresponding author
About the authors
David Kemethofer is Professor at the University College of Teacher Education Upper Austria. His research focus lies in the areas of school management and leadership, quality management in the school system, school inspections and educational standards and standard-based performance tests.
Christoph Weber is Professor at the University College of Teacher Education Upper Austria and researcher at the Research Institute for Developmental Medicine, Johannes Kepler University Linz, Austria. His research interests cover educational inequalities, teacher education and quantitative research methods.
Stefan Brauckmann-Sajkiewicz holds the chair of quality development and quality assurance in education at the Institute of instructional and school development (IUS) of the Alpen-Adria-University Klagenfurt. His main academic fields and interests lie within framework conditions to the education system as well as the different governing mechanisms in educational leadership which affect the development of quality assurance in education.
Petros Pashiardis is Professor of Educational Leadership and Rector of the Open University of Cyprus. His current thinking and research interests take him beyond the WHAT effective/successful school leaders do and into the HOW they do it, which leads to Edupreneurial Leadership.