Testing ZELPH ['sɛlf] – a self-assessment instrument to surface intended and unintended outcomes of work-integrating pedagogies

3.1 Basic characteristics

Lester et al. (2016), Nottingham (2012) and Lillis and Bravenboer (2020), amongst others, offer empirical evidence for the considerable variety of work-integrating programmes in the UK alone, and demonstrate both commonalities and differences in underlying concepts as well as in the modes of delivery (Lillis and Bravenboer, 2020). While Mörth et al.'s (2018) study had echoed the literature's findings relating to the variety of work-integrating educational offers and the range of delivery modes, it had also confirmed both Wall's (2013) observation regarding the variability of the integration of disciplinary knowledge in practice, and Edwards et al.'s (2015) findings concerning the different priorities set by institutions in terms of the integration of academic theory with professional practice. Taking account of these insights, the ZELPH model begins the self-evaluation exercise by providing an opportunity to map the particular type of work-integrated learning of a programme (i.e. cooperative education, degree apprenticeship etc.) and the mode of delivery (i.e. on campus or blended, part-time or full time etc.) (see Table 2). Stating these very basic characteristics of an educational offer helps to typify it, and thus demarcates the limits set by its very type and delivery mode, which, in turn, allows for placing the evaluation in its appropriate context.

3.2 Work as place of learning

Notwithstanding the variety of approaches, they all have in common that work, paid or unpaid, is part of the curriculum and plays a role as a learning site (see e.g. Lillis and Bravenboer, 2020; Wall and Hindley, 2019; Lester et al., 2016; Edwards et al., 2015; Nottingham, 2012; Costley et al., 2010; Cooper et al., 2010). Accordingly, the model's category integration of professional practice into the study programme is designed to shed light on the ways in which work is integrated and valued as a learning site in a systematic manner. Thereby, the category invites details on how integration happens, for example, at curricular level, or at a formal institutional level, such as through a memorandum of understanding with an employer. Drilling down further into the specific conditions of work as a learning site, scholars concur that the students' activities on the job should amount to “a learning enterprise that, while commonly undertaken at work, is not identical to work” (Boud, 2001, p. 50). Accordingly, the category activities at the workplace aims at surfacing whether students complete regular work tasks or whether they are asked to do experiential learning tasks, such as shadowing a senior executive and subsequently reflecting on it in a learning journal. In this context, the literature argues that it also matters when and for how long the students' employment takes place (Lester et al., 2016; Cooper et al., 2010; Billet, 2011). Consequently, ZELPH includes a category entitled time at the workplace, which, despite its ostensible simplicity, reveals much about a programme that, for example, only requires one three-month work placement throughout a four-year programme, compared with another educational offer as part of which students work and study in parallel for four years.

3.3 Value of professional skills and competencies

In the literature, the debate on work as a pertinent learning site within the context of a culture that assumes primacy of subject discipline is reflected, in practice, in the value that educational providers assign to professional skills and competencies. In real life, it directs, for example, developments in negotiated curricular and learning shell frameworks, or in enabling access to HE for non-traditional and lifelong learners (see e.g. Talbot, 2017; Lester et al., 2016; Bravenboer and Lester, 2016; Lillis and Bravenboer, 2020; Wall, 2013; Boud, 2001; Costley et al., 2010). In an effort to operationalise the level of recognition that is afforded to practitioner knowledge, the ZELPH model includes the category entry admission/requirements, which is broad enough to also cater for those programmes that only require students to meet academic requirements. Probing more explicitly into the perceived value of non-academic and/or professional knowledge, the subsequent category accreditation of prior, non-academic competencies is aimed at surfacing whether the programme under scrutiny considers non-academic competencies important enough to translate them into academic credits. In recognition of departments without frameworks for non-academic competencies, the category generic, non-subject specific competencies provides an opportunity to identify learning outcomes that are not discipline-centred and not academic, but relevant to becoming a skilled practitioner.

3.4 Acquiring professional skills and competencies through the academic component

Learning that is relevant to professional practice occurs not only on the job but also in academic components of a work-integrating study programme. While the literature agrees on the importance of defining learning outcomes relating to professional skills and competencies (see e.g. Lillis and Bravenboer, 2020, Lester et al., 2016; Bravenboer and Lester, 2016; Cooper et al., 2010; Boud, 2001), some scholars argue for the relevance of employment- or sector-specific learning (Bravenboer and Lester, 2016; Cooper et al.), while others, especially those discussing learner-centred provisions, make a case for generic learning in an effort to capture transdisciplinary learning that occurs when work is the frame of reference instead of academic disciplines (Boud, 2001; Costley et al., 2010). With a view to reflect the key points of this debate, ZELPH proffers the category approach to interdisciplinarity to reveal whether interdisciplinary learning is achieved, for example, by design (i.e. a compulsory module in another department) or by methods (i.e. tasks that require an interdisciplinary response). ZELPH also features the category approach to integrating themes, topics and issues, which relate to the study field's associated professional practice, into teaching and learning activities on campus with a view to surface how the academic component draws on real-life professional practice, such as asking students to analyse their own practical experience by applying the precepts of academic theory. The subsequent category approach to teaching and learning invites respondents to identify, at a high level, which variant their pedagogy for instilling both academic and professional knowledge falls under, that is, work-based, work-integrating, experiential learning and so forth.

3.5 Personal growth of the reflective student

According to the literature, a key feature of work-integrating programmes is that students are expected to act as self-directed learners and maximise the benefits offered by multiple learning sites (Lillis and Bravenboer, 2020; Lester et al., 2016; Gibbons et al., 1994; Schön, 1983, 1987; Wenger, 1998). In practice, the implication is that work-integrating programmes must instil in students a “strong ethos of reflexivity and practitioner enquiry”, which enables them to cement, consolidate and apply learning that occurs in the classroom and at work (Lester and Costley, 2010, p. 564). In order to achieve such personal growth among students, scholars highlight the importance of supportive activities before, during and after the time at the workplace (Lillis and Bravenboer, 2020; Cooper et al., 2010; Billett, 2011), as well as the role of academic advisers and workplace supervisors in engaging students through reflexive learning activities (Lester et al., 2016; Boud, 2001). Accordingly, ZELPH includes the categories reflexive methods to embed the learning of professional practice, which might involve methods such as learning journals, reflective commentaries and 360 feedback, and approach to developing students' professional identities, which could involve, for instance, career counselling, peer networking and so forth.

3.6 Programme outcomes

Defining learning outcomes is an inevitable part of curriculum development in all HE programmes. It is perhaps slightly more complex in work-integrating programmes, considering that learning outcomes ideally reflect occupation- and sector-specific competencies in addition to academic competencies (Lillis and Bravenboer, 2020; Bravenboer and Lester, 2016; Lester et al., 2016; Nottingham, 2012; Brennan and Little, 1996). Given that ZELPH is designed to assess the pedagogy specific to combining academic and professional learning, it proffers the category outcomes of the programme's particular approach to interlinking academic theory with professional practice. It invites respondents to articulate that the end result is, for example, a loose, complementary juxtaposition of theory and practice, or a formal and systematic fusion of the two. The model also seeks to elicit the outcomes achieved by applying the particular pedagogy to students, which is operationalised through the category outcomes of the programme's particular approach to guiding learners towards achieving learning outcomes, professional competencies and personal maturity. It aims to evoke the impacts on learners, such as an ability to cope on their own in the world of work or an ability to take on leadership roles.

However, not only the formally agreed programme outcomes are pertinent to evaluating the pedagogy of a work-integrating programme. Incidental learning outcomes are equally relevant because they constitute the “side effects” of an educational initiative (see e.g. Merriam and Bierema, 2014; Billett, 2011). They are the result of “what students experience when they engage with what was intended through what is enacted, and how they learn through that experiencing, even that which is unintended by those who plan and enact the curriculum” (Billett, 2011, p. 2). If unintended learning outcomes are found to be undesirable, the circumstances that create them need to be identified and adapted (Portelli, 1993). Correspondingly, if they are viewed as desirable, they can be made explicit and included in the list of outcomes that are formally agreed (ibid). ZELPH reflects these theoretical considerations by inviting respondents to list unintended programme outcomes, such as students' emotional resilience or mistrust of conventional academia.

4.1 A methodological approach that is fit for purpose

As is expected from an effective methodology (Bryman, 2001), all the study's decisions concerning research design, methods and analysis were guided by its overall research question: To what extent is the ZELPH model both a valuable and feasible instrument in surfacing the intended outcomes, as well as the unintended ones, of a university department's particular pedagogic approach to integrating professional practice into an academic programme of study? By suggesting that research design is the glue that bonds a research project together, Trochim (2005) highlights the pertinent role of an effective research design in giving logic and structure to the enquiry. This, in turn, enables the researchers to answer the research question and also assures the readers that resulting insights are robust, valid and worth paying attention to (Denzin and Lincoln, 2011). Accordingly, this research project chose a pilot study design, which allows for evaluating an instrument in a form that is similar to the final one and identifying opportunities for improvement (Denscombe, 2010; Majumdar, 2008). A pilot study design was congruent with this enquiry's objective to conduct a small-scale preliminary test of ZELPH's expediency and its ease of application across cultural, institutional and structural contexts. The research project opted for a qualitative research approach because it allowed for investigating in depth the perceptions and experiences of programme staff piloting the model, while being in line with the project's idealist ontology and constructivist epistemology (Denzin and Lincoln, 2011; Bryman, 2001).

4.2 Collecting data across five countries

The sampling strategy was based on reputation (also called expert or judgemental sampling) with experts being defined as university staff members, who are experienced in the design, delivery or oversight of work-integrating study programmes. Keeping in mind our aim of cross-context applicability, we sought to include experts from a variety of cultural contexts, resulting in five staff members from five countries (Germany, Great Britain, France, Taiwan and South Africa). Although reputation sampling cannot claim to reflect the theoretical population, this was not an issue within the context of this particular project because pilot studies are usually conducted on small, non-probability samples (Northrop and Arsneault, 2008). In a first step, the research participants were sent a document that detailed the ZELPH model, along which they were asked to analyse the pedagogy of their respective work-integrating study programmes. In an effort to illustrate the application of the model, the document also included exemplary analyses of the two good practice cases. In a second step, the respondents were interviewed about their experience of filling in the ZELPH model's categories with a semi-structured topic guide that focused on the perceived value of the model and the feasibility of applying it. As advised by Denscombe (2010), the interview questions were phrased as unambiguously and neutrally as possible to minimise bias and avoid influencing the respondents. All interviews were conducted and audio-recorded with the help of web conferencing tools or video chat applications, and they lasted between 30 and 45 min.

4.3 Data analysis

The recorded interviews were transcribed and subsequently analysed by employing an abductive thematic analysis that enabled the researchers to surface the understandings and misunderstandings of the respondents in applying the model (Blaikie, 2007). In an effort to ensure the validity and credibility of the findings, the researchers took a two-pronged approach: First, two researchers analysed all interviews separately and then conducted a workshop to triangulate their findings. This ensured the verification of emerging themes and ensured that no aspect was overlooked. Second, the results were confirmed by scrutinising the respondents' approach to filling in the ZELPH model categories. It allowed for cross-checking, for example, whether a self-reported ease in filling in a certain category was indeed reflected by the way he or she answered the relevant section in the template. Furthermore, ethical concerns were met by ensuring participants' confidentiality and anonymity and by working in accordance with the university's data protection policy as well as the German Society of Sociology's Code of Ethics.

5.1 1st category of result: conditional value

All participants concurred that the model is essentially a good idea and potentially of practical value as a self-evaluation tool across different contexts, albeit under certain conditions. As is the nature of any instrument, or for that matter of any means to an end (see Rønnow-Rasmussen and Zimmerman, 2005), the analysis showed that the ZELPH model is seen as conditionally, or extrinsically, valuable, with its worth depending upon the conditions under which it is employed. Specifically, two conditions emerged as pertinent to determining the value of ZELPH:

5.2 2nd category of result: practical feasibility

The data analysis also showed that, in general, the participants found ZELPH to be practically feasible and relatively easy to apply. The analysis surfaced two aspects based on which respondents seemed to judge its practical feasibility.

5.3 3rd category of result: programme outcomes

The sample agreed that the ZELPH models Section VII, which asks respondents to identify intended and unintended programme outcomes, stimulated reflection the most.

6.1 The ZELPH model

Following the insights gained from the pilot study, the researchers adjusted the ZELPH model, resulting into the following version.