Blended learning at scale: co-designing a large postgraduate finance course

Abstract

Purpose

This paper aims to identify and evaluate resolutions to key learning and teaching challenges in very large courses that involve practical mathematics, such as foundational finance.

Design/methodology/approach

A design-based research approach is used across three semesters to iteratively identify practical problems within the course and then develop and evaluate resolutions to these problems. Data are collected from both students and teachers and analysed using a mixed-method approach.

Findings

The results indicate that key learning and teaching challenges in large foundational finance courses can be mitigated through appropriate consistency of learning materials; check-your-understanding interactive online content targeting foundational concepts in the early weeks; connection points between students and the coordinator to increase teacher presence; a sustained focus on supporting student achievement within assessments; and signposting relevance of content for the broader program and professional settings. Multiple design iterations using a co-design approach were beneficial to incrementally improve the course and consider multiple perspectives within the design process.

Practical implications

This paper develops a set of design principles to provide guidance to other practitioners who seek to improve their own courses.

Originality/value

The use of design-based research and mixed-method approaches that consider both student and teacher perspectives to examine the design of very large, foundational finance courses is novel.

Citation

Cram, A., Wilson, S., Taylor, M. and Mellare, C. (2024), "Blended learning at scale: co-designing a large postgraduate finance course", Journal of Work-Applied Management, Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/JWAM-10-2023-0101

1. Introduction

Large classes offer a range of challenges for institutions as they seek to leverage economies of scale while also optimising students' learning experiences, and a growing body of literature outlines these challenges. For example, multiple authors have identified and discussed how large classes can inhibit interaction, formative assessment and feedback, and active student involvement, which can lead to feelings of isolation, less depth in thinking, and lower levels of satisfaction and student evaluation scores (e.g. Cuseo, 2007; Hornsby and Osman, 2014; Broadbent et al., 2018). The causes for this include a tendency towards didactic approaches, difficulties in monitoring and tracking students, student and educator attitudes and capabilities, and institutional infrastructure that is unsuited for large classes or policies and practices that inhibit innovation (Hornsby and Osman, 2014; Mantai and Huber, 2021). There is less literature, however, that examines techniques for sustainable and high-quality education at scale, particularly relating to Business degrees or the Finance discipline. Some studies on large classes have highlighted the success of contemporary teaching styles that focus on students and conceptual change (Prosser and Trigwell, 2014), and high-quality formative assessment through the use of carefully designed feedback using audio, exemplars and annotated rubrics that focus students attention on understanding and improving their own performances (Broadbent et al., 2018). Within the business domain, Snowball (2014) found benefits within a large foundational Economics course by moving to a blended learning approach using chunked videos and regular online multi-choice quiz exercises.

This paper seeks to contribute to this literature by identifying and evaluating resolutions to key learning and teaching challenges in a very large Finance course. Additionally, the contribution of this paper is extended using a design-based research approach, which allows the development of shareable design principles through the examination of several iterative cycles of research (McKenney and Reeves, 2018).

The course examined in this study, Capital Markets and Corporate Finance, had between 983 and 1,463 students enrolled in the three semesters of the study, with the recent rapid growth creating challenges for the optimisation of administrative efficiency and student learning and experience. The course is foundational and mandatory for the finance specialisation within a master's degree in business. It introduces the foundation knowledge of a finance major, such as financial mathematics, valuation of financial securities, cash flow construction and discounting, calculation of discount rates, capital structure and dividend policy. It is essential that students have a positive experience and develop a solid conceptual grounding for later finance courses.

The course has traditionally been delivered in person (on-campus) only, however transitioned to fully online delivery in the second iteration of the project due to government-imposed COVID-19 restrictions, and then to both online and on-campus delivery in the final iteration. The contextual challenges and interventions identified in the first iteration moved the course towards a blended approach and this was consolidated over the remainder of the project. Specifically, a flipped classroom approach was implemented, which involved moving information-provision out of class and devoting class time to active and social learning, complemented by out of class activities that enable students to fully benefit from in-class learning opportunities (Abeysekera and Dawson, 2015).

This paper describes a design-based research (DBR) project involving interventions to sustainably improve student experience and outcomes in this large-scale course over three semesters. DBR, also referred to as education design research, involves collaboration between researchers and educators to identify practical problems within an educational program, followed by theory-informed design, implementation and evaluation of course interventions that respond to these problems culminating in reflecting on the theory and the sharing of outcomes (McKenney and Reeves, 2018). DBR projects prioritise understanding of the context of the educational program, and often involve multiple cycles to iteratively make data-driven improvements to the course interventions (Barab and Squire, 2004). DBR offers a flexible approach that is suitable for examining “ecologically complex, technology-enhanced environments” while involving collaboration and co-design (Jacobsen and McKenney, 2023, p. 16).

In this study, we present three cycles of DBR (Figure 1), each involving analysis of the practical problems, development of design solutions to address those problems, testing and refinement of solutions, and reflection on how the design principles and solutions could be implemented more broadly. Barab and Squire (2004) argue that DBR “involves not simply sharing the designed artifact, but providing rich descriptions of context, guiding and emerging theory, design features of the intervention, and the impact of these features on participation and learning” (p. 8). This paper aims to craft a contextualised design narrative in this paper by sequentially describing the study iterations. Relevant literature is introduced in connection with the specific problems and solutions identified, that is, where it contributes to informing progressive developments in each iteration.

The study described in this paper sits in the context of a large-scale strategic priority project at the University of Sydney Business School called Connected Learning at Scale (CLaS) (Bryant, 2022).

1.1 Research questions

A set of high-level research questions was developed to guide the evaluation of the design interventions and dissemination of the findings and design principles:

RQ1.

What are the key challenges related to scale for a foundational postgraduate finance course?

RQ2.

How can these challenges be resolved?

RQ3.

How do DBR and co-design enable and constrain course improvements in this context?

2. Method

Data collected in each DBR iteration is summarised in Table 1. The survey and focus group instruments were designed to collect student and teacher perspectives of the course interventions that were made in each iteration in response to the identified challenges. The instruments were informed by a question bank for the broader CLaS project, which was developed by a multi-disciplinary team of educational developers, learning designers and researchers, based on the CLaS principles and literature on technology adoption and preferences. These questions were contextualised for each iteration in collaboration with the coordinators to ensure calibration with the finance topics and specific interventions. Institutional ethics approval (HREC 2019/892) was received for this research project.

Data were analysed using mixed-method approaches, incorporating both the educator and student voices within each iteration. Analysis began by identifying data that was relevant to each of the course interventions, followed by the development of thematic summaries that evaluated the impact of each intervention through the integration of teacher and student perspectives. These summaries were also used to inform the focus of the subsequent iterations. The range of perspectives incorporated provided a richer picture of the effectiveness of the developments introduced. At the end of the evaluation of Iteration 3, these summaries were used to undertake a cross-iteration analysis to describe and examine recurring themes across the whole project.

3. Results

3.1 Iteration 1

3.1.1 Practical problems

At the beginning of the project, practical problems with the course were identified and analysed by reviewing prior student feedback and consultation with the coordinator. This revealed several issues which became the focus of Iteration 1:

1. More coordinator presence needed to address variability in the experience of students from different streams. The course was divided into “streams” where individual lecturers teach a three-hour class each week. Each stream had approximately 60 students, with upwards of 20 streams. Student feedback highlighted inconsistency across streams regarding level of activity, duration, slide presentation, exam preparation and assessment feedback, and students not taught by the co-ordinator felt their learning experience was compromised. This speaks to the insufficient teacher presence (Garrison et al., 1999) of the course coordinator.

2. More scaffolding needed to support the group assignment. While some students commented that the groupwork assignment was challenging, authentic and effective in supporting social connections, numerous problems were identified. These included issues with group dynamics and the distribution of workload amongst group members; a need to ensure that students' group marks better reflected their perceived level of engagement in the task relative to other members of their group; and concerns about the implementation of peer-review and assessment weightings.

3. More “signposting” needed to orientate students to the content. Students needed further support in understanding the relationship between topics, their level of importance and how they fit into the course, specialisation and profession. This was complicated by the variation in the background of the students. Some students had previously completed an undergraduate finance degree and were well-versed in foundational knowledge, whereas other students had never undertaken finance study. Signposting facilitate learning by assisting students to understand the structure of the field of knowledge and relationships between concepts (Bruner, 1960; cited in Donovan and Bransford, 2005, p. 15).

3.1.2 Development and evaluation of solutions

Guided by the literature, a variety of solutions were developed to address the problems identified above. These were evaluated through student and course evaluation surveys, coordinator interviews and analysis of learning technology data. In this semester, there were 1,372 students.

To increase coordinator presence and reduce variability in the student experience across streams (issue 1), and enhance the level of “signposting” in the course (issue 3), developments included:

1. A high production-quality welcome video to stimulate students' interest and connect them to the course coordinator who shared their own interest in the subject.

2. A single set of slides to be used across all lecture streams to ensure consistency;

3. Beginning weekly online content with a concise summary of how the content connects with other topics and the finance profession;

4. A weekly “wrap-up” podcast recorded by the coordinator to “wrap-up” each week's content and highlight connections with previous and upcoming content; and

5. A course “map” to help students understand how the course fits in with their specialisation.

The welcome video and podcasts are supported by Anderson (2008), who highlights the benefits of “establishing a personalised tone” within online course content and allowing students to see the “appeal that inspires the teacher's interest in the subject”, and the role of creating materials such as “online commentaries” to enhance teaching presence (p. 347). Felten and Lambert (2020) describe this “efficacy-centred approach to online teaching” as an effective way to demonstrate support to students in large courses (pp. 88–89).

The evaluation data indicated generally positive outcomes for the welcome video, podcast and overall consistency. Of the students who watched the welcome video (81% of respondents), over two-thirds indicated that it helped orient them to the course. Of the respondents who engaged with weekly podcasts (88% of respondents) 75.6% agreed that the podcasts helped their learning of key concepts. Qualitative comments suggested that the weekly podcasts were “very helpful” particularly for providing clarity around core concepts. However, the coordinator perceived the podcast production as relatively time-intensive since on average only 238 students listened to the podcasts each week. Complaints about inconsistency were absent from students' course evaluation survey comments, indicating the success of the interventions aiming to improve consistency. Data on student perceptions were not collected on other aspects of signposting, such as the course map; however, the coordinator regarded these interventions positively.

The following strategies were employed to provide a more scaffolded support for the group assignment (issue 2):

1. Online flip card resources to support students with the groupwork process;

2. Formative feedback (Nicol and McFarlane-Dick, 2006) on the first two components of the group assignment;

3. An assessment briefing session by the coordinator for the entire cohort;

4. An exam review session by the coordinator via Zoom webinar;

5. A peer-review process (using a tool called SparkPlus) to help students to reassess their knowledge and beliefs and evaluate group members' contributions and moderate marks; and

6. A marking and feedback workshop for stream lecturers to promote consistency.

The support strategies for the group assignment were well received by the students. Flip cards were engaged with by 80% of respondents, with 73.2% agreement that they provided support for the group assignment. The assessment briefing session had high engagement (92.2% of survey respondents), with 72.3% agreement that it assisted understanding of the major assignment requirements. Formative feedback in the major assignment was largely well received, with 77.6% of respondents indicating that it improved their confidence in completing the task. These findings were reflected in the Unit of Study Survey that showed significant improvement on the feedback item for this semester (increasing from 4.0 to 4.17 out of 5) as determined by a t-test (p < 0.05). A question asking students if they had access to valuable learning resources also showed a statistically significant improvement (increasing from 4.15 to 4.25, p < 0.05).

The peer-review process using SparkPlus was seen to be useful for students but not entirely fit for purpose. Most student survey respondents indicated that self-assessing their groupwork performance increased awareness of their contribution (71.42%), that peer-assessment increased their reflection on team-member contributions (71.1%), that SparkPlus enabled marks for the group assignment to be fairly awarded (72.5%) and that the peer-review process positively influenced their groupwork (69.3%). However, the teaching team reported that the administration process using this technology was burdensome and not effective in addressing problems of scale, e.g. high levels of individual support required for students despite providing tailored support resources. Additionally, the coordinator did not feel it fully addressed freeloading issues or improved participation and group interaction and that equity issues could arise if group members used the mechanism to “punish” a student whose views they disagreed with.

3.1.3 Outcomes for the course and project

Overall, the evaluation indicated that the interventions were effective in supporting student learning and experience while noting potential improvements. All interventions were retained for the following semester, except the peer-review process, which had insufficiently scalable administration processes.

Some interventions (e.g. the weekly podcasts) were optional for students and had varied levels of usage, prompting consideration of whether there was sufficient impact relative to their development effort. Two points were in favour of the interventions: the course size meant that engaging even 20% of the student cohort would impact over 250 students, and the interventions were designed for reuse in the following semesters. Therefore, the evaluation of interventions considered both student feedback and the academic time required to sustain the improvements across semesters.

As the broader CLaS project matured, available resources (including access to learning designers, media support and tools) expanded, with refined development and design processes. This allowed a broadening of the scope of course developments. Rather than further refine smaller interventions, such as the welcome video and course map (where further gains were marginal), subsequent developments focussed on more substantial parts of the course, including expansion of a set of consistent materials for use with all students.

3.2 Iteration 2

3.2.1 Evolution of the practical problems

The second iteration of the project occurred between February and July 2020 and focused on consistency and several other practical problems identified through a combination of student surveys and coordinator feedback:

1. Students struggle if they don't master the financial mathematics content early in the course. The financial mathematics component provides the foundation for all other content. This was known about in Iteration 1 but was not initially targeted as a core challenge.

2. Students have different levels of prior knowledge. Some students were unsatisfied with the pace of delivery or logic of the content. Comments suggested that there was too much content in each lecture and it was aimed at students with prior knowledge of accounting or finance.

3. Limited opportunities for individual feedback and attention due to the current structure and scale of the course. Students felt that smaller classes would allow for more individual attention and feedback, calling for a more traditional lecture-tutorial teaching structure rather than the 3-h lecture stream structure.

3.2.2 Development and evaluation of solutions

The flipped classroom model offered opportunities to address emerging practical problems and further extend previous developments. The literature suggests that higher education students are favourably disposed towards blended and online learning in finance courses (Arbaugh et al., 2009), although this needs to be balanced by augmenting teacher–student interactions and connections (Harjoto, 2017; Alshehri, 2017). While the literature typically examines flipping classrooms in small classes (Duxbury et al., 2016), further research is needed to understand the effectiveness of this approach at scale.

Lopes and Soares (2018) found that the flipped classroom model, including pre-work videos, can increase students' motivation, interest and autonomy in the study of financial mathematics and allow a more effective use of class time. Further, online exercises with immediate feedback in financial mathematics were found to be beneficial for student learning and achievement. In large classes, Solis and Turner (2016) argue that positive teacher–student interactions can be promoted with strategies of self-disclosure, caring leadership and a focus on making the class feel smaller. A flipped learning prototype was introduced in Weeks 2 and 3 of Iteration 2 to support students with early mastery of financial mathematics concepts (addressing issue 1). The original 3-h lectures were reshaped into a 1-h online module (pre-work) and a 2-h workshop.

Online modules included short pencast videos on core financial mathematics topics and check-your-understanding quiz questions (Figure 2). This catered for different levels of prior knowledge (issue 2) and gave students more control over the pace of delivery, by skipping or re-watching videos and using quiz questions to practice applying concepts while receiving immediate feedback (issue 3). Evaluation findings indicated that while students considered a background in mathematics advantageous, they could still gain a good introduction through the videos and quizzes. However, they felt that some quiz feedback could have provided deeper explanations on correct answers.

The workshops were reshaped to build on the pre-work and support students to apply knowledge of key concepts to “real-world” examples or case studies. The volume and sequences of content and tasks were also revised (see Figure 3).

The new online modules were developed for Weeks 2 and 3; however, in Week 3, the COVID-19 pandemic forced a transition to fully online delivery. Therefore, online pre-work content was also developed for the remaining weeks, using the online modules as a “template” with some modifications to enable rapid development. The process of developing the early weeks helped build the coordinators' capability and confidence to implement online content and activities:

So unlike other people who hadn't really had a lot of experience in trying to use some of this … I felt that I was in a much better position because at least I had used a lot of this technology before … I think it's a really, very positive experience.

Students in focus groups indicated that the short online videos helped them maintain interest. Video analytics showed that of the students who viewed them, 72% or more watched them to completion. The post-video quiz questions also supported motivation, and having content in smaller “chunks” helped students understand concepts more easily. The coordinator also felt that students had a generally better understanding of financial mathematics concepts because of the new model, with an acknowledgement that it was difficult to attribute this solely to the new model and COVID-19 introduced difficulty in drawing definitive conclusions.

3.2.3 Outcomes for the course and project

The developments initiated in Iteration 1 provided a foundation to build upon, and the findings from Iteration 2 indicated broad consistency across semesters in students' responses to the original developments.

Iteration 2 aimed to improve students' learning of financial mathematics content for a solid foundation for future learning in the course; further support students with lower levels of prior knowledge; and increase student opportunities to receive individual feedback with less reliance on the lecturer. Aligned with Lopes and Soares (2018), there was some evidence that the flipped classroom model introduced in Weeks 2 and 3 supported student learning and autonomy in the study of financial mathematics, and the provision of immediate feedback after online exercises was beneficial for student learning.

A review by Green et al. (2017) of flipped classroom case studies across a number of disciplines found that most were underpinned by cognitive load theory. Scholars using this theory contend that students' cognitive load can be reduced by chunking content to facilitate more efficient processing by working memory (Banas and Velez-Solic, 2013, Hsieh, 2017). Abeysekera and Dawson (2015) assert that flipped learning can improve student motivation and helps them manage cognitive load. They propose the use of both self-determination theory (SDT) and cognitive load theory to underpin flipped learning research, given that flipped learning strategies rely heavily on students being motivated to independently complete pre-class work and interact with peers in and out of class (Abeysekera and Dawson, 2015). While these theories may help to explain why many students indicated that the flipped classroom strategies trialled in Weeks 2 and 3 supported their learning, not all students engaged in the activities. Students who did engage, however, reported benefits in being able to check their understanding and re-watch material. A further positive impact of rapidly shifting to online learning was more progress towards the final goal than originally anticipated because more content was reshaped into a blended format. This contributed to a further increase in consistency in how material was delivered, and embedded the transformation into the whole course.

3.3 Iteration 3

3.3.1 Evolution of the practical problems

The third iteration occurred between August and December 2020, with several contextual changes: project team changes, with two new coordinators and a new educational developer, bringing new ideas and preferred ways of working; a delivery model of both online and on-campus modes; and re-development of course content in preparation for an upgraded master's program in 2021. Course enrolments also increased to record levels with 1,463 students.

The project team reviewed the outcomes from Iteration 2 and incrementally built on the existing design while re-developing the content. This consolidated the following design features: consistent materials and activities via classroom workshops and online self-paced interactive and multimedia resources; increased support for mastering foundational concepts in weeks 1–3; regular direct engagement by coordinators with all students through weekly summary videos and live assessment briefing webinars; and consistently high levels of signposting related to the course, program and profession.

The key challenges for this iteration were as follows:

1. Existing course materials were not fit for purpose. Materials had been rapidly developed for the emergency pivot to online learning and the upgraded syllabus required some content realignment;

2. Record student enrolments. Numbers put increased pressure on ensuring that all administrative and teaching processes were efficient and scalable; and

3. Sustainability not optimal. The blended learning model needed ongoing refinement to further optimise sustainability and student learning experiences.

3.3.2 Development and evaluation of solutions

The desire for high-quality curriculum materials aligned to the future master's program led to a full review and revision of the curriculum, encompassing all content, practice questions and workshop exercises. This substantial task required much of the project team's available time and included a review of topic sequencing and delineation of content chunks.

Content design changes included development of chunked multi-modal content with text, pencast videos and diagrams, more real-world examples to illustrate concepts and a video (rather than podcast) weekly wrap-up. Signposting was emphasised to highlight the relevance of content chunks to the profession and other finance courses in the program. Almost all interactive activities from Iteration 2 were reused, with a few additional interactives created. Further pre-requisite material was provided in early weeks for students who needed additional mathematical support.

The new curriculum was implemented on a “just-in-time” basis with several weeks published at the start then further content released sequentially. This resulted in tight turnarounds for content development with recurring high levels of workload. The coordinators self-produced high-quality pencast videos using the “do-it-yourself” media studio on campus. Each content element received similar quality assurance as per previous iterations.

3.3.2.1 Blended learning model in general

Evaluation findings reflected students' perspectives on the evolving blended model, which was well-received by students overall. In particular, the chunked content for pre-work was seen as much easier to follow than traditional lectures:

The online content should be used as a guide or inspiration for other courses to follow … I've never had such concise and structured content online.

It was a pleasure to do the work before the tutorials.

Analytics showed that topic videos were watched by between 757 and 966 students (average 846 students, approximately 58% of the cohort), with a slight downward trend in views across the semester. Some students watched the videos multiple times, with videos being loaded between 1,172 and 1,655 times (an average of 1,365 views per video). There was an average viewing duration of almost 87% for the topic videos, with no video under 81%. Students who were not watching videos may have engaged with other texts, diagrams and interactive content, or perhaps had prior experience with the concepts through industry experience or previous completion of a finance major at the bachelor level.

For practice, two sets of exercises were provided each week, one for self-study and one for a workshop discussion. The practice questions were well received and seen to help consolidate students' understanding of core concepts; however, some students desired more questions. The coordinators noted an intention to develop more practice questions that supported students' progress from basic to more complex understandings of the material.

In addition, the live assessment briefing sessions were refined to allow students to submit questions via Padlet prior to the briefing. This approach allowed more time in the briefings on answering students' questions, rather than coordinating the process of students asking questions live. Analytics suggested good engagement with live assessment briefings. The mid-semester assessment briefing had the greatest attendance, with 443 students attending and 36 questions pre-posted. The focus group suggested that some students did not attend review sessions as their questions had been addressed in other environments such as the tutorial. One student commented on the benefits of the practical focus of the content and assessments:

… the way the content was structured and the examples, the assessments, they leaned more towards a very practical approach which I think really makes it easier to visualise how and where you'd use all the content that you've learnt essentially … I can say this confidently because I used to work in a finance firm … if I'd known what I know now … it would have definitely given me a head start. So, I think definitely relatable. And everyone else has also mentioned the way the content is structured being able to follow it at your own pace with plenty of examples, interactive exercises. It kind of really helps you put what you're learning into practice.

3.3.3 Outcomes for the course and project

This semester consolidated many of the major interventions made over the three design iterations. The drive for consistency in Iteration 1 ultimately led to the implementation of a fully blended delivery model, with high-quality centrally managed content alongside targeted supports for assignments and techniques to ensure the coordinators had a strong presence each week despite not having direct contact with all students. The scale of the course also further increased, reaching almost 1,500 students.

Decisions were required on where to direct scarce content development effort, with prioritisation of helping students master introductory concepts in the early weeks, and an emphasis on additional practice and tutorial questions with worked examples to help students master application of the content in later weeks rather than further interactive online activities. Alongside this, content development followed a just-in-time approach each week, with coordinators wanting more access to the learning design tools to develop more advanced interactive activities themselves and “to be able to edit [interactive resources] on the fly”. This reshaped the value added through the co-design process, with less focus on learning designer input and more emphasis on discussions about improving other pedagogical and administrative issues that arose throughout the semester.

Supporting students' assessment preparation also remained a focus, with the live assessment briefing sessions extended to include both written assessments and exams. Collating student questions prior to the briefings was an important intervention, and helped optimise the effective use of time, despite the increased number of students.

4. Discussion

Using a DBR approach, this study set out to answer three key questions:

1. What are the key challenges for this large postgraduate finance course?

2. How can these challenges be resolved?

3. How does design-based research enable and constrain course improvements in this context?

These questions sought to identify effective solutions to practical problems within this course and reflect on how the DBR approach both shaped and was shaped by the unfolding project. Each question is addressed below based on the outcomes of the DBR process, and from this, a set of design principles is presented, followed by conclusions and recommendations for future research.

4.1 Key challenges for the course

Specific challenges were identified as each iteration commenced (summarised in Appendix). These challenges were foregrounded and backgrounded in different iterations. Some solutions that addressed challenges early on were carried forward and refined in future iterations. Other challenges (e.g. relating to groupwork) were not applicable to later iterations due to changes in course design.

All challenges identified were underpinned by problems of scale. The traditional course structure, the 3-h lecture block and stream model, was becoming increasingly unsustainable with inconsistencies in the learning experience across the cohort and students feeling disconnected from the coordinator.

Additional to scale, and compounded by it, were challenges caused by changes to the broader context. The emergence of COVID-19 both added to the challenges in the course and accelerated progress towards the final goal. The imperative to quickly develop a high-quality online experience for emergency remote teaching was strongly aligned with the desire to transition to online modules to address identified issues of scale and consistency. However, the envisaged plans for Iteration 2 required adaptation to fit the new online delivery requirements.

Another example relates to personnel changes in the project team, with several educational developers and coordinators involved throughout the project. As DBR is cumulative, this challenged continuity and the ability to carry over learnings from previous iterations, while sometimes prioritising the need for project inductions and relationship building over development.

A further challenge, common to large-scale educational projects, was that the project matured over time (e.g. with additional educational tools and project resources being developed and tested) and this meant that some initial limitations on project outputs were broadened over time.

The challenges described above arose from the particular context (including time and place) within which the study took place, and highly contextual challenges such as these have been previously underrepresented in the literature on large classes.

In addition to course-specific challenges related to finance education, “standard” educational challenges were present, such as different levels of prior learning, the desire for increased consistency for students' learning experience across the cohort and providing individualised feedback and support to all students. These challenges were modulated by the scale of delivery, and correspond to the challenges identified in prior research into large class teaching (e.g. Cuseo, 2007; Hornsby and Osman, 2014; Broadbent et al., 2018).

An additional challenge experienced by the project team was where to direct scarce development resources when only a portion of students engaged in many of the interventions. Large-scale and sufficiently well-resourced courses have scope to develop interventions that cater for a proportion of students, giving students more choice about how they would like to approach their learning and what sorts of support they need. This suggests a segmented, rather than personalised, approach and echoes the recommendation by Woollacott et al. (2014) of “collectively knowing” your students in large classes. Further research is warranted to investigate specific techniques through which academics teaching large courses can be better supported in making optimal decisions where resources are utilised only by a subset of students.

4.2 How can these challenges be resolved?

Following a DBR approach, the design process for this course was structured and conceptualised over multiple iterations. Mitigation of the challenges identified above required iterative development over three semesters for a full transformation of the course structure. Examining the cumulative outcomes allows the identification of some pivotal resolutions that shaped the project. These are synthesised below as design principles, which offer “informed reusable guidelines” (Herrington and Reeves, 2011, p. 598) that assist the generalisability of the findings of this study by providing design guidance to other researchers and practitioners based on empirically validated educational designs.

Design guidelines for large cohorts involving foundational postgraduate finance:

1. Develop consistent materials for the lecture, tutorials and practice/consolidation activities: Identify which aspects of the curriculum will be centrally managed to ensure appropriate quality controls and refine over multiple semesters based on available resources.

2. Embed “connection points” that involve direct engagement between the course coordinator and students through scalable channels, such as weekly wrap-up podcasts and assessment webinars.

3. Provide resources and activities related to foundational content that allow students to revisit, revise and check their understanding through immediate feedback as required based on their level of confidence and prior learning.

4. Provide ample “signposting” to help students connect concepts within the course, as well as across the program and to the profession.

5. Have a sustained focus on assessment support, and scaffold assessments and exams by providing a live and recorded webinar, with students able to post questions beforehand.

6. Provide multiple options for students to engage with the content and their lecturers to cater for the diversity of student preferences.

7. Reassess the practical problems and evaluation approach with each iteration to accommodate emerging and context-specific issues.

Prior literature on effective designs for large classes has tended to largely focus on one aspect of the design (e.g. formative assessment (Broadbent, 2018), teaching approach (Prosser and Trigwell, 2014), or the introduction of chunked content videos and regular interactive formative quizzes (Snowball, 2014). The design principles above represent a more comprehensive set of guidelines for effective educational design in large classes than has been previously reported in the literature. While these principles are most relevant to the redesign of large-scale courses with similar characteristics to the course described in this study (e.g. postgraduate foundational courses in finance and other disciplines involving moderate levels of applied mathematics), the authors encourage consideration and adaptation of these design principles for large-scale classes in undergraduate programs and other disciplines.

Several design resolutions were found to have a particularly high impact. These included:

1. Assessment and exam briefings, with student questions collated before-hand;

2. Podcast/video summaries each week from the coordinator;

3. Interactives and check-your-understanding activities in foundational weeks; and

4. High-quality content including pencast videos to demonstrate thinking processes for students.

4.2.1 Sustainability of the project

Sustainability is a key challenge in educational development projects and is often exacerbated by environmental factors such as changes in coordinators from one semester to the next. As part of the CLaS project, courses undergo a handover process in which the educational developers and learning designers hand their roles back to the coordinator and the discipline. Coordinators receive training in using educational technology and tools that have been introduced, and key process information, to ensure they can appropriately continue refinements in subsequent semesters. One observation was the need for coordinators to balance design sophistication with ongoing maintenance efforts, considering that some resources may have low proportional use but still be valuable to many students.

4.3 How does DBR enable and constrain course improvements in this context?

The DBR process produced substantial gains, in particular related to the quality of design outcomes, capability development of the co-design team, and opportunities to address educational problems related to learning and teaching in large courses.

4.3.1 Quality of design outcomes

The multiple iterations in the DBR process allowed attention on different focal areas across the project, resulting in a more comprehensive course transformation. The re-analysis of the practical problems at the start of each iteration assisted in addressing unexpected changes in context, such as changes in the teaching team and shifts in the delivery mode due to COVID-19.

The co-design process, which is central to DBR, produced insights through discussion that encompassed multiple perspectives. Similarly, through the iterative evaluation and reflection cycle, data were collected from various stakeholders (students, teachers and the coordinator), promoting a range of perspectives and providing a richer picture of the effectiveness of the developments introduced into the course. This combination of student and staff perspectives, with a mixed-method approach, is rare in blended learning research (Smith and Hill, 2019).

4.3.2 Capability development

One of the unexpected ways DBR enabled improvements in the course was through capability development within the project team. As suggested by Voogt et al. (2015), co-design processes provide an opportunity for those involved to gain skills by interacting with peers with different expertise, and through the negotiation that drives the design and development process. Vallis et al. (2022) explore how co-design and its inherent creative tensions have supported professional learning and more active online teaching practices over time in the broader CLaS project. The skills acquired by the coordinators may also be transferred to other courses they teach for broader impact. In addition to the learning that emerged through the discussion and negotiation involved in the development and design process, new members who joined the co-design team had opportunities to adopt and further refine the previously introduced innovations.

4.3.3 Opportunities to address educational problems at scale

We argue that DBR is particularly suited to addressing scale, because the challenges generated by large-scale courses are complex and require multiple design-based iterations to transform a course, and deal with the interrelated problems and perspectives. These problems often stem from the large, diverse student population and their needs. Furthermore, DBR can mitigate some risks associated with “fast design” with such a large cohort. Poorly executed interventions can cause significant administrative burdens and impact student learning and experience. In contrast, DBR promotes working through developments slowly and reflectively over time with a diverse team. The approach resonates with Strauss and Faud-Luke's principles of “slow design” (2008), which promote social, cultural and environmental sustainability.

5. Conclusions

This study suggests that some core issues require several iterations to fully evolve. These included achieving learning experience consistency, increasing teacher presence, scaffolding assignments, and increasing opportunities for students to check their understanding and receive feedback through different channels. The broad structural change from 3-h face-to-face teaching blocks that focus on content delivery, to a model that includes interactive online learning modules followed by workshops, evolved over three semesters. The gradual approach to testing and refinement allowed early developments to be built upon, with opportunities to unearth what was not working and adjust accordingly. This allowed coordinators to upskill while not being overwhelmed. Overall, findings suggested that the blended model was successful at scale, providing students with opportunities to learn, understand and apply foundational concepts at their own pace while revisiting materials as appropriate.

DBR involves making decisions about what new approaches to keep and refine for subsequent iterations, what to leave behind, what new problems to address and how. This study highlighted the benefits of having criteria to help guide such decisions. For example, while sometimes only a proportion of students engaged with a particular resource, these developments were still deemed valuable, as targeting a small segment of students in large-scale courses can still have a substantive impact. Criteria to support decision making in the DBR process, and assess the impact of an intervention, might consider factors such as uptake by students, workload scalability for the teaching team and reusability across semesters. This could help identify viable interventions that have proportionally low student uptake but sufficiently high reuse and scalability.

The process outlined in this study suggests that to address scale, design needs to focus on both the individual and group levels. For example, considerations of individual students include designing in a way that reduces the cognitive load and provides opportunities for self-determination to support intrinsic motivation, while considerations at the group level relate to segmentation and efficiently designing in ways that cater for different students.

In summary, the study highlighted several implications and opportunities for further research. It provided a more comprehensive set of guidelines for effective educational design in large classes that can be tested and refined in future studies. It demonstrated how design-based research was an effective method for supporting the emergent innovation over time in a large course, and illustrated how transformative education design projects will need to respond to both challenges relating to effective unit design and challenges emerging from the broader context. To assist in the implementation of this method in future studies, some preliminary criteria for decision-making were proposed to better support the design process. These criteria can be further developed and tested in large-class design contexts. Finally, the study demonstrated that in addition to designing at the individual level, designs need to consider the group level to ensure different students are catered for. Innovating within the constraints of large class teaching is a challenge (Dean et al., 2017; Grohs et al., 2019). However, DBR approaches provide an opportunity to progressively develop large courses with the aim of moving away from transactional deliveries and incorporating a more active, collaborative and “high impact” learning practices (Wall, 2017) to better support students with the skills they need for the workplace.

Figures

Figure 1

Phases of design-based research adapted from Reeves (2006) and McKenney and Reeves (2018)

Figure 2

Excerpts from online modules (pre-work)

Figure 3

Resequencing and embedding practice and immediate feedback on sub-topics into weeks 2 and 3

Table 1

Data collected over three iterations

	Iteration 1	Iteration 2	Iteration 3
Total student enrolment	1,372	983	1,463
Student survey: Used to understand students' perceptions of course developments. Included free-text and Likert-scale questions asking level of agreement on statements (from strongly agree to strongly disagree). The survey was distributed to all students within the course	Y (n = 154, 11.2%)	N	N
Student focus group: Used to obtain a more in-depth understanding of students' experiences. Students were recruited via an email that was sent by the CLaS project officer to all students within the course. No students were excluded from participation	N	Y (n = 6)	Y (n = 5)
Coordinator interview: A semi-structured interview at the end of each iteration to understand the coordinators' perspectives on course developments and the co-design process. In the context of the institution associated with the study, a coordinator is someone who leads the teaching team and oversees a subject as a whole (that is, a single course or unit of study). In iteration 3, both coordinators were interviewed together	Y	Y	Y
Staff focus group: Added in the final iteration to elicit the teaching team perspectives on course developments. Staff focus groups included tutors who were part of the teaching team. The teaching staff were recruited via an email sent by the CLaS project officer. No staff were excluded from participation	N	N	Y (n = 2)
Learning analytics: Student usage data of learning technologies to understand how students engaged with online activities in the course	Y	Y	Y
Standard course evaluations: The University's standard Unit of Study Survey, which includes qualitative and quantitative questions, provided an overarching picture of student perceptions across all semesters	Y	Y	Y

Source(s): Table created by authors

Table A1

Summary of challenges within the unit design and broader context

Challenges within the unit design	Financial mathematics/discipline-specific challenges	• More scaffolding was needed to support the group assignment • The need for more “signposting” to orient students to the content, including relevance to the profession and connections between concepts within and across weeks • Students struggle if they do not master the financial mathematics content early in the unit
	General learning and teaching challenges	• More coordinator presence was needed to address variability in the experience of students from different streams • Students have different levels of prior knowledge • Limited opportunities for individual feedback and attention due to the current structure and scale of the unit • Continuing to refine the blended learning model • Decisions about where to direct scarce development resources when only a portion of students engaged with many of the interventions
Challenges emerging from the broader context	Challenges from changing learning and teaching environment (program, institution, sector)	• Existing unit materials were not fit for purpose • Record student enrolments this semester • Institutional decisions about online vs face-to-face delivery models (due to COVID-19 pandemic)
	Project-related challenges	• Changing project team members • Sustainability across iterations and after the project

Source(s): Table created by authors

Appendix

This appendix summarises the main challenges identified within the unit across the three iterations