A case study on narrative structures in instructional MOOC designs

Introduction

With its origins in Northern America, massive open online courses (MOOCs) have conquered not only Europe but educational systems worldwide. The phenomenon can indeed be called global, as Anant Agarwal (2016), the CEO of edX (www.edx.org; one of the most popular American MOOC platforms) has done. He points out “that MOOCs have […] demonstrated potential benefit as a catalyst for change within universities and all over the world” (Agarwal, 2016). Although completion rates are typically fairly low compared to the number of enrolled participants, Agarwal (2016) continues, “MOOC platforms have helped entire countries build their labor forces and create conduits for dramatic social change.” MOOCs can help in making knowledge available for everybody without limitations in time and space, at least theoretically. However, the so-called democratizing process of education that MOOCs were supposed to trigger did not take place in the way experts predicted it almost ten years ago (Dillahunt et al., 2014; Hansen and Reich, 2015), yet Laura Pappano (2012) still declared “the year of the MOOC.”

The MOOC’s dropout problem

In the time that has passed between these first expectations regarding MOOCs and their current standing, scientific research has focused on different aspects within this area. In addition, MOOCs have undergone classification, several subtypes have been identified and still more seem to be forthcoming. Researchers talk about different “MOOC Derivatives” (Hollands and Tirthali, 2014) for slightly modified types of MOOCs, like the personal open online course, the mini-MOOC, or the small private online course, to name just a few. The original dichotomy of xMOOC (based on traditional course structures) and cMOOC (focused on interaction and connectivist principles) was forced to expand. Still, different instructional types and design approaches have led to multiple occurrences of the same phenomenon, like high dropout and hence low completion rates (Colman, 2013; Fidalgo-Blanco et al., 2014; Khalil and Ebner, 2014).

As a result, there is a growing body of recommendations from experts and MOOC providers (usually based on learning analytics, see Veletsianos et al. (2016), results from earlier xMOOCs, specific empirical research as well as literature reviews, such as Bonk et al., in press) with regard to strategies against the decrease in participant activity (Jasnani, 2013; Khalil and Ebner, 2013a, b).

If we emanate from “the typical completion rate of below 10%, approximately 7.5%” (Jasnani, 2013, p. 6) or even less, namely, “from around 3% to 15% of all enrollees” (Hollands and Tirthali, 2014, p. 42), it is not surprising that the investigation of possible reasons for this phenomenon is a pressing concern (Colman, 2013; Khalil and Ebner, 2013a, b). In his online survey, Dan Colman (2013) identified several reasons why participants dropped out instead of finishing a MOOC they had registered for. There are various criteria tightly linked to unfilled expectations of the learner; for example (all quotations cited from Colman, 2013):

• The time component: “Takes Too Much Time.”

• The knowledge component: either the course “Assumes Too Much Knowledge” or it was “Too Basic, Not Really at the Level of Stanford, Oxford and MIT.”

• The instructional (design) component: there might be “Lecture Fatigue” due to the fact that “MOOCs often rely on formal video lectures.” Participants think “MOOCs would be better served if they relied more heavily on interactive forms of pedagogy.” This aspect leads to the second instructional argument, namely, “Poor Course Design” and the wrong decision of tool usage, namely, “Clunky Community/Communication Tools.”

• The financial component: “Surprised by Hidden Costs.”

Besides the learner’s personal expectations, there are criteria that focus on the peer group, its behavior, and the kind of socializing within the MOOC:

• The social component: “Bad Peer Review & Trolls.”

  Finally, there is the participants’ attitude toward the MOOC itself that can be seen as an important criterion whether to attend and complete a MOOC or not:

• The personal component: it was not the primary goal to do a MOOC or to finish it:

“You’re Just Shopping Around” as only parts of it are interesting and important: “You’re There to Learn, Not for the Credential at the End.”

As can be seen, these reasons are pretty heterogeneous and range from very personal, individual components to more general course-related components and the peer group in a wider sense. These components can be seen as important nodes in the sense of a connectivist approach, the “Learning Theory for the Digital Age” as George Siemens defined back in 2004. Crucial principles as described by Siemens (2004) are that “Learning and knowledge rests in diversity of opinions,” as well as “Learning is a process of connecting specialized nodes or information sources.” There are two important core skills, namely, “Nurturing and maintaining connections is needed to facilitate continual learning” and the “Ability to see connections between fields, ideas, and concepts” (Siemens, 2004). The learners themselves get to decide what and how they want to learn, and in what ways they want to deepen their learning experience in specific fields of knowledge.

With these core skills in mind, Hanan Khalil and Martin Ebner (2013a, b) looked at the different possibilities of interaction within MOOCs and the influence these interactions have on the learner’s motivation. They found out that interaction is a crucial aspect in MOOCs when it comes to the learner’s satisfaction, which in turn is a main decision-making aid when deciding on whether to continue or drop out from a MOOC. In their studies, the authors concluded that there were at least three relevant forms of interaction: among learners; between learners and teachers; and between learners and learning resources, i.e. the content. These three actors form the so-called instructional triangle, as shown in Figure 1.

Kelli Nipper and Paola Sztajn (2008) adopted the instructional triangle for mathematics and added the interactions among students to the interdependence between students and teacher, and content, respectively. Students in this case are the learners or “participants” in a more conventional sense, the teacher is the “organizer” and mathematics is the “content” (Nipper and Sztajn, 2008, p. 336). Regarding the contexts, there are several aspects that warrant consideration, for example, the course design (in contrast to the course content), the financial aspects (as mentioned above), and more general institutional aspects. With the triangle in mind, instructional concepts have been aiming at creating a course design that satisfies not only these three stakeholders, but also adapts to the specific MOOC-related context in a way that enhances and the learner’s experiences and his motivation to continue the course.

Instructional designs for MOOCs

Learning contexts also play a role in traditional classroom or conventional online teaching, but it would seem that they are even more important when it comes to designing MOOCs. In their pedagogical guideline for MOOC practitioners, Elke Lackner et al. (2014) listed six categories that have to be taken into account during the different phases of MOOC design and development. These phases are based on ADDIE (Kopp and Lackner, 2014), the most commonly known instructional design framework focused on core requirements, structure, participant requirements, assignments, media design, communication, and resources (Branson et al., 1975). The teacher’s role is redefined and/or enriched to become a teacher-facilitator-moderator, as well as the choice of the content, the course language, and the participants’ role. The course creator, i.e. teacher, has to decide whether they choose a rather specialized and narrow topic that usually only allows them to address a small target group of “experts,” or a topic that might be of interest for a broad and diverse audience. The same holds true with regard to language: if they decide to do a course in English, the target group is potentially larger than that with a national language such as Italian or German. However, if the teachers want to address a non-academically inclined target group in a specific region, it might be important to choose the regionally spoken language.

The audience, i.e. the participants, is the most crucial player when it comes to course concepts. The teacher can neither predict the number of participants that attend a MOOC nor their social, economic or geographical background, and when thinking of a specific target group during the conceptual phase the teacher cannot be sure that the attending audience will match the fictive target. Additionally, as Colman (2013) showed, a bad peer group or trolls are indeed reasons to abandon a MOOC; hence, the social context of the participants, related to a main claim of connectivist learning settings, must not be neglected. Kopp and Lackner (2014) come to the conclusion that MOOCs need a special instructional design that accounts for an unpredictable, heterogeneous audience and their specific expectations and behaviors, in order to foster participation and retention. Last but not least, learners exhibit different types of motivation to attend a course, and they need specific competences, such as the ability to learn in a self-directed way, in order to make the most of a MOOC (Bonk et al., 2015; Song and Bonk, 2016). This learning process may take place inside the MOOC or within a wider context outside the course, e.g. social networks (Veletsianos et al., 2015).

Design patterns for MOOCs

As traditional instructional design strategies seem to struggle with the challenges created by MOOCs, alternative instructional approaches should be considered. One promising approach that has already received a lot of attention is the application of specific instructional design patterns. They have a tremendous influence on the learner’s motivation, which in turn is crucial for the instigation of learning processes and continuation (Lackner et al., 2015). In contrast to traditional instructional guidelines, “patterns support a wider variety of application scenarios” (Bauer and Baumgartner, 2012), since not only the subject itself but also the context has to be considered. As Reinhard Bauer and Peter Baumgartner (2012) sum up: “Applied to a concrete design process this means that it is not possible to design an isolated ‘thing’ […] ignoring the world around this artifact […] and within this artifact […]. The goal is a holistic approach” in course design that does not only integrate immanent factors, as proposed by traditional instructional design models like ADDIE (Branson et al., 1975), but also covers experiences and is based on good practices as well as existing designs and concepts. Hence, an effective instructional design pattern combines “a clear articulation of a design problem and a design solution […] offering a rationale which bridges between pedagogical philosophy, research-based evidence and experiential knowledge of design” (Goodyear, 2005, p. 92). If a design problem or challenge presents itself, specific design patterns may help in finding an adequate solution.

In their paper MOOCs as granular systems: design patterns to foster participant activity Elke Lackner et al. (2015) formulated a design proposal for MOOCs that was based on learning analytics results from preceding xMOOCs and an extensive literature review. This main theme of this design proposal can be summed up under the term granularity. From this rather generic design aspect, three major design patterns were deduced.

The Dr Internet MOOC – a practical example

The existing body of research on MOOC retention highlights many areas that could prove fruitful for improvement efforts. For example, findings with regard to dropout motivation can and should inform didactical models on the one hand and practical aspects of MOOC design on the other. In this paper, we would like to focus on the efficacy of narrative design patterns, and in order to put these theoretical insights into practice and to evaluate their effectiveness in improving MOOC retention, they were implemented in the best possible way during the conceptualization phase, design, and development of a fairly unique xMOOC called “Dr Internet.” This course was the centerpiece of a comprehensive research project, which is the main reason why it was selected for additional testing of some of the described design patterns: there was an interdisciplinary team of researchers and practitioners involved in designing and implementing the MOOC, more extensive data collection, and an attractive topic that was expected to procure large numbers of participants that would help to study activity patterns.

Results from the first run of the Dr Internet MOOC

Due to the fact that the MOOC was only one part of a bigger research agenda, participants were required to fill in a mandatory questionnaire before entering the MOOC for the first time. This questionnaire mainly included questions on health-related matters and previous experiences when looking for health information on the internet, but it also provided a set of basic sociodemographic variables[1]. Overall, the MOOC did not attract a particularly large crowd of users during its first run (October 27-December 6, 2015). Out of 370 registered participants, 206 did in fact enter the MOOC and completed the questionnaire (56 percent), thus qualifying as active learners. The majority of these were female (62 percent) and the average age was 39 years (minimum 15 years, maximum 75). Singles constituted the dominant civil status (58 percent), followed by married (32 percent), divorced (8 percent), and widowed (2 percent) users. With regard to educational level, we see a familiar picture that has been reported from many other MOOCs (Dillahunt et al., 2014; Hansen and Reich, 2015; Neuböck et al., 2015): well-educated participants are heavily overrepresented, with almost 90 percent having completed either high school (33 percent) or a university degree (56 percent).

For an evaluation of the implemented design elements, we obviously have to look at dropout and activity rates throughout the course. Anybody who either took a quiz, read or wrote something in the forum, or watched a video in any given week was classified as an “active participant.” Again, we encounter some fairly typical results: 124 active participants started out in the first week, and 52 still remained in the last week of the course. Figure 2 gives an overview of the decline in the numbers of active participants: as we can see, the biggest drop in participant numbers did not occur in week 4, but between weeks 2 and 3, where there was a loss of almost 40 percent (from 139 down to 84 users). After that, the numbers are stabilizing, and the decline is almost linear. When comparing the small group still active in week 6 (n=52) with the big group that filled in the questionnaire in the beginning (n=206), it can be noted that most sociodemographic variables change very little (gender proportions are now almost equal, single participants drop out at a higher rate than married ones, and the proportion of university-educated is only slightly higher than of those who completed high school). Overall, it can be said that the dropout rates seem to have a weak relationship with sociodemographic characteristics.

The interpretation of these results regarding the effectiveness of the implemented design-based elements is not straightforward. On the one hand, weeks 2 and 3 featured the first two-week video, which coincides with the highest loss in user numbers over the six weeks course duration (see Figure 2). On the other hand, the second two-week video was released during weeks 4 and 5, where we see only a slight drop in numbers. After the initial peak in week 2 and the following week’s loss, the slope of the curve is gradual rather than steep, so there is likely no single effect in the design that caused direct and noticeable dropouts among all participants. When looking only at those participants who completed any quiz in each week, there is a much less pronounced drop in numbers after week 2, and an actual increase between weeks 4 and 5. This could indicate that there is a subgroup of participants that is more interested in the video and quiz features of the MOOC, and that to them the storytelling approach might have been more attractive.

The overall decline is substantial, yet compared with the drastically low completion rates that are usually associated with MOOCs (Hollands and Tirthali, 2014; Jasnani, 2013), the result is quite adequate. Depending on whether the completion rate is calculated as a percentage of registered or active users, the Dr Internet MOOC achieves 8 and 14 percent, respectively. Still, these results do not mark a significant improvement in retention as had been expected from the theoretical assumptions discussed above. In summary, this MOOC did not produce any solid evidence to indicate that the implemented design-based elements were in fact making a difference.

Discussion and conclusion

While the results from the Dr Internet MOOC do not provide any support for the effectiveness of the design patterns in questions, there is also no clear indication that they do not work. The attendance numbers fell short of expectations, so further research with larger sample sizes is needed to produce more data on the subject. While it is hard to discern the reason for the rather small turnout, several factors can be assumed to play a role: the timing of the course dates at the beginning of the school semester, which is generally a good time, but also means that this MOOC was competing for participants with a few other courses on the same platform, the channels used for advertising, in this case mainstream media rather than targeted promotion for an internet-affine audience, and the (academically speaking) low-threshold topic, since the typical MOOC users are disproportionately well educated (Dillahunt et al., 2014). It might also be possible that the compulsory questionnaire at the beginning of the course could have kept some registered users from active participation, although there is no evidence for this assumption when looking at the data – 56 percent of those enrolled qualify as active learners, a fairly average proportion (Koller et al., 2013).

The lack of retention effects could be due to a variety of reasons, for example, the specific way we implemented those principles – the screenplay for the videos might not have delivered the right kind of cliffhanger, and the suspense peaks might not have been perceived as such. The narrator’s framing role could have been developed in a more pronounced and less subtle way – we did not include an introductory video in which the general practitioner introduced himself and his role within the course structure. It might also have been more effective to let the general practitioner activate the forum discussions to strengthen his role as a guide. Additionally, while the MOOC’s topic was fairly attractive in the eyes of the press, it was also unconventional in the sense that it did not convey a lot of factual knowledge, and thus the results may not be representative of more traditional MOOCs. In fact, it is highly plausible that the kind of knowledge conveyed in MOOCs (e.g. competence building vs knowledge gain) determines some of their specific characteristics (like materials and activities offered) and thus also affect the effectiveness of some design patterns.

Based on the data from this course and experiences with previous MOOCs, we have identified another potential design problem in line with the pacing device that could have a significant influence on participants’ dropout: the scheduling of weekly modules. It is tempting to think that the waiting time for either a case resolution or more videos of the same interesting topic is something that persuades participants to log in next week when new content is available. However, this model similar to the suspense peak approach might not work for everyone, and there is evidence that some people actually lose interest when they have to wait for access to the materials they want to see. They might forget about it in the meantime, they are not interested in making the MOOC part of their weekly routine, or they could be unable to take time out of their busy schedules on a regular weekly basis. Data from the Dr Internet MOOC shows that there is only a very small group of users who log in weekly and participate in the available features as they are released. The overwhelming majority has fewer logins, but many of them catch up on material that is new to them.

This phenomenon has inspired some debate about self-paced as opposed to session-based MOOCs. Both approaches come with benefits and disadvantages: self-paced learning without fixed start and end dates cannot provide the same kind of community of many participants learning the same things at the same time, and there are no deadlines, which to some can be a powerful motivation to complete course tasks (Shah, 2015). However, it would seem that the self-paced concept is gaining momentum; a survey conducted among European institutions of higher education revealed that a subsample of MOOC providers attributed higher relevance to the freedom of learning at one’s own pace than to fixed start and end dates (Jansen and Schuwer, 2015).

There are several measures that could help to address this particular variation in learners’ preferences. On the one hand, there is the option of structural changes to the weekly schedules of MOOCs in the direction of self-paced learning, and a higher degree of learner empowerment in general (Guàrdia et al., 2013). On the other hand, the session-based concept could benefit from e-mail reminders for the participants to inform them about new content and to encourage them to return to the course. Even though learner’s structural preferences might vary among the population of MOOC participants, it is reasonable to assume that a stronger focus on these preferences is in any case a promising strategy for future research on MOOC retention.

While course design is among the most important factors for retention in MOOCs, it is far from being the only one. There is always a certain fraction of participants who enroll for MOOCs because of the interesting topic or one specific aspect they are interested in. Earning a credential, an acknowledgment or a statement of accomplishment was never their intention in the first place. They might just want to see the first week of a MOOC out of curiosity for the topic, but once they got an idea of the MOOC’s concept and the extent of the materials offered, their curiosity is satisfied and they drop out early on, as seen with the Dr Internet MOOC. Sadly, there is only so much that MOOC providers can do about this.

In conclusion, it has to be stated that even though the topic as well as the course design and requirements of the Dr Internet MOOC were designed to be low threshold, only a relatively small number of participants enrolled for the course. The narrative design patterns used to develop the MOOC did not have the intended effects on retention that the course creators had hoped they would have. Further research regarding the motivations for dropping out or finishing a course is an urgent desideratum – aspects like participant’s interest, required effort and perceived usefulness are not only important cognitive factors shaping the learning process, they also warrant a more in-depth consideration in the process of developing and implementing suitable design patterns to enhance motivation in the best possible way. However, the lack of sizeable results with regard to the design-based improvement of retention can be seen to raise other questions concerning the current prominent focus on dropout and completion rates in MOOCs. After all, these parameters largely stem from a more traditional educational context and might not always provide a fruitful perspective on inherently different learning platforms that result in diverging learning behaviors. A different, less problematizing focus both in research and practice could draw on the strengths of online learning environments rather than their weaknesses, and retention rates might benefit from this approach nonetheless.