Mobile learning analytics in higher education: usability testing and evaluation of an app prototype

1. Introduction

The utilization of digital technologies in everyday life is constantly growing. For students, digital technologies are mostly indispensable. For example, 95 per cent of 14- to 29-year-old Germans used a smartphone in 2016 (Statista/Bitkom, 2017). However, the potential of mobile devices has not been fully developed in universities. There are numerous possibilities how digital technologies can improve learning in higher education institutions. For example, in the NMC Horizon Report 2017, six key trends were identified to adopt technologies in the higher education sector:

1. advancing cultures of learning (higher involvement in innovation development processes);

2. deeper learning approaches (connection of learning with the real world);

3. growing focus on measuring learning (analytics of data out of learning environments);

4. redesigning learning spaces (improvement of the technical infrastructure);

5. blended learning designs (combination of online and face-to-face learning); and

6. collaborative learning (social interaction and intercultural experiences) (Adams Becker et al., 2017).

The presented study covers two emerging fields of research in higher education:

1. learning analytics; and

2. mobile learning.

Learning analytics (LA) use static and dynamic information for real-time support of students’ learning processes and optimization of learning environments (Ifenthaler, 2015). Besides its flexibility, the main advantages of LA are personalization and the real-time availability of data (Ifenthaler et al., 2014). Lecturers may use rich data for pedagogical decision-making, understand individual performance development of students, identify potential lack of students’ capabilities or the need for curricular improvements (Mattingly et al., 2012). With LA, both students and lecturers can reflect on and improve their communication skills. By capturing, analyzing and visualizing the available information about learning and teaching, lecturers are able to make more reliable predictions about their students’ academic success (Macfadyen and Dawson, 2012; Mah and Ifenthaler, 2018). Furthermore, students at risk can be identified and given support through personalized pedagogical interventions (Lockyer et al., 2013). Successful applications of LA at universities are, for example, Course Signals at Purdue University, aiming to identify students at risk using an approach similar to a traffic light system (green – no risk, yellow – potential risk, red – risky). Students and lecturers can identify needs for action to improve their learning situation. Furthermore, lecturers are able to intervene and help early (Ifenthaler and Schumacher, 2016). At the University of Wollongong, SNAPP (social networks adapting pedagogical practice) is used (Dawson et al., 2011). The main purpose of this system is to increase collaborative learning. For example, conversations in forums are analyzed for investigating the relationship between students and excluded students can be determined. If students have a high learning orientation, they are discussing more frequently in forums about learning and resource sharing. And, students with a high focus on performance are discussing for example about assessments. The interaction delivers data that can be used to get an insight of student’s engagement. At the end of a lecture, the system can also be used for reflection (Sclater et al., 2016). Recent developments include LeAP (LA profiles) which provides students support toward specific learning outcomes in formal learning environments (Schön and Ifenthaler, 2018).

Mobile learning – m-learning – enables learning through personal portable electronic devices across multiple contexts (Delcker et al., 2018; Sampson et al., 2013). Through the use of mobile devices, students have access to learning artifacts more easily. The precondition is the availability of Web-connected devices. In addition, students are independent from locations, time, and they can communicate asynchronously (Lin et al., 2016). Mobile learning supports self-regulated learning (Ifenthaler and Lehmann, 2012) on the one hand, and it is an essential element of blended learning environments on the other hand (Al Saleh and Bhat, 2015). An example for a German mobile application is ARSnova of the THM University of Applied Sciences Gießen. ARS stands for an audience response system. Via this app, students and lecturers can communicate interactively. For example, students can ask questions anonymously during a lecture and the lecturer is able to answer the questions in near real time. Moreover, the communication can take place before or after a lecture. Another function is the evaluation of a lecture directly in the app (ARSnova, 2015).

The app MyLA (My Learning Analytics) of the Cooperative State University Mannheim and University of Mannheim targets to improve learning processes in higher education institutions. MyLA provides ubiquitous communication in form of short messages from students to their lecturer and vice versa. This is especially useful for dual-system courses where students are often away from the campus. Using such data, lecturers can adapt their lectures at university and implement personalized interventions, e.g. adaption of learning materials or a detailed view at topics the students have problems with. Furthermore, the app supports a more personalized and individual way of learning. The students can document their learning motivation through their learning process. The app provides information which enables students to observe their personal progress over time. In summary, MyLA combines LA and m-learning with the extension of personalized learning elements.

The remaining article focuses on the components of the MyLA app prototype and the usability testing of the app. Additionally, the current version of the MyLA app and the MyLA dashboard will be presented.

2. Literature review

Digital media and new technologies have been deployed frequently in the classroom since the 1980s. With the development of wireless internet access, innovative forms of communication and trans-regional collaboration were born. Digitalization and globalization also led to a change in higher education institutions. Particularly, the shift from traditional teaching (i.e. face-to-face teaching) to adaptive and self-regulated learning (SRL) introduced the need for smart learning classrooms. Hence, the benefits for students are a more personalized and independent learning environment (flexibility of time and location) (Hwang, 2014). According to Koper (2014), the following aspects can classify smart learning environments:

• Physical learning environments enhanced by digital devices.

• Consideration of learners’ status quo (e.g. culture, context).

• Digital devices provide additional features for learning, like virtual collaboration or communication, feedback and assessment.

• The monitoring of learners’ development using analytics results of relevant data.

Particularly interesting in this context and for generating a comprehensive understanding of students learning behavior is the use of advanced data sources. Case studies indicated that learning analytics can:

• improve the teaching quality (Ifenthaler et al., 2018a);

• support the learning engaging and motivation (Schumacher and Ifenthaler, 2018a); and

• encourage self-control (Sclater et al., 2016), as well as having potential to support assessment (Ifenthaler et al., 2018b).

The first aspect involves that a lecture can adapt the curriculum toward the students’ requirements. As a predictive tool, LA can help to enhance the students’ retention. Furthermore, the learners can use the tracked, analyzed and often visualized data as a self-control mechanism.

A current study investigated three relevant features of LA from the students’ point of view (Schumacher and Ifenthaler, 2018b):

• Possibility of self-assessment.

• Recommendations for learning.

• Timeline for the evaluation of the students’ status quo.

Moreover, privacy is an important aspect when it comes to LA. Students recommend clear transparency and trustfulness of LA applications (Ifenthaler and Tracey, 2016). By providing the users with access to and control over their personal data, confidence can be enhanced (Prinsloo and Slade, 2015). One further aspect is the possibility of making an interpretation of the learners’ information very simple. By visualization of the data, learners, lecturers, as well as other stakeholders can easily identify a good or poor performer (Ebner et al., 2015). Furthermore, individual information can be collected from different sources. While login information and the frequency of particular websites are classically representing quantitative data sources, entries in forums or blog, for example, have to be interpreted as qualitative resources (Ifenthaler and Schumacher, 2016).

Dashboards including vast amounts of data, for example, performance or user data, are a technical solution to support the teaching and learning process. The majority of dashboards used in the context of learning and teaching consider four stages. First, teachers are getting aware of several activities (e.g. the use of learning materials) in the course by using a dashboard. Next, they can reflect concerning their actual teaching process under consideration of the collected data. As a third step, teachers can identify at-risk students or isolated ones more easily. Last but not least, dashboards can be used to become aware of the current impact, and thereby how the information can help to support the students, e.g. through proving opportunities for re-socialization or providing feedback to poor performers (Verbert et al., 2013).

Usability testing for mobile applications considers different factors. For example, Nielsen (1993) mentioned five attributes: learnability, efficiency, memorability, errors and satisfaction. Learnability covers an easy-to-learn approach, efficiency leads to high productivity, memorability helps to remember the system more easily, errors should not occur or can be revoked and satisfaction means that users like the system. But there are more attributes that can be valuable for usability testing. As an example, the MAUEM (mobile application usability evaluation metrics) model has nine attributes. The five of Nielsen and four added attributes: effectiveness, cognitive workload, interruptability and simplicity (Saleh et al., 2017).

A usability example will be introduced of ME2.0 (Mobile Electronic Personality version 2). This is an application that helps users to manage their personalization attributes, e.g. against identity thefts. The methodology covered two studies. In the first study, users (N = 9) read different scenario uses of the application before they were interviewed. In a last step, the participants illustrated their own desired UI and navigation. The second study consisted of an online survey (N = 16) and an evaluation of an illustrated prototype (N = 4) (Oyomno et al., 2013). Another example is EASEL (Education through Application-Supported Experiential Learning). This is a mobile platform, which can be used by instructors to send reflection, as well as content prompts. After a pre-conducted needs analysis, a prototype was built. The participants (N = 14) were divided in faculty members (N = 8) and students (N = 6). Each group had to solve similar (but different) tasks toward a hypothetical scenario. The screen was recorded for analyzing how participants navigated within the application, and the facilitator observed them. All participants were asked to “talk-aloud” while completing the tasks. For documenting comments or gestures, cameras were used to record all sessions. After each task, the participants answered short questionnaires about their experiences. Furthermore, the usability questionnaire was conducted with questions concerning their general use of technology and application-related questions. In addition, all participants were asked to discuss their own experiences with the facilitator (Schnepp and Rogers, 2017).

3. Usability testing

This section focusses on the methodology, as well as on the findings the of MyLA prototype’s usability testing. This is the basis for the actual app and the additional MyLA dashboard, which will be introduced in Section 4.

4. MyLA app and MyLA dashboard

In this section, the current applications (MyLA app and MyLA dashboard) will be introduced. The navigation and design were adjusted on basis of the conducted usability testing in spring 2017.

5. Discussion and conclusion

Usability testing is a very useful and advantageous method for formative evaluation of the development process (Pirnay-Dummer et al., 2010). The project MyLA can benefit from the valuable input of the potential app user.

With regard to RQ1 (“How intuitive is the MyLA prototype (design, navigation) for students?”), it can be suggested that the prototype is intuitive for the target group. Overall, the results were predominantly positive (all average values were higher than 3); however, some issues were identified for improvement, especially with regard to colors.

In addition, students provided feedback within the scope of open questions. The students evaluated the app’s clarity, simplicity, navigation/structure and features as very positive. Regarding the assessment of the color scheme, the responses were heterogeneous. Critical issues were partly used dark colors, as well as color combinations. Moreover, some students mentioned that the app contained a broad color spectrum. Furthermore, they suggested additional features for future app versions, e.g. a calendar function. With regard to RQ2 (“Is there room for improvement for the development of MyLA prototype?”), it can be summarized that the colors need to be adjusted.

To answer the RQ3 (“How can the empirical results (quantitative and qualitative) help to optimize the MyLA prototype for the main survey?”), the following aspects can be summarized. The involvement of the target group (students) was very important at this early project stage. The reason for that is very simple, because the students are prospective users of MyLA. Therefore, it is inevitable to get them highly involved. The success of a project depends on its acceptance. If the acceptance is high, the potential usage can be high, too. For reaching a large consumption of MyLA, it is necessary to implement the students’ feedback and recommendations. With help of these new insights, future adaptions can be managed. Hence, the findings of the MyLA usability testing provided detailed insights to optimize the app prototype. Some lessons learned of the MyLA usability testing were the following proven statements:

• The handling of the MyLA Web app prototype is intuitive.

• The app’s structure is easy to learn.

• The navigation within the app is clear and user-friendly.

• The students mostly like the idea of MyLA.

• The design and colors can be improved, because the opinions deviate fairly high.

With respect to the usability testing and the students’ recommendations, the app was adjusted. Furthermore, an additional application for lecturers was developed (MyLA dashboard). For guaranteeing the functionality of both tools, they were tested with a small sample. One aim of the research project is to focus on the needs of the major target groups (students and lecturers); thus, several workshops were conducted in 2017 and further events are planned for 2018. With respect to the current status of the project, it has to be mentioned that the panel study started successfully at the beginning of 2018. First prefunded results of the study will be available in summer 2018.

MyLA can be used in different learning settings like in lectures, workshops or practical phases. The lecturer can integrate both applications as they fit hihe/sher needs the best. There are no universal approaches, but it is highly recommended to clarify the general setup of using MyLA to the students in an early process stage. The applications can be used for student feedback, communication with the lecturer, (flexible) time management and for performance or lecture evaluations. Via the MyLA dashboard, the lecturer can manage these functionalities and observe the incoming data. Additionally, the identification of further functionalities needed by the students is an evident part of the project.

According to the literature, the research area of LA is still in its infancy in German-speaking countries (Ifenthaler and Drachsler, 2018; Ifenthaler and Schumacher, 2016). Moreover, with the advance of digital technologies, new skills and competencies are required from students and teachers in higher education institutions, as well as their future workplace (Gibson and Ifenthaler, 2017). In addition, the situation of non-traditional academics will be considered within the project. The combination of LA and m-learning seems to be beneficial to contribute to the individual demands of a diverse group of learners. For example, the project team aims to identify patterns of learning behavior and the management of learning tasks, given a high demand of workload at the university and the workplace (Ifenthaler, 2018). In summary, the project can be seen as an important contribution to the topic of LA for higher education in Germany.

Some limitations and challenges of MyLA (and probably other applications) are as follows:

• Re-designing and implementation of digital tools in curricula.

• Conviction of lecturers to sustainably use digital offers.

• Boost of m-learning concepts in higher education.

Some chances of MyLA are as follows:

• Data collection concerning application usage for research improvement in Germany.

• Privacy-based approach.

• Data collection for lecturers to identify certain problems within a lecture.

• Transfer the Web app to a hybrid app for using device capabilities in future.

As a conclusion of all mentioned points above, there is more research necessary. Some useful research approaches could be: how should a privacy-based application look like in the opinion of students and lecturers, what are overlapping interests of students and lecturers to strengthen and adapt app functions or digital innovations, how can different digital approaches (in one university or cross-university) work together to not “reinvent the wheel”. Furthermore, as already discussed, the additional app features have to be implemented in case of further development. All in all, the project delivers a profound base with many important research questions and approaches for future work.