Big data ethics and its role in the innovation and technology adoption process

1.1 Problem identification

Innovation (new product development) and technology adoption (continuous improvement and production optimization) are integral components of the engineering design process. However, little engineering coursework accounts for the role of ethics within these processes. One author indicates the most common reason for the lack of ethics (research ethics, engineering ethics or otherwise) is that technical engineering coursework offers little room for adding yet another layer of content and focus (Davis, 2006).

As a result, accredited university engineering degree programs typically leverage the traditional Engineering Ethics course to be the all-in-one solution to provide students with the necessary background when considering the ethical treatment of people within the engineering design process (Barry and Ohland, 2012). In addition, some students may choose to get a minor in ethics. This current approach to teaching ethics to engineering students poses two problems. First, Engineering Ethics courses continue to rely on the “tried but true” “oldies but goodies” case studies, such as the Ford Pinto Exploding Fuel Tank, Hyatt Regency Walkway Collapse and Challenger Shuttle O-ring Seal, and fail to account for more updated ethical issues commonly associated with emerging technologies, especially big data. Second, observational evidence suggests the standard Engineering Ethics course exists for the sole purpose of meeting ABET's “f” learning outcome (i.e. an understanding of professional and ethical responsibility). These courses tend to exist in isolation from other courses. As a result, other engineering coursework pays scant or no attention to ethics offering a limited glimpse into the implications of ethics in the various course topic areas.

In summary, the problem exists in that engineering degree programs lack a holistic emphasis on ethics (outside the traditional Engineering Ethics). Because of this, engineering students have few learning opportunities to practice and reflect on ethical decision-making. As a consequence, given technology's potential to promote poor bias and discrimination practices within the innovation and technology adoption process, engineers' lack of awareness, exposure and applied use could prove damaging for companies trying to proactively stay ahead of their competition.

1.2 Proposed solution and contribution

There are many pedagogical approaches to teaching ethics including the following: independent study, extracurricular, super-curricular, guest lecture, elective professional ethics course outside the engineering department, elective professional ethics course inside the engineering department or required ABET (Accreditation Board for Engineering and Technology) ethics course within the engineering department. Yet, one approach, in particular, is posited to have the greatest impact on students. Davis (2006) has proposed the “micro-insertion” strategy as optimal for students to learn about ethics in technical courses. This strategy applies a “small-dose” technique which has the following properties: “(1) It is integrated throughout a course (rather than being presented as a discrete module at only one point in the course). (2) It is based on modifications of small-scale technical problems. (3) It emphasizes ethical issues that professionals confront in the course of their daily activities and are therefore easier for students and novice professionals to imagine encountering (Davis, 2006).”

This paper contributes to the literature by highlighting a curriculum change that integrates big data ethics into an engineering course outside of Engineering Ethics using a modified “micro-insertion” approach. This study is implemented in a way that does include a somewhat substantial change using a five-week module with a focus on big data ethics, as part of a Supply Chain Management Technology course (required for Industrial Engineering Technology majors). The pedagogical interventions, including photo-journaling and information literacy, are delivered with a course context focused on the ethics of emerging technologies. The main assessment for this module is a metacognition reflection assignment. As such, the guiding research question is as follows:

2.1 Learning about big data

Data literacy is the ability to collect, process, understand and communicate numerical information (Fontichiaro and Oehrli, 2016). Data literacy is necessary for students to understand numerical representations of information and be able to both comprehend and judge the statistical arguments made by others (Gould, 2017; Gray et al., 2018; Koltay, 2017). Data literacy has become an important part of every student's education. This knowledge is similar to ethical knowledge in that it should be integrated into every aspect of a student's education (Hunt, 2005; Gould, 2017). Though contributions to the generation and collection of big data are the purview of engineering, many engineers do not actually work with the data themselves and are not necessarily data literate. However, data literacy and ethical literacy should go hand in hand with engineering students. “Ethical and responsible data use is part of knowing how to use data, and that knowledge focuses on how to protect privacy and maintain confidentiality of data (Mandinach et al., 2015).”

There are several proposed methods to teach students about data literacy. D'ignazio (2017) proposes an approach to learning data literacy for nonexperts that is centered around five principles. The first of these is to work with community-centered data. That is to say work with data generated by or directly relevant to the learners themselves. This helps increase their intuitive understanding of the data, as well as keeps them motivated and engaged. The second is to write data biographies (D’ignazio, 2017). This helps learners understand the context and quality of the data. The third is to make data messy or let learners create their own data sets. Faced with the uncertainty and complexity of creating a data set from a specific context, learners start to understand the processes of data collection and how these result in different kinds of data of different qualities. The fourth is to build learner-centered tools. As big data proliferates, so too do the tools to understand it. Most of these, however, are designed for specific audiences, and so learners struggle to know when to employ which tools. The fifth is to make sure data outputs are relevant to the targeted audience. Very often learners are approached with a universal set of principles for data processing, which may not be useful in all circumstances (D’ignazio, 2017).

Kippers et al. (2018) propose teaching data literacy through a five-step process. This process is as follows: (1) define the goal of data collection by clearly formulating the problem to be solved, questions to be answered or hypotheses to be tested. (2) Collect, clean and evaluate the data. (3) Analyze the data. (4) Interpret the data in the context of the data collection goal. (5) Implement measures for improvement of understanding of the data and the data collection process. Mandinach and Gummer (2013) make an argument for a system approach to data literacy. In this approach, data should be viewed as a complex system. Therefore, the interrelationships among data components, the interactions of these components and the hierarchies that form them are the basis for the approach to data literacy.

2.2 Ethics of big data

The complexity of emerging technology issues requires a deep understanding of conventional ethics. These emerging technologies can be recognized as novel technologies that are rapidly being integrated into economic systems or adopted by markets and introduce levels of uncertainty into the socioeconomic domain (Rotolo et al., 2015). One of the prevalent emerging technologies is big data. Moore's law is a principle of computing that posits that because of the constant doubling in the number of transistors on a microchip, the speed and capability of computers will increase commensurately and be accompanied by a drop in cost (Crawford et al., 2014). This increased capacity and lowered cost have created the potential for increasingly greater amounts of data to be collected, compiled, stored, analyzed and used for a multitude of uses (Nunan and Di Domenico, 2013). The power of big data has been touted and recognized such that almost every interaction with information technology now becomes an opportunity to collect more data. This introduces a multitude of ethical challenges. Following questions now emerge Who should be subjected to data gathering and under what conditions? What kinds of data is it acceptable to gather? What are the acceptable uses of these data? The massive collection of data introduces the specter of living in a surveillance state with governments collecting massive amounts of data on the population (Crawford et al., 2014). The extent of this data collection revealed by actions such as the leaks from Edward Snowden demonstrates the extent of these data-gathering efforts and the potential threat they pose to civil liberties. Corporations are also collecting mass amounts of data of their consumers, ostensibly to provide tailored services of better quality. However, given that there are few rules and even less oversight as to how these data can be used, ethical considerations serve as the only guideposts.

The emergence of new technologies amplifies old and/or raises new ethical issues. In the case of big data, these ethical issues are tied to both the nature of the technology and the nature of its focus, namely, information (Hand, 2018; Stahl et al., 2009; Johnson, 2014). The intangible nature of information offers features and possibilities beyond those of many other physically based technologies. Both information and information technology offer affordances making them easily altered, renewable and dynamic. That is to say, data are not fixed. It can be easily changed by the application of a few keystrokes. Data, because of its nature, are also non-rivalrous and difficult to exhaust (Jones and Tonetti, 2020). Access or ownership of data by one individual does not exclude others from equal access and use. These affordances empower the technologies that depend on information levels of malleability and replication effects unseen in other technologies.

Computers are now 300 million times faster than they were thirty-five years ago. Along with this increased speed is significantly decreased cost. These changes are the basis for emerging technologies specifically the evolution of big data. Massive amounts of data can now be collected and processed in fractions of a second. The cost of storage and processing of these data has also decreased significantly. The massive collection of big data opens up a multitude of ethical questions, for example:

1. Information rights and obligations. What information should individuals and companies be able to collect and maintain? Who has the right to access this information and for what purposes? What should this information cost and who should receive benefits? Is the provision of data tied to the right to access products of those data?

2. Property rights and obligations. Who owns data? Do the means of data collection dictate the ownership? Who owns and who should benefit from the intellectual property created from these data? Who is responsible for the accuracy of these data? Do we need new understandings and policy frameworks for these emerging technologies?

3. Accountability and control. What are the harms caused by massive data collection? Who is responsible for downstream effects? How should data crimes be defined? Prosecuted? By whom?

4. System quality. What standards of data and system quality are necessary? Who is responsible for the maintenance of these systems? Should cross network and cross border standards exist?

5. Quality of life. How does the production and use of data technologies change our values? How do these data affect our perception and understanding of people and cultures? What institutions are significantly altered or made obsolete by these emerging technologies? How do technologies like social media and teleworking change the nature of our interactions and how can these be managed? (Laudon and Laudon, 2015; Zwitter, 2014)

The effects of big data and therefore the ethical implications span from the individual (e.g. personal safety, intellectual property, control of personal data, criminal behavior, etc.), to the organizational (liability, data breach disclosure, consumer interactions, etc.), to the societal (e.g. rights of access, disinformation, corporate influence, political participation, etc.), to the global (e.g. cyberwarfare, cross border law enforcement, etc.). These effects are further complicated by the introduction of algorithms. Algorithms are now being deployed to both gather and process data. An understanding of the ethical implications of the design and deployment of these algorithms is critical (Mittelstadt et al., 2016).

2.3 Engineering ethics and big data ethics

Generally, ethics is divided into three categories of morality (Abaté, 2011). Firstly, common morality is an ethical idea shared by people who belong to the same group or culture. Secondly, personal morality is one's personal moral belief acquired from family or religious training. Lastly, professional ethics, a set of ethical principles adopted by professions, guides the professionals in that profession. For example, medical practitioners have professional ethics that guide the practices of the occupation. Lawyers have professional ethics, and engineers have professional ethics that guide the traditions and practices of the professionals.

Engineering ethics is the subset of professional ethics for engineers, and it is the set of standards that guides the practice of the engineering profession (Hamad et al., 2013). The engineering profession goes beyond only the application of technical knowledge; it involves the application of technical expertise ethically. Additionally, like students of other disciplines, engineering students are taught engineering ethics practically. It allows them to learn how to deal with real-life situations that they would encounter in the profession. However, engineers' lack of safety measures and ethical knowledge can cause havoc in the form of significant accidents, which could affect people's lives and properties. Thus, to avoid possible accidents that could arise from the actions and decisions made by engineers in the profession, engineering ethics knowledge would help to provide engineers with the required guidelines of what to do and make sound ethical decisions (Haghighattalab et al., 2019). Additionally, professional ethics are principles that help professionals make the right decisions; also, engineering students would benefit if they learn ethics and ethical theories of their profession in the classroom (Balakrishnan, 2015).

There are different ethical theories related to engineering ethics, and the most common ethical theories are utilitarianism, deontology, care and virtue ethics (Balakrishnan et al., 2020). Utilitarianism is consequentialism, and its primary focus is to maximize utility; it is expressed in terms of pleasure or happiness. In addition, utilitarianism is about acting according to the rules to achieve the desired outcome, and the focus is on maximizing the overall good (Génova and González, 2016). Moreover, deontology ethics is centered on the requirement to carry out duties regardless of the consequences, and it requires the determination of responsibilities and obligations; it focuses on adhering to the engineering rules (Mouton et al., 2015). Likewise, according to Rabin and Smith (2013), care ethics refer to responding to the need of others in a relationship. It focuses on a set of virtues to be acted on in a way that has been predetermined.

Virtue ethics support action that builds character, and it is based on the idea that someone who has moral virtues would act ethically than someone who only follows the rules, as the case with deontology (Frigo et al., 2021). However, virtue ethics differs from utilitarianism and deontology ethics because it focuses on the effects of the action on the person rather than on the results of the action. In other words, virtue ethics focuses on the person who works rather than the action itself. Furthermore, there are different reasons why virtue ethics is unique and preferable to other engineering ethics (Schmidt, 2014). First, virtue ethics focus on developing attitudes, whereas utilitarianism and deontology are algorithmic and attempt to impose principles that govern every action in engineering. Secondly, virtue ethics promotes positivity, and it advocates the use of professional knowledge to promote good for humans. On the other hand, utilitarianism focuses on avoiding technological disaster, and deontology focuses on avoiding professional misconduct. The teaching of big data ethics in the engineering classroom is very important; however, there are challenges associated with the teaching of big data ethics in the engineering classroom.

2.4 Metacognition and big data ethics

Students need both a clear understanding of data literacy and data ethics, and that both should be taught and reinforced consistently across the curriculum. Teaching them separately leads to a disconnect where students fail to understand the importance of ethical reasoning in data applications. An empirical study found that the majority of those working with data (over 90%) could identify whether actions were ethical or not (Levy et al., 2013). However, multiple other studies have found that this knowledge of ethics often did not translate to ethical action. So, the challenge becomes how to integrate approaches to data literacy with approaches to ethics education that take the complexity of big data ethics into account. A suggested solution centers on the introduction of metacognition.

Metacognition is an essential component of successful learning. Metacognition can be defined as thinking about thinking, reflecting on actions and understanding knowledge (Veenman et al., 2006; Mccord and Matusovich, 2019). Metacognition, therefore, forms a key basis for learning about ethics. There are three types of metacognitive knowledge: knowledge of strategy, knowledge of cognitive tasks and knowledge of self (Bürgler et al., 2021). Knowledge of strategy requires that students compare contexts, subjects, dilemmas, outcomes and implications. Knowledge of cognitive tasks is directly leading to improved decision-making. This supports teaching students not only about ethics but also how to make ethical decisions. Knowledge of the self requires students to think critically and therefore develop a deeper understanding of their own learning.

Developing a deeper understanding of their learning (via metacognitive reflections) sets students up for lifelong learning which is necessary for emerging technologies. It also requires students to critically appraise their decision-making which makes them more flexible and adaptive (Pijanowski, 2009). The constant interrogation of their moral thinking forces students to recognize inconsistencies and develop clearer ethical frameworks (Stokes, 2012). Metacognition also increases students' motivation and engagement (Sperling et al., 2004). Metacognition, therefore, helps students understand ethics as dynamic, adaptable and requiring constant reassessment and action; all of these are critical for ethical decision-making. Ethical decision-making, defined as “the process one goes through in determining the behavior one ultimately engages in when faced with an ethical issue,” has strong connections to cognitive and metacognitive processes (Mcmahon and Good, 2016).

One of the most widely cited ethical decision-making frameworks is the Rest (1986) model, which is a four-stage decision-making process including awareness, judgment, intention and behavior. Given the natural connection between this ethical decision-making model and the process of metacognitive reflection, the latter can be used to complement decision-making through planning, monitoring and assessing one's understanding. Together, ethical decision-making and metacognitive reflection form what Langlois (2005) describes as the TERA (trajectory: ethics, responsibility and authenticity) model. Essentially, the TERA model is a dual conceptual framework based on ethical decision-making and grounded in reflection and action. The TERA training model has been validated by educational administrators and leadership using the ethical dilemmas of professional judgment, policies, professional reputation and human resource management (Langlois and Lapointe, 2010). However, to the best of the authors' knowledge, limited ethical decision-making teaching intervention research has been applied to the engineering classroom.

3.1 Participants

The study was conducted at a research-intensive university in the Midwest United States. Participants included 66 sophomore-level students enrolled in a three-credit course called Supply Chain Management Technology. This is considered a convenience sample as it leverages one of the author's access to a specific engineering classroom instead of randomly choosing an engineering classroom (Thomas, 2012). The course is required for students enrolled in the Industrial Engineering Technology (IET) bachelor's degree program, and also serves as an elective for non-IET majors. The course description includes topics such as supply chain design, supply chain strategy, supply chain process mapping and supply chain decision-making with the use of technology, data analysis and performance metrics. The students were required to participate in the research as part of a class assignment. Therefore, the Institutional Review Board (IRB) protocol was approved as Exempt Level 1 (education research).

3.2 Teaching intervention

The applied research study was qualitative and empirical in that it used written evidence (student metacognitive reflections completed in response to a teaching intervention). The study was exploratory in that it aimed to discover themes or patterns in the data. Participants completed a 5-week teaching intervention including five key learning experiences, as summarized in Figure 1.

Focal pedagogical approaches include photo-journaling (Bosman et al., 2019; Walter et al., 2012), a repetitive mix of formative and summative low-stakes and high-stakes assessments (Den Boer et al., 2021; Liu et al., 2015), scaffold module project (Bosman et al., 2019; Herring, 2006) and metacognitive reflections (Section 3.3). These five key learning experiences meet the concepts (explained in Section 2.4 Metacognition and Big Data Ethics) in the integration of data literacy and data ethics whereby students can understand the importance of ethical reasoning in data applications.

The first four weeks were repetitive, following the same flow for the in-class lecture, in-class group work and out-of-class individual homework. This allowed for a series of low-stakes formative assessment opportunities before completing the module project (e.g. summative assessment). During the in-class lecture, students were introduced to the topic and provided with examples (using photo-journaling and responding to prompts) related to Industry X. Then, for the in-class group work, students worked in groups of 4–5 students. They used Google Docs to respond to photo-journaling prompts related to Industry Y. Finally, for the out-of-class individual homework, students used the learning management system's online discussion platform to respond to photo-journaling prompts related to Industry Z. In all three cases, the photo-journaling prompts required students to both find pictures and write a narrative about the picture.

During the fifth week of the module, students were required to complete a project with three major components. A summary of the project directions is provided below.

1. Component 1 – Summarize Key Takeaways of Module: An overview of the module (Big Data Ethics) is shown in Figure 1. You will need to provide a narrative summary (no pictures) of each week's key learning objective using in-text citations to reference the examples you found on the Internet. If you completed each week's online discussion, this should not require finding any new information; instead, you just need to synthesize and summarize the information you have already found.

2. Component 2 – Summarize Articles on Data Ethics Across the Supply Chain: The following articles can be accessed online. Select 3 of the 6 articles and provide a summary responding to these questions: What is the problem the article is attempting the address? What solution does the article propose?

   • Building digital trust: The role of data ethics in the digital age

   • Facilitating ethical decisions throughout the data supply chain

   • Informed consent and data in motion

   • Ethical algorithms for “sense and respond” systems

   • The ethics of data sharing: A guide to best practices and governance

   • Universal principles of data ethics

3. Component 3 – Connect Component 1 and Component 2: Assume you work as a supply chain director for Company X and you are trying to convince the CEO to invest in solutions to build digital trust. Use component 1 to help justify the problem and use component 2 to explain the proposed solution.

Upon completion of the module (via submission of the module project), students were directed to complete the metacognitive assignment. Details are presented in the next section.

3.3 Data collection instrument – metacognitive reflection

The metacognitive reflection instrument (Table 1) is grounded in the work completed by Ambrose et al. (2010). The first three questions require students to apply the cycle of self-directed learning, whereas the latter two questions focus on student motivation to learn with respect to increasing self-efficacy, seeing value and building a supportive environment.

3.4 Data analysis protocol

This study followed a qualitative approach using thematic analysis. According to Braun and Clarke (2006), a thematic analysis is a foundational qualitative method for discovering patterns within the data. It should be conducted using a step-by-step process. Two of the researchers first become thoroughly familiar with the data to generate initial codes. The NVivo 12 qualitative analysis software was used to code the reflections, whereby each researcher created codes specific to the individual five reflection prompts. Then, the two researchers reviewed and analyzed the codes and documents several times. Upon the completion of coding, themes were generated. As a final step, the two researchers revised the themes and wrote the report. Given the qualitative approach applied to coming up with themes, no interrater reliability analysis was conducted.

Due to the qualitative nature of the research, the goal of the analysis was to explore potential themes within the data. The two researchers debated the strengths and weaknesses of strictly conceptualizing themes without quotes and heavily using quotes to provide readers with evidence. It was decided to merge the two philosophies and meet in the middle. Quotes were drawn from the data to allow readers to make their own judgments on credibility, accuracy, and fairness (Corden and Sainsbury, 2006). To be considered a theme, at least 6 student quotes (of the 66 total) were coded as evidence. For the purpose of this study, three quotes were provided to show evidence of each theme.

4.1 Theme #1 connecting ethics to future engineering careers

Some students had a specific career in mind, while other students had a general idea but were still exploring. For the students who had a specific career in mind, they connected the project to future workforce applications, openly articulating the role of ethics within the specified career. However, for students who expressed a desire for obtaining a more general type of career, they expressed the benefits of the assignment as it relates to general employee best practices (e.g. research, report writing, critical thinking). Yet, an important takeaway is that students highlighted the relevance in either case. Example quotes are provided in Table 2.

4.2 Theme #2 ethics-based knowledge extension

In response to considering how the students might extend their knowledge outside the classroom and/or after the class is over, the primary focus is on digitally accessible resources and reaching out to one's personal network. This focus is likely due to one of two reasons. Given the timeline of the project and its intersection with COVID-19, students were in the process of switching from a face-to-face classroom to an online environment. The first half of the module was conducted in a face-to-face manner, and the second half of the module (including submission of the project and metacognition reflection) was conducted online. Thus, students may have been thinking about knowledge extension given the current limitations of being homebound, so naturally, access to resources would have been limited to digital resources and one's personal network. Yet, the focus on digital resources, specifically, could also be due to the Gen Z nature of student participants, known as “technologically astute Post Millennials” (Schwieger and Ladwig, 2018), who have come to rely more on the Internet and technology for knowledge. Example quotes are provided in Table 3.

Based on speculation that the responses could be linked to COVID-19, it is important to note that only a small quantity of students (n = 3) event mentioned the COVID-19 switch to online learning within the metacognition reflection. This is likely because the course structure had minimal changes pre- and post-COVID-19 transition to 100% online, and the consistent use of the learning management system, repetition and scaffolding (mentioned in the study design) kept students on track with minimal disruptions.

4.3 Theme #3 learning in general (bonus theme)

The first bonus subtheme is an overreliance on content knowledge. Chow (2017) describes three types of knowledge: content knowledge (static information and materials related to the topic), process knowledge (“how to” and procedural understanding) and conditional knowledge (knowing how to approach things differently depending on the situation). Metacognition would be helpful here in helping students move beyond content knowledge. In general, the vast majority of the students noted content knowledge with respect to identifying their strengths and weaknesses. This suggests students may be more concerned with the content topic rather than transferable skills including process and conditional knowledge. Given the course is a 200-level, which targets first- and second-year students, this is not surprising as students are still somewhat transitioning into becoming empowered and independent learners.

The second bonus subtheme is related to time management skills. Development and fostering of time management skills are important in preparing undergraduates to enter the workforce. These skills describe the capacity of students to effectively allocate their out-of-class time to the various in-class courses taken throughout the semester (Ahmad Uzir et al., 2020). This is an essential component of metacognition – self-regulation. Through self-monitoring, students can recognize where they fall short and regulate their time management. In general, about one-third of students acknowledged their challenge with time management, noting that next time they would start earlier and/or manage their time better.

Example quotes for both subthemes are provided in Table 4.

4.4 Summary of key takeaways and lessons learned

5.1 Summary of practical contributions

Upon graduating from university, many engineers will work in new product development and/or technology adoption for continuous improvement and production optimization. These jobs require employees to be cognizant of ethical practices and implications for design. However, little engineering coursework (beyond the mandatory ethics course) accounts for the role played by ethics within the design and innovation process. In response to this challenge, this study makes several practical contributions. First, this study showcased one example of how an ethics module can be integrated into an engineering course on Supply Chain Management Technology, which utilizes evidence-based best practices related to photo-journaling, repetition, scaffolding and metacognitive reflections. Second, this study demonstrated how metacognitive reflections can be used to reinforce student self-awareness of the learning process and connections to the workplace. Finally, this study exhibited how metacognitive reflection assignments can be deployed as a teaching and learning assessment tool, providing an opportunity for the instructor to make changes module-to-module within the same semester and semester-to-semester. Although our focus was on designing and evaluating an ethics module for an engineering course, we are confident that similar modules would likely be successful in a variety of courses both inside and outside of engineering.

5.2 Limitations and future research

Three major limitations of the study potentially exist due to the qualitative approach, focus on one classroom and use of one set of metacognitive reflection prompts. However, all three can be overcome in future research. First, the study was qualitative in nature, which provided a more in-depth and richer exploratory understanding of the phenomena. Future research would benefit from a mixed-methods approach integrating quantitative, explanatory analysis to describe the phenomena with greater certainty. Second, the ethics module was implemented in an Industrial Engineering course on Supply Chain Management Technology. Future research would benefit from access to more extensive and more diverse engineering and technology classrooms. In addition, future research should consider the use of the ethics module and metacognitive reflections for assessing student learning with natural science students, social science students and humanities students, to name a few other disciplines. Finally, the metacognitive reflection prompts offer room for improvement, as recommended within this study. Future research should consider the use of modified prompts for deployment and assessment in future classrooms.

5.3 Concluding thoughts

As Albert Einstein is credited with quoting, “The definition of insanity is doing the same thing over and over again, but expecting different results.” All too often, university students are doing the same thing over and over again (e.g. waiting until the last minute to work on a project, avoiding team confrontation/communication or not leveraging university student services, such as writing or tutoring services) and end up surprised by the lower than anticipated grade. Metacognitive reflections provide one way to overcome this challenge by applying the cycle of self-directed learning (Mccord and Matusovich, 2019; Heyes et al., 2020). From an industry and workforce perspective, metacognitive reflections can be thought of as a “debrief” where project leaders and team members identify what went well, what did not go so well and what could be done differently next time. In this study, the metacognitive reflection required participants to think about their strengths and weaknesses at the beginning (pre-assessment), during (mid-assessment) and end (post-assessment).