STEM challenge: two years of community-engaged engineering

Abstract

Purpose

Community-engaged partnerships have the ability to combine expertise and resources to enhance the local STEM learning ecosystem, by engaging the actors in communities that can enhance students' experiences in science, technology, engineering and mathematics (STEM) education. Texas Tech University (TTU) and Lubbock Independent School District (LISD) have partnered to coordinate an annual STEM Challenge to encourage STEM learning and interest among local middle grade students. Each summer, teams of (three to four) students from ten LISD middle schools participate in a week-long engineering design challenge, facilitated by TTU undergraduate mentors and their teachers, structured by the Engineering Design Process (EDP).

Design/methodology/approach

Quantitative (survey) and qualitative (open-ended responses) data from two years of student glider and hovercraft projects offer insight into how 66 students developed STEM knowledge and leveraged 21^st-century skills to accomplish a shared aim (design challenge).

Findings

Findings suggest growth in students' 21^st-century skills, most among underrepresented (racial, ethnic and gender minority) groups. Data from year one (2018) informed year two (2019) in both programming and the research, including enhanced training for mentors and a deeper exploration of students' experiences during each stage of the EDP during the STEM challenge.

Originality/value

Significant and salient findings are discussed along with recommendations for both programmatic and methodological improvements for year three (2020). This study provides insight into how to structure similar community-engaged partnerships in enhancing the community STEM ecosystem through collaborative STEM experiences for diverse, younger learners.

Citation

Hite, R., Spott, J., Johnson, L. and Sobehrad, L. (2020), "STEM challenge: two years of community-engaged engineering", Journal of Research in Innovative Teaching & Learning, Vol. 13 No. 1, pp. 57-82. https://doi.org/10.1108/JRIT-12-2019-0080

Introduction

Partnerships between local universities and school districts have a common goal in designing and implementing informal science, technology, engineering and mathematics (STEM) experiences for K-12 aged students; the fruits of these partnerships are intended to positively impact students' perceptions of STEM as well as their beliefs about their abilities in STEM (Braund and Reiss, 2006; National Research Council [NRC], 2015; Kong et al., 2013). This alliance is important because, together, such community-based actors can communally contribute to their local STEM Learning Ecosystem, defined as STEM experiences to provide students experiences to gain interest, knowledge and skills in the STEM disciplines sourced from their homes, schools, STEM institutions (museums, universities) and after-school and summer programs (Traphagen and Traill, 2014). Given the common aim of a better STEM education for local K-12 students, the stability and respect of these educational institutions within the Lubbock community fosters opportunities for mutually beneficial partnerships and long-term analyses of collaborative K-12 STEM programming. The outcomes of community-engaged collaborations are important, as per the NRC (2015, p. 29) report on Identifying and supporting productive STEM programs in out-of-school settings, they have explicitly stated that “research is needed to better specify and understand the ways in which learning develops across formal and informal settings, [especially in] leveraging community resources and partnerships.” Evaluations of informal STEM programs are typically positive, yet the cognitive and non-cognitive benefits of community-engaged informal STEM programs are less researched, especially the impact of such programs over time.

This paper relates research findings of a partnership made in a midsize city between a tier one research university (Texas Tech University or TTU) and a school district with approximately 30,000 students in school (Lubbock Independent School District or LISD). This collaborative effort sought to reinforce the STEM learning ecosystem by serving the needs of local middle grade students who have interest in STEM, but are not categorized as exemplary STEM students. To that end, these groups co-constructed an out-of-school summer program to improve middle school-aged students' perceptions of STEM through week-long group activities focused on an engineering design challenge. This event also intended to reinforce K-16 relationships by employing undergraduate mentors and classroom teachers to mentor and assist participating students. From this collaboration, the TTU/LISD Middle School STEM Challenge (herein referred to as STEM Challenge) was established as a yearly event collaboratively hosted by TTU's STEM Center for Outreach, Research and Education (STEM CORE, 2019b) and LISD that brings together ten teams of middle-grade students (from each of the ten LISD middle schools) in a week-long, engineering-design competition held each summer. The purpose of this two-year and ongoing research study was to quantify and qualify students' experiences (e.g. learning, enjoyment and interactions with other students and staff), attitudes towards STEM, and use of 21^st-century skills (i.e. creativity, communication, collaboration and critical thinking) as they engaged in the Engineering Design Process (EDP) during the STEM Challenge.

Background and context

The STEM challenge is a week-long outreach program collaboratively developed by Texas Tech University (TTU) and LISD. The specific TTU partner is STEM CORE (2019a), a faculty-led STEM center whose mission is, in part, to transform (student) lives and communities through strategic outreach and engaged scholarship, specializing in facilitating K-12 STEM outreach. LISD is a mid-size school district located within an urban area that according to the Texas Education Agency (TEA, 2018) serves 27,813 students, 6,000 of whom are in the 6th through 8th (middle) grades. Demographics of LISD mirror the greater Texan majority–minority K-12 student population where 59% (16,358) of students identify as Hispanic, followed by 23% (6,475) as Caucasian and 14% (3,779) as African-American. Further, 65% (8,029) of LISD students are categorized as economically disadvantaged and almost half (48%, 13,426) meet the benchmark of being “at-risk,” exceeding the Texas state average of 59%.

The STEM challenge originally developed out of a desire for TTU's STEM CORE to help facilitate productive relationships with LISD students, teachers and administrators. When STEM CORE approached LISD about a collaboration, LISD expressed a need to target and actively engage students who were middle of the road or average STEM students, yet had the potential to be more engaged STEM students through informal, hands-on learning opportunities. Thus, the novelty of this STEM intervention was to focus on improving middle school students' STEM attitudes, affect and their development of non-cognitive skills related to communication, collaboration, creativity and innovation and critical thinking and problem solving (Battelle for Kids, 2019). This facet of the intervention interested TTU given that cognitive affordances are an important outcome of student participation in informal STEM activities, and there may be additional non-cognitive affordances that result from participation that needs to be captured and quantified by empirical research (Kong et al., 2013). Furthermore, these types of informal STEM experiences are particularly important for students from under-sourced backgrounds like rural (Blanchard et al., 2017) and Latinx, (Hite et al., 2018) which comprise the majority of students in LISD. Therefore, students' experiences and interactions that influenced their attitudes and affect as well as 21^st-century skill growth were to be incorporated into the STEM challenge event and captured through research. Previous research has also found that mentors are significant for students' enjoyment and success in K-12 student–university student STEM activity partnerships (Karp and Maloney, 2013), so they too were incorporated into the STEM challenge and studied.

The STEM challenge has been run annually since 2014 during the first full week after the end of the LISD spring semester. From 2014 to 2016, the program was primarily funded by National Science Foundation (NSF), which provided time for TTU and LISD to create sustainable models for continuous internal funding by splitting costs and resources. Notably, LISD selects one teacher to represent each middle school, provides the venue for the week-long build, and shop experts to assist teams. TTU provides faculty and staff to build the STEM Challenge curriculum, develop assessments, conduct research and provide content experts that student groups may utilize as a resource. LISD and TTU each pay their respective employees for their time, but split the costs of lunch, t-shirts and project supplies. A detailed explanation of the STEM challenge is found in the methodology section followed by the research collection, analysis and discussion.

Literature review

Michael Knoll (1991) identified the origins of the project method in 16th-century Italian architectural schools. In these, artisans learned their craft by creating plans from which actual building of their designed structures could occur. These annual design competitions, sponsored by the schools and wealthy patrons, provided avenues for budding architects to gain knowledge and skills that were highly applicable to their discipline. This fundamental concept of the project method has been consistent over the intervening centuries, where the impetus for students to engage in project-based activities is to replicate authentic work scenarios, developing knowledge and skills that reflect practical experiences within a profession like engineering (National Academy of Engineering, 2004). One of the most notable aspects of engineering learning is the EDP, an iterative way of identifying and solving engineering-based (design) problems and so has become a useful pedagogical strategy for teaching and learning engineering (see NRC, 2009, p. 4). Authentic tasks raise student awareness of the relevancy and meaningfulness of classroom work because their tasks mirror tasks undertaken by professionals outside of the classroom (Nicaise et al., 2000), and also undergirds project-based learning (Laur, 2013). Challenge-based learning (CBL) provides a pedagogical vehicle that is learner-driven and mentor-guided to explore authentic questions (Johnson et al., 2009). Akin to both problem-based and project-based learning (PBL), these pedagogies have evidenced success in developing not only content knowledge (Kokotsaki et al., 2016; Savery, 2006) and in the discipline of engineering (Northwood et al., 2003), but also vital non-cognitive skills termed 21^st-century skills (Battelle for Kids, 2019; Bell, 2010) for engineering learners (Sanger and Ziyatdinova, 2014; Zhou et al., 2012). In the context of this study, researchers utilized the National Education Association's (NEA, 2012) 4C framework of 21^st-century skills, which are: creativity, critical thinking, communication and collaboration. Although the focus of this paper is on engineering, soft skills are greatly needed in the American workplace, where growing numbers of workers identify soft skills as equally critical to job success as technical skills (Pew Research Center, 2016).

Therefore, providing opportunities for K-12 students to apply their STEM learning in challenge-based settings allows them to engage in cognitive, non-cognitive and affective learning (Quinn and Bell, 2013). Affective experiences of students engaged in STEM are too important considering “students who choose to participate in engineering-related activities and coursework may become more interested in pursuing careers in engineering” (NRC, 2009, p. 58), which is critically important for underrepresented groups (URGs, as non-white and/or female). These experiences are not confined to the classroom, rather this “dynamic interaction among individual learners, [represents] the diverse settings where learning occurs, and the community and culture in which they are embedded” (NRC, 2015, p. 5). The NRC goes on to describe that students encounter opportunities to learn about STEM-related fields in a variety of spaces from zoos and museums to structured out of school programs or community events. Though consideration of informal settings are outside the purview of the 2009 NRC report on Engineering in formal K-12 education, the NRC does specifically note that informal programs on engineering suggest “increased students' awareness of and stimulated their interest in engineering” (p. 72), suggesting affective affordances for informal engineering learning. In a study of URGs who persisted in STEM, a salient finding was participation in informal, like summer, STEM programs (Palmer et al., 2011). In lieu of major changes to formal K-12 education to better incorporate engineering knowledge and practices, providing opportunities to interact with students in the other sectors of the STEM learning ecosystem (i.e. outside of the formal classroom) provides educators other avenues to impact students' acquisition of important STEM knowledge and skills as well as build affective relationships with students. To these ends, studies exploring out-of-school STEM enrichment programs have been increasingly valued for capturing the positive impact on students' perceptions of STEM and careers (Roberts et al., 2018). But perhaps most important, community partners who invest time, effort and resources in such projects wish to know the impact of these efforts as this is important in regard to programmatic improvement and sustainability. Therefore, careful and concerted research can not only aid in future programming and longitudinal efforts, but also maintain and strengthen partnerships focused on offering enriching STEM experiences.

Research questions

The following research questions guided the research in both 2018 and 2019:

1. What STEM content did students report they learned or used during STEM challenge?

2. What 21^st-century skills did students use during STEM challenge?

3. How did participating students report teachers and/or undergraduate mentors support them during STEM challenge?

Method

For the past two years (2018 and 2019) of the STEM challenge, student participants were surveyed regarding their experiences (e.g. learning, enjoyment and interactions with other students and adults, being undergraduate mentors and teachers), attitudes towards STEM, and use of 21^st-century skills. Because of the community-partnership that originally established the STEM challenge, the research team adapted data collection to the structure of the established STEM challenge format using recommendations sourced from the NRC (2009) report on enhancing K-12 engineering education to:

1. Emphasize engineering design (p. 151);

2. Promote “engineering ‘habits of mind’ [which] align with what many believe are essential skills for citizens in the twenty-first century” (p. 152); and

3. Encourage students from URGs (i.e. non-white and/or female) in engineering activities (p. 161).

Every three years the STEM Challenge cycles through an EDP to build gliders, hovercrafts, or boats. This rotation ensures any student who participates in either 6th, 7th or 8th grade does not participate in a challenge twice. Though the project product may change, there is a prescriptive structure to the STEM Challenge each year, where teams are tasked to design and build a product within defined parameters, such as a timed race or a farthest launch distance. First, the teams of three to four students from grades 6, 7 and 8 are recruited to by their school-based math or science teacher. This teacher also assists and mentors their team with one TTU undergraduate student who is either a STEM or STEM education major. The STEM Challenge week begins on Monday with content-focused mini-challenges for students to design, diagram solutions and calculate what supplies they will need. On Tuesday, teams are given their supplies and allowed to begin their builds. Teams also complete mini challenges focusing on specific STEM concepts designed to help students to think creatively and improve their designs. Wednesdays and Thursdays are dedicated to testing and refining the builds. On Friday morning, students are given a short amount of time to make last-minute adjustments. The public (LISD school administrators, TTU faculty and administrators, family and the local media) is then invited to view the projects and speak with the teams about their experiences. After the public engagement time, the teams put their projects to the test. The winning team is recognized during a brief award ceremony and lunch following the competition.

Year One (2018) Intervention and Participants. The 2018 STEM Challenge was centered on teams designing, building and testing gliders with a 3 to 6 foot wingspan that would be launched from a provided launcher. Gliders should be prototyped, tested and adjusted to reach the maximum gliding distance, before touching the ground.

In 2018, the ten teachers recruited a total of 40 middle school students, of which 28 fully participated in the research protocol. A little over half (n = 15) were male, mainly identified as non-white (n = 16) and generally represented the lower (6th and 7th) grade levels (n = 16). Figure 1 shows one middle school group's initial plan for a glider design whereas Figure 2 shows the tested prototype.

Year Two (2019) Intervention and Participants. The 2019 STEM Challenge topic was a hovercraft. Teams of students were given three leaf blowers and asked to design, test and drive a hovercraft out of the given materials. An added element included each middle school student as a driver, including a turn at the end of the course.

In 2019, ten teachers recruited a total of 40 middle school students, of which 38 fully participated in the research protocol. A little over half (n = 18) were female, mainly identified as non-white (n = 21) and generally represented the lower (6th and 7th) grade levels (n = 16). Figure 3 shows one middle school group's initial plan for a hovercraft design whereas Figure 4 shows the tested prototype.

Data Collection. Students were surveyed pre-and post-participation in the STEM challenge using the middle/high school version of the Student Attitudes toward STEM (or S-STEM) survey (Friday Institute for Educational Innovation [FI], 2012). The S-STEM survey is a quantitative instrument to gauge changes in middle school students' attitudes, confidence and efficacy in the constructs of Technology and Engineering (nine items) and perceived abilities (growth) in 21^st-century learning (11 items). This 5-point Likert style survey is aligned to the NSF evaluation framework for informal science education (Allen et al., 2008) and has a strong construct validity in assessing participants' attitudes, behaviors and skills (NRC, 2010). Inferences made from scores on constructs and items have been validated across grade levels, genders, races and ethnicities (Unfried et al., 2015).

In year one, participating middle school students completed a paper copy S-STEM survey on the first day of the challenge, before engaging in any STEM activities and again (four days later) upon conclusion of the STEM Challenge, but prior to the public project competition held on the last day. Students also completed an open-ended questionnaire, which queried aspects of their experiences during the STEM Challenge week in regard to what they liked, learned, and how they engaged with their team.

In year two, the middle school students completed the modified S-STEM survey on the first day of the Challenge—again, before the students participated in any STEM-based activities. The students then completed the modified S-STEM survey and a newly developed EDP focused open-ended questionnaire after the conclusion of the STEM programming, but again prior to the public competition held on the final day of the challenge week. Lessons learned from the year one STEM Challenge were that there were 21^st-century skills being utilized throughout the EDP process from the framework used in the STEM Challenge (see North Carolina State University [NCSU], 2019). Therefore, in year two, additional questions were posed to students regarding the nature of their 21^st-century skill growth at each step (i.e. ask, imagine, plan, create and improve) within the EDP process, as compared to asking students to reflect holistically on the entire EDP in year one. Also in year two, additional data was collected on the nature of collaboration with the teacher (along with the TTU undergraduate mentor) as research suggests teacher encouragement can be most influential to students, compared to coming from any other adult including the students' parents (Dubetz and Wilson, 2013). Programmatically, mentor training materials made explicit the importance of encouraging individual students and their teams throughout the EDP process.

Analysis. Disaggregating by subgroups (i.e. gender, race and grade levels), analyses were conducted using t-tests for the two construct-level averages and Wilcoxon signed rank tests for significant differences at the item level (FI, n.d.). Students were also asked open-ended questions regarding what they learned and liked about the STEM challenge and in what ways (if any) they interacted as a group and with their undergraduate TTU mentor during the STEM challenge. Students' experiences were open and double coded according to a framework of increasing higher order skills (i.e. understanding, learning, application and creation) per Bloom's Taxonomy of Higher Level Thinking (Overbaugh and Schultz, 2020). Utterances related to affect and non-cognitive skill use were also included. Students' reported interactions with TTU mentors were open coded according to levels of increasing and robust interactions (i.e. general help compared to passive and active collaboration) and if they were viewed as sources of positive affect or encouragement. Last, participants' interactions within school-based teams were coded according to the National Educator's Association Framework of the 4Cs (NEA, 2012), to garner understanding to the extent which soft skills were employed peer-to-peer within the STEM Challenge. Coding was reviewed by four separate coders and frequency counts were made to infer trends in the qualitative data. Validity was ensured by using field-vetted, validated inventories (FI, 2012) appropriate for the student population (middle grades). Trustworthiness was achieved by using extant understandings of the EDP (NCSU, 2019) and 21^st-century skills (NEA, 2012) for a priori coding and double coding data among multiple reviewers (reaching full agreement). Lastly, chi-square analysis of independence explored significance among categorical analyses of qualitative data and a Wilcoxon signed-ranked test explored basic differences between years one and two of STEM Challenge.

Results

In the following section, results are presented of students' experiences in year one (2018) and year two (2019) of STEM Challenge, in both quantitative and then qualitative analyses.

Year One Data, Quantitative Results. In year one, data was collected from 33 students using the full S-STEM survey inventory that measures students' attitudes on four constructs: mathematics, science, 21^st-century skills and technology and engineering. Data was disaggregated by subgroups of gender (male and female) in Table 1, race and ethnicity (white and non-white) in Table 2, and grade level (6th and 7th grade) in Table 3. Paired t-test analyses (at the construct level) and Wilcoxon signed-ranked tests (at the item level) were run to ascertain significance in any items. The latter analyses were run for all students as the sample size did not allow for subgroup analysis (i.e. gender, race/ethnicity and grade level).

In both parametric and non-parametric analyses, there were no significant differences by gender (see Table 1) among the four constructs of students' attitudes towards STEM and 21^st-century skills.

In Table 2, a paired t-test analysis showed a significance difference between scores for white students in the 21^st-century construct (p = 0.005) compared to non-white students in the 21^st-century construct (p = 0.047).

In both parametric and non-parametric analyses, there were no significant differences by grade level (see Table 3). The S-STEM survey had asked students in regard to taking future math and science courses, at the end of STEM Challenge (post-administration of survey), out of 27 responding students that: 24/27 (89%) said they planned to take future courses in mathematics; 23/27 (85%) said they planned to take future courses in science. This is an increase of from 21/27 (78%) in mathematics and 21/27 (78%) for science from the pre-administration of survey.

Year One Data, Qualitative Results. To better visualize the cognitive and non-cognitive affordances of the STEM Challenge, 33 students provided open-ended responses to: What did you like and learn from STEM Challenge?” (Table 4); “How did you interact with your TTU Undergraduate Mentor During the STEM Challenge?” (Table 5); and “How Did you Interact with your Group During the STEM Challenge?” (Table 6). What students liked and learned coding was informed by Bloom's Taxonomy of Higher Level Thinking, where students could identify their thinking as emergent (understanding, learning) to more complex (application and creation), with other non-cognitive affordances reported in Table 4. The majority of student responses (n = 22, 27%) related to STEM content understanding (n = 26) and learning (n = 13), followed by STEM Application (n = 27) and creation (n = 2). Non-cognitive affordances (n = 16, 19%) related to two concepts of productive struggle and working on team.

Table 5 (interactions with mentor/s) was open coded by increasing levels of interactions (passive to active) as well as evidence of positive affect and providing encouragement. The vast majority of students reported passive interactions (n = 17, 36%), followed by indicating their mentor was generally helpful (n = 13, 28%) to them or their group. Ten codes (21%) pertained to positive affect students reported in their interactions, whereas only six utterances (13%) related to having active engagement with the undergraduate mentor.

Table 6 categorizes students' responses, coded to the 4Cs constructs and sub-constructs (NEA, 2012) in how they interacted with their group during the STEM Challenge. The greatest amount of responses related to collaboration (n = 23, 35%) and creativity (n = 22, 34%), where students remarked working as a team (n = 10, 43% of collaboration construct) and working with others (n = 8, 36% of creativity construct) were the most common group experiences.

Year Two Data, Quantitative Results. In year two (2019), 38 students provided S-STEM survey responses on their attitudes pre and post STEM Challenge. Again, the survey data was disaggregated by gender (Table 7), race and ethnicity (Table 8) and grade level (Table 9). With regard to all students, a paired t-test showed significance in the Engineering and Technology inventory (p < 0.02). A Wilcoxon Signed–Ranked Test showed significance in the Engineering and Technology inventory item that asked, “I am good at building and fixing things” (Z = −2.58, p < 0.01) and “I am curious about how electronics work” (Z = −1.98, p < 0.05) for all students.

A paired t-test was significant for all students in the Engineering and Technology inventory (p < 0.02); the Wilcoxon Signed–Ranked Test showed significance in the Engineering and Technology inventory item that asked, “I am good at building and fixing things” for all students (Z = −2.58, p < 0.01) and there were significant findings for male subgroup (p < 0.003). In addition, the 21^st-century inventory was significant for males (p < 0.01).

The paired t-test for the Engineering and Technology inventory was significant for the non-white student subgroup (p < 0.01) in Table 8. Interestingly, for the 21^st-century learning category, white students (p < 0.04) and non-white students (p < 0.0003) were significant between pre- and post-administration. A Wilcoxon signed-ranks test found in the 21^st-century inventory the following items to be significant for non-white students only: “I am confident I can encourage others to do their best” (W = 0, p < 0.05); “I am confident I can manage my time wisely when working on my own” (Z = −2.34, p < 0.02); “When I have many assignments, I can choose which ones need to be done first” (Z = −2.39, p < 0.02), and “I am confident I can work well with students from different backgrounds” (Z = −2.06, p < 0.04).

In Table 9, the paired t-test was significant in both the Engineering and Technology inventory and the 21^st-century inventory for the 8th grade subgroup (p < 0.03) and (p < 0.04), respectively.

Year Two Data, Qualitative Results. Again, to visualize the cognitive and non-cognitive affordances of the STEM Challenge, 39 students provided open-ended responses to what they had liked and learned from STEM Challenge (Table 10); types of interactions they had with their undergraduate mentor and also teacher (Table 11); followed by the types of group interactions (Table 12). As in year one, what students liked and learned coding was informed by Bloom's Taxonomy of Higher Level Thinking. Table 11 was open coded by increasing levels of interactions, passive and active, as well as evidence of positive affect and providing encouragement. Group interactions (Table 12) were coded a priori of the National Educator's Association Framework of the 4Cs (NEA, 2012), to garner understanding to the extent which soft skills were employed peer-to-peer within the STEM Challenge.

Table 10 shows that the minority of student responses (n = 17, 24%) related to STEM content understanding (n = 10) and learning (n = 13), and the majority for STEM Application (n = 14, 26%) in application (n = 11) and creation (n = 9). Non-cognitive affordances (n = 16, 26%) and affect (n = 21, 30%) were also strongly represented among students' responses.

Table 11 describes students' interactions with mentors, both the undergraduates and teachers. For the undergraduate mentor, most students categorized their interactions as passive (n = 18, 34%), followed by generally helpful (n = 14, 26%). Being seen as a source of encouragement (n = 10, 19%) and an active collaborator (n = 9, 17%) was less. For their teachers, students' described their interactions similarly to that of the undergraduate mentor; mainly passive (n = 24, 49%), followed by helpful (n = 13, 27%), with some active collaboration (n = 8, 16%). Affect was minimal among both groups, and teachers were seen as less of a source of encouragement (N = 4, 8%) as compared to the undergraduate mentor (n = 10, 19%). A summary of mentor interactions is found in the far right column of mentor totals.

To explore group interactions in the 2019 STEM Challenge, Table 12 displays 21^st-century skills employed in groups as described by students. In this year, the research protocol had shifted to record data at each point in the EDP, therefore, tallies are greater than 2018 (see Table 6). In 2018, students were asked this question only at the end of the EDP. The largest frequency of reported skills was in critical thinking (N = 215, 31%), followed by creativity (n = 207, 30%), followed by collaboration (N = 157, 22%) and communication (n = 115, 17%).

Since data was collected by each day (and step) of the EDP in year two, data from Table 12 was disaggregated to each step of the EDP. Each figure below represents students' coded utterances at each step of the EDP, from ask (Figure 5), imagine (Figure 6), plan (Figure 7), create (Figure 8) and improve (Figure 9). Figure 10 shows changes in students' coded responses over time, over the progression of the EDP. Collaboration rose steadily over time, reaching an apex at the improve step. Communication had a similar rise, peaking at planning and improvement, yet dipping sharply in the create step. Creativity and critical thinking were most reported at ask and imagine steps, falling at the planning step. There was moderate increases in both at the last improve step, but did not rise back to early EDP levels.

Year One and Year Two Data Comparison. This last section of the results provides a side-by-side comparison of quantitative and then qualitative results from year one (2018) and year two (2019) of STEM Challenge. Notably, the EDP designs were different each year (gliders and hovercraft), number of students participating increased (from 28 to 38), some are and some are not the same students returning from a previous year and data collection varied slightly between years (collecting teacher data and data on the EDP from each step from students in year two). However, this section provides a comprehensive examination of students' growth throughout the program and how collaboration with community partners informed by research, has facilitated improvements to the program and students' outcomes. Therefore, statistical tests (i.e. sign test) were performed to explore significant differences between years.

To explore differences, among student total S-STEM scores between 2018 and 2019, unpaired t-tests showed only significance in pre-survey scores between 2018 and 2019 (p < 0.03) in the engineering and technology construct, where students held higher initial perceptions of their ability in this domain. Table 13 shows differences in S-STEM survey averages for the attitudinal constructs of technology and engineering as well as 21^st-century learning by gender. An un-paired t-test was significant for males pre-survey responses (p < 0.0005) and female post-survey responses (p < 0.02).

Table 14 shows differences in S-STEM survey averages for the attitudinal constructs of technology and engineering as well as 21^st-century learning by race/ethnicity. An un-paired t-test was significant for non-white student post-survey responses (p < 0.007) only.

Table 15 shows differences in S-STEM survey averages for the attitudinal constructs of technology and engineering as well as 21^st-century learning by grade levels. An un-paired t-test was significant for 6th and 7th grade pre-survey responses (p < 0.02) and 8th grade post-survey responses (p < 0.02).

Table 16 shows differences in the frequency of students' responses for what they had liked and learned during their respective STEM Challenge years. Combined, frequencies show relative stability among the four constructs from year one to two, with the exception of an increase of non-cognitive affordances. A 2 × 4 chi-square analysis of independence indicated main codes were not significantly different between years one and two.

Table 17 shows differences in the frequency of students' responses to how they interacted with their mentors, which were only undergraduates in year one and in year two were for both undergraduates and teachers. Combined, frequencies show improvements in encouragement from years one to two, with a decrease in general affective remarks. A 2 × 4 chi-square analysis of independence indicated a significant result between year one and year two among the four sub-code frequencies, X² (3, N = 99) = 13.5, p < 0.009, suggesting fluctuation in encouragement, affective relationships and active collaboration with mentors.

Table 18 shows differences in the frequency of students' responses to how they interacted as a group during the EDP. Since number of utterances collected in year two were much greater than in year one, percentages were used for comparison and analysis. Since uneven sample sizes are accommodated for chi-square analysis (Greenwood and Nikulin, 1996) a 2 × 4 chi-square analysis of independence was significant among 21^st-century skill construct frequencies, X² (3, N = 759) = 9.3, p < 0.03, suggesting fluctuation in the distribution among the 4Cs as reported between years of STEM challenge.

Discussion

In year one (2018), paired t-tests of S-STEM survey data only showed significance (p < 0.05) in the 21^st-century construct for the white students (p < 0.005) and non-white students (p < 0.05) subgroups, although the Wilcoxon signed-ranked test showed no significance at the item level. Moderate gains in 21^st-century skills and no significant gains in the engineering and technology category warranted deeper qualitative study; such as exploring students' experiences of the 21^st-century (4C skill use) within each step of the EDP. In year one, students took the entire S-STEM instrument, which may have led to survey fatigue, so in year two, modification of the S-STEM survey, focusing only on items related to engineering and 21^st-century skills provided better visualization of student's 21st skills and engineering interests. In the open-ended responses, 82 utterances were coded regarding what they had liked and learned from the STEM challenge. Affective experiences (n = 23, 28%) were found the most, followed by STEM understanding and learning (n = 22, 27%), followed by STEM application and creation (n = 21, 26%), and last, non-cognitive affordances (n = 16, 19%). Upon examining their interactions with their undergraduate TTU mentor during STEM challenge (in 46 pieces of coded data), students indicated passive collaboration (e.g. mentor providing ideas, hints and suggestions) was greatest (n = 17, 36%), followed by them being generally or non-specifically helpful (n = 13, 28%), positive affect (being funny, friendly or kind) (n = 10, 21%), with the fewest interactions coded as active collaboration (n = 5, 11%) (e.g. co-creating understanding, working with the group through the EDP). Only one student (2% of sample) indicated that their TTU mentor provided any form of encouragement. This is significant as the quality of mentoring plays a role in students' success in formal STEM programs, especially for students who are considered URGs by gender (Holmes et al., 2012), ability (Powers et al., 2015) or race and ethnicity (Syed et al., 2012) in STEM. This finding was used to facilitate conversation with the community partner and develop encouragement-focused training for undergraduate mentors and teachers in the following year's STEM competition. Lastly, students reported on their team-based interactions within the STEM challenge; 65 coded data showed that students reported collaboration (n = 23, 35%) was most salient in their experience, followed by creativity (n = 22, 34%), followed by a tie between communication (n = 10, 15%) and critical thinking (n = 10, 15%). These findings suggest students had reduced abilities for communication, despite being an embedded skill to be built in engineering-focused PBL (Ganesh and Schnittka, 2014). However, reduced communication has been a similar finding to related work of middle grade aged students engaging in an EDP-based engineering challenge (Hite and McIntosh, 2020). Therefore, structuring avenues for improved communication between mentors and students as well as facilitating communication among students has been points of improvement between STEM Challenge years. Specifically in year two, the EDP process was specifically discussed with teachers before the week. Students were also led in intentional EDP steps throughout the week. In addition, more students were recruited to participate, ensuring URGs were balanced.

In year two of STEM Challenge, paired t-tests of pre- and post-administrations of the S-STEM survey indicated significance (p < 0.05) for all students in the Engineering and Technology inventory (p < 0.02), as well as the male (p < 0.003), non-white (p < 0.01) and 8th grade subgroups (p < 0.03). In the 21^st-century inventory, only the following subgroups were found to be significant: non-white students (p < 0.0003), white students (p < 0.04), males (p < 0.01) and 8th graders (p < 0.04). The Wilcoxon signed-ranked test was significant in the Engineering and Technology inventory item that asked, “I am good at building and fixing things” for all students (Z = −2.58, p < 0.01) and the non-white student subgroup (Z = −2.13, p < 0.03). The item that asked, “I am curious about how electronics work” was significant for all students (Z = −1.98, p < 0.05). These findings suggest that students had a better self-efficacy in their STEM Challenge when building designs as well as greater interest in engineering principles (from pre-to-post STEM Challenge). This is important as the NRC (2009, p. 10) stated the importance for engineering to “excite the interest of students from a variety of ethnic and cultural backgrounds.”

In the 21^st-century inventory, items that were found to be significant were for only non-white students: “I am confident I can encourage others to do their best” (W = 0, p < 0.05); “I am confident I can manage my time wisely when working on their own” (Z = −2.34, p < 0.02); “When I have many assignments, I can choose which ones need to be done first” (Z = −2.39, p < 0.02), and “I am confident I can work well with students from different backgrounds (Z = −2.06, p < 0.04). This illuminates the 21^st-century skill affordances of the STEM Challenge for participating racial/ethnic URGs, whose cultivation of interest and 21^st-century skill development are paramount for future participation in STEM and STEM careers (Kennedy and Odell, 2014). In year three (2020), researchers plan to take a closer qualitative look, through focus groups, of what specific elements of the STEM Challenge aided skill growth in these item areas. Among the open-ended responses, 70 utterances were coded regarding what they had liked and learned from the STEM Challenge experience. Again, affective experiences (n = 21, 30%) were found to be the most, followed next by non-cognitive affordances (n = 18, 26%), STEM understanding and learning (n = 17, 24%) and STEM application and creation (n = 14, 26%). In year two, interactions with TTU undergraduate mentors (n = 53 coded data) were lower in the affective category (n = 2, 4%) compared to year one, which can be explained through the clearer relationship of the mentor to students through mentor training, students were able to better articulate the interactions they had, rather than replying that they were “fun” or “cool” as seen in year one. Notably, students reported being encouraged 10 times as much than in year one (n = 10, 19%), suggesting mentor roles had evolved from the first year of the STEM Challenge. However, passive collaboration (n = 18, 34%) was still the greatest number of coded responses, followed by being generally or non-specifically helpful (n = 14, 26%), and last by active collaboration (n = 9, 17%). This suggests that more scaffolding, through training, is needed for mentors to support students during the STEM Challenge.

When exploring the data of 4C skill use during each step of the EDP (Figure 10), the drop in critical thinking and creativity during the planning phase may have been due to anecdotal observations by the researchers where teacher-mentors “took over” the planning process from students. This will be addressed in future teacher-mentor training, so it can be mitigated in year three of the STEM Challenge. In 2018, the STEM Challenge research focused on exploratory, broader conceptualizations of students' experiences; which was instrumental in fueling the development of the open-ended questionnaire aligned with the EDP from NCSU (2019) in 2019. In 2020, further analysis of students' experiences in each step of the EDP will explore how modification of the intervention further augments' middle grade students 21^st-century skills while engaged in the EDP. Year three of STEM Challenge will be boat building, so researchers, community-partners and program coordinators will use this data to inform day-by-day activities (which mirror the 5 steps of the EDP) to ensure students employ the full complement of 4C skills throughout the STEM Challenge. Establishing and affirming this empirical, research-based feedback loop not only reinforces relationships by sharing of data with LISD but also directly inform future programmatic efforts.

Conclusion

Informal EDP experiences provide middle grade students, especially those from URGs, enhanced engineering and 21^st-century skills and increased their interest in STEM. By focusing on students' experiences, this study provides a research-based voice to these students, who are often underrepresented in program-based research studies of this nature. By conducting research with an educational partner like a local ISD, university researchers may better understand their immediate STEM learning ecosystem, particularly the types of learning students experience outside the framework of formal education (NRC, 2015). Also, by maintaining transparent communication and reinforcing the mutually beneficial aspects of a collaborative projects, both university and school district are able to develop best practices which can be shared across schools and teachers. Though exploration of occasional programs is good for evaluation and measuring short-term impact, longitudinal research of STEM programs is necessary to refine programs to better serve the future needs of K-12 STEM students, especially those elements that offer insight into how students, especially URGs, come to learn and enjoy STEM. This information can provide other university-school district partnerships a roadmap to foster continued collaboration, collegiate pipelines, and prompt conversations about the purpose and design of community-collaborative programs like the STEM Challenge (NRC, 2015).

Figures

Figure 1

One middle school group's plans for a glider

Figure 2

One middle school group's prototype for a glider

Figure 3

One middle school group's plans for a hovercraft

Figure 4

One middle school group's prototype for a hovercraft

Figure 5

Percentages of 4Cs reported by students in year two at the ask step of the EDP

Figure 6

Percentages of 4Cs reported by students in year two at the imagine step of the EDP

Figure 7

Percentages of 4Cs reported by students in year two at the plan step of the EDP

Figure 8

Percentages of 4Cs reported by students in year two at the create step of the EDP

Figure 9

Percentages of 4Cs reported by students in year two at the improve step of the EDP

Figure 10

Line graph of each step of the EDP during year two of STEM challenge

Table 1

S-STEM survey attitudes by construct averages between survey administrations disaggregated by gender (male and female)

Construct	Number of items	Pre-administration of S-STEM survey average			Post-administration of S-STEM survey average
Attitudes about		Total (N = 28)	Males (N = 15)	Females (N = 13)	Total (N = 28)	Males (N = 15)	Females (N = 13)
Technology and engineering	9	4.09	4.24	3.92	3.98	4.10	3.84
21st-century earning	11	4.20	4.11	4.31	4.22	4.28	4.15

Note(s): Responses based on a 5-point Likert scale, strongly agree (5) to strongly disagree (1)

Table 2

S-STEM survey attitudes by construct averages between survey administrations disaggregated by race/ethnicity (white and non-white)

Construct	Number of items	Pre-administration of S-STEM survey average			Post-administration of S-STEM survey average
Attitudes about		Total (N = 28)	Non-Whitea (N = 16)	Whiteb (N = 12)	Total (N = 28)	Non-Whitea (N = 16)	Whiteb (N = 12)
Technology and engineering	9	4.09	4.06	4.14	3.98	3.97	3.99
21st-century learning*	11	4.20	4.05	4.41	4.22	4.22	4.23

Note(s): Responses based on a 5-point Likert scale, strongly agree (5) to strongly disagree (1)

^aNon-White category includes back (N = 1), Hispanic (N = 12), mixed race (N = 3)

^bWhite category includes mixed race (indicating white as part of their identification)

Table 3

S-STEM survey attitudes by construct averages between survey administrations disaggregated by grade levels (6th, 7th and 8th grade)

Construct	Number of items	Pre-administration of S-STEM survey average			Post-administration of S-STEM survey average
Attitudes about		Total (N = 28)	6th and 7th grades (N = 16)	8th grade (N = 12)	Total (N = 28)	6th and 7th grades (N = 16)	8th grade (N = 12)
Technology and engineering	9	4.09	4.24	3.90	3.98	4.21	3.66
21st-century learning	11	4.20	4.32	4.05	4.22	4.30	4.10

Note(s): Responses based on a 5-point Likert scale, strongly agree (5) to strongly disagree (1)

Table 4

Frequencies of codes and sub-codes* among students' responses (N = 33) to “What did you like and learn from STEM Challenge?”

Main codes	Definitions and emergent sub-codes	Coded utterances (N = 82)	Sub-coded utterances (N = 125)
STEM understanding/learning	Utterance was about fundamental understanding of about or from STEM content or learning (processes) of STEM content	22 (27%)	39 (31%)
	Understanding: aerodynamics, angles, balance, EDP (steps of the process), lift, models, precision, physics, scaling, stem knowledge, weight		26
	Learning: concepts of flight, EDP (learning how to make objects), STS (real world application, historical context)		13
STEM application/creation	Utterance was related to what they learned about STEM in a real world context taking learning and applying STEM knowledge to develop new understandings outside of the activity (metacognitive statements)	21 (26%)	29 (23%)
	Application: aerodynamics (N = 6), relationships among STEM disciplines, STEM integration, EDP (process of building, prototyping, sponsored problems, testing, variable outcomes)		27
	Creation: metacognition		2
Affect	Utterance was related to positive feelings derived from the STEM activity	23 (28%)	39 (31%)
	Challenging (N = 10), different, interesting, stimulating		18
	Free choice, fun (N = 12)		14
	Accomplishment, gratitude, pride, success		4
	Frustrating*, too competitive*		3
Non-cognitive affordances	Utterance was related to not content development, but non-cognitive skill growth	16 (19%)	18 (15%)
	Persistence, productive struggle (n = 8), time management		10
	Making friends, sharing with others, working as a team (N = 5)		8

Note(s): *Denotes utterances of negative affect.

At least 1 sub-code qualified a main code, there were no more than 2 sub-codes per main code. Sample sizes are provided in the Sub-codes, if they comprised the majority of the subcategory

Table 5

Frequencies of codes and sub-codes* among students' responses (N = 33) to “How did you interact with your undergraduate mentor during the STEM challenge?”

Main codes	Definitions of emergent sub-codes	Codes and sub-coded utterances (N = 47)
Encouragement		1 (2%)
Affective relationship		10 (21%)
	Cool	3 (30%)
	Funny	3 (30%)
	Friendly or kind	3 (30%)
	Played games, told jokes	1 (10%)
Generally helpful/supportive		13 (28%)
	Answering questions	2 (15%)
	Helping with basic work related to the build	4 (31%)
	Procuring supplies	0 (0%)
	Non-specific in type of help or support	7 (54%)
		17 (36%)
Passive collaboration with mentor	Giving out ideas, hints, advice, suggestions, or telling students how to conduct the EDP	15 (88%)
	Providing help on demand (e.g. making measurements)	0 (0%)
	Rendering assistance when requested (with supplies)	0 (0%)
	Asking questions guiding the process (no direct involvement)	2 (12%)
Active collaboration with mentor		6 (13%)
	Teaching (not telling) how to engage in the EDP	0 (0%)
	Generating ideas and co-creation of understanding	2 (33%)
	Working with supplies to build, prototype, or troubleshoot EDP	2 (33%)
	Leveraging content specific questions to guide the process	1 (17%)

Note(s): At least 1 sub-code qualified a main code, there were no more than two sub-codes per main code.

*Sub-code percentages reflect the percent total of the code, not all main codes

Table 6

Frequencies of constructs and sub-constructs of students' responses (n = 32) to [overall] “how did you interact with your group during the STEM challenge?”

21st-century skills constructs Definitions of a priori sub-constructs	Coded and sub-coded utterances (N = 65)
Collaboration	23 (35%)
Assume shared responsibility	4 (17%)
Ability to work effectively and respectfully in a team	10 (43%)
Flexibility and willingness to compromise for a common goal	5 (22%)
aDifficulty in collaboration	4 (17%)
Communication	10 (15%)
Articulation of thoughts and ideas in various contexts	3 (30%)
Effective listening	1 (10%)
Communication in diverse environments	0 (0%)
Use of a range of communication (e.g. to inform, instruct, persuade)	1 (10%)
Use of media and technologies	0 (0%)
aDifficulty in communication	2 (20%)
bBuilding or reaching consensus	3 (30%)
Critical thinking	10 (15%)
Solve different kinds of problems in conventional and innovative ways	2 (20%)
Identify and ask significant questions that clarify various points of view and lead to better solutions	1 (10%)
Use various types of reasoning appropriate to the situation	0 (0%)
Use systems thinking (analysis of parts to a whole)	0 (0%)
Effectively analyze and evaluate evidence, arguments, claims and beliefs	0 (0%)
Analyze and evaluate major alternative points of view	3 (30%)
Synthesize and make connections between information and arguments	0 (0%)
Interpret information and draw conclusions based on the best analysis	2 (20%)
Reflect critically on learning experiences and processes	1 (10%)
aDifficulty in critical thinking	0 (0%)
bCritical thinking specific to STEM	1 (10%)
Creativity	22 (34%)
Brainstorming and ideation	1 (5%)
New and worthwhile ideas	1 (5%)
Elaborate refine and evaluate ideas	2 (9%)
Being open, responsive, incorporate group input, feedback into work	6 (27%)
Develop, implement, and communicate new ideas to others	8 (36%)
Demonstrating originality, inventiveness to real world limits	0 (0%)
Viewing failure as opportunity to learn	3 (14%)
Acting on creative ideas to create a novel of useful innovation	1 (5%)
aDifficulty in creativity	0 (0%)

Note(s): There was no response from one participant (N = 32).

*Sub-construct percentages reflect the percent total of the construct, not all constructs

^aTo reflect issues reported in the 4Cs, negative codes were established, coded and counted

^bSub-constructs are emergent, not reflective of the NEA's framework (2012) of the 4Cs

Table 7

S-STEM survey attitudes by construct averages between survey administrations disaggregated by gender (male and female)

Construct	Number of items	Pre-administration of S-STEM survey average			Post-administration of S-STEM survey average
Attitudes about		Total (N = 38)	Males (N = 17)	Females (N = 18)	Total (N = 38)	Males (N = 17)	Females (N = 18)
Technology and engineering	9	3.87	3.78	3.98	4.10	4.10	4.09
21st-century learning	11	4.16	3.94	4.31	4.28	4.24	4.31

Note(s): Responses based on a 5-point Likert scale, strongly agree (5) to strongly disagree (1).

Two students chose to not report gender, which are represented in totals, but in the binary male and female categories

Table 8

S-STEM survey attitudes by construct averages between survey administrations disaggregated by race/ethnicity (white and non-white)

Construct	Number of items	Pre-administration of S-STEM survey average			Post-administration of S-STEM survey average
Attitudes about		Total (N = 38)	Non-whitea (N = 21)	Whiteb (N = 17)	Total (N = 38)	Non-whitea (N = 21)	Whiteb (N = 17)
Technology and engineering	9	3.87	3.86	3.95	4.10	4.16	4.03
21st-century learning	11	4.16	4.10	4.28	4.28	4.48	4.04

Note(s): Responses based on a 5-point Likert scale, strongly agree (5) to strongly disagree (1)

^aNon-white category includes black (N = 4), Hispanic (N = 14), mixed race (N = 3)

^bWhite category includes six mixed-race students (white was part of their identification)

Table 9

S-STEM survey attitudes by construct averages between survey administrations disaggregated by grade levels (6th, 7th and 8th grades)

Construct	Number of items	Pre-administration of S-STEM survey average			Post-administration of S-STEM survey average
Attitudes about		Total (N = 38)	6th and 7th grades (N = 16)	8th grade (N = 12)	Total (N = 38)	6th and 7th grades (N = 16)	8th grade (N = 12)
Technology and engineering	9	3.87	3.95	3.76	4.10	4.16	4.01
21st-century learning	11	4.16	4.23	4.09	4.28	4.31	4.24

Note(s): Responses based on a 5-point Likert scale, strongly agree (5) to strongly disagree (1)

Table 10

Frequencies of codes and sub-codes* among students' responses (N = 39) to “What did you like and learn from STEM Challenge?”

Main codes	Definitions and emergent sub-codes	Coded utterances (N = 70)	Sub-coded utterances (N = 109)
STEM understanding/learning	Utterance was about fundamental understanding of about or from STEM content or learning (processes) of STEM content	17 (24%)	27 (25%)
	Understanding: aeronautics, aerodynamics, engineering, EDP (steps of the process), math, lift, stem knowledge		10
	Learning: engineering, EDP (learning how to make objects), pressure effect		13
STEM application/creation	Utterance was related to what they learned about STEM in a real world context taking learning and applying STEM knowledge to develop new understandings outside of the activity (metacognitive statements)	14 (26%)	19 (17%)
	Application: EDP design, EDP building, EDP prototyping, EDP testing		11
	Creation: agency, decision making, detail orientation, metacognition		9
Affect	Utterance was related to positive feelings derived from the STEM activity	21 (30%)	36 (33%)
	Challenging, competitive, inspiring, interesting, stimulating		12
	Free choice, fun (N = 19)		20
	Accomplishment, gratitude, pride, success		4
Non-cognitive affordances	Utterance was related to not content development, but non-cognitive skill growth	18 (26%)	27 (25%)
	Persistence, productive struggle (N = 12), time management		17
	Making friends, sharing with others, working as a team (N = 8)		10

Note(s): At least 1 sub-code qualified a main code, there were no more than two sub-codes per main code

Sample sizes are provided in the Sub-codes, if they comprised the majority of the subcategory

Table 11

Frequencies of codes and sub-codes* among students' responses (N = 39) to how they interacted with their teacher and undergraduate mentor during the STEM challenge

Main codes	Definitions of emergent sub-codes	Teacher mentor (N = 49)	Undergrad mentor (N = 53)	Mentor totals (N = 102)
Encouragement		4 (8%)	10 (19%)	14 (14%)
Affective relationship		0 (0%)	2 (4%)	2 (2%)
	Cool	0	1	1 (50%)
	Funny	0	0	0
	Friendly or kind	0	1	1 (50%)
	Played games, told jokes	0	0	0
Generally helpful/supportive		13 (27%)	14 (26%)	27 (26%)
	Answering questions	0	0	0
	Helping with basic work related to the build	4	4	8 (30%)
	Procuring supplies	0	0	0
	Non-specific in type of help or support	9	10	19 (70%)
Passive collaboration with mentor		24 (49%)	18 (34%)	42 (41%)
	Giving out ideas, hints, advice, suggestions or telling students how to conduct the EDP	14	1630 (71%)
	Providing help on demand (e.g. making measurements)	0	1	1 (2%)
	Rendering assistance when requested (with supplies)	1	0	1 (2%)
	Asking questions guiding the process (no direct involvement)	9	1	10 (24%)
Active collaboration with mentor		8 (16%)	9 (17%)	17 (17%)
	Teaching (not telling) how to engage in the EDP	3	2	5 (29%)
	Generating ideas and co-creation of understanding	1	2	3 (18%)
	Working with supplies to build, prototype or troubleshoot EDP	3	4	7 (41%)
	Leveraging content specific questions to guide the process	1	1	2 (12%)

Note(s): At least one sub-code qualified as main code, there were no more than two sub-codes per main code.

*Sub-code percentages reflect the percent total of the code, not all main codes

Table 12

Frequency counts of constructs and sub-constructs among students' responses (N = 39) to how participants interacted with their group in each EDP step during STEM challenge

21st-century skills constructs Definitions of a priori sub-constructs	Coded utterances (N = 694)
Collaboration	157 (22%)
Assume shared responsibility	51 (32%)
Ability to work effectively and respectfully in a team	68 (43%)
Flexibility and willingness to compromise for a common goal	38 (24%)
aDifficulty in collaboration	0 (0%)
Communication	115 (17%)
Articulation of thoughts and ideas in various contexts	22 (19%)
Effective listening	10 (9%)
Communication in diverse environments	0 (0%)
Use of a range of communication (e.g. to inform, instruct, persuade)	13 (11%)
Use of media and technologies	5 (4%)
aDifficulty in communication	0 (0%)
bBuilding or reaching consensus	65 (57%)
Critical thinking	215 (31%)
Solve different kinds of problems in conventional and innovative ways	17 (8%)
Identify and ask significant questions that clarify various points of view and lead to better solutions	23 (11%)
Use various types of reasoning appropriate to the situation	22 (10%)
Use systems thinking (analysis of parts to a whole)	26 (12%)
Effectively analyze and evaluate evidence, arguments, claims and beliefs	12 (5%)
Analyze and evaluate major alternative points of view	22 (10%)
Synthesize and make connections between information and arguments	13 (6%)
Interpret information and draw conclusions based on the best analysis	38 (18%)
Reflect critically on learning experiences and processes	14 (6%)
aDifficulty in critical thinking	13 (6%)
bCritical thinking specific to STEM	15 (7%)
Creativity	207 (30%)
Brainstorming and ideation	71 (34%)
New and worthwhile ideas	2 (1%)
Elaborate refine and evaluate ideas	51 (25%)
Being open, responsive, incorporate group input, feedback into work	28 (14%)
Develop, implement and communicate new ideas to others	7 (3%)
Demonstrating originality, inventiveness to real world limits	25 (12%)
Viewing failure as an opportunity to learn	11 (5%)
Acting on creative ideas to create a novel of useful innovation	0 (0%)
aDifficulty in creativity	12 (6%)

Note(s): *Sub-code percentages reflect the percent total of the construct, not all constructs

^aTo reflect issues reported in the 4Cs, negative codes were established, coded and counted

^bCodes are emergent, not reflective of the NEA's framework (2012) of the 4Cs

Table 13

S-STEM survey attitudes by construct averages between survey administrations disaggregated by gender (male and female) for year one and year two

Construct	Number of items	2018 S-STEM survey averages			2019 S-STEM survey averages
Attitudes about		Total (N = 28)	Males (N = 15)	Females (N = 12)	Total (N = 38	Males (N = 17)	Females (N = 18)
Technology and engineering	9	4.09 3.98	4.24 4.10	3.92 3.84	3.87 4.10	3.78 4.10	3.98 4.09
21st-century learning	11	4.20 4.22	4.11 4.28	4.31 4.15	4.16 4.28	3.94 4.24	4.31 4.31

Note(s): Responses based on a 5-point Likert scale, strongly agree (5) to strongly disagree (1)

Two students chose to not report gender, which are represented in totals, but in the binary male and female categories

Table 14

S-STEM survey attitudes by construct averages between survey administrations disaggregated by race/ethnicity (white and non-white) for year one and year two

Construct	Number of items	2018 S-STEM survey averages			2019 S-STEM survey averages
Attitudes about		Total (N = 38)	Non-whitea (N = 21)	Whiteb (N = 17)	Total (N = 38)	Non-whitea (N = 21)	Whiteb (N = 17)
Technology and engineering	9	4.09 3.98	4.06 3.97	4.14 3.99	3.87 4.10	3.86 4.16	3.95 4.03
21st-century learning	11	4.20 4.22	4.05 4.22	4.41 4.23	4.16 4.28	4.10 4.48	4.28 4.04

Note(s): Responses based on a 5-point Likert scale, strongly agree (5) to strongly disagree (1)

^aNon-White category includes black (N = 4), Hispanic (N = 14), mixed race (N = 3)

^bWhite category includes six mixed-race students (white was part of their identification)

Table 15

S-STEM survey attitudes by construct averages between survey administrations disaggregated by grade levels (6th-7th grades and 8th grade) for year one and year two

Construct	Number of items	2018 S-STEM survey averages			2019 S-STEM survey averages
Attitudes about		Total (N = 38)	6th and 7th grades (N = 16)	8th grade (N = 12)	Total (N = 38)	6th and 7th grades (N = 16)	8th grade (N = 12)
Technology and engineering	9	4.09 3.98	4.24 4.21	3.90 3.66	3.87 4.10	3.95 4.16	3.76 4.01
21st-century learning	11	4.20 4.22	4.32 4.30	4.05 4.10	4.16 4.28	4.23 4.31	4.09 4.24

Note(s): Responses based on a 5-point Likert scale, strongly agree (5) to strongly disagree (1)

Table 16

Frequencies of codes and sub-codes* among students' responses (N = 72) to “What did you like and learn from STEM challenge?” from year one and year two of STEM challenge

Codes	Definitions and emergent sub-codes	Year one (N = 82)	Year two (N = 70)
STEM understanding/learning		22 (27%)	17 (24%)
	Understanding	26	10
	Learning	13	13
STEM application/creation		21 (26%)	14 (26%)
	Application	27	11
	Creation	2	9
Affect		23 (28%)	21 (30%)
	Positive affect	36	36
	Negative affect	3	0
Non-cognitive affordances		16 (19%)	18 (26%)
	Formed individually	10	17
	Formed within a group	8	10

Note(s): *Denotes utterances of negative affect.

At least one sub-code qualified a main code, there were no more than two sub-codes per main code

Sample sizes are provided in the Sub-codes, if they comprised the majority of the subcategory

Table 17

Frequencies of codes and sub-codes* among students' responses (N = 72) to how they interacted with their undergraduate mentor (N = 99) during year one and year two

Main codes	Definitions of emergent sub-codes	Year one (N = 46)	Year two (N = 53)	Year one and two (N = 99)
Encouragement		1 (2%)	10 (19%)	11 (11%)
Affective relationship		10 (21%)	2 (4%)	12 (12%)
	Cool	3	1	4 (33%)
	Funny	3	0	3 (25%)
	Friendly or kind	3	1	4 (33%)
	Played games, told jokes	1	0	1 (9%)
Generally helpful/supportive		13 (28%)	14 (26%)	27 (27%)
	Answering questions	2	0	2 (7%)
	Helping with basic work related to the build	4	4	8 (30%)
	Procuring supplies	0	0	0 (0%)
	Non-specific in type of help or support	7	10	17 (63%)
Passive collaboration with mentor		17 (36%)	18 (34%)	35 (36%)
	Giving out ideas, hints, advice, suggestions or telling students how to conduct the EDP	15	16	31 (89%)
	Providing help on demand (e.g. making measurements)	0	1	1 (3%)
	Rendering assistance when requested (with supplies)	0	0	0 (0%)
	Asking questions guiding the process (no direct involvement)	2	1	3 (8%)
Active collaboration with mentor		5 (11%)	9 (17%)	14 (14%)
	Teaching (not telling) how to engage in the EDP	0	2	2 (14%)
	Generating ideas and co-creation of understanding	2	2	4 (29%)
	Working with supplies to build, prototype or troubleshoot EDP	2	4	6 (43%)
	Leveraging content-specific questions to guide the EDP	1	1	2 (14%)

Note(s): At least one sub-code qualified as main code, there were no more than two sub-codes per main code.

*Sub-code percentages reflect the percent total of the code, not all main codes

Table 18

Frequencies of constructs and sub-constructs among students' responses (N = 72) to how participants interacted with their group during the stem challenge from year one and year two with a sign test comparison

21st-century skills constructs abbreviated a priori sub-constructs	Year one (N = 65)	Year two (N = 694)	Sign test
Collaboration	23 (35%)	157 (22%)	+
Shared responsibility	4 (17%)	51 (32%)	+
Work in a team	10 (43%)	68 (43%)	+
Compromise for a common goal	5 (22%)	38 (24%)	+
aDifficulty in collaboration	4 (17%)	0 (0%)	−
Communication	10 (15%)	115 (17%)	+
Articulation of thoughts and ideas	3 (30%)	22 (19%)	−
Effective listening	1 (10%)	10 (9%)	−
Communication in diverse environments	0 (0%)	0 (0%)	NA
Use of a range of communication	1 (10%)	13 (11%)	+
Use of media and technologies	0 (0%)	5 (4%)	+
aDifficulty in communication	2 (20%)	0 (0%)	−
bBuilding or reaching consensus	3 (30%)	65 (57%)	+
Critical thinking	10 (15%)	215 (31%)	+
Solve problems: conventional and innovative ways	2 (20%)	17 (8%)	−
Solve problems: identify and ask significant questions	1 (10%)	23 (11%)	+
Reason effectively: use various types of reasoning	0 (0%)	22 (10%)	+
Use systems thinking	0 (0%)	26 (12%)	+
Make judgement and decisions: claims and evidence	0 (0%)	12 (5%)	+
Make judgement and decisions: alternative P.O.V.	3 (30%)	22 (10%)	−
Make judgement and decisions: synthesize arguments	0 (0%)	13 (6%)	+
Make judgement and decisions: draw conclusions	2 (20%)	38 (18%)	−
Make judgement and decisions: reflect critically	1 (10%)	14 (6%)	−
aDifficulty in critical thinking	0 (0%)	13 (6%)	+
bCritical thinking specific to STEM	1 (10%)	15 (7%)	−
Creativity	22 (34%)	207 (30%)	−
Thinking creatively: brainstorming and ideation	1 (5%)	71 (34%)	+
Thinking creatively: new and worthwhile ideas	1 (5%)	2 (1%)	−
Thinking creatively: elaborate refine and evaluate ideas	2 (9%)	51 (25%)	+
Working creatively: being open and incorporating ideas	6 (27%)	28 (14%)	−
Working creatively: communicate new ideas to others	8 (36%)	7 (3%)	−
Working creatively: understand real world limits	0 (0%)	25 (12%)	+
Working creatively: productive failure and mistakes	3 (14%)	11 (5%)	−
Implement innovation: useful innovation or contribution	1 (5%)	0 (0%)	−
aDifficulty in creativity	0 (0%)	12 (6%)	+

Note(s): ^aTo reflect issues reported in the 4Cs, negative codes were established, coded and counted

^bCodes are emergent, not reflective of the NEA's framework (2012) of the 4Cs