Improving the impact of remote Playing Lean workshops through action inquiry and critical reflexivity

Abstract

Purpose

The purpose of this study is to share how the learning impact of a remote workshop was improved through action research practices, especially action inquiry and critical reflexivity.

Design/methodology/approach

Research design detailed herein consists of one complete cycle of action and reflection. Methods used include: journaling into four territories of experience, free-form journaling, individual and joint reflection using four parts of speech and extended ways of knowing.

Findings

Action inquiry and critical reflexivity have shown themselves to be a potent means of improving the learning outcomes of remote Playing Lean workshops. Drawing on early insight, the author suggests several venues for further inquiry.

Originality/value

This paper contributes a novel combination of action research practices that can be used for improving other learning initiatives as well, and an example of how to question the veracity of qualitative findings.

Citation

Pešec, B. (2023), "Improving the impact of remote Playing Lean workshops through action inquiry and critical reflexivity", International Journal of Lean Six Sigma, Vol. 14 No. 6, pp. 1144-1167. https://doi.org/10.1108/IJLSS-04-2022-0088

1. Introduction

How can I deliver the best possible remote experience, without sacrificing learning outcomes?

In early 2020, when the world was forced to digitise at an unprecedented pace, the above question became a burning issue for the author. Workshops with Playing Lean, a board game for teaching innovation and entrepreneurial skills, were designed for in-person delivery. Everything, from table setup, to participant distance, to physical positioning of the facilitator, was designed to maximise the learning impact. It was as far as possible from “digital-first” design paradigm as one can imagine.

Adjusted, remote-friendly version of Playing Lean was an attempt to replicate what worked well in the offline edition using technology, whilst discarding whatever was inappropriate in the digital setup.

This paper details how the author used action inquiry and critical reflexivity to continuously improve the impact of remote Playing Lean workshops. It introduces the theory behind the game – lean startup methodology, gamification, effective learning strategies and game mechanics – as well as the action research paradigm. The latter enunciates that:

• the research should be done with, instead of on, people; and

• the researcher should embrace being within the researched system.

Research design is coupled with a detailed account of one whole action–reflection cycle. Every point is expanded on and supported by an example from practice. Validity of conclusions is questioned through three specific quality criteria: plurality of knowing, practicality of research outcomes and extent of critical reflexivity.

2. Theoretical background

2.1 Lean startup

Lean Startup is a methodology for developing businesses, products and services. The term was coined by Eric Ries in 2008, after he had experienced several failures in his software company. The method is rooted in the “lean thinking” paradigm, stressing the importance of customer, quality, quick learning and waste elimination. Nowadays, business model design, agile engineering and customer development are fundamental parts of the method.

Core principles of the Lean Startup method are:

• Entrepreneurs are everywhere. In a free society, everybody should have the option to propose ideas, be they simple improvements or radical innovations. Creativity is a human trait that has very little to do with corporate roles. Everybody’s entrepreneurial ideas should be given a fair hearing.

• Entrepreneurship is management. Developing ideas into products, services or businesses is a difficult undertaking that does involve some luck. That is not to say that we should relinquish ourselves purely to chance. Deming, Juran, Ishikawa and their contemporaries revolutionised quality management by introducing the notion that one can improve quality by actively managing it. The same line of thinking applies to entrepreneurship although the tools might be different.

• Validated learning. Failing fast is about learning, not failure. The latter just happens to be a potent ground for the former – as long as the failee has reflective capabilities. Designing falsifiable experiments is key for generating trustworthy data.

• Innovation accounting. This principle is about recognising that the traditional accounting practices and financial measures are a poor fit for the level of uncertainty inherent in innovative entrepreneurial ventures. Each venture will require their own specific set of measures, relevant for their context and ambitions. In the early stages of venture development, learning velocity is as important as traction. Whatever measures end up selected, all should be actionable, addressable and auditable.

• Build–measure–learn loop. Inspired by the PDCA loop. In practice it begins with “Learn”, asking what do we need to learn to proceed? Then figuring out what the appropriate measure(s) of that would be, and then what is the smallest thing that needs to be built. Execution then proceeds as listed – build, measure, learn.

Key concepts introduced by the Lean Startup method are:

• Business model hypothesis testing. Despite scientific-sounding name, this is not the same as traditional hypothesis testing usually seen in sciences. A business model describes how a venture creates, delivers and captures an idea. It usually consists of customer value proposition, resources and processes required to create and deliver it, and a profit formula. Once a business model has been described, the entrepreneurs should identify critical assumptions – those that if proven wrong, would invalidate the whole business model – and test them using minimum viable product tests.

• Minimum viable product. A learning tool for testing assumptions or hypotheses. The smallest, cheapest and fastest thing an entrepreneur can “build” to test a critical assumption or hypothesis. It is not a smaller version of a final product. It has constrained functionality and operations, and addresses only a limited set of customer needs.

• Pivoting. After every experiment, entrepreneurs should ask themselves if they should persevere or pivot. Eric Ries described this as changing strategy while retaining one’s original vision (Eisenmann et al., 2011). In other words, we might change our pathways, but destination should remain the same. It is essential that any change is grounded in validated learning and are not simply whims of the entrepreneur.

Despite its name, the Lean Startup method is used by startups, traditional businesses, governments and educational institutions.

2.2 Gamification

Gamification, a “process of enhancing a service with affordances for gameful experiences in order to support user’s overall value creation” (Huotari and Hamari, 2012, p. 19), can improve the learning outcomes (Hays, 2005; Kapp, 2012; Ke, 2011) while making learning more fun (Jakubowski, 2014; Kapp, 2012). Deif (2017) found that physical games generate higher levels of involvement, especially in collaborative problem-solving, than computer-based games. Ramos et al. (2013) emphasize that using games as teaching tools can be beneficial for: introducing difficult concepts, developing problem-solving and decision-making skills, promoting an active participation of the student, increased interest among students, developing each student’s talents, which welcomes students at different learning levels, and helping the teacher identify each student’s difficulties.

2.3 Dunlosky’s five most effective learning strategies

Dunlosky et al. (2013) compared 10 learning techniques (elaborative interrogation, self-explanation, summarisation, highlighting, the keyword mnemonic, imagery use for text learning, rereading, practice testing, distributed practice and interleaved practice) across six factors (utility, learners, materials, criterion tasks, issues for implementation and educational context).

According to them, the five most effective learning strategies are:

1. Distributed practice. When practice schedule is spread out to increase information retention.

2. Practice testing. When learners are either asked, or on their own initiative, recall information.

3. Elaborative interrogation. Asking the learner to explain why a stated fact or concept is true.

4. Self-explanation. Asking the learner to explain how new information relates to their existing knowledge; or to detail their problem-solving process.

5. Interleaved practice. When practice schedule includes a number of different types of materials and problems in a single learning session.

2.4 Playing lean

Playing Lean is a turn based board game for teaching the Lean Startup methodology. It is intended to be played by 4–16 players, divided into four teams, each taking a role of a fictional start-up. They all compete in the same industry (social media, ridesharing or hospitality), with the winner being the first team to reach the “early majority”. Market is divided into three types of customers [inspired by Moore’s (2014) technology adoption curve]: innovators (“techies”), early adopters (“visionaries”) and early majority (“pragmatists”).

Each customer tile on the game board has specific requirements, which need to be met to sell to them successfully. It is possible to over-deliver, but under-delivering will result in a failed sale and a very disappointed customer. Players can learn what the customer wants by conducting experiments, market research and customer development. Whenever they do so, the facilitator pulls an experiment card at random (see example in Figure 1), explains the concept presented on the card and narrates the outcome.

The game turns are divided into two phases:

• (1)

  Planning phase. Every team has a number of employees, which can be assigned one of the following tasks:

  • develop or remove product feature;

  • conduct market research and customer development;

  • attempt to sell the product to customer; or

  • attend training, if available.

For the sake of simplicity, all employees can do all tasks. Players usually have limited time to decide how to assign their people.

• (2)

  Execution phase. Once every team has assigned tasks to their employees, the game facilitator goes in clock-wise order, and queries them “What would you like to do first?” While product development is always successful, the same is not true for experiments – which can range from learning nothing to identifying four different customer’s needs – and sales – which often result in failure when players go in “blindly”, i.e. attempt to sell without knowing what the customer wants.

The game incorporates a number of gamification elements, including: storytelling, social learning, motivation and reward structures, competition and use of facilitator (game master). Playing Lean workshops deploy four, out of five, Dunlosky et al.’s most effective learning techniques:

1. Practice testing. During and after the game, facilitator will probe the players to recall some of the introduced concepts. For example, if there was an experiment on “functional jobs” that was explained in turn 2, the facilitator might ask one player to explain the concept again in turn 9. Alternatively, players often bring up previous concepts to each other when planning their turn.

2. Elaborative interrogation. Once the facilitator has finished narrating the example from the experiment card, they can ask the players “Why do you think this was the outcome?”

3. Self-explanation. Facilitators will often ask” how does this manifest in your business/context”, especially during post-game reflection. Although the players will start making the connections during the game, they are mostly preoccupied with winning, instead of learning. The best time for self-explanation is once the game has been concluded, and passions have died down – a bit. Players are asked to reflect on two distinct things:

   • how did they make decisions during the game, and how is that similar or dissimilar to how they make the same decisions in their business; and

   • how do various concepts encountered during the game (e.g. different experiments, innovation accounting, pivoting, etc.) manifest in their business.

4. Interleaved practice. The game itself includes a number of different situations; and each game is different, because every player has their own play style. Learning is coming from the experiment cards, facilitators and other players. There is learning in player’s own actions, observing others actions and then reflecting at the end.

Playing Lean depends on a facilitator, so-called game master, to act as an impartial arbiter, storyteller and teacher. Although experiment cards have a story on them (see example in Figure 1), facilitators are encouraged to share their own, and explicitly connect it to the player’s experience. Post-game retrospection is important for improving the learning outcomes. Facilitators are encouraged to break up the players into duos, and ask them to discuss why did their respective team win or lose, and what was their overall strategy. By doing so, the facilitators aid the learners in positioning their new experience relative to their overall lived experience.

Due to the random nature of experiment cards and player behaviour, no two workshops are the same.

3. Research design

Action research is a participatory paradigm where research is done with, and not on, people. It is grounded in participatory world-view (Reason and Bradbury, 2001b), with a subjective–objective ontology (Coghlan and Brannick, 2005; Denzin and Lincoln, 2000; Heron and Reason, 1997) and extended epistemology (Heron and Reason, 1997, 2008).

The locus is on practical issues, knowledge-in-action, human flourishing, participation and democracy, all within an emergent developmental form. Action inquiry, a set of research practices under the action research umbrella, is “inspired by the primitive sense that all our actions, including those we are most certain about and are most committed to, are in fact also inquiries” (Torbert, 2001, p. 250).

If the researcher does not participate in the experience itself, nor do they involve the participants as co-researchers to some extent; if they do not take action, reflect on it and reflect on the reflection itself; if they do not pay attention to the quality of inquiry and its impact on individuals and the broader community, then they will have a weak claim to action research.

3.1 Action research cycle design

Action research is often conducted in a spiralling action–reflection cycling pattern. Three steps proposed by Lewin (1946) are: plan action, take action and reflect on the results of action. These steps are preceded by a research question, and succeeded with a modification or adjustment for further action research. Figure 2 depicts these steps.

The research question for this project was:

RQ1.

How can I deliver the best possible remote experience, without sacrificing learning outcomes?

There were three planned activities for the action phase:

1. Before workshop. Imagine situations that might significantly impede or contribute to the learning outcomes of the workshop.

2. Workshop. Conduct the Playing Lean workshop remotely.

3. After workshop. Record situations that might have significantly impeded or contributed to the learning outcomes of the workshop.

Furthermore, three reflection activities were planned as well:

1. Individual reflection. Individual sense-making and reflection on the differences between what was expected and what actually had happened.

2. Joint reflection. Joint sense-making and reflection on the differences between what was expected and what actually had happened.

3. Closing. Formulate and record improvement ideas for the workshop design and delivery.

Each activity was matched with a specific method, as outlined in the following section. Table 1 and Figure 3 summarise and illustrate the activities, methods and order in which they were conducted.

3.2 Methods

This section briefly introduces the following methods:

• four territories of experience, an action inquiry practice;

• free-form journaling;

• four parts of speech, another action inquiry practice; and

• one-page summary.

3.2.1 Four territories of experience.

Torbert’s and Cook-Greuter’s (2004) four territories of experience are:

1. 1st territory: the world and events outside of me.

2. 2nd territory: my own sensed performance, what I am doing.

3. 3rd territory: my thoughts, emotions and action-logics.

4. 4th territory: my bare attention or post-cognitive consciousness.

More detailed descriptions of each territory are presented in Table 2. The more we focus our attention and awareness on these four territories, the more we can observe and react to, hence increasing the range of our insights and options to act upon. 4th territory is the most difficult one, because thinking about it immediately takes us back to the 3rd territory. The researcher has prepared a structured journaling template which facilitates capturing all four territories on a single sheet of paper.

3.2.2 Free-form journaling.

Journaling involves capturing personal experiences, observations and thoughts. Free-form implies there is no specific structure or practice to follow. The researcher simply sits down and writes into the matter at hand.

3.2.3 Four parts of speech.

Another method by Torbert and Cook-Greuter (2004) is complimentary to their four territories of experience. Authors argue that “speaking is the most influential medium of action in the human universe” (2004, p. 27). Furthermore, they break down speech into four parts:

1. Framing. It is rare that everybody shares the same understanding or view of the situation, hence it is useful to be explicit in such matters. Examples of that would be explaining the context of a present situation, being explicit regarding issues being faced, ambitions to be achieved and underlying assumptions and biases.

2. Advocating. Explicitly stating a course of action, feelings, thoughts and preferences are all forms of advocacy. “We must enter new markets to grow” and “I don’t like that idea” are two examples of advocacy.

3. Illustrating. Storytelling can be used to flesh out other parts of speech, providing further context and motivation. Building on a previous example might look like “We have been market leaders in our industry for a decade, and it seems like our profit margins have plateaued. We must enter new markets to grow”.

4. Inquiring. Asking with genuine intent to learn about or clarify an issue. Leading questions, rhetorical questions and automatic social questions do not constitute inquiring. Torbert and Taylor (2008) caution us that raw inquiry can make others defensive, hence one should provide proper framing and illustrating prior to inquiring.

3.2.4 One-page summary.

It is relatively easy to come up with ideas during the research project; the challenge is capturing their context and important specifics. Failure to do so is a form of waste which Ward and Sobek (2014) call discarded knowledge. They propose using “knowledge brief”, a single sheet of paper on which all information relevant to the issue at hand is recorded. Toyota Problem Solving A3 (Sobek and Smalley, 2008) and Design for Six Sigma one-page summary (Gremyr and Fouquet, 2012) would be good examples.

For this research project, the author has developed a one-page template with following constraints:

• one idea per page;

• brief description of the observed phenomena, including outcomes; and

• brief description of proposed intervention of change, including expected outcomes.

3.3 Data collection

Journal entries were the primary data type collected during this research. They were written and documented by two co-researchers, of whom one is the author of this paper. Six students, and their lecturer, participated in the remote workshop delivered by the author. Subsequent section provides more details about the participants and their context.

3.4 Research quality and rigour

Reason and Bradbury (2001b) invite action researchers to evaluate their work across five interconnected question sets:

1. questions of emergence and enduring consequence;

2. questions of outcome and practice;

3. questions about plural ways of knowing;

4. questions of relational practice; and

5. questions about significance.

The quality of this research project will be evaluated by asking the following:

• To which extent was plurality of knowing addressed?

• To which extent were outcomes practical?

• To which extent was critical reflexivity practised?

3.4.1 Extended epistemology.

Action researchers acknowledge that there are multiple ways of knowing, saliently captured in Heron and Reason’s extended epistemology. They argue for four ways of knowing (Heron and Reason, 1997, 2008):

1. experiential: knowing through first-hand experience;

2. presentational – knowing through showing;

3. propositional – intellectual knowing of ideas and theories; and

4. practical – knowing how to do something.

Experiential knowing is not the same as a positivist understanding of worldly things. Because experiencing includes participation, which in turn leads to both shaping and encountering, authors propose that experiential reality is “always subjective–objective, relative both to the knower and to what is known” (Heron and Reason, 2008, p. 368). Other forms of knowing are grounded in experiential knowing. Although it informs both of them, experiential knowledge cannot be reduced to solely propositional or presentational knowledge.

In extended epistemology, level of congruence between experiential, presentational, propositional and practical ways of knowing is a measure of validity. Heron and Reason (2008, p. 378) explain how each way of knowing contributes to the overall quality of knowing:

• quality in experiential knowing is based on the receptiveness and willingness to engage with new experiences in the world we are in;

• quality in presentational knowing comes from expressing the experiential knowing in a multitude of ways, broadening and deepening the inquiry;

• quality in propositional knowing is reflected through the fidelity of translating presentational forms to textual ones, critical engagement with existing theories and attentiveness to issues of power; and

• quality in practical knowing is observed through the ability of “individuals, organizations and communities” to enact desirable change, supported by sufficient evidence of doing so.

3.4.2 Practical outcomes.

Action research is deeply concerned with practical outcomes, posing questions such as “is this work useful” and “is it helpful” (Reason and Bradbury, 2001a, p. 448) Consequently, then, the degree to which the research participants can leverage newly created knowledge and insight to improve their own practice becomes the measure of validity. What constitutes practical, and how does the measurement scale look like, is up to each individual action researcher to establish and argue for.

3.4.3 Critical reflexivity.

Reflexivity, a rather important aspect of any good action research, seems to be mystified in a myriad of ways. Lincoln and Guba (2000, p. 183) describe reflexivity as:

The process of reflecting critically on the self as … both inquirer and respondent, as teacher and learner, as the one coming to know the self within the processes of research itself.

They point out that is both good and bad, as it might make way for more “dynamic, problematic, open-ended, and complex forms of writing and representation” (Lincoln and Guba, 2000, p. 184).

Grey and Sinclair (2006, p. 447) offer a cautionary description of how reflexivity is often misused as a:

Masquerade for transparency, a self-flagellating defence against criticism or simply as a chance to flex their theoretical muscle — rather than offering a reasonably lucid and decently honest statement of authorial position.

Czarniawska’s (2016) argues that reflexivity should supplant scientific rigour, a position the author of this paper finds as pointless as arguing about creative versus analytical mindset, something Richardson (2000) labelled as a “dinosaurian belief”.

What do we end up with if we attempt to deconstruct reflexivity? There is an act of observation, followed by analysis and then interpretation. What makes it different from reflection is the object (or focus) of these activities. In the action–reflection action research cycle, reflection is focused on the action itself (processes, outcomes, etc.). In reflexivity, the focus of reflection is on that very process of reflection in the action–reflection cycle, or, as Lather (1993, p. 675) eloquently wrote, noticing what “frames our seeing”.

Putting it all back together, one can claim that reflexivity is reflection on reflection-in-action (Schön, 1983, 1987). When done well, it enhances the quality of research (Reason and Bradbury, 2001a), supports its validity (Czarniawska, 2016) and humanises the researcher (Cunliffe, 2018). And, as Doane (2003) and Burden and Steghöfer (2019) have experienced, there is great value in approaching reflexivity as a shared, rather than solitary, activity.

The author is not making any claims that there is a firm and obvious boundary between being reflective and reflexive, nor that one guarantees the other. The claim is that the distinction is useful – at least to the author – in understanding how they are – in thought, action and emotion – as an action researcher.

3.5 Research site

Because the research was concerned with remote workshops, all participants conducted their respective activities from their home.

4. An in-depth example of one action research cycle

This section presents one complete action research cycle, using the previously described design.

4.1 Background

Playing Lean board game was designed to be used in a workshop setting, with all players being close to each other. Due to exogenous circumstances, such workshops were not possible from 2020 to 2022. Pešec (2021) explained six specific issues which arose when conducting Playing Lean workshops in a remote setting. The author also proposed a set of countermeasures for each issue. Both are summarised in Table 3.

Remote Playing Lean workshop that was the focus of this specific research was conducted at a Nordic university, for a mixed group of students (master’s and bachelor’s) with predominantly technical background. The students were unfamiliar with the Lean Startup methodology. Their lecturer was present during the workshop. The workshop was conducted by the author of this paper (hereinafter: Facilitator), who was assisted by one person from the Playing Lean team (hereinafter: Assistant).

4.2 Journaling into the imagined workshop situations

Before the workshop Facilitator and Assistant individually answered the following prompt: Imagine situations that might significantly impede or contribute to the learning outcomes of the workshop. They did so using the four territories of experience as follows:

1. 1st territory was used to describe the situation.

2. 2nd territory was then used to describe how is the individual responding.

3. 3rd territory was then used to describe what was the individual sensing and feeling.

4. 4th territory was then used to capture raw intention.

All in all, they described 13 different situations. Four excerpts, two from the Facilitator and two from the Assistant, are presented in Tables 4 and 5.

4.3 Journaling into the actual workshop situations

After the workshop had been conducted, Facilitator and Assistant journaled into situations that might have significantly impeded or contributed to the learning outcomes of the workshop, using the four territories of experience as described in previous section. They captured 20 situations, of which five are presented in Tables 6 and 7.

4.4 Individual reflections

During this step Facilitator and Assistant reflected on their “before” and “after” journal notes, paying attention to emergent patterns, incongruences and deviations from the expected Playing Lean workshop norms. Both reflected a day after the workshop, although the Assistant also took notes during the workshop itself. Each captured their reflections in their personal journal.

Facilitator’s reflections were focused on the adjustments to the workshop process, as illustrated by the following excerpt:

The students were quick to learn the rules, so I adjusted the customer tiles to be more demanding on the fly. I did the same with experiment cards. That created several memorable teaching moments, which all came to life during post-game reflections. Since the students were so fast, I decided to have two rounds of guided reflections, and split the students into duos.

Unsurprisingly, a bulk of their reflection is directed at questioning the veracity of learning outcomes:

[…] students failed to differentiate between the game (Playing Lean) and method (Lean Startup). My judgement is based on their responses to my invitation to define the lean startup method in their own words. I asked that at the beginning and end of the workshop, and in both cases the students were first responding about the game.

That left me puzzled, especially since I’ve explained the difference a couple of times. Furthermore, they showed good understanding of various lean startup concepts during the game and reflective conversations. So how could they get the concepts right, without comprehending the difference between the method and the game? And on what basis then do I say if this was a good or bad workshop? […]

Given the students had zero previous knowledge of the lean startup method, and could speak about build-measure-learn, experimentation, innovation accounting, and pivoting in their own words by the end of the workshop, I judge that it has delivered on the educational dimension as well. Had I not had two reflective rounds, I would be much more hesitant to draw the same conclusion.

During the workshop Facilitator usually has to manage multiple browser and application windows (e.g. board, chat, team dashboards and video call), which makes it difficult to see the participants. Assistant, on the other hand, was free to observe both the students and the Facilitator. Hence, the Assistant’s notes have more detail on students’ actions:

[…] all players were in the same room and team members were close to each other, they interacted between themselves outside the game planning phase and were not fully focused on the lessons the facilitator was teaching. For a facilitator it is hard to control these situations or even notice them in an online setting.

[…] some students had to get up in the middle of the game and plug in their laptops because their battery was running out. When you add this interruption to their speaking amongst themselves during lessons, it causes them to miss out on some lessons.

Both had notes on how this workshop turned out to be a hybrid one, instead of fully remote as expected.

4.5 Joint reflections

Joint reflection was conducted as an unstructured, open-ended dialogue between the Facilitator and Assistant. They both read all journal entries, as well as individual reflections, prior to the meeting. In fact, all the previous steps have created proper framing and generated a number of illustrations, useful for joint reflection. As described in the methods section, jumping straight into inquiring or advocating can be counterproductive, especially in a low-trust or low-context environments.

In this case both Facilitator and Assistant know each other, and have delivered a number of workshops together. Still, they both appreciated rich data generated during the action phases. Their first move was to open by inquiring into each other’s experience, asking if there was anything that was left out of the journals. That was followed by inquiring into similarities between entries and experiences. Both focused on the learning outcomes and their respective tasks (teaching for Facilitator, technical support for Assistant).

Questioning the differences came as the natural next step. Assistant observed how different the tone was between the journal entries, labelling Facilitator’s notes as “less empathetic” and in “stern, authoritative, and borderline autocratic voice”. Furthermore, the Facilitator was more focused on progressing the game, workshop and teaching, whilst the Assistant paid much more attention to the students’ actual physical behaviour. One could argue that was somewhat expected due to the division of labour, but both found it relevant enough to flag it.

Final topic of inquiry was related to issues and learning outcomes. Both found technical issues like echo, cacophony, breakout room adjustment, and similar, manageable and non-disruptive. Playing Lean workshop is intended to serve as an introduction to the Lean Startup methodology, hence the expectations are that participants are able to talk about key concepts in their own terms. Ultimately, based on all the collected data and their personal experience, Facilitator and Assistant concluded that the learning goals had been met. All students were able to provide coherent descriptions of various concepts.

Advocacy and inquiry intermingled when discussing potential improvements and changes for future remote workshops. Questions like” “what was that really about”, “how might that look like” and “does it make sense” were fielded quite often.

4.6 Documenting improvement ideas

At this moment a reminder might be in place – improvement ideas captured in this stage are supposed to build on all the previous ones. That is why they are deliberately kept small, almost atomic and brief. Although research design lists one-page summary as the method, the truth is that each improvement idea fit on a single index card. They were:

• Check with the workshop organiser where from will each participant join.

• Include information about headset in the initial information package.

• Include reminder about power outlets in the initial information package.

• In case of hybrid workshop: ask players to limit the communication between themselves to the planning phase.

• In case of hybrid workshop: inform players of their teams in advance.

• Include an explicit statement that Playing Lean is the game (tool) and Lean Startup is the method in the initial information package.

• Explicitly state that Playing Lean is the game (tool) and Lean Startup is the method at the beginning of the workshop.

• Increase the number of in-game and workshop reflections from one to at least two.

• Create a dedicated area on the digital whiteboard where participants can capture their reflections.

5. Reflection and discussion

This section provides a critical look at the process of, and the conclusions derived from, the presented action research cycle.

5.1 Evaluating plural ways of knowing

All four ways of knowing – experiential, presentational, propositional and practical – were heavily featured in this action research cycle. Experiential knowing was prominently featured in all phases, from leveraging past lived experience, to living through the new experience. The four territories of experience framework has proven to be useful in capturing rich notes that include observed actions, thoughts, responses, emotions, sensations and intentions.

Presentational knowing was perhaps least deliberate. It manifested mostly through use of various visual methods and tools, and storytelling, to some extent. Propositional knowing was suppressed during most of the phases, and was most accentuated when formulating and documenting improvement ideas. That was intentional, because the researcher was aware of their bias for this way of knowing. They made the decision to create space for other ways of knowing to come to the fore.

Practical knowing, like experiential, was quite prominent in all phases of this action research cycle. It is important to remind ourselves that practice here stands for the researcher’s practices like imagining potential situations and scenarios, being attentive to different experiences, critical reflexivity and using self as an improvement instrument.

Tables 8 and 9 map each way of knowing to each respective phase of the conducted action research cycle. Although more space could have been given to different forms of presentational knowing, the researcher confidently concludes that this cycle satisfied the requirements of plural ways of knowing.

5.2 Evaluating practical outcomes

Action researchers accept there are at least three territories of research: first-person (“me”), second-person (“us”) and third-person (“them”). The questions of practical outcomes in this research pertain to improving the impact of remote Playing Lean workshops through action inquiry and critical reflexivity. Second-order outcomes would consider the workshop participants, especially in regard to their understanding of the Lean Startup methodology.

Prior to evaluating the outcomes, we have to establish participants’ territories. Both Facilitator and Assistant engaged in first-person research, followed by second-person co-inquiry. There were also elements of extended first-person research (first-person with others), but that was mostly accidental. Third-person research will be only possible after the publication of this paper, when other facilitators, educators, coaches and training professionals, attempt to use methods, techniques and practices described herein.

All improvement ideas proposed earlier in the paper were implemented in subsequent Playing Lean workshops conducted by the Facilitator and Assistant. Between the publication of this research and concluding the first action research cycle, the author has not had any recurrences of participants confusing the game for the method, nor surprise hybrid workshops. Increasing the number of reflective sessions during the workshop, coupled with providing an easy way for participants to document their conclusions, has resulted in several positive outcomes:

• Participants find it easier to recollect their experience and learnings.

• Participants find it easier to aggregate and disseminate their experience and learnings.

• Participants feel more ownership of their experience and learnings.

Based on above, the researcher concludes that the short-term practical outcomes were worth it. Evaluating the long-term consequences requires more time, practice and reach.

5.3 Evaluating critical reflexivity

As elaborated before, there are many takes on what is critical reflexivity and how does it look like. The author considers it to be a reflection on reflection-in-action. To phrase it differently, reflection is about differences between an action and an outcome, while reflexion is about differences, commonalities and patterns between (and within) the reflections. See Figure 4 for the visual representation.

Research design included ample opportunities for both:

• Four territories journals included brief reflections on situations.

• Individual journals built on that, deepening the reflective process.

• Joint discussion using the four parts of speech mixed reflection and reflexion.

Perhaps the most valuable contribution to the quality of above processes was reflecting in pairs. Doane (2003) shares how she tried to create space for reflexivity in her research project by having one researcher dedicated to actively observe, document and interpret what is happening in the moment, whilst the other researcher (her) focused on being fully present, entering a consciousness-without-content mode. That way, she argues, one of the researchers is able to create relational knowing “centred in experience rather than language, cognition, and discourse” that “expands and heightens the depth of understanding and knowledge we are gleaning through our research process” (2003, pp. 100–101).

Similar effect was achieved in this research, with the Facilitator being fully immersed in the workshop experience, whilst the Assistant was free to observe, document and interpret in real time. With that being said, reflexivity does benefit from additional time and distance (Czarniawska, 2016), which was not necessarily present in abundance during this research.

The researcher concludes that because critical reflexivity was practised at multiple levels – individual, joint, between actions and outcomes and between reflections – it satisfies the quality criterion set forth for this research. The author acknowledges the infinitude of the continuous improvement process, as well as the reflexive potential in revisiting the learning from this action research cycle in the future.

5.4 Closing the loop

The author highlights following theoretical contributions.

5.4.1 Gamification, learning strategies and lean games.

Storytelling, social learning, motivation and reward structures, competition and use of facilitator (game master) – gamification techniques used during Playing Lean workshops – have shown to engage students in a remote setting as well. Research by Deif (2017) indicates that computer-based games generally underperform physical games when it comes to engagement and learning outcomes.

Although Playing Lean is a physical board game, remote workshops are conducted with a digital replica and students play using their personal devices. Therefore, it would be fair to expect the remote version to underperform the in-person version. Because that has not happened, the author suggests that the matter might not be computer versus non-computer based games, but rather the degree and appropriateness of gamification techniques used in the game itself. For example, while folding planes is a valid physical simulation, what makes it a game?

In other words, adopting a gamification lens whilst evaluating the learning effectiveness of lean games might uncover new improvement opportunities.

5.4.2 Action inquiry and critical reflexivity.

This paper presents a novel protocol for journaling using the four territories of experience, an action inquiry method and a structured table (as shown in Tables 4–7). Each column represents a specific territory, while each bar one particular situation. Descriptions in the table header help with quick categorisation of notes.

The author has experienced that aforementioned design enables co-researchers untrained in action research to participate with minimal training. For example, the Assistant quickly grasped how to fill in the journal template, and found it helpful to have descriptions they can reference. This approach is complimentary to the methods described by Torbert and Cook-Greuter (2004), the originators of action inquiry.

Reflection protocol consisting of: individual sense-making and reflection; free-form journaling; joint sense-making and reflection using four parts of speech; and joint formulation and documentation of further action is another contribution. While none of the individual elements are novel, the combination, sequence and execution are.

Such layered approach has been devised in an attempt to attain genuine reflexivity by cycling through individual and joint reflection, whilst documenting both the process and outcome. As Czarniawska’s (2016, p. 618) wrote, the “wider public needs to see the results; it does not need to watch the painful process”. The protocol shared herein captures both for one simple reason – one will find it nearly impossible to reflect on data that has been lost. And given that additional reflection cycles can lead to new insight, it is only prudent to ensure data-capture is part of the regular protocol.

6. Conclusion

“How can I deliver the best possible remote experience, without sacrificing learning outcomes?” might have been the opening question, but everything described on previous pages is accessible to all to replicate.

Action research provides researchers with the means to take on a more involved, almost interventionist, role, without sacrificing their research. This paper outlined how the author deployed a gamut of action research practices – from different forms of journaling to access rich, multi-layered data, to ways to be reflective and reflexive to how to critically question conclusions in this paradigm – to improve the learning outcomes of a remote Playing Lean workshop.

Even though the focus was on “remote” and “Playing Lean workshop”, the author suspects that the described process could be applied to improving any workshop which is intended to teach anyone new skills. They took great care in detailing specific practices, so the inspired reader has all they need to take action.

Research conclusions held up to the scrutiny. Plurality of knowing was upheld, practical outcomes achieved and critical reflexivity was present at multiple levels.

The author would like to offer following interesting questions for further inquiry:

Q1.

What could be learned by zooming out, and reflecting on multiple improvement cycles? What recurring patterns and behaviours could be identified, and to what result?

Q2.

What would be good measures of “enduring consequence” and how could one go about establishing them? What would make for a “good” mix of quantitative and qualitative metrics?

Q3.

How could the action–reflection protocol described herein be applied to improving the learning outcomes of other workshops? Is the design too dependent on the researcher, or is it generalisable? Would the effects be worth the effort?

Q4.

How could the methods and techniques described herein be disseminated to, and employed by, other Playing Lean facilitators to achieve the same, or better, outcomes? In what way could that be a participative, collaborative, and enduring action research?

Regardless of what line(s) of inquiry are explored further, the author hopes this paper inspires other lean educators and practitioners to experiment with action research methods in their own work. Their participatory, collaborative and practical nature creates ample opportunity for furthering both theory and creating a meaningful impact in the participants’ lives.

Figures

Figure 1.

An example of a Playing Lean experiment card

Figure 2.

Spiralling action research cycles

Figure 3.

Research design phases, activities and methods

Figure 4.

A visualisation of reflective and reflexive gaps

Table 1.

Research design phases, activities and methods

Phase	Activity	Method
Action	Imagine situations that might significantly impede or contribute to the learning outcomes of the workshop	Describe them using the four territories of experience
Action	Conduct the workshop	N/A
Action	Record situations that might have significantly impeded or contributed to the learning outcomes of the workshop	Describe them using the four territories of experience
Reflection (individual)	Individual sense-making and reflection on the differences between what was expected and what actually had happened	Free-form journaling
Reflection (joint)	Joint sense-making and reflection on the differences between what was expected and what actually had happened	Dialog using four parts of speech
Closing	Formulate and record improvement ideas for the workshop design and delivery	One-page summary

Table 2.

Four territories of experience

Territory	Torbert and Cook-Greuter (2004)	Torbert and Taylor (2008)	Morris (2020)
1st	Outside events: results, assessments, observed behavioural consequences, environmental effects	The outside world: objectified, discrete, interval units, of which “I” am actively aware when “I” notice the colour and manyness of what “I” see or the support the outside world is giving me through the soles of my feet (focused attention)	Outside of me: Noticing and observing what is happening around me. This is partly about people – what they are doing/saying; but it’s also impersonal – the physical environment, data on a page, etc.
2nd	Own sensed performance: behaviour, skills, pattern of activity, deeds, as sensed in the process of enactment	One’s own sensed behaviour and feeling: processual, ordinal rhythms in passing time, of which “I” am actively aware when I feel what I am touching from the inside, or when I listen to the in-and-out of my breathing or the rhythms and tones of my own speaking (subsidiary, sensual awareness)	What I am doing. This includes physical sensations I am experiencing from the inside (movement, breathing, taste, etc.).
3rd	Action-logics: strategies, schemas, ploys, game plans, typical modes of reflecting on experience	The realm of thought: eternal nominal distinctions and interrelations, of which I can be actively aware if my attention “follows” my thought, if I am not just thinking, but “mindful” that I am thinking (witnessing awareness)	My thoughts, emotions and action-logics
4th	Intentional attention: presencing awareness, vision, intuition, aims	Vision/attention/intention: the kind of noumenal vision/attention/intention that can simultaneously interpenetrate the other three territories and experience incongruities or harmonies among them	My bare attention or post-cognitive consciousness. This kind of attention is rarely cultivated, but through lifelong practice can come to hold the other three territories in a way that is objective, impartial and simultaneous. It’s hard to do because thinking about it takes us quickly back to the 3rd Territory!

Table 3.

Select issues with taking the playing lean experience online and their countermeasures

Issue	Countermeasures
Attention: students are easily distracted during an online session	Set the expectations in advance: no mobile phones, close all browser and windows tabs unrelated to the workshop, respect the time schedule. Introduce specific roles with clear accountability for each team member.
Storytelling: the facilitator does not control what the student sees; lack of unified experience by the students	Limit the storytelling to speech, use more vivid and less abstract examples, speak slower and more eligible, make sure to have attention of everyone before sharing.
Improvisation: technology can hinder impromptu exercises or adjustments	Gather student profiles and desired learning outcomes in advance and adjust the workshop design as necessary. Disseminate all student materials in advance, and have all relevant URLs listed in one document.
Communication: only one person can speak at a time, group discussions are difficult	Use video-conferencing solution (e.g. Zoom) for voice and white-board solution (e.g. Miro or Mural) as virtual classroom. Be explicit where communication is supposed to happen. Prepare a dashboard for each team. Ask how did students communicate with each other.
Social learning: team and intra-group discussion is important for overall learning, which is hindered due to communication issue listed above	Use breakout rooms to facilitate social learning within teams, modify the reflection session to include discussion on each other’s strategies and actions. Ask how did teams learn from each other.
Technological aptitude: it is very difficult to troubleshoot technical issues due to many possible combinations of software and hardware	Select the least amount of technologies needed to deliver the workshop and achieve desired learning outcomes. Prioritise technologies with wide adoption and perceived ease of use. Include use of these technologies in the workshop agenda.

Source:

Pešec (2021)

Table 4.

A sample of imagined situations from the facilitator’s journal

1st territory the world and events outside of me	2nd territory my own sensed performance, what I am doing	3rd territory my thoughts, emotions and action-logics	4th territory my bare attention or post-cognitive consciousness
Students start playing on their phones; they lose attention; don’t come back when they should…	(1) I say out loud what I’m seeing and experiencing, and explicitly say that’s unwelcome and disrespectful behaviour. (2) I send a message to the lecturer, whom is physically present with the students, asking for help	Playing Lean is a legitimate game, and I never had a player that was disengaged. I’d probably be quite annoyed if that happened, and might be less than graceful in my rebuking	Label unwanted behaviour and move on
Students or the lecturer ask me to display experiment cards on the screen	(1) Refuse, explaining that I’d rather have students focusing on the story I’m telling them than the slide I might be showing them. (2) Re-invite students to ask clarifying questions for any concept they find confusing. (3) Use whiteboard feature to illustrate the concept live. (4) Ask students to explain back the concept I’ve just introduced	(1) I dislike using too much technology in workshops, even if they are being done remotely. Stories are more important than showing them the slide. (2) Previous experience tells me that students will only care about the result on the slide (number of customer tiles turned), and not so much about the concept presented. (3) That’s why I’d rather explain and show it in different ways, than displaying the card as a slide	(1) How do I say no without diminishing reputation of the lecturer? (2) How do I help student understand the matter at hand without making them feel excluded and/or ignored?

Table 5.

A sample of imagined situations from the assistant’s journal

1st territory the world and events outside of me	2nd territory my own sensed performance, what I am doing)	3rd territory my thoughts, emotions and action-logics	4th territory my bare attention or post-cognitive consciousness
A player can’t turn on his video and sound for the workshop, and can only communicate with his team via chat	I will wait for the team with the muted player to call me into their breakout room for consult, and tell them to work it out themselves. I will also ask the Facilitator for advice	I fear that the muted player will be left out of the decision making in his team, and that he will not learn as much as the others since his involvement is less	Having a player that can’t turn on his video and sound is out of my hands, and I need to let the situation play out on its own
A team calls me into their breakout room. I am assuming that it is a technical question since the Facilitator tells them that they can call me in for that reason, but they actually have a game related questions – a question on their next move	I will not give them a direct answer that will give them an advantage in the game, but rather ask them what they want to do and tell them to try it out	The learning in the game comes through the mistakes teams make as well. The move they made could cost them their win, but it will not harm the learning process	I want everyone to have an even chance at winning, and trust in the playing and learning process to deliver maximum results

Table 6.

A sample of actual situations from the facilitator’s journal

1st territory the world and events outside of me	2nd territory my own sensed performance, what I am doing)	3rd territory my thoughts, emotions and action-logics	4th territory my bare attention or post-cognitive consciousness
6 students show up, which is fewer than announced. They arrive as two groups of three, and are unwilling to be split up.	I tell the lecturer that I’ll break up the students’ groups, but they tell me they “really, really don’t want that”. … I ask Assistant, via private message, to delete one of the game teams off the digital table.	No point in arguing about groupings now, in this medium. … If this was an in-person workshop, I’d probably be less inclined to satisfy their desires.	As this is happening … I slowly become aware of the fact that this isn’t a remote workshop. No, this is in fact a hybrid workshop, with the lecturer and participants in a shared physical space, while Assistant and I are facilitating remotely.
On-the-fly delivery adjustment: stop using break-outs.	I tell Assistant to stop using breakouts, and that I’ll use the timer on the whiteboard instead.	(1) Break-outs make a lot of sense when everybody is remote—it’d be nearly impossible to have team discussions otherwise. (2) Since all the participants are physically together, they can discuss their activities live.	As soon as I noticed this procedure doesn’t add to the process any more.
For the last question of the workshop, I ask the participants to share their current understanding of the lean startup methodology in the chat. They write about Playing Lean, the game, instead. Again. They write about the method after I remind them of the difference.	I point out that they are writing about the game, and that they should answer about the method, instead.	This moment left me perplexed for a second. How could they be drawing all the proper lessons, as proven by their reflection notes, and yet get this one so wrong?	Everything is brought into question. I feel relief knowing that I’ll capture my experience in a structured way, and so will Assistant. That should enable us to assess the learning value of workshop.

Table 7.

A sample of actual situations from the assistant’s journal

1st territory the world and events outside of me	2nd territory my own sensed performance, what I am doing)	3rd territory my thoughts, emotions and action-logics	4th territory my bare attention or post-cognitive consciousness
The students are all in the same room and there is an echo, which is distracting to everyone when someone speaks	At first, I pull them into breakout rooms but then Facilitator decides to ditch the breakout rooms, but gives no instructions to them how to organize themselves	Since I can’t see the layout of the room and if team-mates are sitting close to each other, I worry that if we don’t take a minute and explain how to best organize themselves, the teams won’t communicate well with each other which will cause someone to be left out of the discussion	(1) Is it better to have a hybrid workshop where all the players are in the same room, but the facilitator is not? (2) How will this affect the discipline, and the facilitator’s control of the room/players?
The students are talking amongst themselves while Facilitator is explaining an experiment	(1) I’m not mentioning this to Facilitator because I don’t want to break his concentration. (2) I’m not telling the students to be quiet because I’m expecting this to pass	(1) I fear that they are missing out on valuable lessons, but hope that they will concentrate harder as the game progresses. (2) I fear they are more concentrated on the game and winning, or even worse, they don’t understand what Facilitator is talking about	I let things play out on its own, and manage my expectations

Table 8.

Evaluation of extended epistemology in the action phases

Phase	Activity	Experiential knowing	Presentational knowing	Propositional knowing	Practical knowing
Action	Imagine situations that might significantly impede or contribute to the learning outcomes of the workshop	Previous lived experience informs the imagination, especially the 3rd territory	Situations are fleshed out in a visual table, using vivid language	Knowledge of various facilitation, teaching, and domain theories is used when imagining the responses to the situations	Imagined responses are predominately rooted in historic practice of the Facilitator and Assistant
Action	Conduct the workshop	Living the experience, generating first-hand data during the workshop	Facilitator used a number of visual and storytelling methods to convey different lessons	Facilitator leverages their knowledge of relevant theories to answer participants’ questions, and to adjust workshop delivery in order to maximise learning outcomes	Putting in practice other forms of knowing; noticing own thoughts, emotions, sensations, and actions, as well as those of others
Action	Record situations that might’ve significantly impeded or contributed to the learning outcomes of the workshop	Adding new lived experience to the existing one while it is still fresh; deliberate effort to capture all four territories of experience	Situations, responses, and intentions are captured in a visual table	Propositional knowing is to some extent suppressed, although it shows in entries that try to justify or reason what might’ve been the root cause of some behaviour or situation	The practice of capturing four territories of experience while they are still fresh

Table 9.

Evaluation of extended epistemology in the reflection and closing phases

Phase	Activity	Experiential knowing	Presentational knowing	Propositional knowing	Practical knowing
Reflection (individual)	Individual sense-making and reflection on the differences between what was expected and what actually had happened	Making sense of the experience	Although free-form journaling allows one to use presentational written forms like poems, stories and similar, neither the Facilitator nor the Assistant did so	Still suppressed; the locus is on dwelling in the lived experience	The practice of being intentionally reflective and making sense of four territories of experience
Reflection (joint)	Joint sense-making and reflection on the differences between what was expected and what actually had happened	Moving from “my” experience to “your” and “our” experience	Facilitator and Assistant used a number of visual methods to express and communicate experiences and situations to each other	Ideas and theories were featured when discussing potential causes for certain situations, as well as participants’ behaviours, and Facilitator’s responses	The practice of being intentionally reflective together, and using four parts of speech to have a richer dialogue
Closing	Formulate and record improvement ideas for the workshop design and delivery	Cumulative lived experience informs the improvement ideas	Arguably, presentational knowing is present in the use of a specific template that goes beyond words, and includes visual elements like diagrams, clear structure and uniformity	Formulating and documenting improvement ideas informed by experience, practice and theory	The practice of data- and experience-informed continuous improvement