MASTER-PM: ADDRESSING REMOTE TEACHING CHALLENGES IN PROJECT MANAGEMENT THROUGH A SERIOUS GAME APPROACH

ABSTRACT

The Covid-19 pandemic provoked a reevaluation of teacher-student dynamics, prompting the adoption of innovative approaches to ensure student engagement, participation, academic performance, and retention. To effectively address the emerging challenges in this context, mixed methods research was conducted, anchored by the Design Science Research approach, where artifacts were developed, evaluated, and systematized in a collaborative serious game, MASTER-PM. The findings demonstrated the game’s positive impact on teamwork learning outcomes, and valuable student feedback facilitated iterative enhancements to the game and its support artifacts. This paper describes the MASTER-PM application in an undergraduate administration course at a Brazilian university and evaluates its viability and usefulness in project management education.

Keywords
Serious Games; Active Learning; Remote Teaching; Engagement; Project Management

INTRODUCTION

The Covid-19 pandemic compelled various establishments, including companies, shops, cultural centers, and leisure spaces, to undergo adaptation processes in compliance with health guidelines such as social distancing, mask usage, and sanitation protocols. Similarly, educational institutions like schools and universities had to swiftly adopt remote teaching methodologies, leveraging technology to sustain academic activities while mitigating the risk of virus transmission. The Universidade Federal de Pernambuco (UFPE) was no exception, implementing remote teaching during the 2020,3 semester under (Resolution 08-2020 - CEPE) to pursue its institutional mission amid these unprecedented challenges.

The transition to remote teaching has brought about notable changes in the dynamics between professors and students. Multiple authors have identified engagement as a common obstacle that significantly affects the quality of learning. In this remote context, students require heightened encouragement and stimulation compared to face-to-face interactions ( (ALMAIAH; AL-KHASAWNEH; ALTHUNIBAT, 2020); (RASHEED; KAMSIN; ABDULLAH, 2020); (WIDODO et al., 2020). Previously less noticeable issues have been magnified as the absence of immediate teacher supervision in indirect student interactions can lead to a sense of lethargy and impact their level of participation.

Failure to acknowledge the impact of this phenomenon in the teaching process can result in monotonous classes, leading to decreased student concentration and increased distractions with parallel activities. These issues contribute to student discouragement, reduced class engagement, negative effects on academic performance, and, ultimately, student evasion from the course. In light of these challenges, this research aims to provide valuable insights for fostering an innovative teacher-student relationship by applying adaptable teaching methodologies in the remote teaching context. With the expectation of this initiative, it is anticipated that implementing stimulating approaches will lead to increased student participation, heightened engagement, improved academic performance, and reduced likelihood of student evasion.

In today’s dynamic organizational landscape, professionals are increasingly required to possess various skills. The use of games in educational and organizational settings has emerged as a growth strategy, as it enhances crucial abilities such as attention, focus, logical reasoning, and strategic thinking, all essential for effective decision-making. This comprehensive set of cognitive, social, emotional, and technological skills is increasingly demanded to navigate the evolving demands of organizations ( MANYIKA et al., 2017). The ever-changing organizational environment is intricately linked to the transformative power of Information Technologies, shaping an increasingly interconnected world and influencing communication and global economies ( ISF JOURNAL, 2021).

In this broad framework, the research question of this study emerges:

RQ: To what extent can the combination of active learning approaches and game elements contribute to improving engagement in the remote teaching process?

Considering these insights, this research adopts the methodological framework of Design Science Research (DSR) ( HEVNER; CHATTERJEE, 2012) to address the challenges associated with remote learning engagement.Termed the “difficulties faced in engaging in the remote learning process,” this study focuses on generating a viable solution through ideation, design, development, and evaluation of instruments that facilitate organizational learning.These instruments aim to foster active learning in governance and project management, thereby contributing to a more effective learning experience for remote learners.

This study combines various active learning techniques to provide a comprehensive solution to the research question, including Flipped Classroom, Gamification, and Problem-Based Learning (PBL), complemented by the immersive elements of Serious Games within the context of Business Games. The development and evaluation of these instructional artifacts were conducted with a group of 60 students enrolled in the AD413 - MANAGEMENT OF PROJECTS AND SERVICES course in the Administration program at the Universidade Federal de Pernambuco (UFPE) during the academic semester of 2020,3.

This article follows a structured approach to present the research findings. Section 2 establishes the theoretical foundation by delving into relevant concepts and prior studies. Section 3 outlines the adopted methodological framework, providing insights into the research design and approach.In Section 4, the game’s development process is comprehensively described, offering a detailed account of its creation. Subsequently, Section 5 focuses on the application of the game and provides a thorough analysis of the main results obtained from its implementation. Finally, in Section 6, the article delves into a critical discussion of the main contributions and limitations of the work, along with the potential avenues for future advancements and development.

BACKGROUND

This section briefly introduces several key themes that provide crucial support to the realization of this study. In light of the ongoing transformational changes experienced by organizations, both in terms of technology and culture, it becomes increasingly imperative to prioritize and promote initiatives to equip managers with the necessary skills to navigate these challenges. This includes emphasizing the significance of incorporating such initiatives within their foundational education and ongoing professional development ( ALMEIDA; BUZÁDY, 2019.

Active Learning stands out as an approach that holds significant value, which entails creating real or simulated learning environments that empower students to take charge of their learning journey.This approach encourages students to identify problems, develop strategies and solutions, and reflect on their learning experiences ( YOST HAMMER; GIORDANO, 2012). An exemplary Active Learning modality is the Flipped Classroom, as described by Valente (2014). In the Flipped Classroom model, students engage with the content and instructions before class, allowing the classroom time to work on the assimilated knowledge. This approach enables teachers to design activities centered around the pre-studied content while class time is utilized to deepen conceptual understanding, explore problemsolving scenarios, and foster collaborative learning ( TUCKER, 2012).

According to Bishop and Verleger (2013), the Flipped Classroom concept is rooted in studentcentered learning theories and methodologies, which can be divided into two distinct components.Firstly, students independently engage with teacher-provided content before class, acquiring foundational knowledge individually. Secondly, during class time, students actively participate in interactive learning groups, where they can transcend the acquired content and engage in stimulating dynamic, creative, and collaborative activities. The authors emphasize that the flipped classroom approach goes beyond merely discussing pre-studied material; it allows students to delve deeper into the subject matter, fostering engagement and encouraging further exploration of the content in the preparatory phase.

Furthermore, Mitre et al. (2008) highlight the significance of Problematization and Problem- Based Learning (PBL) as active approaches that facilitate teaching and learning processes. When confronted with a proposed problem, students are prompted to pause, analyze, reflect, establish connections with their experiences, and reframe their discoveries.

Originally defined as the integration of game design elements into non-game contexts, Gamification has evolved into a focused strategy that applies game elements to non-game activities, intending to influence and modify individual and group behaviors to achieve desired organizational outcomes ( DETERDING, 2012; SCHÖBEL et al., 2020). Three fundamental elements underpin the essence of Gamification: dynamics, mechanics, and game components, each playing a pivotal role in shaping the overall experience. Dynamics encompass the underlying rules, goals, and challenges that drive engagement and motivation within a game. Mechanics, on the other hand, are the tangible building blocks that operationalize the dynamics. These mechanics include points, levels, rewards, and feedback systems, guiding and incentivizing desired behaviors ( COSTA; MARCHIORI, 2015). Additionally, game components act as connective tissue, interlinking mechanics, and dynamics. Each mechanic is intricately linked to one or more dynamics, while each component aligns with mechanics or dynamics, forming a cohesive approach to influence team attitude.

Serious Games offer a unique approach beyond pure entertainment, utilizing game elements to enhance participant engagement and performance. According to Hookahm and Nesbitt (2019), these games create a memorable experience, leading to more effective learning outcomes. The term “serious” typically characterizes the application of this approach in various industries such as defense, education, scientific exploration, healthcare, administration, emergency management, urban planning, engineering, and politics ( MICHAEL; CHEN, 2006; ZUMBACH; RAMMERSTORFER; DEIBL, 2020). Within this landscape, Business Games, a specialized form of serious games tailored for the management learning context, have gained widespread recognition as innovative pedagogical tools. These games provide Business Administration students with experiential learning opportunities, enabling them to acquire valuable practical skills and knowledge ( DE ARAÚJO et al., 2019).

The Role-playing Game (RPG) genre is a cooperative gaming experience where players assume character roles and collectively craft an interactive story to accomplish goals guided by a “game master” ( JONES et al., 2020). The game master sets objectives or missions while the players, adhering to predetermined rules, employ their knowledge and skills to achieve them. When applied to organizational settings, this game genre can be considered a gamification of organizational tasks ( BOUCHILLON; STEWART, 2021). Hitchens and Drachen (2009) identified key game elements that characterize an RPG, including the Game World (the immersive setting), Participants encompassing the roles of Characters (player avatars) and the Game Master (facilitator), Interaction, and Narrative.RPGs offer unique opportunities to enhance engagement and problem-solving skills by fostering creativity and collaboration.

When delving into gameplay, Hunicke, Leblanc, and Zubek (2004) highlight the significance of mechanics, dynamics, and aesthetics in shaping game experiences. Mechanics refer to the set of rules that define the possibilities and limitations within the game, encompassing the actions, behaviors, and control mechanisms available to players. On the other hand, dynamics pertain to how these rules are interpreted and applied, encompassing the underlying concepts and social structures that drive the game. Dynamics describe the evolving behaviors of players and the runtime interactions with the mechanics over time. Aesthetics, the experiential aspect, encapsulate the desirable emotional responses evoked in players when they engage with the game system.

The MDA framework, proposed by Hunicke, Leblanc, and Zubek (2004), introduces an interconnected relationship among mechanics, dynamics, and aesthetics. Mechanics form the foundation upon which dynamics are built, and dynamics when executed, give rise to aesthetics. As players engage with the game, they experience aesthetics and infer knowledge about the underlying mechanics and dynamics. MDA is useful to aid game designers in crafting immersive and engaging experiences, enabling players to derive meaningful and enjoyable interactions within the game system.

Experiential Learning Theory (ELT), formulated by Kolb (1984), places significant emphasis on the role of experience in fostering successful learning outcomes. ELT defines learning as the transformative process through which knowledge is created by engaging with experiences. At the core of this theory lies the learning cycle, which encompasses two modes of experiencing - Concrete Experience (CE) and Abstract Conceptualization (AC) - and two modes of transforming experience - Reflective Observation (RO) and Active Experimentation (AE). Grund (2015) further elaborates on these stages, associating them with distinct activities (feeling, observing, thinking, doing) that correspond to different learning styles (diverging, assimilating, converging, and accommodating).

The diverging learning style caters to individuals who prefer to engage in feeling and observing, allowing them to grasp concrete situations from multiple perspectives. Assimilating involves observing and thinking, enabling learners to comprehend a wide range of information and synthesize it concisely and logically. Converging pertains to the adeptness of doing and thinking, empowering individuals to effectively apply ideas and theories to practical scenarios. Lastly, the accommodating learning style encompasses doing and feeling, enabling learners to acquire knowledge primarily through hands-on experiences ( KOLB, 1984).

Experiential learning, as a knowledge-building process, thrives on the dynamic interplay among these four learning styles, adapting to the demands of the context. By understanding these concepts, educators and learners alike can harness the power of experiential learning to improve their full learning potential.

Within organizational learning, Agile Governance emerges as a crucial capability that empowers organizations to perceive, adapt, and respond to environmental changes in a coordinated and sustainable manner, surpassing the rate of these changes ( LUNA; MARINHO; MOURA, 2020).This organizational capacity hinges upon developing and preserving individual and collective competencies within the organizational ecosystem, encompassing both technical and behavioral aspects (LUNA et al., 2016). To effectively analyze and describe phenomena occurring within environments undergoing frequent or significant transformations, the Agile Governance Theory (AGT) has emerged, aiming to enhance understanding of these phenomena and identify the necessary organizational capabilities that must be nurtured based on each organization’s specific context. These capabilities are essential for effectively navigating changes and achieving or maintaining sustainable competitiveness ( LUNA, 2015 ).By understanding and cultivating this capability, organizations can embrace adaptability as a strategic advantage, enabling them to thrive in rapidly evolving environments.

Hookham and Nesbitt (2019) conducted a systematic literature review to define and measure engagement in serious games. As a result, they introduced a comprehensive three-dimensional framework that captures the multidimensional nature of engagement. This framework encompasses three key dimensions: (1) behavior, (2) cognition, and (3) affect, which collectively contribute to the overall engagement experience. Moreover, the authors identified three key constructs that are intricately connected to these dimensions, shedding light on the underlying mechanisms of engagement: (4) flow, (5) immersion, and (6) presence. Based on their framework, we framed Figure 1.

Figure 1
Engagement Model. Based on UML Model by Hookham and Nesbitt (2019).

Flow (4) is intricately tied to the user’s experience in finding the optimal balance between their skill level and the challenges presented in the game. Reaching the state of flow (4) favors cognition (2) and can be observable in player behavior (1). Immersion (5) is closely linked to the player’s emotional involvement, interest, and focused participation in the game activities. On the other hand, presence (6) pertains to the player’s sense of belonging and connection to the game world. Both immersion (5) and presence (6) are indicators of the affect component (3) and are experienced according to the game’s flow (4). The combined occurrence and the interplay of behavior (1), cognition (2), and affect (3) facilitates the engagement manifestation ( HOOKHAM; NESBITT, 2019).

Hence, it is imperative to consider behavior, cognition, affect, and the associated constructs of flow, immersion, and presence to ascertain student engagement. Evaluating user behavior involves examining factors such as time on task, which aims to gauge the frequency and duration of a player’s participation in the interactive dynamics. This information can be derived from game metrics, such as score history and participation or observation techniques. Meanwhile, the dimension of affect is captured by assessing the satisfaction level and overall sense of engagement experienced by the players. This valuable insight can be obtained through satisfaction surveys administered to students (players) at the culmination of each synchronous meeting. Lastly, the dimension of cognition entails scrutinizing the players’ focus and interest in the content explored throughout the course, with academic performance as an objective indicator.

Based on these perceptions, within the scope of the methodological approach adopted, this study seeks to address a reasonable solution to the class of problems: “difficulties faced in engaging in the remote learning process”. Therefore, this article describes the ideation, design, development, and evaluation of tools to aid the organizational learning process, which were systematized through a serious game, considering elements from different genres, such as Role-Playing, Puzzles, Simulation Games, Visual Novels, and Management ( HOOKHAM; NESBITT, 2019), for active learning of governance and project management, targeting trainees, be them, practitioners, or students.

METHOD

The research project adopted a mixed methods approach ( CRESWELL, 2009) and was guided by the Design Science Research (DSR) approach ( HEVNER; CHATTERJEE, 2012), which facilitated the iterative development of artifacts through DSR cycles. The selection of DSR was motivated because both areas of Information Systems and Management Sciences, where this study takes place, are interested in leveraging scientific knowledge to design and construct practical solutions for organizational challenges ( LACERDA et al., 2013).

DSR is an iterative process with three cycles: Relevance, Design, and Rigor. The original Information Systems research framework by Hevner and Chatterjee (2012) was adapted to the context of this research. Figure 2 presents the cycles characterizing each practical issue to be addressed by developing the artifacts.

In short, the Relevance Cycle, where the research begins, involves defining the problem to be addressed, the research requirements, and the criteria for evaluating the research results. The problems addressed by the present study concern the “difficulties faced in engaging in the remote learning process”. Such difficulties were organized as the “class of problems” (practical and theoretical), for which this research seeks to address a reasonable solution through the ideation, design, development, and evaluation of learning instruments.

The class of problems addressed by this research emerges as a consequence of the adversities arising from the interaction mediated by technologies in the context of remote teaching, having a significant impact on the quality of learning and having recently been evidenced in the scenario brought about by the Covid-19 pandemic, as students need more encouragement and stimulation in this (remote) context than in face-to-face interactions (ALMAIAH; AL-KHASAWNEH; ALTHUNIBAT, 2020 ; RASHEED; KAMSIN; ABDULLAH, 2020; WIDODO et al., 2020).

Figure 2
Research Framework based on Hevner and Chatterjee (2012).

Considering the problems’ class above, instruments were developed to help in remote teaching, seeking to meet the following requirements:

[R1] Behavior: encourages and inspires students’ appreciative and collaborative attitudes in interactions with the teacher and their peers.

[R2] Cognition: provides stimulating learning situations and builds a critical perception of the topics explored in the course’s syllabus.

[R3] Presence: provides the necessary ambiance to favor the students’ sense of belonging in the context of teamwork and the proposed learning scenarios, fostering participation, attendance, and punctuality of students.

Such requirements were defined based on the literature and the researchers’ experience. For example, the characterization of [R1] was prompted by the issues raised by Widodo et al. (2020) on the cognitive-behavioral context capable of providing a more successful experience in remote teaching. Likewise, the characterization of [R2] was influenced by the perception of the significant impact on the quality of learning in remote teaching, which was recently even more evident in the scenario brought about by the Covid-19 pandemic, as pointed out by Almaiah, Al-Khasawneh, and Althunibat (2020). In comparison, the characterization of [R3] was inspired by the insights brought by Rasheed, Kamsin, and Abdullah (2020) on concerns and influences of “student isolation” (and its challenges) in remote learning, as students need more encouragement and stimulation in this (remote) context than in face-to-face interactions.

In addition to being useful as a criterion for evaluating the artifacts produced, these requirements also help to guide the extent to which the MASTER-PM can be adopted to mitigate the challenges of a remote teaching in project management. An artifact must be considered viable if it can be applied according to its description if it produces what it proposes to deliver, and if its application requires effort considered adequate. In addition, an artifact must be considered useful if its application adds value (or brings benefits) to mitigate or solve problems of the class that motivated its development. In this research, we will adopt the following feasibility and usefulness indicators suggested by Peffers et al. (2008) and Trinkenreich, Santos, and Barcellos (2018). The first criterion, feasibility, is described as the balanced combination of applicability, effectiveness, and adequate effort. Where: (i) applicability: evaluates whether the MASTER-PM can be applied as planned/designed; (ii) effectiveness: evaluates whether the approach produces what it should deliver; and (iii) adequate effort: evaluates whether the application of the game requires efforts considered adequate (not excessive effort for its use). In addition, the second evaluation criterion, usefulness, assesses whether using MASTER-PM contributes to mitigating the challenges of a remote teaching in project management.

In the Design Cycle, the researchers’ team conceived, developed, systematized, and evaluated artifacts to address the characterized class of problems. In this study, several artifacts were developed and evaluated through six design cycles, generating progressive and incremental game versions. The evaluation of each incremental version of the MASTER-PM was carried out through case studies delimited in the context of synchronous activities specific to the course (Thematic Meetings - TM), where the version of the game was applied and evaluated by the class.

The Rigor Cycle used and generated knowledge, connecting design science activities to the knowledge base through foundations, experiences, and scientific knowledge produced by research. Rigor is achieved by properly using fundamentals and methodologies from a knowledge base that supports the research while adding the knowledge generated by the research to contribute to the growing knowledge base ( TRINKENREICH; SANTOS; BARCELLOS, 2018).

Considering what was exposed in Section 2, it is opportune to consider the knowledge related to Experiential Learning Theory (ELT) ( KOLB, 1984), Agile Governance Theory (AGT) ( LUNA; MARINHO; MOURA, 2020), and the proposed engagement model by Hookham, Nesbitt (2019) in the design of instruments that provide “new ways of experimenting, making decisions and acting in the context of project management”. Considering that learning is a holistic and continuous process based on the experience of adapting the individual to the world to create knowledge, in addition to the aforementioned references, the main foundations of this research were composed of knowledge related to the following topics: Active Learning, PBL, Gamification, Serious Games, Business Games, and RPG. In addition to the DSR, methods such as exploratory review and case study were also adopted in the research stages.

The DSR cycles were carried out in synergy with two research projects of the Institutional Scientific Initiation Scholarship Program (PIBIC/UFPE/CNPq), where students worked in collaboration, guided by the same professor, aiming to establish synergy between research, as well as to foster a culture of collaboration and scientific cooperation among students and stimulate the development of 'lifelong learning skills' ( DUNLAP, 2005).

Initially, through an exploratory review, the bibliography related to the main fundamentals of the research was studied, comprising the phase of formalization of concepts, updating and enhancement of knowledge, and associated processes. As a starting point, an exploratory literature review was carried out based on keywords extracted from the research objectives based on related topics, still using the “snowball” sampling technique. After going through initial learning cycles to establish the conceptual bases that supported the research, the research went through cycles of development and evaluation of the artifacts and consequent systematization of the game. Finally, refinement cycles were important for analyzing and improving the artifacts, intending to improve them through feedback and class perception gradually.

As a result, the artifacts produced were systematized in the context of a collaborative game called MASTER-PM (Master Project Manager), using game elements for active learning of project governance and management. This analog game, mediated by the technologies used in remote teaching, was developed and evaluated in the context of the experimental (extraordinary) academic semester 2020.3 planned by the institution at the height of the COVID-19 pandemic so that teachers and students could adjust to practices of teaching and learning in the remote teaching modality.

DEVELOPMENT

The development of MASTER-PM started with incremental learning cycles. Each cycle consisted of an investigative study to obtain relevant knowledge for game development. Two studies were carried out. First, an exploratory literature review was carried out to identify theories, models, and constructs that support the theoretical conjectures considered in the research, helping, for example, to establish the requirements presented in the previous section.

Then, a new exploratory literature review was performed to identify correlated artifacts, similar applications, techniques, and design principles; and to characterize the problem to be mitigated by the research, seeking to denote the causes, consequences, frequency of occurrence of the problem, those interested in the resolution, the relevance in solving the problem. These two incremental learning cycles were essential investigative activities to obtain useful knowledge for game development.

The activities of the course were organized into synchronous and asynchronous activities in the context of remote learning to allow the alignment of this research during the school semester. Asynchronous activities made the syllabus contents and their respective fixation exercises available to the students in advance. Such knowledge packages were worked on through synchronous meetings with the support of mechanics, dynamics, and other game elements, seeking to favor student engagement.

Following the DSR approach, the MASTER-PM was developed in six design cycles, as shown in Figure 3. Each design cycle produced and evaluated, iteratively and incrementally, a set of artifacts, which systematized in the context of its application, make up the first version of the collaborative game described in this article. These “sets of artifacts” were worked on in the realm of self-contained “learning contexts” and planned according to “packages of knowledge” arising from the course syllabus. The synchronous meetings where the “sets of artifacts” that comprise the game were evaluated, and produced in each cycle, were named Thematic Meets (TM) and conducted under the active approach of a Flipped Classroom ( VALENTE, 2014).

For example, the first design cycle was carried out during the 3-week planning period, which preceded the start of the course and was dedicated to the game’s general design, considering its main characteristics and core systematics, such as evaluation, role-playing, rewards, acknowledge, and awards; and the identification of other dynamics, mechanics, strategies, and game elements that could be used in its construction.

Figure 3
MASTER-PM development and refinement: Design Cycles.

Its evaluation took place in the 1st MT with the students, where the approach overview was presented to the class in alignment with the course’s Teaching Plan. Consequently, we obtained data from the students’ profiles (players), expectations about the experience to be lived, and feedback regarding the proposed approach and planned activities. This information was useful as guidelines for subsequent cycles.

The sessions from the 2nd MT onwards provided the opportunity to evaluate both the artifacts applied at that time and those developed for previous dynamics that were gradually improving. From these evaluations, the artifacts were again exposed to improvement cycles and applied in the next PM: Addressing remote teaching challenges in project management through a serious game approach 11 MT. Figure 4 depicts the main elements and strategies of the MASTER-PM version reported in this article.

RESULTS AND DISCUSSIONS

To evaluate the game, we adopted an exploratory strategy based on a holistic multiple case study (type 3) ( YIN, 2015), with the 'course class' as the unit of analysis, in six distinct phases characterized by the respective design and evaluation cycles employed in iterative and incremental game development. The studies had as propositions, respectively: (1st cycle) “how the students would receive the proposal, what contributions they could provide”; (from 2nd to 6th cycles) “how and why the approach described by the incremental versions of the game contributes to engagement in remote teaching, considering the requirements of behavior [R1], cognition [R2] and presence [R3]”. The following sections briefly describe the application of the game and the results produced.

MASTER-PM Application

The course class consisted predominantly (53/60) of representatives of Generation Z (born between 1991 and 2010), mostly graduating students (54/60), of which 35/60 were already working in the job market or doing internships (15/60), in public and private companies in the secondary (8/60) and tertiary (42/60) sectors of the economy.

Figure 4
Overview of the MASTER-PM components.

The class received the game proposal very well, and the students presented relevant contributions and expectations during the 1st MT. This feedback influenced the game's development, such as the suggestion that the final result could generate a bonus in the PBL Project grade, conducted by them to practice the course knowledge.

In the role-playing context, the game provoked a weekly rotation among the students of each team, of the role of Project Manager (PM), in conducting the real project during the course related to elaborating a Seminar about an emerging topic inProject or Service Management. This approach aimed to provide the opportunity for all students to live the experience of the leadership role and the challenges of a PM while seeking to integrate the course syllabus with the inclusion of students in real challenges, in which these knowledge, skills, and attitudes were essential.

Various interactive activities, such as quizzes, puzzles, and quests, were implemented throughout the course. These activities involved engaging question and answer dynamics, utilizing card draws and the quiz platform Mentimeter.com. The platform generated scores and rankings based on participants' accuracy and response speed. To enhance the learning experience, elements from Simulation Games, Visual Novels, and Simulated Management Scenarios, as described by Hookahm;Nesbitt, (2019), were incorporated into the development of MASTER-PM. These elements helped contextualize the teaching case narrative of the Hotel, simulating strategic problems faced by a fictitious hotel enterprise. The challenges were presented by non-playable characters (NPCs) portrayed by the employees, encouraging lively debates and collaboration among players and their teams to find solutions for the given situations.

The teaching-learning cycle projected from the 2nd to the 6th MT, and depicted in Figure 4, consists of three stages: (1) Thematic Pre-Meet, (2) Thematic Meet, and (3) Thematic Post-Meet. The experiential learning process idealized in the design of the game seeks to provide a learning spiral where the learner is encouraged to experience the ELT dialectic modes: concrete experience (EC), reflective observation (RO), abstract conceptualization (AC), and active experimentation (EA). Thus, experiencing, reflecting, thinking, and acting in a recursive process, the student is encouraged to respond to the learning situation and what is being learned. Immediate or concrete experiences are the basis for observations and reflections. These reflections are assimilated and distilled into abstract concepts from which new implications for action can be drawn. These implications can be actively tested and serve as guides in creating new experiences in subsequent iterations. The experiential learning approach adopted emphasizes practical (hands-on) experiences and experimentation activities, which must be explicitly complemented with reflective observation and abstract conceptualization (minds-on) exercises to complete the learning cycle and provide meaningful conceptual understanding.

The Pre-Meet started by carrying out asynchronous activities in Google Classroom, considering the exploration of each theme in the syllabus of the course, and carrying out knowledge fixation exercises, in preparing students for the Thematic Meet. At the end of this stage, a warm-up activity was carried out for the Meet, where students described for the period under analysis: (i) the level of immersion in the subject content; (ii) qualified and quantified interactions with team members about the project; (iii) reported the experience or their expectations, in the role of PM; and, (iv) suggested points of improvement to improve the material made available and the asynchronous activities applied.

The application and evaluation of the game by the 60 students took place in the context of a Flipped Classroom during synchronous classes called Thematic Meets. The subjects introduced in the asynchronous activities were explored within the game's scope in these meetings. During the Thematic Meet, the teams used the knowledge assimilated in the Pre-Meet, and their experiences, in three dynamics: (D1) Engaged team - which evaluated the cohesion, attendance, and punctuality of the team, and assigned +1 point when all members of the team were present at the time of the Meet verification; (D2) PM mode on! - which evaluated the participation of the leadership of the week, being able to attribute +1 point to the team in the participation of the GP, or -1 in case of its absence; (D3) Knowledge challenge - in which the knowledge previously explored in asynchronous activities was worked through gamified narrative and progression dynamics, through group discussion, simulations of real situations, sociotechnical experiments, PBL, quests, puzzles, challenges, and quizzes, often contextualized by a teaching case, among other techniques, which seek to encourage the student to reflect on the syllabus while providing the necessary conditions for the development of critical thinking on the themes worked on.

Depending on the team's performance and planned activities, D3 could attribute up to 7 points to each Meet. At the end of each Thematic Meet, the students also evaluated the session: (a) scoring their satisfaction on a 5-point Likert scale, (b) pointing out what they liked most, (c) what they least liked, and (d) what they could be done to improve the next sessions (levels) of the game. The mechanics, dynamics, and other game components were gradually improved over the Meets based on feedback from the class, suggestions for improvement, and analysis by the researchers.

In the Post-Meet, at the end of each MT, individual and collective rankings were produced, and a scoring system was applied, which allowed the teams to advance on the game board, and game monitoring panel. At the end of the semester, the students from the three teams best placed on the board received a bonus on the project PBL grade, and all students who completed the course received a certificate of participation in the Assessment Workshop for version 1.0 of the MASTER-PM, with the final placement of your team on the game board.

Considering the elements of an RPG, the (i) treatment of space and interaction with the game world and (ii) narrative support, in the context of MASTER-PM, were characterized through the adoption and progressive adaptation of ateaching case of a hotel and the characters that were part of this picturesque and fictitious organizational context. Within the scope of the RPG above elements, the treatment of space was important to delimit the players' action area, which was configured in decisions about mitigating the hotel's problems. The investigation of the imaginary world is made possible through the treatment of space. The narrative element is associated with the sequence of events in the game world, allowing the immersion of players ( HITCHENS; DRACHEN, 2009). This game element, often used in role-playing games, helped build the Hotel's game world.

In terms of (iv) participants, characters, and roles, the use of characters in RPGs is essential since it is through them that players can interact with the game world ( HITCHENS; DRACHEN, 2009). Pontes (2017) expands the concept of RPG's role for teams, in which members start o have defined characteristics and established roles. In this sense, in the case of teaching at the hotel, the teams established themselves as consultancies to solve the problems of this client company. Despite this, the members also had roles and objectives to fulfill, such as the Project Manager (PM) and the team members who had specific roles due to the challenge proposed by the game. The PM, selected weekly by the team, organized and led the activities, setting goals, meetings, and tasks for him/herself and his/her colleagues, while the other members, who were waiting for their week to lead, acted as consultants.

The professor and the research team acted as Masters of the Game, organizing the dynamics, helping the teams, and making the content available. According to Hitchens and Drachen (2009), the main role of the Game Master is to elaborate on the elements of the story and present the scenario of the game world, adjusting the rules when necessary. Among the characters, the concept of Non- Playable Character (NPC) was also used, adapting the teaching case to create dialogues related to the hotel’s problems since interaction via dialogue is an important aspect in role-playing games, as the same authors.

Data were collected through questionnaires, observation during the MTs, and document analysis referring to the deliveries of asynchronous activities. The TMs were also recorded, and when there was any doubt about the students’ reactions and feedback in the context of the applied dynamics, the research team watched and discussed the videos.

Due to the unavailability of data on the completion of the course in the remote teaching modality without applying a gamified approach, it was necessary to use the previous data obtained in the face-to- face teaching modality as a reference to evaluate the feasibility and usefulness of the game.

Concerning the feasibility of the central artifact of this research, the MASTER-PM, we can say that the game resulting from the systematization of the various artifacts produced and gradually evaluated throughout two cycles of incremental learning and six cycles of the iterative and incremental design was applied according to the description of the set of artifacts for each thematic, in six Thematic Meet sessions sized for 100 minutes each.

The research requirements described in Section 3 were used as criteria for evaluating the research results in the following analysis. From the viewpoint of fulfilling the artifact’s purpose, i.e., verifying whether the game produced what it set out to deliver during the TM sessions, evidence of compliance with the research requirements could be observed and collected.

About the behavior [R1] favored and stimulated by the dynamics of the game, during the TMs, we observed the manifestation of appreciative and collaborative attitudes of the students in interactions with the teacher and with their peers for the resolution of puzzles, the achievement of the objectives of the quests, progress in quizzes, as a result of the balanced implementation of mechanics of challenges, cooperation, and dispute, seeking to provide aesthetic experiences of collaboration and competition, as well as establishing bonds of companionship and the exercise of leadership to help the team progress in the game. Under the lens of Hookham’s engagement model, the balance established between the challenge provided by the game, and the student’s skill to deal with situations arising from this challenge, provides a state of focus, attention or concentration characterized as 'flow', in the search for success in carrying out an activity. This state, when achieved through a ludic activity, is observable in the player’s behavior and favors the cognitive process involved in the activity.

Manifestations of the 'flow' state were observed on multiple occasions during the game’s implementation, as evidenced by the students’ intense concentration in the activities, their sense of control while performing tasks (even in the face of distractions caused by other team members or intentionally introduced as obstacles in tasks), a diminished self-awareness as individuals prioritized building a collective team consciousness to achieve success, and an altered perception of time due to the students’ deep immersion in activity discussions. In the latter case, the teacher was crucial in coordinating the sessions to ensure they were completed within the allotted time.

Considering the opportunity to develop cognition [R2] expected by the application of the game, we observed that the situations provided by the game, which sought to stimulate student learning through the construction of a critical perception of the topics explored in the syllabus of the course, presented satisfactory learning outcomes. Even when compared with results obtained in the face-to-face teaching modality in the two previous academic semesters, the result of the application of the MASTER-PM expressed a slightly higher academic performance in the course, as depicted in Table 1.

Table 1-
Comparison of the metrics of the academic semesters.

Metrics	2019.1	2019.2	2020.3
Global Approval	12/15	35/42	54/60
Average Approval	4/15	24/42	52/60
Failure	0/15	1/42	1/60
Absenteeism	3/15	6/42	5/60
Class Average Grade	7,00	7,35	8,49
Students (n)	15	42	60

Sixty students were enrolled in the course, of which 54 completed the requirements and were approved. Among the 54 approved students, 52 out of 60 achieved an average passing grade, while the remaining 2 out of 60 were able to pass the course after taking the final exam. Only one student did not pass the final exam. Additionally, 5 out of 60 students dropped out of the course during the semester and were classified as absenteeism. Of these dropout students, 3 out of 5 attended at least one synchronous class, while the rest did not participate in synchronous meetings or submit any asynchronous activities.

By analyzing the data presented in Table 1 and scaling it proportionally to the size of each class, Figure 5 was constructed. It is important to exercise caution when comparing the research data, as the semesters 2019.1 and 2019.2 were conducted in a face-to-face format without utilizing the approach introduced by the game. Nevertheless, when considering the proportional size of each class, the following observations can be made: (1) there was an increase in overall approval rates; (2) there was a significant increase in approval rates based on average grades; (3) there was a notable reduction of the absenteeism, even in an experimental remote semester; and (4) there was a slight decrease in the failure rate compared to the 2019.2 semester. Furthermore, the data reveals a significant improvement of 1.14 points (15.51%) in the final overall class average when comparing the results of the 2020.3 semester with those of the 2019.2 semester.

Regarding the evaluation of the presence [R3] provided by the game, we can point to satisfactory evidence of the adoption of Simulation Games, Visual Novel, Management, PBL, and RPG strategies employed in the development of sets of artifacts associated with the teaching case adaptation that simulated a hotel endeavor with strategic problems, in order to stimulate debate between the players and their team to resolve the situations presented, as well as carry out the integration of the syllabus of the course with the inclusion of students in challenges, in which those knowledge, skills, and attitudes were essential.

Figure 5
Comparison of academic performance and absenteeism indicators per semester.

The adoption of these strategies provided the construction of (i) students’ interaction space with the game world; (ii) the narrative associated with the sequence of events in the game world (narrative backing), and (iii) characters that were part of this fictitious organizational context. The students’ sense of belonging to the world of the game was stimulated by the interaction with the Non-Playable Characters (NPC) that helped to conduct the game’s progression narrative (based on the proposed challenges), as well as by exercising or changing roles associated with these activities, for example, in the case of the weekly rotation of the PM role.

According to the model by Hookham and Nesbitt (2019), engagement immersion, and presence are considered flow components. While 'flow' is considered a cognitive state, the notions of immersion and presence suggest the affective foundations needed to achieve the flow condition. In other words, the affection aroused by the experience provided by the game is due to the player’s feeling of belonging to the game world and the immersion provided by their experience. This belonging and immersion, in turn, contribute to the establishment of behavior that provides a state of focus, attention, or concentration (flow), contributing to the player’s cognitive development (cognition) while facing the challenges proposed by the game.

The students’ engaged behavior in this state of flow was observed within the context of the 'presence' construct through reactions and feedback to the setting designed by the game in the investigation of the imaginary world, in the delimitation of the players’ area of action, favoring the sense of belonging of the students to the context of teamwork and the learning scenarios of the proposed challenges, encouraging student participation, attendance, and punctuality.

In addition to that, data regarding student participation, attendance, and punctuality during the TM sessions were also collected. Only the Meets from the 2^nd session onwards were considered to calculate these metrics, as the 1^st TM session was dedicated to introducing the course on the first day of the semester. To calculate participation, three students who did not engage in any activities throughout the course were excluded, resulting in a total of n=57 students. During each TM session, three key metrics were monitored to assess student participation: (A) initial engagement, which measured the number of students who completed the warm-up activity before each session to prepare for the meeting; (B) participation, which tracked the number of students who actively participated in the session; and (C) effective participation (players on task), which recorded the number of students who engaged in all activities during the TM session until its conclusion.

The overall average for initial engagement (A) was 83.5%, with the highest initial engagement recorded at 91.2% during the 6^th TM session and the lowest at 80.7% during the 3^rd, 4^th, and 5^th TM sessions. As for overall participation (B), the average was 97.2%, with the highest participation observed at 100.0% during the 4^th TM session and the lowest at 96.5% in the remaining TM sessions.The global average for effective participation (C) was 96.5%, with the highest recorded at 100.0% during the 4^th TM session and the lowest at 94.7% in the 5^th and 6^th TM sessions. Regarding attendance and punctuality (D), the average attendance rate was 96.8%, with the highest recorded at 100.0% during the 4^th TM session and the lowest at 94.7% during the 2^nd TM session.

In general, the level of student participation was significantly high in all Thematic Meets, reaching full participation throughout the 4^th MT. Such data point to the importance of the control planned by the mechanics designed and implemented by the dynamics of warm-up, cohesion (D1), leadership (D2), and knowledge (D3) since they produced mechanisms that stimulated the students’ participation and commitment in the activities developed during the synchronous meetings. These results corroborate researchers’ observations about student participation during the application of the game.

CONCLUSIONS

We can argue that the game was implemented as planned regarding MASTER-PM’s applicability. In evaluating its effectiveness, the collected evidence indicates that the game successfully fulfills the research requirements outlined in Section 3, effectively delivering the intended outcomes. Additionally, the research team reports that the game did not require excessive resources or effort for its implementation. The analysis of these criteria strongly supports the game’s feasibility, as it demonstrates a well-balanced combination of applicability, effectiveness, and reasonable effort.

Referring to the usefulness of MASTER-PM, the evaluation aimed to assess whether its implementation brings forth contributions or benefits in addressing the challenges of a remote teaching in project management. Based on the obtained results, it can be inferred that the integration of research fundamentals, such as Active Learning and game elements, along with RPG elements like role-playing, randomness, challenges, and game progression, proved to be valuable in overcoming obstacles associated with the specific class of problem that drove this research. The produced artifacts received positive evaluations and demonstrated improvement throughout the semester. The assessed metrics, including pass rate, class average, participation, and absenteeism rate, indicate the significant value of the game in fostering class engagement and achieving positive outcomes in the teachinglearning process, even within the context of remote teaching.

The analysis of these results also suggests that the application of the game was able to provide experiences of immersion and presence as an indication of affection ( affect) of the students for the activities they performed, which led them to a state of focus, attention, or concentration ( flow), providing the behavior conducive to a positive cognitive use ( cognition) of the syllabus that was being worked on. Thus, we can infer that the engagement phenomenon was evidenced by the application of the MASTER-PM, adding value to mitigate problems of the class that motivated its development.

Although the basis for comparing the results is not ideal for a broader analysis, the positive repercussions of applying the game on the academic performance and dropout of students in the class, as well as on attendance and punctuality, participation, and engagement of students during activities, indicate that the developed game is a reasonable solution for the class of problems: “difficulties faced in engaging the remote learning process”. Consequently, it is reasonable to consider that the research question established in Section 1 can be affirmatively answered in terms that the combination of active learning approaches and game elements can contribute to improving engagement in the remote teaching process satisfactorily, according to the results obtained.

Despite the encouraging results, some questions remain. For example, could a game like MASTER-PM be easily adapted to teach another course? In another Academic program? What would be the way to instantiate the system proposed by the game for the organizational environment? Questions like these could stimulate future work. Likewise, new questions can be formulated related to the characterization of the scientific knowledge produced by the application of the artifact, seeking to answer questions such as: “What can we theorize from the use of the artifact, thus enabling the advancement of scientific-theoretical knowledge in addition to technical development?, that is “What ‘new understanding/perception emerged’ about the phenomena related to the class of problems that motivated the research?”

We also analyzed research limitations and threats to its validity. Considering the construct validity, which is related to the measures used to evaluate the object of study, the main threat refers to the indicators used to evaluate the game, as they may not be able to represent the properties they operationalize fully. In order to minimize this threat, we define indicators for the properties that we consider relevant to the game’s application to mitigate the problems for which it was conceived. These properties were used as acceptance criteria in the MASTER-PM evaluations, as described in Section 3.

Considering the available data related to academic performance for comparison with the study findings, it is important to acknowledge that the data from 2019.1 and 2019.2 were obtained in traditional face-to-face teaching without using the game. In contrast, the study data were collected during a remote teaching scenario in 2020.3, incorporating the use of the game. Despite these differences, it is reasonable to conclude that the development, implementation, and the students’ subsequent evaluation of the game produced positive outcomes. It was evidenced by the achievement of good class academic results, active participation, and sustained engagement throughout the entire academic semester, even in a more challenging context than what is typically encountered in face-toface teaching environments. The scenario in which the game’s design and evaluation took place was novel and uncharted territory for the teachers and students involved. It was a time of learning and adaptation for everyone coping with the challenges established by the Covid-19 pandemic.

As for internal validity, the main threat is that the researchers conducted the studies carried out during the development and evaluation of the MASTER-PM. It keeps in mind the concern about how the researchers’ participation may have affected the results of the studies. To minimize this treatment, the researchers’ participation was limited to the teacher’s roles and those established in the game, as well as to activities in which their intervention was necessary (e.g., managing TM duration).In addition, more than one researcher analyzed and interpreted the data obtained from the studies.

The strategy adopted to evaluate the MASTER-PM was based on a case study. Wieringa and Daneva (2015) point out that one of the biggest threats in this context is the ability to generalize the specific findings of each case to different cases. Thus, the main threat to external validity in this study concerns the generalizability of the results. According to the same authors, the generalizations produced in the DSR are considered medium range, allowing to generalize beyond the case level, even if not intended to be universal. Therefore, although the specific problems may vary according to the question to be faced by each artifact produced, the proposed solutions for a certain “class of problems” should produce a satisfactory answer when applied to specific problems of that same class, allowing that solution to be “generalizable” in the context of the class in question. For example, in research based on case studies, once the results of specific case studies are obtained, it is possible to generalize to similar cases YIN, 2012.

Game artifacts were conceived as learning objects which can be recombined and reused in different configurations. Thus, it is reasonable to consider that the artifacts produced and evaluated in this specific context can be useful tools to help students and teachers face similar difficulties related to remote teaching in their own contexts. In other words, the artifacts and knowledge produced and the experience endured in conducting this research once applied to similar contexts, could be potentially useful instruments to contribute to people’s engagement in these new organizational contexts. Given that engagement is an ongoing and significant challenge that impacts organizational transformations and the learning process, future research could further explore how systematic approaches based on serious games for teaching and learning can enhance engagement within organizational learning processes. Additionally, such studies can shed light on how the knowledge generated from these endeavors can foster engagement during organizational transformations, whether technological or cultural.

• ALMAIAH, M. A.; AL-KHASAWNEH, A.; ALTHUNIBAT, A. Exploring the critical challenges and factors influencing the E-learning system usage during COVID-19 pandemic. Education and Information Technologies, [S. l.], v. 25, p. 5261–5280, 2020.
• ALMEIDA, F.; BUZÁDY, Z. Learning entrepreneurship in higher education through flow theory and FLIGBY game. International Journal of Virtual and Personal Learning Environments (IJVPLE), [S. l.], v. 9, n. 1, p.1–15, 2019.
• BISHOP, J.; VERLEGER, M. A. The Flipped Classroom: A Survey of the Research Paper. In: BEST OF COMPUTER IN EDUCATION DIVISION 2013, Atlanta, Georgia. Anais [...]. Atlanta, Georgia: 2013 ASEE Annual Conference and Exposition, 2013. p. 23–1200.
• BOUCHILLON, B. C.; STEWART, P. A. Games-based trust: Role-playing the administrative experience of immigrants. New Media &Society, [S. l.], v. 23, n. 3, p. 479–496, 2021. DOI: https://doi.org/10.1177/1461444819900557. Disponível em: https://journals.sagepub.com/doi/abs/10.1177/1461444819900557
  » https://journals.sagepub.com/doi/abs/10.1177/1461444819900557
• CEPE-UFPE, Resolução No 08-2020 - CEPE: Regulamenta o Calendário Acadêmico Suplementar para os cursos presenciais de graduação da Universidade, 2020.
• COSTA, A. C. S.; MARCHIORI, P. Z. Gamificação, elementos de jogos e estratégia: uma matriz de referência. InCID: Revista de Ciência da Informação e Documentação, [S. l.], v. 6, n. 2, p. 44, 2015. DOI: 10.11606/issn.2178-2075.v6i2p44-65.
• CRESWELL, J. W. Research design: Qualitative, Quantitative, and Mixed Methods Approaches. 3. ed. Los Angelis, CA: SAGE Publications Inc, 2009.
• DE ARAÚJO, S. A. et al. Applying computational intelligence techniques to improve the decision making of business game players. Soft Comput, [S. l.], v. 23, p. 8753–8763, 2019. DOI: https://doi.org/10.1007/s00500-018-3475-4. Disponível em: https://link.springer.com/article/10.1007/s00500-018-3475-4
  » https://link.springer.com/article/10.1007/s00500-018-3475-4
• DETERDING, S. Gamification: designing for motivation. Interactions, [S. l.], v. 19, n. 4, p. 14–17, 2012. DOI: https://doi.org/10.1145/2212877.2212883
  » https://doi.org/10.1145/2212877.2212883
• DUNLAP, J. C. Changes in Students’ Use of Lifelong Learning Skills During a Problem-based Learning Project. Performance Improvement Quarterly, [S. l.], v. 18, n. 1, p. 5–33, 2005. DOI: https://doi.org/10.1111/j.1937-8327.2005.tb00324.x
  » https://doi.org/10.1111/j.1937-8327.2005.tb00324.x
• GRUND, C. K. How games and game elements facilitate learning and motivation: A literature review. Lecture Notes in Informatics (LNI), Proceedings - Series of the Gesellschaft fur Informatik (GI), [S. l.], v. 246, p. 1279–1293, 2015.
• HEVNER, A.; CHATTERJEE, S. Design Science in Information Systems. [s.l: s.n.]. v. 28 DOI: https://doi.org/10.1007/978-1-4419-6108-2
  » https://doi.org/10.1007/978-1-4419-6108-2
• HITCHENS, M.; DRACHEN, A. The Many Faces of Role-Playing Games. International Journal of Role-Playing, [S. l.], v. 1, n. 1, p. 3–21, 2009.
• HOOKHAM, G.; NESBITT, K. A systematic review of the definition and measurement of engagement in serious games. In: PROCEEDINGS OF THE AUSTRALASIAN COMPUTER SCIENCE WEEK MULTICONFERENCE 2019, Sydney. Anais [...]. Sydney: Association for Computing Machinery, 2019. p. 1–10. DOI: 10.1145/3290688.3290747. Disponível em: https://dl.acm.org/doi/10.1145/3290688.3290747.
• HUNICKE, R.; LEBLANC, M.; ZUBEK, R. MDA: A formal approach to game design and game research. AAAI Workshop - Technical Report, [S. l.], v. WS-04-04, p. 1–5, 2004.
• ISF JOURNAL. Information Systems Frontiers - Editorial. 2021. Disponível em: http://www.springer.com/business+%26+management/business+information+systems/journal/10796. Acesso em: 3 nov. 2021.
  » http://www.springer.com/business+%26+management/business+information+systems/journal/10796.
• JONES, S. et al. Designing Seamless Learning Using Role-Playing Experiences. In: INFORMATION RESOURCES MANAGEMENT ASSOCIATION (org.). Mobile Devices in Education: Breakthroughs in Research and Practice (2 volumes). 1. ed. Hershey, PA: IGI Global, 2020. p. 392–420. DOI: 10.4018/978-1-7998-1757-4.ch024. Disponível em: https://www.igi-global.com/chapter/designing-seamless-learning-using-role-playing-experiences/242622
  » https://www.igi-global.com/chapter/designing-seamless-learning-using-role-playing-experiences/242622
• KOLB, D. A. Experiential Learning: Experience as The Source of Learning and Development. Prentice Hall, Inc., [S. l.], n. 1984, p. 20–38, 1984. DOI: https://dl.acm.org/doi/10.1016/B978-0-7506-7223-8.50017-4
  » https://dl.acm.org/doi/10.1016/B978-0-7506-7223-8.50017-4
• LACERDA, D. P. et al. Design Science Research: A research method to production engineering. Gestao e Producao, [S. l.], v. 20, n. 4, p. 741–761, 2013. DOI: https://doi.org/10.1590/S0104-530X2013005000014
  » https://doi.org/10.1590/S0104-530X2013005000014
• LUNA A. J. H. de O. et al. Foundations for an Agile Governance Manifesto: a bridge for business agility. Proceedings of the 13th CONTECSI International Conference on Information Systems and Technology Management, [S. l.], v. 13, n. June, p. 4391–4404, 2016. DOI: 10.5748/9788599693124-13contecsi/ps-4228.
• LUNA, Alexandre J. H. de O. Agile Governance Theory. 2015. 601 f. - Ph.D. Thesis. Informatics Center (CIn). Universidade Federal de Pernambuco (UFPE). Recife, Pernambuco, Brazil., [s. l.], 2015. Retrieved from: https://repositorio.ufpe.br/handle/123456789/15494.
• LUNA, A. J. H. de O.; MARINHO, Marcelo L. M.; DE MOURA, H. P. Agile governance theory: operationalization. Innovations in Systems and Software Engineering, [S. l.], v. 16, n. 1, p. 3–44, 2020. DOI: https://doi.org/10.1007/s11334-019-00345-3
  » https://doi.org/10.1007/s11334-019-00345-3
• MANYIKA, J. et al. Jobs lost, jobs gained: workforce transitions in a time of automation. San Francisco, California. Disponível em: http://hdl.voced.edu.au/10707/444873
  » http://hdl.voced.edu.au/10707/444873
• MICHAEL, D.; CHEN, S. Serious Games: Games that Educate, Train and Inform. 1. ed. Boston: Cengage Learning, 2006. Disponível em: https://anagroudeva.files.wordpress.com/2013/06/serious_games__games_that_educate__train__and_inform.pdf.
  » https://anagroudeva.files.wordpress.com/2013/06/serious_games__games_that_educate__train__and_inform.pdf.
• MITRE, S. M. et al. Metodologias ativas de ensino-aprendizagem na formação profissional em saúde: debates atuais. Ciências &saúde Coletiva, [S. l.], v. 13, n. 2, p. 2133–2144, 2008.
• PEFFERS, K. et al. A Design Science Research Methodology for Information Systems Research. Journal of Management Information Systems, [S. l.], 2007. DOI: 10.2753/MIS0742-1222240302. Disponível em: https://jyx.jyu.fi/bitstream/handle/123456789/63435/Design Science Research Process.pdf?sequence=1
  » https://jyx.jyu.fi/bitstream/handle/123456789/63435/Design Science Research Process.pdf?sequence=1
• PONTES, E. C. Objective Team - Role Playing Game para o Sistema de Apoio a Decisão da Escola de Aperfeiçoamento de Oficiais (OT-RPG / SAD-ESAO). 2017. Universidade Federal da Bahia, Salvador, 2017.
• RASHEED, R. A.; KAMSIN, A.; ABDULLAH, N. A. Challenges in the online component of blended learning: A systematic review. Computers \&Education, [S. l.], v. 144, p. 103701, 2020.
• SCHÖBEL, S. et al. A research agenda for the why, what, and how of gamification designs: Outcomes of an ecis 2019 panel. Communications of the Association for Information Systems, [S. l.], v. 46, n. 30, p. 706–721, 2020. DOI: https://doi.org/10.1016/j.infsof.2018.04.006
  » https://doi.org/10.1016/j.infsof.2018.04.006
• TRINKENREICH, B.; SANTOS, G.; BARCELLOS, M. P. SINIS: A GQM+Strategies-based approach for identifying goals, strategies and indicators for IT services. Information and Software Technology, [S. l.], v. 100, n. October, p. 147–164, 2018. DOI: 10.1016/j.infsof.2018.04.006. Disponível em: https://doi.org/10.1016/j.infsof.2018.04.006
  » https://doi.org/10.1016/j.infsof.2018.04.006
• TUCKER, B. The flipped classroom. Education next, [S. l.], v. 12, n. 1, p. 82–83, 2012.
• VALENTE, J. A. Blended learning e as mudanças no ensino superior: a proposta da sala de aula invertida. Educar em Revista, [S. l.], v. 30, p. 79–97, 2014.
• WIDODO, A. et al. From face-to-face learning to web base learning: How are student readiness? Premiere Educandum : Journal Pendidikan Dasar dan Pembelajaran, [S. l.], v. 10, n. 2, p. 149, 2020. DOI: https://doi.org/10.25273/pe.v10i2.6801
  » https://doi.org/10.25273/pe.v10i2.6801
• WIERINGA, R.; DANEVA, M. Six strategies for generalizing software engineering theories. Science of Computer Programming, S. l., v. 101, p. 136-152, 2015. DOI: 10.1016/j.scico.2014.11.013. Disponível em: http://dx.doi.org/10.1016/j.scico.2014.11.013
  » http://dx.doi.org/10.1016/j.scico.2014.11.013
• YOST HAMMER, E.; GIORDANO, P. J. Active Learning. In: Effective College and University Teaching: Strategies and Tactics for the New Professoriate. 2455 Teller Road, Thousand Oaks California 91320 United States: SAGE Publications, Inc., 2012. p. 99-106. DOI: 10.4135/9781452244006.n11. Disponível em: http://sk.sagepub.com/books/effective-college-and-university-teaching/n11.xml
  » http://sk.sagepub.com/books/effective-college-and-university-teaching/n11.xml
• YIN, R. K. Estudo de Caso: Planejamento e Métodos. 5a. Edição ed. Porto Alegre: Bookman Editora, 2015. https://hugepdf.com/download/estudo-de-caso-planejamento-e-metodos_pdf
• ZUMBACH, J.; RAMMERSTORFER, L.; DEIBL, I. Cognitive and metacognitive support in learning with a serious game about demographic change. Computers in Human Behavior, [S. l.], v. 103, p. 120–129, 2020. DOI: https://doi.org/10.1016/j.chb.2019.09.026 Disponível em: https://www.sciencedirect.com/science/article/abs/pii/S0747563219303607?via%3Dihub.
  » https://doi.org/10.1016/j.chb.2019.09.026

Data availability

Data citations

ISF JOURNAL. Information Systems Frontiers - Editorial. 2021. Disponível em: http://www.springer.com/business+%26+management/business+information+systems/journal/10796. Acesso em: 3 nov. 2021.

Publication Dates

• Publication in this collection
  18 Dec 2023
• Date of issue
  2023

History

• Received
  07 Jan 2023
• Accepted
  23 Aug 2023