Learning lean with lego: developing and evaluating the efficacy of a serious game

Abstract

This study presents the use of a serious game developed to teach Lean philosophy. The structure of this game was built from theoretical elements and predefined learning events. Learning outcomes and student motivation were considered in the evaluation of the efficacy of the game. This serious game was applied to four groups of students with different profiles. The evaluation results were compared among these groups of students. It can be concluded that the serious game developed showed positive results in learning and motivation demonstrated by the students, regardless of the group analyzed. The main contributions to the literature presented in this article were the serious game (named 3L) that was developed and the efficacy evaluation method, considering the learning and motivation demonstrated in different profiles of students.

Keywords
Active learning; ARCS; Business simulation game

1 Introduction

The current job market has been looking for high qualified professionals with competencies that go beyond technical knowledge. Soft skills, or transversal competencies, such as communication, teamwork and leadership, have become a key part of students’ curriculums (Geithner & Menzel, 2016). However, such skills can be hardly developed in current traditional teaching environments (Ramazani & Jergeas, 2015; Córdoba & Piki, 2012) characterized as lecture-based delivery models (Fernandes et al., 2014). Experiential learning, on the other hand, makes students balance the deep meaning of ideas with the skills of applying them (Kolb et al., 2014).

In order to improve the existing educational and instructional systems, innovative learning approaches are being designed and tested. For instance, some teachers are already designing serious games (SG) to create a more interactive, participative, inductive, reflective and exploratory learning environment (Tao et al., 2015). During an SG, students can learn based on their own experience (Lopes et al., 2013).

This research aimed to develop and evaluate the efficacy of an SG for the teaching of Lean philosophy. For this, a set of theoretical elements and a set of learning events (conditions which have the function of improving student learning) have been defined. After, the structure of an SG developed for teaching Lean philosophy is presented. This structure is designed according to the theoretical elements and defined learning events. To evaluate the efficacy of SG, two instruments were applied. The first was developed and applied to measure the level of learning achieved with the SG. The second instrument is a combination of ARCS Motivation Model and Instructional Material Motivational Survey (IMMS), used for the analysis of motivation of the student participating in the SG.

The results of this evaluation were compared with 4 groups of students with different profiles. The aim was to check whether the SG presented different assessment results according to the profiles of students considered in this research.

This article is divided into seven items: introduction, literature review, methods used, development and application of SG, evaluation of the efficacy of SG, discussion of results and conclusions.

2 Background

2.1 Lean manufacturing theories and tools

Lean manufacturing has been widely adopted since the Japanese industry, led by Toyota, reached the top of the automotive industry (Womack et al., 1992). Even today, all sorts of companies see the lean manufacturing philosophy as a way to remain competitive in the market (Lindo-Salado-Echeverría et al., 2015).

Lean manufacturing, or the Toyota Production System, focuses on the absolute elimination of waste (Ohno, 1997). To eliminate all kinds of waste and its causes, a set of tools and concepts were developed, such as the 5 senses (5S), pull systems, kanbans, takt time, total productive maintenance (TPM) and single minute exchange of die (SMED).

An effective production system requires an organized, clean and sustainable work environment. The 5S is an organizational tool for sorting, stabilizing, cleaning, standardizing and sustaining work environments, and 5S applications can even raise systems’ productivity (Osada, 1992).

Next, it is also important to understand the difference between push and pull systems. In push systems, production happens independently and individually in each process, and there is no communication among these processes regarding their demands. This lack of communication can result in overproduction (Ohno, 1997). In pull systems, on the other hand, communication is established throughout the process flow, so that production takes place “just in time” (JIT) (Rother & Shook, 1999).

Process communication can be established with the use of kanbans. A kanban is a piece of information (a card, a space on the floor, an electronic message) automatically sent and used in production control (Lage Junior & Godinho Filho, 2008). The kanban sends information about the demand (type and amount of parts), and where and when it should be delivered.

In fact, demand plays a key role in lean systems. Pull production and rhythm must be triggered by the final customers’ demands. For this reason, the output rate of each process should remain as close as possible to the takt time, which is the available production time divided by the demand within the same period (Rother & Shook, 1999).

The real available production time is often compromised by two issues: process failure and setup execution. To overcome the first issue, total productive maintenance (TPM) is applied. TPM is the general involvement of the staff in the productive maintenance procedure, which involves failure identification and study, empowerment and continuous improvement (Nakajima, 1989).

The second issue is related to the system’s flexibility. Today, there is an increasing demand for customized products. In addition, many workstations are designed to produce more than one product. Thus, process setups must be executed as quickly as possible. The Single Minute Exchange of Die (SMED) is a theory and a set of techniques used to obtain setup time reductions (Shingo, 1985). Setup time minimization is necessary to obtain a low volume and balanced production mix.

2.2 Teaching lean manufacturing with serious games

Lean manufacturing is driven by a culture of problem solving that values the learning process as an essencial output (Badurdeen et al., 2010). For this reason, the authors state that the problem-based learning method (PBL) is the ideal instructional approach to teach lean manufacturing. PBL is a student-centred and active learning method based on interdisciplinary integration (Lima et al., 2012).

Several different PBL techniques were already developed and tested. Among them, serious games (SG), also known as problem-based games (Kiili, 2007), appear to be promising instructional approaches in different areas including manufacturing and engineering education (Pourabdollahian et al., 2012). Wouters et al. (2007) define the SG as the application of games with the aim of educating and learning. SGs place students in situations in which they take actions, make decisions, reflect and evaluate themselves (Geithner & Menzel, 2016).

There are different SGs designs to teach lean manufacturing, as those presented by Bicheno (1995), Billington (2004), Ozelkan & Galambosi (2008). For a higher number of lean SGs, see the survey conducted by Badurdeen et al. (2010).

Additionally, manipulative tasks and three dimensional (3D) objects can be included in SGs to enhance the learning process (Castro-Alonso et al., 2015). LEGO® is considered a valuable learning tool because it raises the level of interaction and engagement (Roos et al., 2004) and promotes reflective thinking (James & Brookfield, 2014).

2.3 Evaluating the efficacy of learning methods

Although innovative techniques are shown as promising learning methods, it is important to register students’ feedback, and observe how instruction materials impact their motivation, engagement, and the learning process (Huang & Hew, 2016). When it comes to SGs, it is also natural to question if they are really worth the effort, considering the costs associated with their development and management (Pasin & Giroux, 2011).

There are different efficacy evaluation procedures proposed in the literature. The Kirkpatrick evaluation model (Kirkpatrick, 1976) is recognized as a valuable evaluation model (Mollahoseini & Farjad, 2012). The Kirkpatrick model measures four aspects: reaction, learning, behavior and results (Kirkpatrick & Kirkpatric, 2006). So far in this study, only the first two aspects evaluated the SG. The behavior can only be measured after the students start using the knowledge, skills and attitudes learned from the SG, and the aspect related to the results corresponds to the changes brought to the organization or work environment of the students (Kirkpatrick & Kirkpatric, 2006). Since the SG was recently applied, the evaluation of the last two aspects is suggested for future research.

The first aspect, reaction, is related to how the participants felt about the SG. One of the ways to observe participants’ reaction is to identify whether their motivation increased after the SG. This can be achieved by the application of the ARCS model. According to Keller (1987), the level of motivation relies on four instructional principles: attention, relevance, confidence, and satisfaction (ARCS). Keller (1987) also developed the Instructional Materials Motivation Survey (IMMS), an integral part of the ARCS model used to measure students’ motivation (Huang & Hew, 2016). The IMMS is a standardized form with 36 statements. Participants choose the alternative that best describes their experience for each statement within a Likert scale. Applications of the IMMS can be found in Pittenger & Doering (2010) and Huang (2011).

The second aspect of the Kirkpatric’s evaluation model, the “learning” aspect, can be measured by tests taken before and after the SG, as in the work of Geithner & Menzel (2016). Significant results regarding possible learning outcomes can be highlighted by hypothesis tests, such as the 2-sample t or ANOVA. The Wilcoxon Signed-Rank Test can also be used for comparison purposes, especially when the data is not normally distributed, as shown by Pourabdollahian et al. (2012). The hypothesis tests previously mentioned are described by Montgomery & Runger (2012).

3 Methods

This research is divided into three steps. The first relates to the development and application of a serious game (SG), focusing on teaching some of the lean manufacturing concepts. The SG is called “3L” and reads as “triple L” (Learning Lean with Lego). The second stage concerns the evaluation of the efficacy of this SG, with regard to the student's learning level and motivation of the student to participate in this game. In the third and final stage of the evaluation, results among the 4 groups of students with different profiles will be compared.

In the step of creation and application of this SG, the experiences required were first defined. These experiences were divided into two groups: theory that must be inserted in the game (T group) and events that must be inserted in the game (E group). The SG was designed in four levels (S group) and applied to groups of students with specific characteristics (G group). These groups are described in section 4. The authors aim to verify whether theories (T group) and learning events (E group) are well-distributed over levels (S group).

At the stage of efficacy evaluation, two instruments were used to measure the level of learning and motivation. These results were compared among the four groups of students. In the measurement of learning level, a list of statements (^{Appendix A}) was prepared by the authors regarding the practical effects of some tools and concepts of Lean. Each student was asked to choose, on a scale, their level of agreement with statements. This list was delivered before the SG and immediately after the SG, keeping the same statements. To measure student motivation, the ARCS Motivation Model and Instructional Material Motivational Survey (IMMS) were used, as described in section 2.3.

4 Development and application of the SG

This step begins by defining the theory that should be inserted into the game (T group). The T group was defined in the following manner: 5S (T₁), Push System (T₂), Pull System (T₃), TPM (T₄), SMED (T₅), Kanban (T₆) and Takt Time (T₇). The elements of this group were defined based on consultations with academics working on teaching lean manufacturing concepts and tools. The authors of this study also work with this theme.

Some events were inserted in the game (E group). These events aim to increase the student learning with respect to the T group elements. Thus, the E group was defined in the following manner: the perception of the misalignment between demand and production (E₁), the perception of the effect of 5S (E₂), decision-making in relation to the positioning of the kanban (E₃), the perception of the effect of positioning the kanban (E₄), the perception of the effect generated by the high setup time (E₅), the perception of the difference between the process cycle time and takt time (E₆).

For this study, the SG was applied in four groups of students. In this paper, the term “student” was used to represent all participants. The groups are:

a) Group 1 (G₁) - This group was composed of associates of the management sector of a high-tech company. Men and women aged between 21 and 50 years. Some participants of this group had studied the Lean philosophy previously.

b) Group 2 (G₂) - This group was composed of students of business undergraduate course. Men and women aged between 21 and 23 years. None of the students had studied Lean philosophy before participating in this SG.

c) Group 3 (G₃) - This group was composed of students of industrial engineering undergraduate course. Men and women aged between 21 and 23 years. Before participating in this SG, all these students had studied Lean philosophy.

d) Group 4 (G₄) - This group was composed of students of an industrial engineering graduate course. Men and women aged between 25 and 40 years. Some students of this group had already studied Lean philosophy. The G₂, G₃ and G₄ refer to courses at the same University in Brazil.

To approach all the elements defined in groups T and E, a serious game initially published by Pinho et al. (2005) was chosen. The authors of this study adapted this SG with new levels. Thus, this new SG was named 3L (Lean Learning with Lego). Four levels designed for 3L were named S₁, S₂, S₃ and S₄ (S group). Each level had a duration of four minutes.

This SG was applied to each group of students separately on different days. The location where the SG was applied and the team of educators were the same for the four groups of students.

This SG was designed to be about three hours long and to be applied to approximately 20 students. During this SG four teams of at least three, and no more than five students, were formed. LEGO® pieces were used in this SG to simulate raw material, work in process, and finished goods. The SG is conducted by one or more people who will be called educators this article.

The functions of the students of each team are: operator A, operator B, finished goods (these three work stations form an assembly line), controller, and manager. The last two functions can be performed by the same student who has already been appointed to carry out either the function of operator A, operator B, or finished goods.

LEGO® bricks with different shapes and colors were mixed and delivered to each team. The teams are informed that they should assemble 3 different products. The only difference between three products is their color (white, red or blue). Each product has two assembly steps. Operator A performs the first stage of assembly and operator B completes the assembly. When the product is assembled, it is sent to the finished goods area to be shipped on a truck (trucks are drawn on sheets of paper). At the end of each level (four minutes), the controller must complete a worksheet that guides the team to calculate their inventory costs and their profit or loss on the sale of shipped products. The following will describe each of the four levels of SG (S₁, S₂, S₃ and S₄).

4.1 Level 1 (S1)

a) Rules for customer service

Each team is informed that they should ship 30 products (10 white, 10 red, and 10 blue). In each team, the student responsible for the finished goods area receives a card with 6 trucks (numbered from one to six), where each truck should receive 5 products according to the color indicated on the truck itself. The teams are informed that the six trucks must be fully completed in four minutes. This information allows the calculation of takt time (240 seconds / 30 products = 8 seconds per product).

b) Rules for the production

Operator A receives a list of production orders which is different from the list of operator B. These lists report the amount and color of the product that is to be assembled, composed of alternating batches of different numbers and different colors. Operators A and B are informed that they must follow the list in the correct sequence. The sequence of the products of the trucks is different from the list of production orders.

Operator B is the only one who must execute a setup operation before production of the lot of each color. This setup is done by assembling four extra LEGO® bricks. This assembly was purposely designed, in the SG, to consume a considerable amount of time.

c) Occurrences in this level

Because the LEGO® bricks are disorganized (region 1 of Figures 1a and 1b), operators A and B spend a lot of time finding the correct bricks for assembling the products. During this first level, operator B is suddenly interrupted by the educator for 20 seconds. This event simulates a broken machine (maintenance corrective). In addition, students realize setup operation causes significant delays in production.

Figure 1a
Level 1 of 3L game (1. Raw material disorganized; 2. Few shipped finished products). Source: Authors.

Figure 1b
Level 1 of 3L game (1. Raw material disorganized; 2. Few shipped finished products; 3. Work in process). Source: Authors.

Every 8 seconds (takt time) the educator announces that a product should be shipped in the truck. However, teams can not keep this pace. At the end of four minutes, the teams can not fill the trucks (region 2 of Figures 1a and 1b). Moreover, teams generate a large work in process inventory (region 3 in Figure 1b). As the lists of production orders are not synchronized with the demand represented in the trucks, few finished products are shipped. Figures 1a and 1b show students playing the level 1 of 3L game.

d) Financial results recorded by the controller

The teams record financial loss after S₁. This loss can be explained by three reasons: the high cost of work in process, the high cost of finished products not shipped, and low revenue obtained due to the shipment of only a few finished products.

e) Interval between S₁ and S₂

At the end of S₁, the teams discuss and the manager of each team announces to the educator the problems observed. One of the reported issues was that the students complained about the short available time to complete the task. In addition, some solutions are proposed to the teams. The educator manages the discussion among the teams. After, the educator provides some explanations, highlighting the reported problems and presenting the concept of 5S and TPM. These two theories are used in S₂.

4.2 Level 2 (S2)

a) Rules for customer service

The rules are the same as S₁.

b) Rules for the production

The educator distributes a new set of LEGO® bricks, which are organized and previously separated by each team. Now, only the parts necessary for assembly of the product are delivered (5S implemented). Furthermore, the interruption of operator B no longer happens (TPM implemented). The lists of production orders are the same as those used in S₁. The proposal at this level is to check the effect of 5S and TPM on the production line. The setup is not changed in this level.

c) Occurrences in this level

After implementing 5S (region 1 in Figure 2) and TPM, operators are able to produce more parts. A small increase in the number of products shipped is observed. However, students also observe an increase in the work in process and an increase of unshipped finished products. Students realize that 5S eliminates part of the waste, but there is still a misalignment between demand and production.

Figure 2
Level 2 of 3L game (1. Raw material organized after the 5S). Source: Authors.

d) Financial results recorded by the controller

The teams still report financial loss after S₂. The reasons are the same as indicated in S₁.

e) Interval between S₂ and S₃

One more time, the managers of the teams report the remaining problems. The educator introduces the concept of kanban and pull system. In addition, the educator explains the concept of production leveling and tells the teams that in S₃ the sheets containing the trucks will be replaced. The same number of products has to be shipped. However, the new shipping sheets provide a sequence with more batch interchanges. As a result, students predict major difficulties due to the setup. This finding is important for the process of active learning, since the students could reflect on the setup problem outcomes.

4.3 Level 3 (S3)

a) Rules for customer service

At this level, there is a new sequence of shipments composed of more batch interchanges.

b) Rules for the production

The bricks are rearranged, keeping the implemented 5S. The setup is not changed at this level. The educator presents the teams a sheet that represents a visual kanban, formed by three rectangles in the colors of three products. Each rectangle must contain only one product of the respective color. Each team must decide whether to put this visual kanban in the operation A (replacing the list of production orders in operation A and keeping the list of production orders in operation B) or in the operation B (replacing the list of production orders in operation B and keeping the list of production orders in operation A). The educator waits two minutes for teams to decide. Each team makes its own decision.

If the decisions made by all teams are equal, the educator does not interfere and accepts the decisions. The educator hopes the teams make different decisions, to compare the effects of decisions among the teams. However, the educator should not control the decisions of the teams. Regardless of the decision, it is important that the teams realize the effect of their decisions. After the effects are observed, the educator reveals the best decision and explains why it is in fact, the best choice. Therefore, regardless of decisions made by the teams, all students learn the best decision according to the lean principles.

c) Occurrences in this level

Teams which opted for the kanban to be positioned in operation B (closer to the end customer) have much better results than the other possible choice. This perceived difference in results is important for the active learning process, because the students could feel the impact of their own decisions.

With the choice of replacing the list of production orders of operation B with the kanban (region 2 in Figure 3), a greater number of finished products is shipped. This is because the pull system programming occurs in the area of finished goods, thus aligning production in B with customer demand (defined by trucks). However, there is also the formation of work in process inventory between operations A and B, due to the permanence of the list of production orders in operation A (region 1 in Figure 3).

Figure 3
The choice of the kanban positioned in operation B in Level 3 (1. List of production orders of operation A; 2. Kanban used in operation B). Source: Authors.

d) Financial results recorded by the controller

The recorded results show the effects of choices made by teams. The teams that had chosen to place the kanban in the operation A had a low income due to the small number of shipped products. Moreover, this choice generated high costs in inventory of not shipped finished products.

The teams that had chosen to place the kanban operation in B recorded profit. This is because of the revenue increase due to the larger number of products shipped. Although this choice also results in a large cost in the work in process inventory, there is a great reduction in the cost of finished products not shipped (only products that were in the kanban). In fact, all finished products were produced according to demand.

e) Interval between S₃ and S₄

At this point some information on the theory is presented to explain the differences between the choices made by teams, as alignment of production with demand. In S₃, the push and pull systems were working together.

Additionally, the concept of SMED and the unwanted effects of high setup time for production are explained. It is announced to the teams that two kanban controls will be used in S₄, so that production becomes fully converted to the pull system. In addition, improvements are announced in the setup.

4.4 Level 4 (S4)

a) Rules for customer service

There is no change of rules for customer service in S4.

b) Rules for production

A new visual kanban is implemented. Thus, the four teams start using the kanban in operations A and B, making the system totally converted to the pull process. Furthermore, the assembly of the setup operation is replaced by a much faster one (by applying the theory of SMED).

c) Occurrences in this level

Since all teams are now following the same rules, the facts observed between them are the same. The kanban positioned in Operation A (region 1 in Figure 4) decreases the work in process inventory (only 3 units now). Likewise, the kanban in operation B (region 2 in Figure 4) decreases the inventory of products not shipped. The setup operation becomes a quick activity (region 3 in Figure 4), allowing operator B to assemble different batches without a significant loss of time. With the production converted to the pull system and in line with customer demand, the teams are able to ship 30 products requested by the customer in less than 4 minutes (region 4 in Figure 4).

Figure 4
Level 4 of 3L game (1. Kanban of operation A; 2. Kanban of operation B; 3. Setup operation with quick assembly; 4 Full trucks). Source: Authors.

d) Financial results recorded by the controller

At the end of S₄, it is clear that the teams had profit due to the shipment of all products and the applied control to the formation of inventory. This financial result is the best of all the levels.

e) The end of the game

After S₄, the educator summarizes the events during the levels. Typically, during S₄, teams completely fill the trucks in less than four minutes. Nevertheless, the educator must wait a period of four minutes to complete the level. The teams that finish production in less than four minutes remain waiting, idly, until the end of the level. This waiting is important in this process of active learning because students realize that the cycle time of the production line became much smaller than the takt time.

To increase student awareness of the difference between the cycle time and takt time, the educator asks the teams to unload the shipped products on the first truck. After this, a rule is set for the reassembly of the products of this first truck. The student in the area of finished goods must pull a finished product kanban B only when the educator orders. This order will be given every 8 seconds (takt time). So every 8 seconds operators A, B and the student responsible for the finished goods area must start and complete their tasks. This way, it is possible to observe that the production rate established by the takt time is much less than the rhythm maintained by students in S₄. This discovery is an important part of this active learning process, since it makes students reflect on the results of S_1. As opposed to what the students felt in the end of S₁, it is totally feasible to deliver all products within four minutes.

4.5 The relationship among the sets T, S, E

After presenting the levels of this SG, it is possible to see when the theories (T) and events (E) were explored in this active learning process. Figure 5 shows this.

Figure 5
The use of theories, T_n, and occurence of events, E_n, within the levels S_n. Source: Authors.

Where:

• •

  3L structure game (S): S₁ = level 1; S₂ = level 2; S₃ = level 3; S₄ = level 4;

• •

  Theory inserted in the 3L game (T): T₁ = 5S; T₂ = Push System; T₃ = Pull System; T₄ = TPM; T₅ = SMED; T₆ = Kanban; T₇ = Takt Time;

• •

  Events in the 3L game (E): E₁ = the perception of misalignment between demand and production; E₂ = the perception of the effect 5S; E₃ = decision-making in relation to the positioning of the kanban; E₄ = the perception of the effect of kanban positioning; E₅ = the perception of the effect generated by the high setup time; E₆ = the perception of the difference between the process cycle time and takt time.

5 Efficacy evaluation of 3L game

For efficacy evaluation of the 3L game, two instruments were applied to four groups. The first instrument assessed the learning aspect of the students and the second evaluated the students’ reaction through IMMS/ARCS.

5.1 Learning evaluation

A list of five statements about the practical effects of some tools and concepts of Lean was developed and applied. This list was given to the students before they participated in the 3L game. Each student was asked to choose, on a scale, their degree of agreement or disagreement with each statement. After students choose the options, the educator takes the list.

After the 3L game, the list was handed back to students. Each student could at this time change their degree of agreement or disagreement with each statement. The goal is to verify that the student's experience participating in the 3L game changed his or her knowledge related to the Lean philosophy.

In each statement on this list (^{Appendix A}), the participant chose a unique alternative to express their opinion. Before the 3L game, a range of responses with scores from 1 to 5 was on the list. A value of 1 (undesirable) was assigned when the student completely disagreed with a true statement or when the student agreed completely with a false statement. On the other hand, the value 5 (desirable) was assigned when the student completely disagreed with a false statement or when the student agreed completely with a true statement.

The scale was changed in the list delivered after the 3L game. To be able to measure the increase of the student's conviction to agree or disagree with the statement, the range of answers was changed to scores between 0 and 6. The value 0 was assigned when the student, after the 3L game, further increased his conviction agreeing completely with a false statement or disagreeing completely with a true statement. In these cases, the 3L game would have further induced the student to the misinterpretation of the theory. Similarly, the value 6 was assigned when the student increased his conviction about the right choices.

The responses of groups, before and after the 3L game, were compared in order to test whether there was a significant increase in the knowledge of the students.

The responses of the participants in each of the 4 groups of students were grouped as follows: responses before 3L game (G₁B, G₂B, G₃B and G₄B) and responses after the 3L game (G₁A, G₂A, G₃A and G₄A). When analyzing the responses before and after each group, it was observed that their average had increased after the application of 3L game, as can be seen in Figure 6.

Figure 6
The interval Plot of group responses. Source: Authors.

To confirm whether each group had a significant increase in the score of right answers it was necessary to perform a Wilcoxon Signed-Rank Test (Miller & Miller, 1993; Walpole et al., 2009.). The justification for the use of this test is because samples are paired and nonparametric. The difference in the scores of responses from each group, before and after the 3L game, was called DG₁, DG₂, DG₃ and DG₄. The hypothesis test was defined as:

• -

  H₀: The difference between the score obtained after and before the 3L game on group n (DG_n), is less than or equal to zero;

• -

  H₁: The difference between the score obtained after and before the 3L game, of group n (DG_n), is greater than zero.

The results of the statistical test are presented in the Table 1.

Table 1
Wilcoxon Signed-Rank Test of the group difference.

	N	N for Test	Wilcoxon Statistic	P-value	Estimated Median
DG1	85	79	2670.5	0.000	1.500
DG2	95	92	4198.0	0.000	1.500
DG3	90	83	3090.5	0.000	1.000
DG4	75	68	2079.0	0.000	1.500

• Source: Authors

It is observed that in all groups analyzed the P-value was less than 5%. Thus, for all groups, the null hypothesis (H₀) must be rejected. This result shows that all groups showed a significant increase in learning. That is, in general, the groups improved their scores significantly, after participating in the 3L game. This change indicated greater knowledge of the subjects covered.

5.2 Motivation evaluation

The authors used the ARCS Motivation Model and Instructional Material Motivational Survey (IMMS), as described in section 2.3.

Each student received the IMMS form only at the end of the 3L game. The answers are separated into predetermined groups in accordance with the following criteria: Attention (A), Relevance (R), Confidence (C) and Satisfaction (S). The answers are scored using a Likert scale from 1 to 5 (1 - strongly disagree to 5 - completely agree). The maximum possible score in each criteria is obtained by multiplying the number of statements of this criteria by 5, and the minimum by 1.

With the results, the values of Cronbach's alpha were calculated (Cronbach & Shavelson, 2004), in order to have an estimate of the internal consistency of the data. The Alpha calculation of all respondents was 0.8672. When the analysis was done by group (G₁, G₂, G₃, G₄), the following respective alpha values were obtained: 0.8853, 0.7508, 0.8797 and 0.9303. Considering a desired Cronbach's alpha above 0.70, the form is classified as reliable.

Next, for the different criteria (Attention, Relevance, Confidence and Satisfaction) and for the group of students (G₁, G₂, G₃ and G₄), the average sum of the students’ scores was obtained using Equation 1.

$p{G}_X,y=\frac{{{\displaystyle \sum }}_{i=1}^{m{G}_X}{{\displaystyle \sum }}_{J=1}^{n_y}{s}_{ij}}{m{G}_x}$ (1)

Where:

$G_X$ = group of students (G1, G2, G3 and G4);

y = ARCS criteria (A,R,C, S);

$p{G}_X,y$ = average of G_x group in the criteria y;

$m{G}_x$ = number of students who participated in group x;

$n_y$ = number of statement in the criteria y;

$s_{ij}$ = score of the student i from group G_x in the question j of the criteria y.

The authors of this study normalized these values, thereby generating a motivation parameter (MP), as shown in Equation 2.

$M{P}_{G_x,y}=\frac{p_{G_x,y}-{\min }_y}{{\max }_y}$ (2)

Where:

${\min }_y$ = minimum possible sum of scores that a student can obtain in criteria y (the worst scenario);

${\max }_y$ = maximum possible sum of scores that a student can obtain in criteria y (the best scenario).

Table 2 shows the MP values obtained.

Table 2
MP values related to the four criteria and to the group of students.

		Attention	Relevance	Confidence	Satisfaction
	n	12	9	9	6
	Maximum	60.0	45.0	45.0	30.0
	Minimum	12.0	9.0	9.0	6.0
Group 1	Average(p)	55.2	39.6	40.2	28.0
	MP	90%	85%	87%	92%
Group 2	Average	57.9	40.6	40.1	28.8
	MP	96%	88%	86%	95%
Group 3	Average	53.8	42.1	41.3	28.4
	MP	87%	92%	90%	93%
Group 4	Average	55.9	40.1	40.2	27.9
	MP	91%	86%	87%	91%

• Source: Authors.

In all cases the MP values were greater than 85%. Thus, there has been a satisfactory degree of motivation as shown in Figure 7.

Figure 7
ARCS analysis. Source: Authors.

6 Discussion of results

In the learning evaluation, the group G₃, formed by students who had studied Lean philosophy before the 3L game, showed higher scores before the game. However, their increase in the score was also significant.

The group G₂, formed by students who had not studied Lean philosophy before the 3L game, had the highest increase in scores (Figure 6). This result showed that the game developed was able to successfully transmit the theory, even for students who had no prior knowledge on the subject.

Regarding motivation, the results presented in the Table 2 show MP values above 85%. These groups had different profiles, different age ranges and different levels of prior knowledge. Even with these different profiles, the results to motivation were similar and highly favorable for educators. The authors of this research attribute the positive results presented in the tests of learning and motivation to the following reasons:

• •

  The 3L game features a playful learning environment, mainly due to the use of Lego® pieces, as also observed by Roos et al. (2004) and James & Brookfield (2014);

• •

  The 3L game presents the theory (T group) and learning events (E group) in a well-distributed manner over the levels (S group), as shown in Figure 5;

• •

  Learning events of the 3L game do not generate obvious results to students, requiring reflection and team discussions on the experiences they had;

• •

  Teamwork is a strong point of the 3L game. Students have well-defined functions and can see the impact of their actions on the team result.

7 Conclusions

This research presented a detailed description of an SG developed for teaching Lean philosophy. The preparation of the levels of this game came from the need to include theoretical elements and predefined learning events. Once developed, the SG was evaluated according to the learning (the list of statements about Lean philosophy) and motivation aspects (IMMS/ARCS) displayed by students. Four groups of students were considered in this study. Each group had different characteristics, such as the age and the level of prior knowledge of Lean philosophy.

The results indicated that the four groups showed positive results in learning and motivation. Regarding the learning evaluation, all groups showed a significant improvement, since the P-values were less than 5% in the Wilcoxon Signed-Rank Test. In the motivation evaluation, the MP values obtained were higher than 85%, which can be interpreted as a satisfactory degree of motivation.

Finally, this study highlights that the use of an SG can improve the process of teaching and learning, allowing a good level of student learning in a pleasant and motivating environment. Similar results were reported by Geithner & Menzel (2016) and Pourabdollahian et al. (2012).

As proposals for future studies, the authors suggest: (i) analysing learning outcomes, according to Bloom´s Taxonomy; and (ii) evaluating the SG by all aspects presented in Kirkpatrick model measures (reaction, learning, behavior and results).

Appendix A List of statements to measure the level of learning achieved with the SG.

1) The 5s methodology can increase the production rates in operations.

(Score: 6) It increased my conviction that it is true.

(Score: 5) I am convinced that it is true that statement.

(Score: 4) I think it's true that statement.

(Score: 3) I do not know.

(Score: 2) I think it's false that statement.

(Score: 1) I am convinced that it is false that statement.

(Score: 0) It increased my conviction that it is false.

2) The decrease of the setup time does not allow the production in small batches.

(Score: 0) It increased my conviction that it is true.

(Score: 1) I am convinced that it is true that statement.

(Score: 2) I think it's true that statement.

(Score: 3) I do not know.

(Score: 4) I think it's false that statement.

(Score: 5) I am convinced that it is false that statement.

(Score: 6) It increased my conviction that it is false.

3) In a process “half pull, half push” is better put the pull control near to the final customers.

(Score: 6) It increased my conviction that it is true.

(Score: 5) I am convinced that it is true that statement.

(Score: 4) I think it's true that statement.

(Score: 3) I do not know.

(Score: 2) I think it's false that statement.

(Score: 1) I am convinced that it is false that statement.

(Score: 0) It increased my conviction that it is false.

4) The operator does not have how to know what and when he needs to produce using only the Kanban visual control.

(Score: 0) It increased my conviction that it is true.

(Score: 1) I am convinced that it is true that statement.

(Score: 2) I think it's true that statement.

(Score: 3) I do not know.

(Score: 4) I think it's false that statement.

(Score: 5) I am convinced that it is false that statement.

(Score: 6) It increased my conviction that it is false.

5) The use of visual Kanban induces the operator to produce only the necessary and at the right time.

(Score: 6) It increased my conviction that it is true.

(Score: 5) I am convinced that it is true that statement.

(Score: 4) I think it's true that statement.

(Score: 3) I do not know.

(Score: 2) I think it's false that statement.

(Score: 1) I am convinced that it is false that statement.

(Score: 0) It increased my conviction that it is false.

That is the after version. In order to obtain the before version must remove the highlight options for each statement (Scores 0 and 6).

• How to cite this article: Leal, F., Martins, P. C., Torres, A. F., Queiroz, J. A., & Montevechi, J. A. B. (2017). Learning lean with lego: developing and evaluating the efficacy of a serious game. Production, 27(spe), e20162227. http://dx.doi.org/10.1590/0103-6513.222716.

References

• Badurdeen, F., Marksberry, P., Hall, A., & Gregory, B. (2010). Teaching lean manufacturing with simulation and games: a survey and future directions. Simulation & Gaming, 41(4), 465-486. http://dx.doi.org/10.1177/1046878109334331
  » http://dx.doi.org/10.1177/1046878109334331
• Bicheno, J. (1995). The Buckingham lean game: manual. Buckinghamshire: PICSIE Associates.
• Billington, P. J. (2004). A classroom exercise to illustrate lean manufacturing pull concepts. Decision Sciences Journal of Innovative Education, 2(1), 71-76. http://dx.doi.org/10.1111/j.0011-7315.2004.00021.x
  » http://dx.doi.org/10.1111/j.0011-7315.2004.00021.x
• Castro-Alonso, J. J., Ayres, P., & Paas, F. (2015). Animations showing Lego manupulative tasks: three potential moderatores of effectiveness. Computers & Education, 85, 1-13. http://dx.doi.org/10.1016/j.compedu.2014.12.022
  » http://dx.doi.org/10.1016/j.compedu.2014.12.022
• Córdoba, J. R., & Piki, A. (2012). Facilitating project management education through groups as systems. International Journal of Project Management, 30(1), 83-93. http://dx.doi.org/10.1016/j.ijproman.2011.02.011
  » http://dx.doi.org/10.1016/j.ijproman.2011.02.011
• Cronbach, L. J., & Shavelson, R. J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64(3), 391-418. http://dx.doi.org/10.1177/0013164404266386
  » http://dx.doi.org/10.1177/0013164404266386
• Fernandes, S., Mesquita, D., Flores, M. A., & Lima, R. M. (2014). Engaging students in learning: findings from a study of project-led education. European Journal of Engineering Education, 39(1), 55-67. http://dx.doi.org/10.1080/03043797.2013.833170
  » http://dx.doi.org/10.1080/03043797.2013.833170
• Geithner, S., & Menzel, D. (2016). Effectiveness of learning through experience and reflection in a project management simulation. Simulation & Gaming, 47(2), 228-256. http://dx.doi.org/10.1177/1046878115624312
  » http://dx.doi.org/10.1177/1046878115624312
• Huang, B., & Hew, K. F. (2016). Measuring learners’ motivation level in massive open online courses. International Journal of Information and Education Technology (IJIET), 6(10), 759-764. http://dx.doi.org/10.7763/IJIET.2016.V6.788
  » http://dx.doi.org/10.7763/IJIET.2016.V6.788
• Huang, W. H. (2011). Evaluating learners’ motivational and cognitive processing in an online game-based learning environment. Computers in Human Behavior, 27(2), 694-704. http://dx.doi.org/10.1016/j.chb.2010.07.021
  » http://dx.doi.org/10.1016/j.chb.2010.07.021
• James, A., & Brookfield, S. D. (2014). Engaging imagination: helping students become creative and reflective thinkers. San Francisco: Jossey-Bass.
• Keller, J. M. (1987). Development and use of the ARCS model of instructional design. Journal of Instructional Development, 10(3), 2-10. http://dx.doi.org/10.1007/BF02905780
  » http://dx.doi.org/10.1007/BF02905780
• Kiili, K. (2007). Foundation for problem-based gaming. British Journal of Educational Technology, 38(3), 394-404. http://dx.doi.org/10.1111/j.1467-8535.2007.00704.x
  » http://dx.doi.org/10.1111/j.1467-8535.2007.00704.x
• Kirkpatrick, D. L. (1976). Evaluation of training. In R. L. Craig (Ed.), Training and development handbook: a guide to human resource development. New York: McGraw-Hill.
• Kirkpatrick, D. L., & Kirkpatric, J. D. (2006). Evaluating training programs: the four levels. San Francisco: Berrett-Koehler Publishers.
• Kolb, A. Y., Kolb, D. A., Passarelli, A., & Sharma, G. (2014). On becoming an experiential educator: the educator role profile. Simulation & Gaming, 45(2), 204-234. http://dx.doi.org/10.1177/1046878114534383
  » http://dx.doi.org/10.1177/1046878114534383
• Lage Junior, M., & Godinho Filho, M. (2008). Adaptations of the kanban system: review, classification, analysis and evaluation. Gestão & Produção, 15(1), 173-188. http://dx.doi.org/10.1590/S0104-530X2008000100015
  » http://dx.doi.org/10.1590/S0104-530X2008000100015
• Lima, R. M., Silva, J. M., Hattum-Janssen, N., Monteiro, S. B. S., & Souza, J. C. F. (2012). Project-based learning course design: a service design approach. International Journal of Services and Operations Management, 11(3), 292-313. http://dx.doi.org/10.1504/IJSOM.2012.045660
  » http://dx.doi.org/10.1504/IJSOM.2012.045660
• Lindo-Salado-Echeverría, C., Sanz-Angulo, P., De-Bendito-Martín, J. J., & Galindo-Melero, J. (2015). Aprendizaje del lean manufacturing mediante Minecraft: aplicación a la herramienta 5S. Revista Ibérica de Sistemas y Tecnologías de Información, 16, 60-75.
• Lopes, M. C., Fialho, F. A., Cunha, C. J., & Niveiros, S. I. (2013). Business games for leadership development: a systematic review. Simulation & Gaming, 44(4), 523-543. http://dx.doi.org/10.1177/1046878112471509
  » http://dx.doi.org/10.1177/1046878112471509
• Miller, J. C., & Miller, J. N. (1993). Non-parametric and robust methods in statistics for analytical chemistry (3rd ed.). London: Ellis Horwood PTR Prentice Hall.
• Mollahoseini, A., & Farjad, S. (2012). Assessment effectiveness on the job training in higher education. Case study: Takestan University. Procedia: Social and Behavioral Sciences, 47, 1310-1314. http://dx.doi.org/10.1016/j.sbspro.2012.06.817
  » http://dx.doi.org/10.1016/j.sbspro.2012.06.817
• Montgomery, D. C., & Runger, G. C. (2012). Estatística aplicada e probabilidade para engenheiros. Rio de Janeiro: Editora LTC.
• Nakajima, S. (1989). Introdução ao TPM. São Paulo: IMC Internacional Sistemas Educativos.
• Ohno, T. (1997). O sistema Toyota de produção: além da produção em larga escala. Porto Alegre: Artes Médicas.
• Osada, T. (1992). Housekeeping, 5S’S: Seiri, Seiton, Seiso, Seiket Su, Shitsuke; cinco pontos-chaves para o ambiente da qualidade total. São Paulo: Iman.
• Ozelkan, E. C., & Galambosi, A. (2008). Lampshade game for teaching lean manufacturing. In 2008 Industrial Engineering Research Conference. Vancouver, Canada.
• Pasin, F., & Giroux, H. (2011). The impact of a simulation game on operations management education. Computers & Education, 57(1), 1240-1254. http://dx.doi.org/10.1016/j.compedu.2010.12.006
  » http://dx.doi.org/10.1016/j.compedu.2010.12.006
• Pinho, A. F., Leal, F., & Almeida, D. A. (2005). Utilização de bloquinhos de montagem LEGO para o ensino dos conceitos do Sistema Toyota de Produção. Produção Online, 5(4), 5525-5532.
• Pittenger, A., & Doering, A. (2010). Influence of motivational design on completion rates in online self-study pharmacy-content courses. Distance Education, 31(3), 275-293. http://dx.doi.org/10.1080/01587919.2010.513953
  » http://dx.doi.org/10.1080/01587919.2010.513953
• Pourabdollahian, B., Taisch, M., & Kerga, E. (2012). Serious games in manufacturing education: evaluation of learners’ engagement. Procedia Computer Science, 15, 256-265. http://dx.doi.org/10.1016/j.procs.2012.10.077
  » http://dx.doi.org/10.1016/j.procs.2012.10.077
• Ramazani, J., & Jergeas, G. (2015). Project managers and the journey from good to great: The benefits of investment in project management training and education. International Journal of Project Management, 33(1), 41-52. http://dx.doi.org/10.1016/j.ijproman.2014.03.012
  » http://dx.doi.org/10.1016/j.ijproman.2014.03.012
• Roos, J., Victor, B., & Statler, M. (2004). Playing seriously with strategy. Long Range Planning, 37(6), 549-568. http://dx.doi.org/10.1016/j.lrp.2004.09.005
  » http://dx.doi.org/10.1016/j.lrp.2004.09.005
• Rother, M., & Shook, J. (1999). Aprendendo a enxergar: mapeando o fluxo de valor para agregar valor e eliminar o desperdício. São Paulo: Lean Institute Brasil.
• Shingo, S. (1985). A revolution in manufacturing: the SMED system. Portland: Productivity Press.
• Tao, Y. H., Yeh, C. R., & Hung, K. C. (2015). Validating the learning cycle models of business simulation games via student perceived gains in skills and knowledge. Journal of Educational Technology & Society, 18(1), 77-90.
• Walpole, R. E., Myers, R. H., Myers, S. L., & Ye, K. (2009). Probabilidade & Estatística para engenharia e ciências (8th ed.). São Paulo: Pearson Prentice Hall.
• Womack, J. P., Jones, D. T., & Roos, D. (1992). A máquina que mudou o mundo. Rio de Janeiro: Campus.
• Wouters, P., Tabbers, H. K., & Paas, F. (2007). Interactivity in video-based models. Educational Psychology Review, 19(3), 327-342. http://dx.doi.org/10.1007/s10648-007-9045-4
  » http://dx.doi.org/10.1007/s10648-007-9045-4

Publication Dates

• Publication in this collection
  2017

History

• Received
  09 Sept 2016
• Accepted
  13 Mar 2017