Effectiveness of a serious game for medical education on insulin therapy : a pilot study

Objective: We report the preliminary assessment of InsuOnline©, a serious game designed for medical education on insulin therapy. Materials and methods: We conducted a pilot study with 41 undergraduate medical students and Internal Medicine residents to assess the educational effectiveness of InsuOnline©, as compared to a traditional educational activity (lecture, cases discussion). Knowledge, skills and beliefs on insulin therapy were evaluated by a questionnaire applied before, immediately after, and 3 months after both interventions. Results: Mean knowledge/skills score was improved from 68% to 89% in traditional education group (n = 23; p < 0.001), and from 61% to 90% in game group (n = 18; p < 0.001). After 3 months, mean score decreased (to 80% in traditional education group, and to 78% in game group; p < 0.001 for both) but remained significantly higher than at baseline in both groups (p < 0.001 for both). Although mean score was lower in game group than in traditional education group at baseline (p = 0.04), no difference remained between groups either immediately or 3 months post-intervention. Score increment was better with the game (29%) than with traditional education (21%; p = 0.04). Beliefs improved in the game group only. Conclusions: InsuOnline© is at least as effective as a traditional educational activity for medical education on insulin therapy, and it can a good option for large-scale continuing medical education on diabetes. Arch Endocrinol Metab. 2015;59(5):470-3


INTRODUCTION
P rimary care physicians (PCPs) are the main health- care providers for most diabetic patients worldwide (1), but these professionals usually lack knowledge and confidence on several aspects of DM management (2), especially regarding insulin use (3), which contributes to clinical inertia ("the failure to advance therapy when indicated") (4), underuse of insulin (5), and poor glycemic control.Continuing medical education (CME), often advocated to optimize PCPs knowledge and practice (6), have small and short-lasting efficacy (7), demanding new educational methods.Digital games are promising tools for medical education (8), since they are well-acepted (9,10), promote experiential (active) learning (11), and are compatible with principles of Adult Education (12), but evidence on their effectiveness is still scarce (13).

MATERIALS AND METHODS
InsuOnLine © is a serious game developed by a multidisciplinary team, composed by endocrinologists, experts in medical education, and game designers, as described elsewhere (14).In the game, the player takes on the role of a medical doctor in a primary healthcare unit, who must evaluate an increasing-complexity series of patients with diabetes and choose the best therapeutic option to improve their glycemic control, usually requiring insulin initiation or adjustment (14).
To preliminarly assess the educational effectiveness of this game, we conducted a pilot study with a convenience sample of 41 undergraduate medical students (from third to sixth year) and Internal Medicine residents at Londrina State University.Allocation was made by subject preference.In the game group (13 students and 5 residents, 78% female, mean age 24 ± 4), subjects played a Web ver-A game for medical education on insulin Arch Endocrinol Metab.2015;59/5 sion of InsuOnLine © from a browser, in their own computers and free time.In the traditional education group (11 students and 12 residents, 56% female, mean age 26 ± 2), subjects attended a presential lecture and clinical cases discussion.Duration (3-4h) and contents were similar for both activities.Clinical problems were designed to depict common situations in primary care.Recommendations were drawn from Brazilian (SBD), ( 15) American (ADA), (16) and European (EASD) (17) diabetes guidelines, and adapted to be applicable in Brazilian primary health care.
Knowledge and skills on insulin therapy were measured by the score on a questionnaire with 32 multiple-choice items, applied at 3 time points: baseline, immediately after interventions, and 3 months after interventions (to assess content retention).Beliefs about insulin were assessed, at the 3 time points, by 13 Likertscale questions, freely adapted from Lakkis and cols.(18).In addition, some group-specific questions (15 Likert-scale questions about user satisfaction and perceived usefulness of the traditional activity, and 27 Likert-scale questions to assess game playability) were asked immediately after the corresponding interventions.
Mean knowledge/skills scores were compared within each group by ANOVA, with Bonferroni correction, and between groups at each time point by Students' t test, using Epi-Info 7 (CDC, Atlanta), with significance level p < 0.05.Beliefs were compared within each group by chisquare test, and between groups by Fisher exact test.Internal consistency (reliability) of the subscales on knowledge/skills on insulin, beliefs regarding diabetes/insulin, game playability, and impressions on the traditional activity was measured by estimation of Cronbach's alpha coefficient, using SPSS 14.0 (SPSS Inc., New York).
Research procedures were conducted in accordance with the Declaration of Helsinki.Research protocol was previously approved by Londrina State University review board (#15/2014).

RESULTS AND DISCUSSION
Knowledge and skills on insulin therapy, as measured by mean score on the questionnaire, were significantly improved immediately after both interventions (p < 0.0001 as compared to baseline, in both groups).Three months later, mean scores presented a small but significant decrease, compared to immediately after intervention (p = 0.0008 for both groups), but the 3-month scores were still significantly higher than at baseline (p = 0.0003 for both groups) (Figure 1), which demonstrates similar content retention with both activities.
In the comparison between groups, we observed a lower baseline score in the game group (p = 0.04 compared to traditional education group), that could be at least partly explained by the larger proportion of residents in the traditional education group.However, no difference between groups remained at immediate or 3-month posttests (Figure 1).This finding suggests that the game was at least as effective as the traditional activity, regarding knowledge/skills acquisition, or slightly more effective.In fact, mean absolute increment from baseline to immediate post-intervention score was significantly better with the game (29% versus 21% with traditional instruction; p = 0.04).
Beliefs about insulin were significantly improved only in the game group, where subjects abandoned the wrong belief that insulin initiation should be delayed until it is absolutely essential, and started believing (correctly) that insulin will be necessary at some point for most patients with diabetes.Data on the most relevant questions on insulin beliefs is presented in table 1.
All subjects, in both groups, said that the intervention has increased their knowledge, and that it would have impact on their professional practice.The traditional educational activity was considered "pleasant" by all subjects in that group, and the game was considered "fun" by all but one (95%) of subjects in the game group.All subjects in game group considered playing the game more effective for learning than seeing a lecture.
Reliability of the subscales used in this pilot study was estimated as "good" for the 32-item insulin knowledge/skills subscale (Cronbach's alpha = 0.823), "good" for the 15-item traditional activity impressions subscale (alpha = 0.815), "acceptable" for the 27-item game playability subscale (alpha = 0.778), and "poor" subscale reliability was not much improved by the deleting worst items (maximum alpha = 0.517 with 6 items), but the authors preferred keeping those questions anyway, because they evaluate personal opinions on insulin, deemed relevant, and because those questions were very alike the ones used in previous studies on PCPs beliefs (3,18), allowing comparison.In fact, despite the difference of subjects characteristics (the surveys from Middle East (18) and United States (3) were made with primary care physicians), insulin beliefs of our subjects at baseline were strikingly similar to those two other studies, except that insulin initiation was more often considered difficult in our study.

CONCLUSIONS
This is the first report on the effectiveness of a digital game for medical education on insulin therapy.Our preliminary results suggest that a well-designed game can be at least as effective as traditional instruction to improve knowledge, skills and beliefs on insulin.Games also present other advantages, such as easy dissemination, customizable content, and flexibility of use, that make them excellent tools for large-scale continuing medical education.In order to evaluate InsuOnLine © actual effectiveness, an adequately powered randomized controlled trial with PCPs, using refined questionnaires, is already in progress.We hope our game can contribute to reduce clinical inertia and improve quality of care for patients with diabetes worldwide.
Disclosure: The authors LAD and PAG are co-owners of the game described in this study.The other authors have no potential conflicts of interest relevant to this article.for the 13-item beliefs subscale (alpha = 0.280).Aiming to refine those instruments for a further randomized controlled trial, we found that deleting the worst items would generate shorter and improved subscales, with internal consistency considered "good" for a new 20-item knowledge/skills subscale (alpha = 0.839), "good" for a new 10-item traditional activity impressions subscale (alpha = 0.862), and "good" for a new 16-item playability subscale (alpha = 0.871).Beliefs

Figure 1 .
Figure 1.Mean scores on insulin knowledge/skills (in percentage of correct answers) at baseline, immediate post-intervention, and 3 months after intervention (traditional instruction or game).

Table 1 .
Insulin beliefs at baseline, immediately after, and 3 months after educational interventions (traditional instruction or game) p < 0.05 compared to traditional instruction.† p < 0.05 compared to baseline. *