Effectiveness of a serious game for medical education on insulin therapy: a pilot study

Abstract

Objective

We report the preliminary assessment of InsuOnline©, a serious game designed for medical education on insulin therapy.

Materials and methods

We conducted a pilot study with 41 undergraduate medical students and Internal Medicine residents to assess the educational effectiveness of InsuOnline©, as compared to a traditional educational activity (lecture, cases discussion). Knowledge, skills and beliefs on insulin therapy were evaluated by a questionnaire applied before, immediately after, and 3 months after both interventions.

Results

Mean knowledge/skills score was improved from 68% to 89% in traditional education group (n = 23; p < 0.001), and from 61% to 90% in game group (n = 18; p < 0.001). After 3 months, mean score decreased (to 80% in traditional education group, and to 78% in game group; p < 0.001 for both) but remained significantly higher than at baseline in both groups (p < 0.001 for both). Although mean score was lower in game group than in traditional education group at baseline (p = 0.04), no difference remained between groups either immediately or 3 months post-intervention. Score increment was better with the game (29%) than with traditional education (21%; p = 0.04). Beliefs improved in the game group only.

Conclusions

InsuOnline^© is at least as effective as a traditional educational activity for medical education on insulin therapy, and it can a good option for large-scale continuing medical education on diabetes. Arch Endocrinol Metab. 2015;59(5):470-3

Video games; medical education; insulin; diabetes mellitus; computer-assisted instruction

INTRODUCTION

Primary care physicians (PCPs) are the main healthcare providers for most diabetic patients worldwide (¹), but these professionals usually lack knowledge and confidence on several aspects of DM management (²), especially regarding insulin use (³), which contributes to clinical inertia (“the failure to advance therapy when indicated”) (⁴), underuse of insulin (⁵), and poor glycemic control. Continuing medical education (CME), often advocated to optimize PCPs knowledge and practice (⁶), have small and short-lasting efficacy (⁷), demanding new educational methods. Digital games are promising tools for medical education (⁸), since they are well-acepted (⁹,¹⁰), promote experiential (active) learning (¹¹), and are compatible with principles of Adult Education (¹²), but evidence on their effectiveness is still scarce (¹³).

MATERIALS AND METHODS

InsuOnLine^© is a serious game developed by a multidisciplinary team, composed by endocrinologists, experts in medical education, and game designers, as described elsewhere (¹⁴). In the game, the player takes on the role of a medical doctor in a primary healthcare unit, who must evaluate an increasing-complexity series of patients with diabetes and choose the best therapeutic option to improve their glycemic control, usually requiring insulin initiation or adjustment (¹⁴).

To preliminarly assess the educational effectiveness of this game, we conducted a pilot study with a convenience sample of 41 undergraduate medical students (from third to sixth year) and Internal Medicine residents at Londrina State University. Allocation was made by subject preference. In the game group (13 students and 5 residents, 78% female, mean age 24 ± 4), subjects played a Web version of InsuOnLine© from a browser, in their own computers and free time. In the traditional education group (11 students and 12 residents, 56% female, mean age 26 ± 2), subjects attended a presential lecture and clinical cases discussion. Duration (3-4h) and contents were similar for both activities. Clinical problems were designed to depict common situations in primary care. Recommendations were drawn from Brazilian (SBD), (¹⁵) American (ADA), (¹⁶) and European (EASD) (¹⁷) diabetes guidelines, and adapted to be applicable in Brazilian primary health care.

Knowledge and skills on insulin therapy were measured by the score on a questionnaire with 32 multiple-choice items, applied at 3 time points: baseline, immediately after interventions, and 3 months after interventions (to assess content retention). Beliefs about insulin were assessed, at the 3 time points, by 13 Likert-scale questions, freely adapted from Lakkis and cols. (¹⁸). In addition, some group-specific questions (15 Likert-scale questions about user satisfaction and perceived usefulness of the traditional activity, and 27 Likert-scale questions to assess game playability) were asked immediately after the corresponding interventions.

Mean knowledge/skills scores were compared within each group by ANOVA, with Bonferroni correction, and between groups at each time point by Students’ t test, using Epi-Info 7 (CDC, Atlanta), with significance level p < 0.05. Beliefs were compared within each group by chi-square test, and between groups by Fisher exact test. Internal consistency (reliability) of the subscales on knowledge/skills on insulin, beliefs regarding diabetes/insulin, game playability, and impressions on the traditional activity was measured by estimation of Cronbach’s alpha coefficient, using SPSS 14.0 (SPSS Inc., New York).

Research procedures were conducted in accordance with the Declaration of Helsinki. Research protocol was previously approved by Londrina State University review board (#15/2014).

RESULTS AND DISCUSSION

Knowledge and skills on insulin therapy, as measured by mean score on the questionnaire, were significantly improved immediately after both interventions (p < 0.0001 as compared to baseline, in both groups). Three months later, mean scores presented a small but significant decrease, compared to immediately after intervention (p = 0.0008 for both groups), but the 3-month scores were still significantly higher than at baseline (p = 0.0003 for both groups) (Figure 1), which demonstrates similar content retention with both activities.

Figure 1
Mean scores on insulin knowledge/skills (in percentage of correct answers) at baseline, immediate post-intervention, and 3 months after intervention (traditional instruction or game).

In the comparison between groups, we observed a lower baseline score in the game group (p = 0.04 compared to traditional education group), that could be at least partly explained by the larger proportion of residents in the traditional education group. However, no difference between groups remained at immediate or 3-month posttests (Figure 1). This finding suggests that the game was at least as effective as the traditional activity, regarding knowledge/skills acquisition, or slightly more effective. In fact, mean absolute increment from baseline to immediate post-intervention score was significantly better with the game (29% versus 21% with traditional instruction; p = 0.04).

Beliefs about insulin were significantly improved only in the game group, where subjects abandoned the wrong belief that insulin initiation should be delayed until it is absolutely essential, and started believing (correctly) that insulin will be necessary at some point for most patients with diabetes. Data on the most relevant questions on insulin beliefs is presented in table 1.

Table 1
Insulin beliefs at baseline, immediately after, and 3 months after educational interventions (traditional instruction or game)

	Baseline (% agree)	Immediate (% agree)	3 Months (% agree)
I prefer to delay the initiation of insulin until it is absolutely essential
Traditional	48	18	43
Game	78	44	6*†
I believe that insulin therapy should be initiated by an endocrinologist
Traditional	4	0	4
Game	33*	0	6
I believe that the initiation of insulin therapy is one of the most difficult aspects of managing diabetes
Traditional	70	73	87
Game	83	83	94
I believe that most patients with type 2 diabetes will eventually need insulin, regardless of their adherence to treatment
Traditional	56	91	65
Game	39	100†	78
I believe that for most patients, the benefits of insulin therapy outweigh the risks
Traditional	70	95	96
Game	94	94	100
I believe that most patients would benefit from insulin therapy prior to developing diabetes complications
Traditional	74	91	91
Game	83	100	100
I believe that for most patients, training on the proper usage of insulin is not complicated
Traditional	9	18	30
Game	28	61*	56

• * p < 0.05 compared to traditional instruction. † p < 0.05 compared to baseline.

All subjects, in both groups, said that the intervention has increased their knowledge, and that it would have impact on their professional practice. The traditional educational activity was considered “pleasant” by all subjects in that group, and the game was considered “fun” by all but one (95%) of subjects in the game group. All subjects in game group considered playing the game more effective for learning than seeing a lecture.

Reliability of the subscales used in this pilot study was estimated as “good” for the 32-item insulin knowledge/skills subscale (Cronbach’s alpha = 0.823), “good” for the 15-item traditional activity impressions subscale (alpha = 0.815), “acceptable” for the 27-item game playability subscale (alpha = 0.778), and “poor” for the 13-item beliefs subscale (alpha = 0.280). Aiming to refine those instruments for a further randomized controlled trial, we found that deleting the worst items would generate shorter and improved subscales, with internal consistency considered “good” for a new 20-item knowledge/skills subscale (alpha = 0.839), “good” for a new 10-item traditional activity impressions subscale (alpha = 0.862), and “good” for a new 16-item playability subscale (alpha = 0.871). Beliefs subscale reliability was not much improved by the deleting worst items (maximum alpha = 0.517 with 6 items), but the authors preferred keeping those questions anyway, because they evaluate personal opinions on insulin, deemed relevant, and because those questions were very alike the ones used in previous studies on PCPs beliefs (³,¹⁸), allowing comparison. In fact, despite the difference of subjects characteristics (the surveys from Middle East (¹⁸) and United States (³) were made with primary care physicians), insulin beliefs of our subjects at baseline were strikingly similar to those two other studies, except that insulin initiation was more often considered difficult in our study.

CONCLUSIONS

This is the first report on the effectiveness of a digital game for medical education on insulin therapy. Our preliminary results suggest that a well-designed game can be at least as effective as traditional instruction to improve knowledge, skills and beliefs on insulin. Games also present other advantages, such as easy dissemination, customizable content, and flexibility of use, that make them excellent tools for large-scale continuing medical education. In order to evaluate InsuOnLine^© actual effectiveness, an adequately powered randomized controlled trial with PCPs, using refined questionnaires, is already in progress. We hope our game can contribute to reduce clinical inertia and improve quality of care for patients with diabetes worldwide.

REFERENCES

• ¹
  DeWitt DE, Hirsch IB. Outpatient insulin therapy in type 1 and type 2 diabetes mellitus. JAMA. 2003;289:2254-64.
• ²
  Rubin D, Moshang J, Jabbour S. Diabetes knowledge: are resident physicians and nurses adequately prepared to manage diabetes? Endocr Pract. 2006;12:17-21.
• ³
  Hayes RP, Fitzgerald JT, Jacober SJ. Primary care physician beliefs about insulin initiation in patients with type 2 diabetes. Int J Clin Pract. 2008;62:860-8.
• ⁴
  Peyrot M, Rubin RR, Lauritzen T, Skovlund SE, Snoek FJ, Matthews DR, et al.; on behalf of the International DAWN Advisory Panel. Resistance to insulin therapy among patients and providers: results of the cross-national Diabetes Attitudes, Wishes, and Needs (DAWN) study. Diabetes Care. 2005;28:2673-9.
• ⁵
  Riddle MC. The underuse of insulin therapy in North America. Diabetes Metab Res Rev. 2002;18:S42-9.
• ⁶
  Peyrot M, Rubin RR, Khunti K. Addressing barriers to initiation of insulin in patients with type 2 diabetes. Prim Care Diabetes. 2010;4(Suppl.1):S11-8.
• ⁷
  Sharp LK, Lipsky MS. Continuing medical education and attitudes of health care providers toward treating diabetes. J Contin Educ Health Prof. 2002;22:103-12.
• ⁸
  The New Media Consortium. NMC Horizon Report: 2014 Higher education edition. 2014. Available from: http://cdn.nmc.org/media/2014-nmc-horizon-report-he-EN-SC.pdf. Accessed on: Jan 2, 2015.
  » http://cdn.nmc.org/media/2014-nmc-horizon-report-he-EN-SC.pdf
• ⁹
  Kron FW, Gjerde CL, Sen A, Fetters MD. Medical student attitudes toward video games and related new media technologies in medical education. BMC Med Educ. 2010;10:50.
• ¹⁰
  Diehl LA, Souza RM, Gordan PA, Esteves RZ, Coelho ICM. Gaming habits and opinions of Brazilian medical school faculty and students: what’s next? Games Health. 2014;3:79-85.
• ¹¹
  Gee JP. What video games have to teach us about learning and literacy. New York: Macmillan; 2003.
• ¹²
  Bryan RL, Kreuter MW, Brownson RC. Integrating adult learning principles into training for public health practice. Health Promot Pract. 2009;10:557-63.
• ¹³
  Akl EA, Sackett KM, Pretorius R, Bhoopathi PSS, Mustafa R, Schünemann H, et al. Educational games for health professionals. Cochrane Database of Systematic Reviews. In: The Cochrane Library. 2011;09, Art. No. CD006411.
• ¹⁴
  Diehl LA, Souza RM, Alves JB, Gordan PA, Esteves RZ, Jorge MLSG, et al. InsuOnline, a serious game to teach insulin therapy to primary care physicians: design of the game and a randomized controlled trial for educational validation. JMIR Res Protoc. 2013;2(1):e5.
• ¹⁵
  Brazilian Diabetes Society (SBD). Guidelines from the Brazilian Diabetes Society 2013-2014. [In Portuguese.] São Paulo SP; AC Farmacêutica; 2014.
• ¹⁶
  American Diabetes Association. Standards of medical care in diabetes – 2014. Diabetes Care. 2014;37(Suppl 1):S14-80.
• ¹⁷
  Inzucchi SE, Bergenstal RM, Buse JB, Diamant M, Ferrannini E, Nauck M, et al. Management of hyperglycemia in type 2 diabetes: a patient-centered approach: position statement of the American Diabetes Association (ADA) and the European Association for the Study of Diabetes (EASD). Diabetes Care. 2012;35:1364-79.
• ¹⁸
  Lakkis NA, Maalouf GJ, Mahmassani DM, Hamadeh GN. Insulin therapy attitudes and beliefs of physicians in Middle Eastern Arab countries. Fam Pract. 2013:30:560-7.

Publication Dates

• Publication in this collection
  Oct 2015

History

• Received
  31 Jan 2015
• Accepted
  31 Aug 2015