Effectiveness of teaching strategies on the development of critical thinking in undergraduate nursing students : a meta-analysis

5 Universidade Federal de Viçosa, Viçosa, MG, Brasil. AbstrAct Objective: To evaluate the effectiveness of teaching strategies used for development of critical thinking (CT) in undergraduate nursing students. Method: Systematic review with meta-analysis based on the recommendations of the Joanna Briggs Institute. Searches were conducted in the following databases: PubMed, CINAHL, EMBASE, Web of Science, SCOPUS, LILACS, Cochrane CENTRAL, PsycINFO, ERIC, and a database of theses from four continents. The initial selection and evaluation of studies and assessment of methodological quality was performed by two reviewers independently. Results: Twelve randomized clinical trials were included in the study. In the metaanalysis of the four studies included that evaluated the strategy of problem-based learning (PBL), compared to lectures, the effectiveness of PBL was demonstrated with statistical significance (SMD = 0.21 and 95% CI = 0.01 to 0.42; p = 0.0434) for the development of CT in undergraduate nursing students, and the studies were homogeneous (chisquare = 6.10, p = 0.106). Conclusion: The effectiveness of PBL was demonstrated in the increase of overall CT scores. Further studies need to be conducted in order to develop, implement and evaluate teaching strategies that are guided in high methodological rigor, and supported in theoretical models of teaching and learning.


INTRODUCTION
It is increasingly important to train professionals that are critical, reflective, and able to make complex decisions based on knowledge of scientific evidence (1) .In this context, development of critical thinking (CT) skills has become a prerequisite for safe and effective clinical practice (2) , given the demonstrated positive and significant correlation between CT and problem solving (3) .
Critical thinking is an essential tool for nursing education, and its acquisition is one of the most desired results of higher education today (7) .CT should not be confused with intelligence; instead, it should be understood as a skill that can be taught (4) by developing logical and consistent thinking (5) that involves both cognitive and behavioral components (6) .
Critical thinking has been much discussed in nursing literature (8)(9)4) , and is considered one of the most important concepts in education (10) . Seeral studies (3,7,(10)(11) have emphasized the importance of preparing educators to teach CT skills through implementing active and innovative teaching strategies.Failure to teach CT at the university level can produce health care professionals devoid of this skill, which carries drastic consequences for the safety and proper treatment of patients in health care settings (11) .
Experimental studies have shown favorable results from strategies for Problem-Based Learning (PBL) (12)(13)(14) and the concept map (15)(16) in developing CT in nursing students.However, no systematic reviews (SR) of quantitative studies that sought to evaluate the effectiveness of these strategies were identified in the literature; thus, the need was indicated for an SR on qualitative studies of CT (9) .The authors of the present study identified SR on specific teaching strategies such as PBL (17)(18) and simulation (19) , which considered CT one of the outcomes assessed.In addition, only one SR (17) was identified that assessed the effectiveness of a teaching strategy for development of CT according to the variables and effect size in a meta-analysis.
Therefore, given the importance of training nurses capable of thinking critically, the Latin American Network for Research in Nursing Education (Red Iberoamericana de Investigación en Educación en Enfermería -RIIEE) has addressed this issue in a multicenter research project carried out in 16 countries in Latin America, and the present study is part of this project.Investigation of CT in an international research network enabled the authors to identify the need to synthesize evidence available in the literature on effective teaching strategies for development of CT.Thus, this study aimed to evaluate the effectiveness of teaching strategies for the development of CT in undergraduate nursing students.The following research questions were considered: What teaching strategies have been used and tested by randomized clinical trials to promote CT in undergraduate nursing education?and What is the effectiveness of teaching strategies to develop CT in undergraduate nursing students?

METHOD
Systematic review and meta-analysis guided by recommendations of the Joanna Briggs Institute (20) and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) (21) .The SR protocol was published in the JBI Library (22) , and registered in the database International Prospective Register of Systematic Reviews (PROSPERO) (23) .

databaSe SearcheS and Selection of StudieS
A preliminary search in the databases Cochrane Database of Systematic Reviews, Joanna Briggs Institute Library of Systematic Reviews, JBI COnNECT+, Database of Abstracts of Reviews of Effects (DARE), PubMed and PROSPERO yielded two SR (14,19) and one SR protocol (24) on the effectiveness of teaching strategies for the development of CT in nursing students.However, none of these studies was designed based on the aims of the present study.
Inclusion criteria for the present SR were: studies of randomized clinical trials (RCT); published in full in Portuguese, Spanish or English; and, having the aim of researching the effectiveness of teaching strategies on development of CT.Studies that did not show the abstract in full online in the databases searched, and those for which the full text was not obtained online or in the bibliographic archive, or after direct contact with the authors of the manuscript were excluded.No time limit was set for the search.
The PICOS strategy (P -population; I -intervention; C -comparison; O -outcomes and S -study) guided the development of the leading question of the SR, and was the basis for development of search strategies, so that the synonyms of each PICO component were combined using the Boolean operator "OR" and the components of the acronym were combined with the Boolean operator "AND".Undergraduate nursing students in any stage in the course were considered the study population.Teaching strategies used to develop CT were considered the study intervention.Studies that used a specific teaching strategy or traditional teaching or no specific intervention on the control group were considered the comparison.CT was considered the outcome, which was analyzed through skills and dispositions, and evaluated by means of CT measurement scales (25) .RCTs were considered for inclusion in the study.
Preliminary searches in the PubMed portal and CINAHL database were conducted to identify controlled and uncontrolled terms contained in the titles and abstracts of the studies.The searches were carried out by two reviewers independently, and the terms and search strategies developed by the primary reviewer were validated by the secondary reviewer and a librarian.
The search was conducted from September to December 2014.The databases used were PubMed, CINAHL, EMBASE, Web of Science, SCOPUS, LILACS, Cochrane CENTRAL, PsycINFO and ERIC.Thesis and dissertation databases from four continents (Africa, America, Europe and Australia) were used to search the gray literature: ProQuest Dissertations and Theses, Cybertesis, Digital Library of Theses and Dissertations of the University of São Paulo (Biblioteca Digital de Teses e Dissertações da Universidade de São Paulo -Brazil), Australian Digital Theses (ADT), Open Access Scientific Repository of Portugal (Repositório Científico de Acesso Aberto de Portugal -RCAAP), Academic Archive On-Line (DiVA), Theses Canada, Electronic Theses Online Service (EThOS), DART-Europe E-Theses Portal and the National ETD Portal.The terms used for the search included: P -"nursing student", "undergraduate", "baccalaureate"; I and C -"teaching methods", "learning, problem-based", "simulation", "case study", "concept maps", "lectures"; O -"thinking", "critical thinking"; S -"randomized controlled trial", "experimental", "evaluation study".
Selection of studies by title, abstracts and full reading was performed by the primary and secondary reviewers independently.The studies that generated some disagreement between the reviewers for inclusion were discussed with a third reviewer.
aSSeSSment of methodological Quality and data extraction CREMS version 5.02C and JBI-MAStARI were used to analyze the methodological rigor of the studies included, which was performed by two reviewers independently.The studies were evaluated using the Critical Appraisal Tools Randomized Control Trial/Pseudo-randomized Trial (MAStARI), the instrument for critical evaluation of randomized and controlled clinical trials.The instrument assessed whether the study participants were randomized; if the researcher who allocated the subjects and what was evaluated was blinded, as well as the study participants; analysis by intention to treat; if the subjects of the intervention and control groups at baseline were compared; if the groups were identically treated, except for the intervention group; if measurement of results in the control and intervention groups was identical and reliable; and finally, whether statistical analysis was appropriate.Studies that obtained a score of seven or higher of ten items evaluated were included, since the JBI leaves definition of this cut-off point for inclusion of studies to the researcher's criteria (20) .
After assessment of methodological quality, data extraction of studies selected for the SR was performed by two reviewers using the MAStARI data extraction instrument, in order to ensure the accuracy of the findings.The extracted data included details of the study population, intervention, inclusion and exclusion criteria and study results.Data that were not identified in the studies were obtained through contact with authors.

Summary of data and StatiStical analySiS
The results were presented descriptively, and studies evaluating the same teaching strategies and that presented homogeneity in their design were combined to carry out the statistical meta-analysis using the MIX 2.0 Pro software.
The standardized mean difference (SMD) is the difference between two means, divided by an estimate of the standard deviation between the groups, and is used when the outcome is assessed by different scales (20,26) .In the analysis of continuous variables with outcomes considered desirable, an SMD>0 indicates that the intervention was favorable (statistically significant), and that the results appear to the right of the forest plot (20,26) .
The effect size (Overall Z) is a generic term for estimating the effect of a study, so that researchers can say that the effect of the intervention is significant when it has a value of p less than 0.05 (20,26) .In contrast, the heterogeneity of the studies was assessed using the standard Chi-square, which is estimated by the method of inverse variance.Thus, studies are considered homogeneous when the value of p is greater than 0.05 (20) .
Since the studies included evaluated CT through different scales and were considered homogeneous, the effect size was assessed by the fixed effects model, standardized mean difference (SMD), and 95% confidence intervals were calculated for analysis.

RESULTS
A total of 2,949 studies were found in the selected databases and by means of cross-reference strategy.557 duplicates were excluded, and 78 studies were completely evaluated. first-and second-year students Concept map in the development of nursing care plans

Traditional method using column format 1 semester CCTDI
There was no statistically significant difference between the groups. students in their last semester Animated Pedagogical Agents (APA).Using an online tool, students performed three case studies, which they received in writing and that were narrated by APA.

Critical Thinking Process Test (CTPT)
There was no statistically significant difference between the overall CT scores in the groups after the intervention (p=0.561)continued...  (34) 95 secondyear students Concept map + Lectures Lectures 15 weeks Critical Thinking Scale (CTS) No statistically significant differences were found between the groups Brown (2008) (USA) (35) 67 finalyear students

High-fidelity simulation
with the use of HPS + Debriefing + Role Modeling High-fidelity simulation with the use of the HPS + Debriefing.

Professional Judgment Rating Form (PJRF)
No statistically significant differences were found between the groups The teaching strategies used and tested in the RCTs included: PBL, concept map, simulation, reflective writing, Role Modeling and Animated Pedagogical Agents.In the control group, lectures were used in 58.3% of the studies.The "traditional" strategy already used in the discipline appeared as a resource used in the control group in 25% of studies.In the remaining 16.7%, the control group's strategy was the same as that used in the intervention group, except for the model that the researcher wanted to test, which were the APA (32) and Role Modeling (35) .
It is notable that in only three studies was the intervention proposed based on a conceptual model, such as the 5E Learning Model (29) , Richard Paul's CT Instructional Model (30) and the theoretical model of Role Modeling (35) .It is also notable that in only three studies (14,28,31) was it mentioned that the people who implemented the intervention were trained or had experience with the strategy applied.

biaS riSk
Bias risks were identified in the studies based on JBI criteria (20) and the Cochrane Collaboration tool (Figure 2) to assess bias risk in RCTs (36)(37) .
With regard to selection bias, this was uncertain in 50% of studies, due to the paucity of data regarding generation of the random sequence, and in only four studies did the authors explicitly state that they maintained the randomization sequence hidden throughout the study period (27)(28)31,35) .
In regard to performance bias, in none of the studies was it possible to blind the participants, because educational interventions were being investigated.Regarding blinding of the professionals who applied the intervention, in only one study (29) was this described.Blinding of the researcher who analyzed the data was reported in only two studies (31,35) .
In regard to attrition bias, 33.4% of the studies did not perform analysis by intention to treat, and most had no losses in adherence.In reporting bias, most studies showed a low risk of bias.

meta-analySiS
Four RCTs were included using PBL as the intervention and lectures on the control group (14,27,29,33) .The combination of these studies in a meta-analysis, including 362 participants, demonstrated the effectiveness of PBL (SMD=0.32,95% CI=0.08 to 0.56; Overall Z=2.67, p=0.0078) when compared to lectures for the development of overall CT in undergraduate nursing students (Figure 3), with homogeneity being demonstrated (Chi-square=6.1, p=0.1068) between the studies.Three RCTs were also included that used the concept map as intervention and traditional teaching on the control group (15)(16)34) . The ombination of these studies in one metaanalysis, including 251 participants, did not demonstrate effectiveness of the concept map (SMD=-0.53,CI 95% -0.79--0.28;Overall Z=4.07, p<0.0001) when compared to traditional education for development of overall CT in undergraduate nursing students (Figure 4).It is noteworthy that statistical heterogeneity was identified between the studies (Chi-square=10.82,p=0.0044).

DISCUSSION
To the best of the authors' knowledge, this is the first SR that sought to identify the most effective teaching strategies for development of CT in undergraduate nursing students.Other SR have been developed on this topic, however they sought to: evaluate one strategy in isolation, such as PBL (17- 18) and simulation (19) ; to identify, in two recent SR, the relationship between learning styles and CT (38) ; and, one sought to explore how CT is perceived by teachers and nursing students through a SR of qualitative studies (9) .
From the results of studies that evaluated teaching of PBL and the meta-analysis performed of the RCT that used this teaching strategy, it can be said that PBL demonstrated effectiveness in increasing overall CT scores.This finding corroborates the results of the other two SR conducted that evaluated PBL (17)(18) .However, the SR for the present study advances knowledge with respect to the previous ones, since it included only the RCT; identified a statistically significant increase in CT skills using PBL, and not only abilities, as suggested in one SR that evaluated the effectiveness of PBL in development of CT (17) .In addition, the meta-analysis for the present study demonstrated greater homogeneity between the studies included in relation to this (17) .
The concept map is a widely used teaching strategy and discussed in nursing education (39)(40) as a promising tool for development of CT skills, however, there are few experimental studies testing its effectiveness.Meta-analysis of the three studies included that used the strategy of concept map compared to the usual teaching method did not demonstrate its benefit, although the studies were identified as heterogeneous, corroborating the findings of another study (41) which performed an uncontrolled trial with the same goal.
In addition to the PBL developed as an isolated teaching strategy, studies have been conducted evaluating the use of PBL associated with concept maps (42)(43) and even in webbased learning environments (44) .Both demonstrate the benefit of this association for the development of CT in students.Therefore, further studies need to be developed with the combination of different teaching strategies, since, although in this SR the strategy of concept map did not demonstrate effectiveness in the development of CT, studies have shown promising results of its association with PBL (42)(43) .
As regards other teaching strategies used (simulation, Role Modeling, reflective writing and APA), none of them proved to be effective to develop CT in undergraduate nursing students.However, isolated studies have been conducted with small samples, and the authors of the present study encourage future research on these strategies that is supported in theoretical models of teaching and learning.
It is no longer acceptable for a nursing instructor to tell their students to think critically, without first providing a detailed explanation of the process and its assumptions (45) , since in order to improve CT skills among university students, CT should be taught in the classroom (46) .Explicit teaching of CT has been studied in-depth by means of instructional models, in which the teacher teaches the theoretical concepts related to the development of CT capacities and skills, in addition to stimulating the development of these skills.Although the only study included in this SR (30) that used an instructional model for teaching CT did not show a significant increase in CT scores, studies have shown the effectiveness of this model in the development of CT abilities and skills in college students (46)(47) .Thus, the authors of the present study encourage research on the effectiveness of the instructional model combined with other teaching strategies, and highlight the need for a theoretical model of CT that better instructs teachers on instruction of CT for undergraduate nursing students.
Regarding instruments to evaluate CT, although all of those which were used in this SR have proven their validity and reliability according to statistical tests, only two studies used specific instruments for nursing students (29,32) , given the shortage of scales available in the literature.Therefore, there is a pressing need to develop tools to evaluate CT for undergraduate nursing students, and to consider the aspects inherent to the science and profession of nursing.
Another important issue discussed is the influence of culture in the use of PBL to promote CT (48) .In this SR, of the four studies that tested the use of PBL for the development of CT, three were developed in China and one in South Korea.This seems to be an important aspect to consider with regard to the external validity of the studies included in this SR, which demonstrated the effectiveness of PBL, since the cultures of Asian countries greatly differ from those in the West.Thus, the authors of the present study encourage the development of further experimental studies in Western countries to assess the most effective strategies for teaching CT.In this sense, we believe that the next steps of the multicenter project of the RIIEE will contribute to these demands, especially in Latin America, because it will enable approximation of different cultures in the context of an international network.
For teaching in nursing, the authors believe that this SR enables reflection on teaching practice in the training of nurses capable of thinking critically.Thus, the authors encourage teachers to implement, develop and research new teaching strategies to promote CT in nursing students.The findings of the present study demonstrated the effectiveness of PBL when the JBI guidelines were used as a methodological reference, because JBI is an organization internationally recognized for development of SR, since a comprehensive search of the published and unpublished literature was performed, and only studies with the highest level of evidence (RCT) were included, reflecting the credibility of the findings.
As a secondary study, the SR is limited to the availability of data from the primary studies included.In this way, the following limitations of this SR were identified: the difficulty in finding adequate information about the type of study design in the abstracts, as the authors put very different nomenclatures to designate experimental studies, particularly RCT, which may have produced a risk of search bias in this SR; there are several tools used to evaluate CT, few of which were specific to nursing, so that the validity and reliability of different instruments may have influenced the measurement of the outcome; finally, the difficulty of contacting authors to answer questions that would determine the inclusion of studies in the SR, which led to the exclusion of some studies due to lower methodological quality.It should be noted that for this SR, contact was attempted with six authors to obtain further information, though responses were obtained from only two.
As implications for future research, the authors of the present study believe the following to be necessary: clarification of the concept of CT in nursing literature; development and evaluation of teaching strategies that are based on instructional models of teaching CT; development of studies with larger, high-quality sample sizes, to withstand the effects it proposes to prove, using a pre-and post-test randomized format to control internal and external threats to the validity of the study; researchers who apply the intervention must be trained, and a standardized intervention needs to be applied, so that nurses need to develop the intervention method according to their actual educational context; and, it is urgent to develop an instrument capable of evaluating the development of CT, which considers the aspects inherent to the science and profession of nursing.
writing, Role Modeling and Animated Pedagogical Agents.Furthermore, based on the meta-analysis of the studies included, the effectiveness of PBL was demonstrated in the increased overall CT scores from homogeneous studies.This result confirmed our hypothesis that the teaching strategies that show better efficacy in RCT are those that involve the use of active methodologies.
Further studies need to be undertaken in order to develop, implement and evaluate new teaching strategies, as well as to test existing strategies in different cultures, to develop CT in undergraduate nursing students that are guided by high methodological rigor, supported in theoretical teaching and learning models, and developed in multicenter research projects.

Figure 2 -
Figure 2 -Summary of the assessment of risk of bias -São Paulo, Brazil, 2015.

Figure 3 -
Figure 3 -Meta-analysis of overall CT scores of studies that evaluated the strategy of PBL compared to lectures -São Paulo, SP, Brazil, 2015.

Figure 4 -
Figure 4 -Meta-analysis of overall CT scores of studies that evaluated the strategy of the concept map compared to traditional teaching -São Paulo, SP, Brazil, 2015.
All studies were published in English and had a level of evidence II, according to the JBI classification of quantitative studies(20); 50% were carried out in the United States and 33% in China; 75% were published in article format; 83% had grade 7 according to the JBI-MAStARI instrument; and, 66% of the studies were developed in the last five years.The complete characterization of the studies included is described in Chart 1. 1 -Characterization of the studies included -São Paulo, Brazil, 2015.