Validation of the Brazilian Portuguese version of Evidence-based Practice Knowledge Assessment in Nursing (EKAN)

Validação da versão Brasileira da Evidence-based Practice Knowledge Assessment in Nursing (EKAN)

Validación de versión brasileña de la Evidence-based Practice Knowledge Assessment in Nursing (EKAN)

ABSTRACT

Objective

The Evidence-based Practice Knowledge Assessment in Nursing (EKAN) is an objective measure of evidence-based practice (EBP) knowledge. The aims of the project were to translate the EKAN instrument into Brazilian Portuguese, test its psychometric properties, and document a baseline assessment of students’ EBP knowledge.

Method

This study used methodological design. The EKAN-Brazilian Portuguese was administered to nursing students from two baccalaureate programs. Data were analyzed using Rasch procedures.

Results

123 students participated (87% ranged 19-25 years old; 60% were juniors; 38% perceived having no exposure to EBP). Validity indices showed the mean difficulty index = -2.48 to 2.07, the Infit (WMS) mean was.985 (SD = 2.13), and the Outfit (UMS) mean was.975 (SD = 0.17). Reliability indices showed sufficient item separation index (4.47); moderate person separation index (.47); high item reliability (.95); and low person reliability (.18). The mean EKAN sum score (max = 20) was 7.6 (SD = 2.13).

Conclusion

This study on the EKAN-Brazilian Portuguese version provided sufficient evidence for validity and reliability; the person reliabilities showed the need for improvement in incorporating EBP concepts and content in nursing programs.

Evidence-Based Practice; Health-Knowledge, Attitudes, Practice; Validation Study; Psychometrics

RESUMO

Objetivo

A Evidence-based Practice Knowledge Assessment in Nursing (EKAN) é uma medida objetiva do conhecimento sobre prática baseada em evidência (PBE). Os objetivos do projeto foram traduzir a EKAN para o português do Brasil, testar suas propriedades psicométricas e documentar avaliação de linha de base do conhecimento de estudantes sobre a PBE.

Método

A EKAN–Versão Brasileira foi aplicada a estudantes de dois programas de bacharelado em enfermagem e os dados foram submetidos a análise de Rasch.

Resultados

Participaram 123 estudantes (87%-idades entre 19-25 anos; 60%-primeiro ano; 38% sem exposição prévia a conteúdo de PBE). Índices de validade: dificuldade média = -2,48 a 2,07; Infit (WMS) médio = 0,985 (DP = 2,13); Outfit (UMS) médio = 0,975 (DP = 0,17). Índices de confiabilidade mostraram índice de separação suficiente (4,47); de separação de pessoa moderado (0,47); alta confiabilidade dos itens (0,95); baixa confiabilidade de pessoas (0,18). O escore médio na EKAN (máximo = 20) foi 7,6 (DP = 2,13).

Conclusão

Esse estudo produziu evidências de validade e confiabilidade da EKAN-Versão Brasileira; a confiabilidade de pessoas mostrou a necessidade de incorporar conceitos e conteúdos sobre a PBE nos programas de enfermagem.

Prática Clínica Baseada em Evidências; Conhecimentos, Atitudes e Prática em Saúde; Estudo de Validação; Psicometria

RESUMEN

Objetivo

La Evidence-based Practice Knowledge Assessment in Nursing (EKAN) es una medida objetiva del conocimiento sobre práctica basada en la evidencia (PBE). Los objetivos del proyecto fueron traducir el EKAN al portugués brasileño, testear sus propiedades psicométricas y documentar una evaluación de línea de base del conocimiento de estudiantes sobre la PBE.

Método

El EKAN-Versión Brasileña fue aplicado a estudiantes de dos programas de licenciatura en enfermería, los datos fueron sometidos a análisis de Rasch.

Resultados

Participaron 123 estudiantes (87%-edades entre 19-25 años; 60%-primer año; 38% sin contacto previo con material sobre PBE). Índices de validez: dificultad promedio = -2,48 a 2,07; Infit (WMS) promedio = 0,985 (SD = 2,13); Outfit (UMS) promedio = 0,975 (SD = 0,17). Los índices de confiabilidad mostraron índice de separación suficiente (4,47); de separación de persona moderado (0,47); alta confiabilidad de ítems (0,95); baja confiabilidad de personas (0,18). El puntaje promedio en la EKAN (máximo = 20) fue 7,6 (SD = 2,13).

Conclusión

El estudio produjo evidencias de validez y confiabilidad de la EKAN-Versión Brasileña; la confiabilidad de personas mostró necesidad de incorporar conceptos y contenidos sobre PBE en los programas de enfermería.

Práctica Clínica Basada en la Evidencia; Conocimientos, Actitudes y Práctica en Salud; Estudio de Validación; Psicometría

INTRODUCTION

Evidence-based practice (EBP) is a significant global movement and is pertinent to all health professions. Research shows the relationship between best practice and patient outcomes(¹-²). These results demonstrate the need to teach EBP in the academic setting. Since EBP is an international movement, a global instrument that objectively measures EBP knowledge is important to combat variability in assessment practices and diminish the use of subjective measures as self-ratings of knowledge correlate poorly with actual knowledge(³-⁵).

The EKAN instrument, originally developed in English as an objective measure of EBP knowledge in nurses and has been tested across levels of education in academic and practice settings(⁵-⁶). The EKAN was selected because it is an objective instrument with evidence of strong validity and reliability when tested in student nurse populations from different levels of nursing degree programs(⁶). It is a 20-item multiple choice examination that provides three answer options and requires a single-correct response. The EKAN focuses on measuring not only knowledge but also application of EBP principles and was developed from the Essentials for Baccalaureate Education for Professional Nursing Practice(⁷), and the Quality and Safety Education for Nurses Competencies(⁸). Items evaluate students’ knowledge of steps of the EBP process, interpretation of statistical results, and phases in a global quality improvement cycle.

The EKAN has already been validated in Spanish(⁴), English(⁶), and French(⁹). Validation in Japanese and Korean languages has been completed but not yet published. The team chose to develop and test the EKAN in the Portuguese language since, according to the World Health Organization, Portuguese is the sixth most natively spoken language in the world(¹⁰). Portuguese is the official or co-official language in Angola, Brazil, Cape Verde, Guinea-Bissau, Mozambique, Portugal, Timor-Leste, Macau, São Tomé and Príncipe(¹¹-¹²).

Establishing instrument validity in multiple languages will facilitate a standardized, global approach to documenting the state of nursing’s knowledge of EBP. Therefore, the objectives of this research were to translate the EKAN instrument into Brazilian Portuguese, and test the psychometric performance using Rasch model(¹³). An additional aim was to document a baseline assessment of baccalaureate nursing student’s knowledge of EBP at two universities in Brazil.

METHOD

DESIGN OF STUDY

This methodological study is part of a global research project that has translated and tested the EKAN in several languages.

SCENARIO

The data collection took place in the classroom of a private university (Institution A) and a public university (Institution B), both located in São Paulo, Brazil. The private university is a faith-based institution that enrolls 100 baccalaureate nursing students annually. Students complete 4,440 hours (theory and clinical) over ten semesters. The public university is maintained by the State of São Paulo and enrolls 80 full-time students each year. Students complete 4,170 hours (clinical and theory) over eight semesters. In both institutions, EBP concepts are taught, but not formally in any one course, nor threaded systematically throughout the whole curriculum. Both campuses offer internet access for searching research.

SAMPLE DEFINITION

Junior and senior nursing students from two baccalaureate programs were recruited and responses from the two groups were combined for psychometric analysis. The recommended sample size with Rasch models is between 100 responses but less than 200 responses per test item; when the size falls in this range, the model provides stable results(¹⁴).

ETHICAL ASPECTS

Institutional Review Board (IRB) approval for exempt research was granted at the home institution of the principal investigator in the US, and of the Escola de Enfermagem da Universidade de São Paulo (CAAE:63582316.0.1001.5392 / Report Nº 1.969.806, 2017). Both participant institutions authorized data collection by providing letters of support. All investigators hold current certifications in human subject’s protection training, either through the US-based Collaborative Institutional Training Initiative (CITI) or the Brazilian Comissão Nacional de Ética em Pesquisa (Conep). To protect student identity, neither the demographic nor the EKAN forms asked for any personal or identifying information. However, a consent form was required by both study sites. This study followed all the requirements of the Resolution nº 466 of the Conselho Nacional de Saúde (CNS) of the Brazilian Health Ministry(¹⁵).

PREPARATION OF THE INSTRUMENT AND OTHER FORMS

EKAN instrument. The original 20-item EKAN-English is an objective instrument that uses multiple-choice items to measure EBP knowledge as a unidimensional trait. Using Rasch analysis, psychometric testing on the EKAN-English provided strong evidence for validity with infit and outfit statistics centered around 1.0; and a high reliability index of.98 and person reliability of.66 when tested in baccalaureate nursing students(⁶).

The team took the validated EKAN-English and adapted it, using the forward and back-translation method proposed by Brislin(¹⁶). First, the Brazilian co-investigators translated the EKAN into Portuguese. Then, two bilingual individuals, of which one was a professional translator, independently back translated the EKAN to English. Neither of these two were involved in the original translation from English to Brazilian Portuguese. Finally, one of the original EKAN authors and the principle investigator compared the back translations to the original EKAN-English version. Five questions, noted below, required clarification or verification of single words in the stem or answer options.

In Question 2, the back translation used the trans-literated “distributed randomly” like in the Portuguese version “distribuídos aleatoriamente” rather than the original English term “random assignment.” The term used in the Portuguese version was verified as contextually correct. In Question 5, the back translated version used “What property are you evaluating” which is correct, while the Portuguese version used “se referindo” or “What property are you referring to.” The word was changed to “está avaliando” which more closely aligns with the original version. In Question 9, the back translation used “allocate,” as the Portuguese version also used the term “alocar.” Yet the original English term used was “allow,” which has a different connotation. This term was changed to “disponibilizar” which is contextually similar to the original meaning. In Question 13, the back translation used the term “affect the decision” rather than “influence the decision” as is in the original. The Portuguese version was checked and it correctly used “influenciarão,” and was retained as is. Finally, in Question 16, the back translation used “self-assessment” which is different contextually than the original version’s “self-report.” When checked, the Portuguese version used “self-report,” matching the original, and was not changed. In summary, two of the five questions needed word refinement to correlate more closely to the original.

With those changes, the EKAN-Brazilian Portuguese was finalized in June 2017. The Brazilian co-investigators also translated the Demographic Questionnaire and SIS from English to Brazilian Portuguese, but those forms did not require back translation.

Demographic questionnaire. The seven-item IRB approved Demographic Questionnaire asked subjects a series of questions about characteristics such as age, gender, and the completion of specific courses (EBP, statistics, research). It also asked subjects to rate the extent of their agreement with the following statement, “I am sure I can deliver evidence-based care” on a 5-point, Likert-type scale (1= Strongly Disagree, 5= Strongly Agree). This last question enabled data to assess correlations between student’s knowledge of EBP and self-rated confidence in providing evidence-based care. The demographic form did not require back translation.

Study Information Sheet (SIS). The SIS provided information regarding the study, stressed that participation was voluntary, the student could leave the room before finishing and actions would not negatively affect the student’s course grade or status within the program. The researchers posted an advertisement using the IRB approved “Study Information Sheet” (SIS) in the school of nursing buildings where students had classes. The co-investigators also provided verbal announcements about the study a few months prior to, the week of, and the morning of data collection. In addition, at Institution B, students received an email invitation along with the SIS. The SIS form did not require back translation.

DATA COLLECTION

Data collection occurred during scheduled, proctored sessions with the principal investigator and onsite co-investigators. Co-investigators communicated with the students using a standardized protocol to ensure information and distribution of study materials was similar in each session at both study sites. The protocol reiterated details of the study purposes, voluntary status of participation, and the multiple-choice format. Subjects were further instructed to not put any identifying information on the forms, and to not talk or access any electronic resources, books, or notes during the session. After the instructions, interested students signed the informed consent and received the study instruments. To reduce the threat of crosstalk among subjects, multiple sessions were scheduled in rapid sequence on the same day at each site.

Upon finishing each data collection session, the principal investigator collected the forms, placed them in an envelope, and transported the forms to a locked office for data entry at a later time. MS Excel was used for demographic management and Winsteps v. 4.5.0 software was used for Rasch Analyses(¹⁷). Two researchers performed data entry and data cleaning together to ensure reliability in these two processes. During inspection of the EKAN data, two subjects did not complete the last four questions; this was the only overlap of unanswered questions. One additional subject had seven non-responses but there was no overlap in any of the unanswered questions with the other two. The non-overlap indicated randomness to the non-responses which is important, as randomness of missing data provides unbiased item responses in the Rasch model(¹⁸). Therefore, the records from the three individuals were retained, making the total sample size N = 123 for all analyses.

DATA ANALYSIS

The team used Rasch model for dichotomous data to analyze data because it enables researchers to assess student’s ability and the instrument’s item difficulty and discrimination(¹⁹). Rasch models provide “information-weighted fit statistic (Infit)” and “outlier-sensitive fit statistic (Outfit”). While common Infit and Outfit statistics are considered local fit testing measures for the Rasch model, it is also recommended to perform global fit testing including determination Type I α error since the model is sensitive to sample size in relation to the number of items in the instrument(²⁰). The x²/df of Anderson’s Conditional Likelihood Ratio is recommended since this fit test is the most invariant to changes in sample size. Based on the instrument’s length of 20 items, a maximum cutoff of 1.32 would minimize the possibility of a Type 1 error using 68% CI. The mean x²/df for the Anderson’s Conditional Likelihood Ratio goodness of fit for the EKAN-Brazilian Portuguese is 1.22, supporting appropriate use of the Rasch model analytics.

RESULTS

DEMOGRAPHICS

A total of 123 students volunteered to participate. Most students at both study sites were between the ages of 19 to 25 years and female. All students self-identified as in the 5^th, 7^th, or 9^th semester. No student took more than 30 minutes to complete the forms. Sample characteristics are summarized in Table 1.

Table 1
– Sample characteristics of BSN students – São Paulo, SP, Brazil, 2018.

Characteristics	Institution A (n = 77)	Institution B (n = 46)	Total (N = 123)
Age
19-25 26-30 31-35 36-40	63 (82.0%) 6 (8.0%) 6 (8.0%) 2 (2.0%)	44 (96.0%) 2 (4.0%) - -	107 (87.0%) 8 (6.0%) 7 (6.0%) 2 (1.0%)
Gender
Female Male	66 (86.0%) 11 (14.0%)	39 (85.0%) 7 (15.0%)	105 (85.0%) 18 (15.0% )
Progression in nursing program
5th semester 7th semester 9th semester	37 (48.0%) 17 (22.0%) 23 (30.0%)	37 (80.0%) 9 (20.0%) -	74 (60.0%) 26 (21.0%) 23 (19.0%)
Country of origin
Brazil Angola South Korea	74 (96.1%) 2 (2.6%) 1 (1.3%)	46 (100.0%) - -	120 (97.6%) 2 (1.6%) 1 (0.8%)
Exposure to statistics course
Completed Enrolled Have not taken	25 (32.0%) 16 (21.0%) 36 (47.0%)	46 (100.0%) - -	71 (58.0%) 16 (13.0%) 36 (29.0%)
Perceived exposure to EBP content (number of days for EBP content exposition)*
Not taken any content < one 1-2 > 2 Took a whole course	15 (19.5%) 3 (4.0%) 5 (6.5%) 8 (10.0%) 46 (60.0%)	32 (69.5%) 5 (11.0%) - - 9 (19.5%)	47 (38.0%) 8 (6.5%) 5 (4.0%) 8 (6.5%) 55 (45.0%)
Self-rated confidence in providing EBP care
Strongly agree Agree Neither agree or disagree Disagree Strongly disagree	29 (38.0%) 41 (53.0%) 6 (8.0%) - 1 (1.0%)	4 (9.0%) 1 8 (39.0%) 20 (43.0%) 4 (9.0%) -	33 (27.0%) 59 (48.0%) 26 (21.0%) 4 (3.0%) 1 (< 1.0%)

• Note: *Neither institution has an entire course on EBP
• Abbreviation: EBP = Evidence-based Practice

Of the sample, 32% of students from Institution A already completed a statistics course versus 100% of students from Institution B. Since some of the questions on the EKAN involve interpretation of statistical results, this could have affected scores in one institution over the other. There was great variation in students’ reported exposure to EBP content. At Institution A, 60% perceived they had already taken an entire EBP course though co-investigators at the site reported there was no content in the curriculum yet. At Institution B, almost 70% perceived they had not taken any EBP coursework and co-investigators confirmed that students were not systematically exposed to this content. When asked to rate their confidence in providing evidence-based care, 91% of students from Institution A strongly agreed or agreed they could provide evidence-based care compared to 48% of students from Institution B.

PSYCHOMETRIC PERFORMANCE

Since students from both institutions had similar cultural background, age and gender distribution, in the same type of nursing program, and both programs provided limited exposure to EBP, the team pooled data from the two institutions and treated them as one group for psychometric testing. A description of the 20 EKAN items and corresponding fit statistics are presented in Table 2. All items fell within recommended infit WMS (Weighted Mean Square) limits while all, but one item of the outfit UMS (Unweighted Mean Square) parameters fell within the recommended boundaries. The one item that did not was.79, missing the recommended range by.01. When standardized, all items met recommended infit WMS and outfit UMS parameters. The team analyzed response patterns for each item in the EKAN. One item had 120 responses, two items had 121 responses, seven items had 122 responses, and 10 items had 123 responses; all items met the recommended number of responses (between 100-200 responses), thus increasing validity of results.

Table 2
– Descriptions of each EKAN item and Rasch fit parameters – São Paulo, SP, Brazil, 2018.

Item and Content Description	Difficulty (SE)	Infit		Outfit
		WMS	SWMS	UMS	SUMS
1. Purpose of regression versus other tests	1.57 (0.29)	0.92a	-0.31b	0.79	-0.79b
2. Sampling and study design	0.37 (0.21)	0.97a	-0.30b	0.98a	-0.14b
3. Purpose of Institutional Review Board	0.01 (0.19)	0.97a	-0.47b	0.96a	-0.46b
4. Distinguishing measures of central tendency	0.01 (0.19)	0.97a	-0.50b	0.96a	-0.45b
5. Distinguishing validity, reliability, and generalizability	0.64 (0.22)	0.96a	-0.30 b	0.91a	-0.61b
6. Proper use of pre-appraised evidence	0.25 (0.20)	0.99a	-0.07b	0.99a	-0.04b
7. Role of judgement in EBP decision making	-0.27 (0.19)	1.01a	0.17b	1.02a	0.41 b
8. Steps of the EBP process	-2.48 (0.26)	0.99a	0.02b	1.05a	0.31b
9. Facilitating EBP in practice settings	1.67 (0.30)	0.97a	-0.08b	1.02a	0.18b
10. Interpreting odds ratio	-0.05 (0.19)	1.01a	0.10b	1.00a	0.08b
11. Understanding credibility and bias	-0.74 (0.19)	0.94a	-1.23b	0.95a	-0.89b
12. Identifying steps in plan-do-study-act cycle for quality improvement	0.11 (0.20)	1.10a	1.48b	1.10a	1.06b
13. Priority of evidence, patient values, and clinical judgement in EBP decision making	0.01 (0.19)	1.01a	0.16b	1.05a	0.67b
14. Distinguishing causation from correlation in regression	0.10 (0.20)	0.97a	-0.49b	0.97a	-0.36b
15. Ranking of evidence quality (hierarchy)	-1.11 (0.19)	1.07a	1.08b	1.06a	0.78b
16. Strength of measurement approaches	-0.76 (0.19)	0.96a	-0.84b	0.94a	-0.98b
17. Role of PICOT question in evidence searching	0.15 (0.20)	0.96a	-0.58b	0.93a	-0.76b
18. Nurse-sensitive quality indicators	2.07 (0.35)	0.92a	-0.19b	0.87a	-0.27b
19. Understanding effect sizes	-0.23 (0.19)	1.05a	0.90b	1.03a	0.48b
20. Statistical v. clinical significance	-1.33 (0.20)	0.92a	-1.02b	0.93a	-0.80b

• Note: ^a Fit statistic falling within acceptability range 0.8 to 1.2, recommended by Bond & Fox⁽²¹⁾; ^b Standardized fit statistic falling within an acceptability range of -3.0 to 3.0, recommended by Meyer⁽²²⁾
• Abbreviations: EBP = Evidence-based Practice; PICOT = Population, Intervention, Comparison, Outcome, Timeframe; WMS = Weighted Mean Square; SWMS = Standardized Weighted Mean Square; UMS = Unweighted Mean Square; SUMS = Standardized Unweighted Mean Square

To test validity properties, WMS and UMS statistics were used. Reliabilities were tested, using the item and person separations and reliabilities. Results are seen in Table 3.

Table 3
– Pooled Validity and Reliability Indices for the EKAN-Brazilian Portuguese version – São Paulo, SP, Brazil, 2018.

Rasch Validity Indices	EKAN-Brazilian Portuguese	*EKAN-English	Benchmark
Difficulty index range θ (in logits)	-2.48 to 2.07	-2.00 to 2.8	-3.0 to 3.0
Infit Weighted MS Mean (SD)	.985 (2.13)	1.01
Infit Weighted MS Range	0.69 - 1.43	-0.7 – 1.6	0.7 to 1.3**
Infit Weighted MS Interquartile Range	0.23
Outfit Unweighted MS Mean (SD)	.975 (0.17)	1.02
Outfit Unweighted MS range	0.57 - 2.11	0.93 – 1.14	0.7 to 1.3**
Outfit Unweighted MS Interquartile Range	0.34
Rasch Reliability Indices	EKAN-Brazilian Portuguese	*EKAN-English	Benchmark
Item Separation Index (ISI)	4.47	7.05	> 2***
Person Separation Index (PSI)	.47	1.66	> 2
Item Reliability (IR)	.95	.98	>.80
Person Reliability (PR)	.18	.66	>.80

• Note: *Results from Spurlock & Wonder⁽⁶⁾; **Validity benchmarks recommended by Bond & Fox⁽²¹⁾; ***Reliability benchmarks recommended by Tennant & Conaghan⁽²³⁾

Validity indices. In reviewing parameters for dimensionality, EKAN-Brazilian Portuguese showed all 20 items with infit WMS ranging from 0.92 to 1.10 with a mean of 0.983 and median of 0.97. The outfit UMS ranged from 0.93 to 1.10 with one student outside the range at 0.79. The mean and median of the outfit UMS without the outlier was 0.98 and 0.98; with the outlier included, the outfit UMS was 0.97 and 0.97 for both mean and median. The included outlier shifted the mean slightly lower. The scale quality statistics with an adjusted variance of 0.98 for the items (SD = 0.99), centers on 1.0, confirming the newly translated EKAN instrument was unidimensional in measuring EBP knowledge. The fit statistics are within parameters by Linacre(¹⁷) of 0.5–1.5 as well as other researchers who use more conservative parameters of 0.7-1.3. These results show support for the items to fit the model, assessing different aspects of EBP knowledge as a unidimensional trait.

The mean difficulty index (θ = -2.48 to 2.07) met benchmark parameters set by Bond and Fox(²¹). The Infit WMS mean and standard deviation was 0.985 (SD = 2.13) with a range of 0.69 to 1.43. The Outlier-sensitive fits (outfit UMS) should also center on 1.0; those that do not, show either predictability (values less than about 1) or randomness/guessing of answers (values more than about 1). For this study, outfit UMS mean was 0.975 (SD = 0.17) (range of 0.057–2.11) and fell outside the benchmark of 0.6 to 1.4. This indicates some students were too predictable in their responses (e.g., A, A, A, A for each item) while other students did not have the knowledge and simply guessed at the answers.

Reliability Indices. Reliability, tested via item and person separations and item and person reliabilities, can also be seen in Table 3. The item separation index of 4.47 showed items could separate knowledge into categories; meanwhile the person separation index was 0.47 indicating the instrument could not distinguish between high and low performers.

Item reliability (IR) was.95 and met standard benchmarking of >.80, indicating items will perform similarly again; IR meeting benchmark also indicate the sample size was sufficient(²²). Person reliability (PR) was very low at 0.18, indicating there was very little variance in the sample’s knowledge (homogeneity) and has a narrow range in responses(¹⁶,²³). Therefore, the person reliability parameter could not be used to inform certain aspects of total reliability but did clearly indicate a need for the addition of EBP content in the curriculum. The homogeneity in scores was logical, given that EBP has not been taught consistently covered in either of the curricula.

PERFORMANCE OF ITEMS RELATED TO STUDENTS’ ABILITY

A Rasch Wright map (Figure 1) that of the 20 items on the EKAN, 13 items were at or above the mean difficulty (harder to answer); 10 of the 13 items fell within one standard deviation (S), being moderately difficulty to answer, and three items were very difficult to answer. Seven items fell below the mean, indicating there was a higher probability that students with average EBP knowledge were able to answer these items correctly. The distribution of students’ ability displays a normal parametric curve. Half the students (n = 60, 49%) were above the mean (M) and half were below the mean (n = 63, 51%). The mean (M) item difficulty is half a logit higher than mean (M) student ability.

Figure 1
– Rasch Wright map of Performance of 20 EKAN-Brazilian Portuguese items in relation to 123 students’ ability – São Paulo, SP, Brazil, 2018.

EBP KNOWLEDGE

Sum scores and ranges are seen in Table 4. Site A had lower scores (Note: one-third of students from site A had completed a statistics course); site B had higher scores (100% from site B had completed a statistics course). While minimum scores varied among the sites, the maximum score of 13 points was the same for both institutions.

Table 4
– Sum scores for EKAN-Brazilian Portuguese version (max score = 20) – São Paulo, SP, Brazil, 2018.

Statistics	Site A (n = 77)	Site B (n = 46)	Combined Sites (N = 123)	*EKAN-English
Mean (SD)	6.8 (2.09)	8.6 (1.92)	7.6 (2.13)	10.4 (2.13)
Median	6	9	8	*
Min - Max	3–13	5 - 13	3 - 13	5
Inter-quartile range	3.0	3.0	3.0	*

• Note: *No information available

DISCUSSION

The sample size provided sufficient responses for each item, and therefore delivered valid psychometric results for the EKAN-Brazilian Portuguese version. Results (item separation index and item reliability) showed strong item performance in the newly translated instrument. Person reliability showed limited consistency in student responses from one item to the next and person separation index reflected the restricted trait range of students’ EBP knowledge(¹⁶,²³). These results, while expected because EBP has not been systematically integrated into the curriculum at either study site, will be helpful to gauge the impact of teaching enhancements in courses and across curriculum. While the EKAN is intended for single administration, it has been used as a pre/post measure to evaluate the effectiveness of an EBP-related course(²⁴), which may be particularly helpful as programs begin to develop class content, teaching strategies, and assignments. The EKAN can also be used across levels of a program(²⁵) to provide feedback on whether the curriculum is supporting progressive knowledge development versus stagnation or decline. Results from the current study can be used as a baseline measure and, when repeated, can help gauge the impact of targeted interventions (e.g., educational exposures, faculty development) on students’ EBP knowledge development.

The one outlier score that fell outside the acceptable range in the Outfit UMS could have happened for various reasons. Since the Outfit UMS measures excessive predictability in choosing the same response (below the range) or excessive guessing of answers (above the range), one explanation is the student answered several questions with the same option without reading or taking care to answer the question based on knowledge or “best guess”. This is the most plausible answer since the other students’ scores fell within the range, and they all had the same level of exposure to the content.

Testing of dimensionality showed support for the items included in the EKAN and good fit with the Rasch model—the items measured the same dimension of EBP knowledge. These results are consistent with the original EKAN(⁶). The inability of the instrument to distinguish between high and low performers (as evidenced by the low person separation index) can be explained by the variance; achieving a higher person separation index occurs when variance in their knowledge ability is wider. In general, students’ knowledge ability was more narrow and lower since coverage of content was lower. An increase in ability variance would occur as knowledge increases; and this would occur when EBP content is increased in the curriculum.

The Wright map showed students’ difficulty to correctly answer more difficult EBP questions. When more EBP content is added to the curriculum, a closer alignment between mean item difficulty and student ability will occur. The EKAN will enable ongoing program evaluation to gauge the impact of course and curriculum enhancements.

The newly adapted instrument showed a mean EKAN sum score lower or similar when compared to other validation studies using the EKAN(⁴,⁶,⁹). The lack of correspondence between objective (EKAN) and subjective (confidence rating on ability to provide evidence-based care) measures is a phenomenon that has been shown in other disciplines(¹), in nursing students(⁴,⁶-⁷), and in practicing nurses(⁵). The low knowledge scores yet high confidence is consistent in all the EKAN validation studies. In the study to validate the EKAN-Spanish version, although 84% of 139 nursing students agreed or strongly agreed they were “confident in providing EBP care,” while the EKAN-Spanish mean score was only 6.5 (SD = 2.03) out of 20(⁴). In the original EKAN validation study in the US, 80% of 200 nursing students felt “confident in providing EBP care,” yet the mean score in this population was 10.4 (SD = 2.31)(⁶). In the EKAN-French validation study, 81% of 105 Haitian nursing students agreed or strongly agreed they were “confident in delivering evidence-based care” despite the mean score of the EKAN-French score was 5.9 (SD = 1.93) out of a maximum score of 20(⁹). These studies, as well as the current study, show student confidence in delivering EBP care is poorly correlated with EBP knowledge.

Results from the current study also showed incongruence or student’s confusion between their concept of research content (in this case primary research content), and EBP content. Each element of these results should encourage institutions to acknowledge the need for a more systematic approach to introducing EBP concepts in courses across the curriculum.

LIMITATIONS

One of the limitations of cross-sectional research is that results provide a snapshot or an assessment of a population at one point in time. Hence, the design makes it difficult to establish a strong association between EBP knowledge and exposure to EBP content. Additionally, the timing of the snapshot does not guarantee a representative sample, especially when sampling is non-random as was the case with this study, making a potential source of bias. The great variation in percentages of students who had completed a statistics course between both institutions also shows the impact of timing. This bias may have negatively contributed to the almost two-point difference in mean EKAN sum scores between sites, since some items required statistical interpretation.

GENERALIZABILITY

Positive psychometric outcomes were obtained due to four factors, showing evidence of generalizability. Sample size requirements for the model were met, the instrument was tested in two different academic institutions, and minimal data were missing. Further, the randomness of the missing data showed no serious nonresponse issues. All of these factors support representativeness and the ability to use the translated EKAN instrument in the target population of junior- and senior-level, Brazilian nursing students. Testing the EKAN-Brazilian Portuguese with students in other degree levels should also be conducted. As EBP knowledge and abilities are a growing expectation, this instrument will enable fruitful educational research in Brazil.

CONCLUSION

This study provides baseline results on baccalaureate nursing students’ EBP knowledge in Brazil and contributes to a larger body of work aimed at promoting the global use of evidence-based care. Validity results for the newly translated EKAN-Brazilian Portuguese contributes performance data for an instrument that was originally developed in the US and then translated and tested in Colombia, Dominican Republic, Japan, Korea, and Haiti. Ongoing, rigorous evaluation will be needed as Brazil’s faculty develop and test curricular content and teaching strategies for EBP. The EKAN enables a consistent approach for evaluation, allowing faculty to collaborate globally on nursing education and EBP.

REFERENCES

• ¹
  McAdam-Marx C, Tak C, Petigara T, Jones NW, Yoo M, Briley MS, et al. Impact of a guideline-based best practice alert on pneumococcal vaccination rates in adults in a primary care setting. BMC Health Serv Res. 2019;19(1):474. https://doi.org/10.1186/s12913-019-4263-2
  » https://doi.org/10.1186/s12913-019-4263-2
• ²
  Shulman LN, Browner AE, Palis BE, Mallin K, Kakade S, Carp N, et al. Compliance with cancer quality measures over time and their association with survival outcomes: the Commission on Cancer’s Experience with the quality measure requiring at least 12 regional lymph nodes to be removed and analyzed with colon cancer resections. Ann Surg Oncol. 2019;26(6):1613-21. https://doi.org/10.1245/s10434-019-07323-w
  » https://doi.org/10.1245/s10434-019-07323-w
• ³
  Zell E, Krizan Z. Do people have insight into their abilities? A metasynthesis. Perspect Psychol Sci. 2014;9(2):111-25. https://doi.org/10.1177/1745691613518075
  » https://doi.org/10.1177/1745691613518075
• ⁴
  Nick JM, Wonder AH, Suero AR, Spurlock D. A global approach to promoting evidence-based practice knowledge: validating the translated version of the Evidence-Based Practice Knowledge Assessment in Nursing into Spanish. Worldviews Evid Based Nurs. 2018;15(6):440-6. https://doi.org/10.1111/wvn.12327
  » https://doi.org/10.1111/wvn.12327
• ⁵
  Hagedorn Wonder A, McNelis AM, Spurlock D Jr, Ironside PM, Lancaster S, Davis CR, et al. Comparison of nurses’ self-reported and objectively measured evidence-based practice knowledge. J Contin Educ Nurs. 2017;48(2):65-70. https://doi.org/10.3928/00220124-20170119-06
  » https://doi.org/10.3928/00220124-20170119-06
• ⁶
  Spurlock D Jr, Wonder AH. Validity and reliability evidence for a new measure: the Evidence-Based Practice Knowledge Assessment in Nursing. J Nurs Educ. 2015;54(11):605-13. https://doi.org/10.3928/01484834-20151016-01
  » https://doi.org/10.3928/01484834-20151016-01
• ⁷
  American Association of Colleges of Nursing. The essentials of baccalaureate education for professional nursing practice [Internet]. Washington, DC: American Association of Colleges of Nursing; 2008 [cited 2020 Feb 22]. Available from: http://www.aacnnursing.org/portals/42/publications/baccessentials08.pdf
  » http://www.aacnnursing.org/portals/42/publications/baccessentials08.pdf
• ⁸
  Cronenwett L, Sherwood G, Barnsteiner J, Disch J, Johnson J, Mitchell P, et al. Quality and safety education for nurses. Nurs Outlook. 2007;55(3):122-31. https://doi.org/10.1016/j.outlook.2007.02.006
  » https://doi.org/10.1016/j.outlook.2007.02.006
• ⁹
  Nick JM, Roberts L, Adrien FM, Wonder AH. Establishing a global EBP tool to measure Evidence-Based Practice Knowledge: translating and testing the french version of EKAN. J Nurs Meas. Forthcoming 2020. https://doi.org/10.1891/JNM-D-19-00003
  » https://doi.org/10.1891/JNM-D-19-00003
• ¹⁰
  World Health Organization. ePORTUGUESe Programme celebrates 10 years [Internet]. Geneva: World Health Organization; 2015 [cited 2020 Feb 22]. Available from: https://www.who.int/eportuguese/en/
  » https://www.who.int/eportuguese/en/
• ¹¹
  Bouchard ME. Language shift from Forro to Portuguese: language ideologies and the symbolic power of Portuguese on São Tomé Island. Lingua. 2019;228:102712. https://doi.org/10.1016/j.lingua.2019.06.013
  » https://doi.org/10.1016/j.lingua.2019.06.013
• ¹²
  Sevilla-Pavón A. Computer assisted language learning and the internationalisation of the Portuguese language in higher education contexts. Dig Educ Ver [Internet]. 2015 [cited 2020 Mar 10];28(28):37-44. Available from: https://revistes.ub.edu/index.php/der/article/view/12033/pdf
  » https://revistes.ub.edu/index.php/der/article/view/12033/pdf
• ¹³
  Rasch G. Probabilistic models for some intelligence and attainment tests. Copenhagen: Danmarks Paedagogiske Institut; 1960.
• ¹⁴
  Jones P, Smith RW, Talley D. Developing test forms for small-scale achievement testing systems. In: Haladyna TM, Downing SM, editors. Handbook of test development. New York: Routledge; 2011. p. 487-525.
• ¹⁵
  Brasil. Ministério da Saúde. Conselho Nacional de Saúde. Resolução nº 466, de 12 de dezembro de 2012. Diretrizes e normas regulamentadoras de pesquisas envolvendo seres humanos [Internet]. Brasília, DF: Conselho Nacional de Saúde; 2013 [cited 2020 Feb 23]. Available from: https://conselho.saude.gov.br/resolucoes/2012/Reso466.pdf
  » https://conselho.saude.gov.br/resolucoes/2012/Reso466.pdf
• ¹⁶
  Brislin RW. Back-translation for cross-cultural research. J Cross Cult Psychol. 1970;1(3):185-216. https://doi.org/10.1177/135910457000100301
  » https://doi.org/10.1177/135910457000100301
• ¹⁷
  Linacre J. A user’s guide to WINSTEPS: Rasch model computer programs. Chicago: MESA; 2011.
• ¹⁸
  Waterbury GT. Missing data and the Rasch Model: the effects of missing data establishing global nursing mechanisms on item parameter estimation. J Appl Meas [Internet]. 2019 [cited 2020 Mar 10];20(2):154-66. Available from: https://www.ncbi.nlm.nih.gov/pubmed/31120433
  » https://www.ncbi.nlm.nih.gov/pubmed/31120433
• ¹⁹
  Tavakol M, Dennick R. Psychometric evaluation of a knowledge based examination using Rasch analysis: an illustrative guide: AMEE guide no. 72. Med Teach. 2013;35(1):e838-48. https://doi.org/10.3109/0142159X.2012.737488
  » https://doi.org/10.3109/0142159X.2012.737488
• ²⁰
  Baghaei P, Yanagida T, Heene M. Development of a descriptive fit statistic for the Rasch model. North Am J Psychol [Intenet]. 2017 [cited 2020 Mar 10];19(1):155-68. Available from: https://hal-hprints.archives-ouvertes.fr/hprints-01654099/document
  » https://hal-hprints.archives-ouvertes.fr/hprints-01654099/document
• ²¹
  Bond T, Fox C. Applying the Rasch model. 3rd ed. New York: Routledge; 2015. Chapter 1, Fundamental measurement in the human sciences; p. 274. https://doi.org/10.4324/9781315814698
  » https://doi.org/10.4324/9781315814698
• ²²
  Tennant A, Conaghan PG. The Rasch measurement model in rheumatology: what is it and why use it? When should it be applied, and what should one look for in a Rasch paper? Arthritis Rheum. 2007;57(8):1358-62. https://doi.org/10.1002/art.23108
  » https://doi.org/10.1002/art.23108
• ²³
  Meyer JP. Applied measurement with jMetrik. New York: Routledge; 2014. https://doi.org/10.4324/9780203115190
  » https://doi.org/10.4324/9780203115190
• ²⁴
  Cosme S, Milner KA, Wonder A. Benchmarking of prelicensure nursing students’ evidence-based practice knowledge. Nurse Educ. 2018;43(1):50-3. https://doi.org/10.1097/NNE.0000000000000398
  » https://doi.org/10.1097/NNE.0000000000000398
• ²⁵
  Serfass RL, Hagedorn Wonder A. You’re teaching evidence-based practice to BSN Students… But are they learning? Nurs Educ Perspect. 2018;39(3):172-4. https://doi.org/10.1097/01.NEP.0000000000000283
  » https://doi.org/10.1097/01.NEP.0000000000000283

• Financial support: Loma Linda University School of Nursing Seed Fund Grant.

Publication Dates

• Publication in this collection
  11 June 2021
• Date of issue
  2021

History

• Received
  12 Apr 2020
• Accepted
  19 Jan 2021