Reliability and validity of the Lasater Clinical Judgment Rubric – Brazilian Version Confi abilidade e validade da Lasater Clinical Judgment Rubric – Brazilian Version Confi abilidad y validez de la Lasater Clinical Judgement Rubric – Brazilian Version

DOI http://dx.doi.org/10.1590/19820194201800038 Universidade Federal de Pernambuco, Recife, PE, Brazil. Escola de Enfermagem, Universidade de São Paulo, Ribeirão Preto, SP, Brazil. OH SU School of Nursing, Portland, USA. Escola Paulista de Enfermagem, Universidade Federal de São Paulo, São Paulo, SP, Brazil. Confl icts of interest: nothing to declare. Abstract Objective: To evaluate the reliability and validity of the Lasater Clinical Judgment Rubric Brazilian Version in a sample of nursing students. Methods: Methodological design study conducted at a public institution of higher education in the southeast region of Brazil. Preliminarily, was analyzed the clarity of behaviors described by eight students in each level of the 11 dimensions of the instrument. Next, was collection of data on the evaluation of psychometric properties with self-application by 179 students. Participants were grouped in junior (1 and 2 terms n=115) and senior (3 and 4 terms n=64) students. The following psychometric properties were analyzed: discriminant validity, reliability and dimensionality. Results: Through the Brazilian version of the rubric were differentiated the two groups of students (p-value <0.05) in the 11 dimensions evaluated. Stability was verifi ed by test-retest (Intraclass Correlation Coeffi cient – ICC: 0.88). Internal consistency was obtained for the global instrument (Cronbach’s alpha: 0.899) and for phases of Noticing (α=0.75), Interpreting (α=0.64), Responding (α=0.78) and Refl ecting (α=0.63). Dimensionality validity by confi rmatory factorial analysis (CFA) obtained results of composite reliability (CR) above 0.7, and average variance extracted (AVE) higher than 0.5 in all phases. The discriminant validity of the factorial model by the Fornell-Larcker criterion and cross loadings confi rmed the theoretical structure of the rubric original version. Conclusion: The evaluation of the psychometric properties of the Brazilian version of the rubric showed evidence of reliability and construct validity of the instrument for measuring the development of nursing students’ clinical judgment.


Introduction
Nursing training for accurate care practice and safe care requires that professionals have assertive mental attitudes in clinical reasoning, judgment and decision making.The development of cognitive and technical ability and improvement of these actions are fundamental to identify individuals' needs through diagnostic reasoning and the direction of the care plan for therapeutic reasoning. (1,2)hen dealing with clinical reasoning, particularly the processes by which nurses and other professionals make their judgments, in the process of generating hypotheses, Tanner (3) considers the deliberation on evidences and the use of reasoning by recognition of pattern and intuitive understanding.Clinical judgment is a complex activity that requires professionals' flexibility and ability for recognizing important aspects in an undefined clinical situation for a proper interpretation of findings and a satisfactory answer.It also requires nurses' knowledge of pathophysiology, of patients' clinical manifestations, and the understanding of patients' and family's experiences of illness, their physical, social and emotional strengths, and coping resources.
Thus, when nurses initiate care, their clinical judgment is influenced by previous experiences, knowledge about patients and their pattern of response, the context in which the situation occurs, and the culture of the health unit.Moreover, by standards of reasoning, whether analytical, intuitive and/or narrative, and reflection on practice. (3)According to the Clinical Judgment model proposed by Tanner (3) , the actions developed by nurses go through four phases, namely: Noticing ; Interpreting; Responding, and Reflecting.
The complexity of cognitive ability, whether from clinical reasoning and/or clinical judgment, has generated some questions in the academic community about which teaching strategies favor the acquisition of this skill and which types of assessment tools can be used.These concerns have encouraged researchers from different teaching contexts and countries to investigate this issue. (4- 6)Thinking about an assertive nurse care practice leads to a review of the way of teaching, monitoring and evaluating the cognitive development process.A view on how nursing students identify and interpret individual needs for directing actions is critical for safe care.
The interest in nursing graduates' training and the need to use instruments for performance evaluation of the teaching-learning process, explains the choice for the Lasater Clinical Judgment Rubric (LCJR).It was developed by Lasater (7) and adapted to the Brazilian culture (8) as Lasater Clinical Judgment Rubric-Brazilian Version (LCJR-BV).This instrument allows a continuous and formative evaluation of the level of development of nursing students' clinical judgment.
The formative evaluation method allows the monitoring of learning in an individualized way, identification of gaps, self-assessment, and regulation of the knowledge acquisition process. (9)For this purpose, the monitoring and evaluation of learning requires the use of a reliable and valid instrument.
In Brazil, some studies on validation of instruments in the nursing area have been directed to health education in clinical practice (10) and educational practices in clinical simulation, (11) among others.However, there is still lack of research on valid instruments for the Brazilian culture that evaluate clinical judgment.
The aim of this study was to evaluate the reliability and validity of the LCJR-BV in a sample of nursing students.

Reference model of Tanner and the LCJR
In the LCJR, was used the Tanner model (3) that assesses clinical judgement performance in four phases, namely: Noticing; Interpreting; Responding; and Reflecting.The first three are part of thinking-in-action skills, and the last one comprises thinking skills about the action, which is a reflection occurring after responding to the situation.
In the Noticing phase, are evaluated the focused observation, recognizing deviations from expected patterns, and information seeking.In the Interpreting phase, is considered the prioritization and understanding of data.The Responding phase reflects dimensions targeted for calm and confident performance, clear communication, well planned intervention/flexibility and being skillful.In the last phase, Reflecting, are considered aspects related to evaluation/self-analysis and commitment to improvement. (7)or each phase, the LCJR describes two to four dimensions, totaling 11.In each dimension, students' behaviors can be evaluated during the learning process, in the following four levels: beginning, developing, accomplished and exemplary.At each level, 1 point is assigned for beginning; 2 points for developing, 3 points for accomplished; and 4 points for exemplary.The minimum score is 11 and the total score is 44 points. (7)n several studies, the Tanner model (3) and the LCJR (7) have been applied as a guide for structuring students' reflection in the development of clinical judgment skills and for monitoring their progress throughout their clinical experience.(14)(15) The rubric is a method of guided reflection for evaluating the development of clinical judgment and an instrument used by students for assessment of their own progress by identifying areas that need improvement in order to be successful. (16)dditionally, using the LCJR allows a common language between teacher and student, the collaborative work for improving the performance of clinical skills, and helps students to feel more confident and competent for initiating care practices. (15)n the professional environment, the LCJR allows nurses' performance evaluation in the conclusion of an educational activity by favoring the definition of performance criteria and self-assessment of strengths and weaknesses regarding the competence in the ability of clinical judgment during reflective practice. (17)he cultural and semantic validation of the LCJR was performed in Brazil, and the instrument was named LCJR-BV.It was used to analyze five videos in a situation of high-fidelity simulation regarding performance of nursing students' clinical judgment.Three independent judges participated in this analysis and obtained satisfactory results in both intraobserver concordance (Kappa= 0.834; p≅00.000;Kappa = 0.764; p≅0.00; kappa 0.823; p≅0.00, respectively) in two analyzes with 15-day intervals, as in interobserver concordance (Kappa= 0.828; p≅ 0.00).However, the author recommends tests for evaluation of other psychometric properties of the instrument, the purpose of this study. (18)he assessment and follow-up of the development of nursing students' clinical judgment is essential to ensure consistent clinical training.To this end, it is necessary to guarantee the availability of an instrument with accurate psychometric characteristics.If such an instrument is valid, it is expected to contribute with students' self-assessment and reflection in relation to their performance, and be a sign of what is expected from them in terms of developing clinical judgment of excellence throughout their training.

Methods
This is a methodological design study addressing the development, validation and evaluation of research instruments and methods (19) with the aim of evaluating the psychometric properties of the LCJR-BV instrument.It was initiated after the agreement of using the Lasater Clinical Judgment Rubric by the Brazilian version authors (8) and the American original. (7)Nursing students of a public institution located in the southeastern region of Brazil participated in the study.
In order to evaluate a suitable instrument for participants, nursing students' comprehension of the rubric items was preliminary investigated regarding the clarity of behaviors described in each level of the 11 dimensions of the LCJR-BV.Eight students participated in this process (two by term), and this was a convenience sample.
For each of the 11 LCJR-BV dimensions, the participant was asked to mark 'yes' or 'no' for clarity, and include a suggestion or comment for the corresponding item in case of a negative answer.They following responses emerged: in the dimension 'recognizing deviations from expected patterns', the term 'obvious deviations' was considered unclear by a student, and it was changed to 'evident deviations'.Other students suggested changes in domains definitions: 'calm confident manner' and 'well planned intervention/flexibility' at developing level; 'being skillful', 'evaluation/self-analysis' and 'commitment to improvement' at beginning level.However, these suggestions were disregarded because they reflected changes in content and contained personal interpretations of value judgments.In view of these findings, the LCJR-BV was considered suitable for the investigation of psychometric properties.
Data collection with the purpose of evaluating the discriminant validity, reliability and dimensionality of the LCJR-BV occurred in October and November 2016, after approval of the institution and the ethics committee.The invitation to participants was made in the classroom, when were presented the study objectives, the instruments for demographic characterization, and the Lasater Clinical Judgment Rubric -Brazilian Version.Eligibility criteria were age above 18 years, and having some practical experience with outpatient or hospital nursing care.Participants involved in the previous phase of the study did not participate.
For descriptive and statistical data analysis, was used the SPSS software, version 18.0.The Shapiro-Wilk test was used to check the normality of data.Participants were included in two groups, namely: juniors, those enrolled in 1 st and 2 nd terms; and seniors, those in 3 rd and 4 th terms.Discriminant validity was assessed using the non-parametric Mann-Whitney test by comparing the scores of junior and senior students.A significance level of 5% was adopted.
The instrument trustworthiness or reliability was assessed through internal consistency by Cronbach's alpha and simple correlation by test-retest.For the test-retest, was adopted the period of two weeks after the first application, and it was measured by intraclass correlation coefficient (ICC). (20)Values higher than 0.75 are indicative of excellent agreement; (21) values 0.4≤ICC<0.75,satisfactory agreement; and ICC< 0.4 weak agreement. (22)or evaluation of the LCJR-BV dimensionality, was used the Confirmatory Factor Analysis (CFA) for verification of the number of latent traits (theo-retically defined).Structural equation models were used by considering Partial Least Squares (PLS) as an estimation method, and using the Smart PLS 2.0 software. (23)The factorial model analysis comprised the following two steps: analysis of convergent and discriminant validity of the LCJR-BV.
In the convergent validity analysis of the factorial model, were evaluated the AVE (Average Variance Extracted) results for each of the model factors.AVE values higher than 0.5 indicate that the model converges to a satisfactory result. (23)Subsequently, were evaluated the values of factorial load between items and their respective factors.Items with loads lower than 0.5 are considered as candidates to leave the factorial model.It is defined that loads should be at least greater than 0.5 and ideally greater than 0.7. (24)Another precision indicator used was the composite reliability that evaluates the quality of an instrument structural model.It was a more robust indicator when compared to Cronbach's alpha. (25)he result above 0.7 is considered as satisfactory. (23)n the evaluation of discriminant validity, was adopted the Fornell-Larcker method (26) .It compares the square roots of AVEs with the correlation values between factors.This model has discriminant validity if the square roots of AVEs are larger than correlations between factors.Another criterion to evaluate discriminant validity was cross loadings analysis.In this case, it was observed if the factorial load of a given item was higher in the factor in which it was initially allocated than in the other factors of the model.
The study was approved by the institution and the Research Ethics Committee under CAAE protocol number 56124216.1.0000.5505.The ethical and legal aspects of Resolution number 466/2012 were respected.

Results
Of the 179 participants, 161 (89.94%) were female, 115 were classified as juniors (1 st and 2 nd terms) and 64 were seniors (3 rd and 4 th terms).The mean age was 22.1 years (minimum of 18 years old; maximum of 49 years old).Of the nursing students, 76.54% experienced practical activities in the outpatient clinic and 95.53% in the hospital setting.
Discriminant validity is used to verify if the instrument differentiates the two distinct groups.In the comparison test between the evaluated grades/ terms (Table 1), the difference (p-value <0.05) in all evaluated dimensions became evident.The distribution comparison test was significant in all domains evaluated (p-value <0.001), which indicates the evaluation of the senior group is significantly higher than that of the junior group in all domains under study.
The AVE values of each of the model factors were greater than 0.5 (Noticing=0.66;Interpreting=0.73;Responding=0.60;and Reflecting=0.73),and composite reliability results were higher than 0.7 (Noticing=0.85;Interpreting=0.84;Responding=0.86;and Reflecting=0.84),which indicates the model converges to a satisfactory result.
In the analysis of discriminant validity of the factorial model by means of the Fornell-Larcker criterion, square root values of the AVEs were higher than the correlations between factors (Table 2) thereby showing a satisfactory result.The overall internal consistency of the LCJR-BV obtained Cronbach's alpha value of 0.889, and values in the phases were the following: Noticing (α=0.75),Interpreting (α=0.64),Responding (α=0.78) and Reflecting (α=0.63).Regarding stability verified by the test-retest with 27 participants of the 2 nd term, there was a total correlation of 0.88.In the different phases of development of clinical judgement, results were the following: Noticing (ICC=0.57);Interpreting (ICC=0.61);Responding (ICC=0.85);and Reflecting (ICC=0.88).
In order to evaluate the dimensionality validity of the LCJR-BV, the measurements of composite reliability (CR) and average variance extracted (AVE) were initially calculated with the purpose of evaluating the convergent validity of the factorial model.evidenced that the LCJR instrument differentiated nursing students into three levels of proficiency (low expectation, in expectation and above expectation) when evaluated by the examiner in simulation scenarios.The Brazilian version described above was considered valid when differentiating the level of performance of clinical judgment between junior and senior students (Table 1).(29)(30)(31) In the LCJR version for the Korean culture (K-LCJR), was obtained a Cronbach's alpha value for the total instrument (0.910), and for phases of Noticing (α=0.736),Interpreting (α=0.722),Responding (α=0.807) and Reflecting (α=0.683). (31)The Korean study also pointed that 152 students participated in three simulation scenarios (nurses-father-son interaction; febrile child and emergency measures for newborn).Of these, the febrile child scenario was used for data analysis.Students watched the videos recorded during the simulation for self-assessing their performance by using the K-LCJR. (31)In the initial validation study for the Brazilian version, (8) authors obtained a Cronbach's alpha of 0.892, and the following values in the phases: Noticing (α=0.816),Interpreting (α=0.714),Responding (α=0.795) and Reflecting (α=0.655).In the present study, Cronbach's alpha was 0.899 for the total value of the instrument.In the phases, values were the following: Noticing (α=0.75),Interpreting (α=0.64),Responding (α=0.78), and Reflecting (α=0.63).Findings of this study and those of the other studies allow to conclude the existence of a satisfactory level of internal consistency for the measurement instrument. (23)he analysis of composite reliability was not available in the aforementioned studies, but indicated the LCJR-BV has good quality in the proposed structural model In the present study, in the test-retest, were obtained ICC values below 0.75 in the clinical judgment phases (observation and interpretation).This fact may be related to the influence of internal or external factors of subjects by considering these domains require cognitive ability for the analysis and synthesis of objective and subjective data, which are still in progress in this phase of participants' academic development.However, the total ICC value of 0.88 indicated an excellent reliability score. (21)n the confirmatory factorial analysis, was confirmed the original theoretical framework of the LCJR proposed by Lasater (7) in four phases of clinical judgement development.Three dimensions were confirmed in the Noticing phase, namely: focused observation, recognizing deviations from expected patterns and information seeking; in the Interpreting phase, two dimensions: prioritizing data and making sense of data; in the Responding phase, four dimensions: calm confident manner, clear communication, well-planned intervention/ flexibility and being skillful; and in the last phase, Reflecting, two dimensions: evaluation/self-analysis and commitment to improvement. (8,18)n the face of results of the present study and those of Nunes, (8) the LCJR-BV can be considered validated for the Brazilian culture and recommended as an instrument to monitor the evolution of clinical judgment development.This can be done in an observational way by the teacher or self-applied by students themselves in activities of the training process in the different care contexts.
In spite of the good results evidenced in the present study, and given the statistical analyzes, coherence of the proposed instrument and the theory adopted in its construction, in our field, studies correlating the results of the rubric with another instrument measuring the same phenomenon have not been identified yet.In this study, students used the LCJR-BV in the mode of self-assessment of clinical practice performance.

Conclusion
The analysis of the LCJR-BV psychometric properties, that is, discriminant validity, reliability and dimensionality, has demonstrated evidence of reliability and validity of the instrument for evaluating the development of nursing students' clinical judgment.

Table 1 .
Distribution of scores (mean and standard deviation) of LCJR-BV dimensions according to nursing students grouped in juniors and seniors (n=179) p-value obtained by Mann-Whitney test; # Shapiro-Wilk test p≥0.05;§ Shapiro-Wilk test p<0.05

Table 2 .
Discriminant validity of the LCJR-BV by the Fornell-Larcker criterion Note: Diagonal values highlighted in bold indicate the square root of the average variance extracted (AVE)

Table 3 .
Discriminant validity of the LCJR-BV by cross loadings