Validity and reliability of the Brazilian version of the Johns Hopkins Fall Risk Assessment Tool to assess the risk of falls

REV BRAS EPIDEMIOL 2019; 22: E190037 RESUMO: Objetivo: Avaliar a validade e a confiabilidade da escala Johns Hopkins Fall Risk Assessment Tool ( JH-FRAT) para avaliação do risco de quedas em pacientes hospitalizados. Método: Estudo retroativo com 297 pacientes de um hospital privado de São Paulo usando dados de 2014. A validade foi avaliada por meio da acurácia (sensibilidade, especificidade, valores preditivos positivo — VPP e negativo — VPN), da validade de critério, verificando-se as associações entre a classificação do risco pelas escalas JH-FRAT e Morse, e da análise discriminante (comparação dos pacientes com e sem quedas em relação aos itens da escala e verificação de situações prévias de risco em relação ao dano ocorrido). Foram utilizados os testes de associação χ2. A confiabilidade foi avaliada por meio da reprodutibilidade teste-reteste interobservadores, usando as estatísticas Kappa ponderado quadrático e prevalence-adjusted and bias-adjusted Kappa (PABAK). Resultados: A sensibilidade foi de 97,0%, a especificidade, 14,6%, o VPP, 36,2%, e o VPN, 90,6%. A análise de critério mostrou associação (p < 0,0001) entre as avaliações pelas duas escalas. Cinco dos oito itens da escala e a classificação geral mostraram capacidade de discriminação do risco (p < 0,050). A reprodutibilidade interobservadores variou entre itens (PABAK de 0,25 a 1,00) e foi substancial na classificação do risco global (PABAK = 0,71). Conclusão: A JH-FRAT apresentou validade e confiabilidade esperadas para um instrumento de triagem do risco de quedas, podendo contribuir na aplicação de estratégias para a gestão de quedas em hospitais.

Validity and reliability of the Brazilian version of the Johns Hopkins Fall Risk Assessment Tool to assess the risk of falls

INTRODUCTION
Falls that occur with hospitalized patients are adverse events with negative repercussions for patients and institutions [1][2][3][4][5] , whose prevention should adopt evidence-based practices such as having the call bell within the patient's reach, hospital bed in low position and treatment of the causes of mental confusion 2,3,[6][7][8] .Among the recommended practices is the use of scales that identify risk conditions for the occurrence of falls, either in the form of specific risks or in risk stratification, directing interventions compatible with the identified needs [2][3][4]7,8 .
Among the instruments available in the international literature, two scales went through the cross-cultural adaptation process for use in Brazil, the Morse Fall Scale 9 and the Johns Hopkins Fall Risk Assessment Tool ( JH-FRAT) 5 ; both did not report the measurement properties of the Brazilian version.This study chose to evaluate the measurement properties of the JH-FRAT scale as an instrument whose content's validity was already demonstrated in the Brazilian version, showing good applicability in systematized protocols and simple operationalization 5 .

Palavras-chave:
The scale's measurement properties were evaluated in patients hospitalized in South Korea 12 and in the elderly in American nursing homes 13 , showing acceptable sensitivity and specificity 12,13 .These studies, however, present methodological limitations that make it difficult to generalize the results to other populations.
Considering that previous studies have shown that JH-FRAT has acceptable measurement properties [10][11][12][13] and that there are no assessments of risk scales for falls in Brazil, this study aimed to analyze the measurement properties of JH-FRAT for assessing the risk of falls in hospitalized adult patients.

METHOD
The study was approved by the Research Ethics Committee of Hospital Samaritano and followed the principles of the Declaration of Helsinki.Due to the use of secondary data, the Informed Consent was not necessary.Authorization was obtained from the authors of JH-FRAT and of the Brazilian version of the Morse scale for use in this study.
The target population was composed of adult patients hospitalized in a high complexity private hospital in the city of São Paulo, which uses a protocol for the management of falls since 2005, with the implementation of JH-FRAT at the end of 2013.The protocol provides for the daily assessment of individual risks through JH-FRAT, with the adoption of preventive measures according to each patient's risks and of universal precautionary measures, such as maintaining night lights activated, signaling risks and freeing the environment of obstacles.The evaluation was performed by nurses and filed in an electronic medical record.This study used data from 2014, after a period of training and routine consolidation by the nursing team.This is a case-control study including patients from the adult hospitalization areas (clinical, surgical, oncology, gynecology and obstetrics and intensive care), with a wide variety of clinical conditions.We selected 100.0% of patients with reported falls in 2014 (99 cases) and 198 patients without reported falls (two controls for each case), resulting in a sample of 297 individuals.The controls selected were patients admitted to the room next to a patient with a reported fall event, in the same hospitalization sector and on the same date of the fall, in order to homogenize the characteristics of the care received.No other pairing criteria were used in order to preserve differences regarding other potential risk factors that are part of JH-FRAT's operational structure.Some patients had more than one reported fall event, with each being assessed independently.For patients with falls, the last evaluation was considered within the 24 hours before the fall.For patients without falls, the same date for patients with falls was used.Individuals aged less than 18 years and/or whose hospitalization lasted for less than 24 hours were excluded.
Five researcher nurses collected the data retroactively from the electronic medical record, using a form composed of identification fields (chart number, attendance number, age, date of evaluation, hospitalization sector), for items from the JH-FRAT scale and Morse scale.Data were also obtained from the fall notification forms, filled out by nursing assistants.
The JH-FRAT scale is composed of eight aspects of fall risk 11 : 1. previous risk-defining situations: complete paralysis/immobilization; history of two or more falls in the six months prior to admission; history of one fall during hospitalization; and patient considered as high-risk according to specific protocols.
If the patient has complete paralysis/immobilization, they are classified as low-risk; if they present one of the other conditions, they are classified as high-risk.In these cases, the assessment is terminated and the rest of the scale is applied only to patients who are not classified under these conditions; 2. age at admission: 0-3 points; 3. history of one fall in the six months prior to admission: 0-5 points; 4. intestinal and urinary eliminations (increased frequency, urgency, incontinence): 0-4 points; 5. use of drugs with high risk for falls: 0-7 points; 6. presence of care equipment that may compromise patient mobility: 0-3 points; 7. impairment of mobility: 0-6 points; 8. impairment in cognition: 0-7 points.
The sum of points from items 2 to 8 provides a score of 0 to 35 points, which is classified as low risk (0-5 points), moderate risk (6-13 points) and high risk (≥ 14 points) 11 .Item 1 of the scale is not included in the computation, only in the risk classification.Further details on the scale are described in the literature 5,6,11 .
The Morse scale 14 showed satisfactory accuracy in the identification of patients at high risk of falls in international studies 15,16 .It is composed of six aspects (history of falls, secondary diagnoses, walking aid, intravenous therapy/devices, gait, and mental state), with scores varying from 0 to 30, providing a score ranging from 0 to 125 points categorized as low risk (0-24 points), medium risk (25-44 points) and high risk (≥ 45 points) 9 .
In this study, the validity was evaluated through the analysis of accuracy, criterion (concurrent analysis) and discriminant, and reliability was evaluated through the analysis of reproducibility.
Scales for the assessment of fall risk are considered diagnostic tests for the event risk (fall) 17 .The validity of the diagnostic tests is analyzed by means of accuracy studies, in which the results of a test applied to a population are compared to those obtained by a gold standard [17][18][19] .In this case, the test is the JH-FRAT and the gold standard is the occurrence of fall.Accuracy analysis verified sensitivity, specificity, positive (PPV) and negative (NPV) predictive values and positive (PLR) and negative (NLR) likelihood ratios Sensitivity evaluates the probability of a test being positive when the disease is present and consists of calculating the prevalence of exposure among events; specificity evaluates the probability of a test being negative when there is no disease and consists in calculating the prevalence of non-exposed among non-events; the PPV evaluates the probability of having the disease when the test was positive and consists of calculating the prevalence of events among those exposed; and the NPV evaluates the probability of not having the disease when the test was negative and consists of calculating the prevalence of non-events among non-exposed individuals.PLR informs the probability of a positive result occurring in a person with the disease in relation to the probability of occurrence in a person without the disease.NLR informs the likelihood of a negative result occurring in a person with the disease in relation to the probability of occurrence in a person without the disease 3,20,21 .The fall risk was considered as exposure ( JH-FRAT scale), and the analyzes considered two scenarios: 1. cutoff point > 6 points (moderate/high risk versus low risk); 2. cutoff point > 13 points (high risk versus moderate/low risk) in JH-FRAT.
The criterion validity compares the analyzed scale to a valid standard that predicts the same performance 22,23 , being concurrent when the instrument and criterion are applied simultaneously 22 .The concurrent criterion validity investigated the associations between JH-FRAT and Morse risk classification using the χ 2 test.
The discriminant analysis verifies whether a test is capable of identifying differences in a characteristic of subjects 22 .It was verified whether each of the eight items of the scale was able to discriminate the presence of the specific risk among patients with and without falls using the χ 2 test.
Also in the discriminant analysis, it was verified whether previous risk-defining situations (item 1 of the scale) were associated with the severity of the injuries between the fall events, through the χ 2 test.The severity of the injuries was classified, from an adaptation of Schwendimann et al. 1 , in: 1. no injury; 2. mild injury: minor cuts, mild bleeding, skin excoriation, pain, edema, mild traumatic brain injury (TBI), hyperemia and hematoma; 3. moderate injury: excessive bleeding, laceration requiring suturing, moderate TBI, dislocation and sprain; 4. severe injury: fracture, subdural hematoma, severe TBI, death.
Reproducibility analysis was performed using the interobserver test-retest comparison, with a random sub-sample of 60 patients.Two researchers independently assessed the same patient using the JH-FRAT scale, based on the same reference date, from data recorded in an electronic medical record.Quadratic weighted Kappa (PABAK) and prevalence-adjusted and bias-adjusted Kappa statistics were used.For the interpretation of the results, the criteria of Landis & Koch were used 24 .
A significance level of 5% was considered in all analyzes.The software SPSS (version 23), STATA (version 14) and WinPepi (version 11.65) were used.

RESULTS
Table 1 shows that the study population consisted of 297 patients, 50.8% females and 49.2% males; 55.9% aged 60 years or older; 63.0% married/had a partner; 46.1% had complete superior education; 14.1% in intensive care units; and 85.9% in open units.Patients with and without falls did not present differences as to gender (p = 0.870), age range (p = 0.200), marital status (0,159), educational level (0,189) and reason for hospitalization (p = 0.677).It was observed that 10.8% of the patients had low risk of falls, being 3.0% among those with falls and 14.6% among those without falls.Patients with high risk of falls had a higher percentage of occurrence of events (62.6%) than the others (44.9%) (p = 0.002).
Table 2 shows that, in the concurrent criterion validity, there was a statistically significant association between the two scales (p < 0.001).The highest proportion of patients classified as low risk by JH-FRAT was classified as low risk by Morse (46.9%); the highest proportion of patients classified as moderate risk by JH-FRAT was classified as moderate risk by Morse (53.5%); and the highest proportion of patients classified as high risk by JH-FRAT was classified as high risk by Morse (57.6%).When moderate and high risk are grouped, the similarity in results between the two scales rises to 90.2%.
The discriminant analysis showed which items of the scale had a statistically significant difference in the presence of the specific risk (Table 3).In the item "history of falls", 18.2% of the patients with events had a history of one fall in the six months prior to hospitalization against 2.5% of those without a fall (p < 0.001).In the item "use of drugs with high risk for falls", the category "2 or more drugs" was observed in 70.7% of patients with events and in 53.5% of those without events, and sedation in the last 24 hours occurred in 3.0% of patients with falls and in 9.1% of those without falls (p = 0.027).The results of the category "sedation in the last 24 hours" are different than expected, but the small number of cases of falls (3 patients) does not have statistical stability.The item "equipment impairing mobility" showed that ≥ 3 devices were present in 5.1% of patients with events and in 11.1% of those without events (p = 0.040).Again, the results may suffer bias from the small number of subjects with falls (five patients).In the item "impaired mobility", patients with events had higher percentages  than those without events in all the categories evaluated (p = 0.003).The same occurred in the item "cognition" (p < 0.001).Also in the discriminant validity analysis, the previous situations that defined the risk (item 1 of the scale) were not associated with the injury(ies) due to the fall (p = 0.557).
Table 4 shows the results of the reliability analysis through interobserver reproducibility.Quadratic weighted Kappa values ranged from 0.15 to 1.00.There was improvement REV BRAS EPIDEMIOL 2019; 22: E190037 after adjustment for prevalence and bias, with PABAK ranging from 0.25 to 1.00.The item "prior risk situations" presented reasonable reproducibility (PABAK = 0.24).For the other items, substantial or almost perfect reproducibility (PABAK of 0.60 to 1.00) was observed, and the overall risk classification presented substantial reproducibility (PABAK = 0.71).

DISCUSSION
The study analyzed whether the JH-FRAT scale is valid and reliable for assessing the risk of falls in adult inpatients.The results suggest that JH-FRAT has a better sensitivity at the score's cutoff point > 6, grouping moderate and high-risk patients in the same category.An explanation for this is the existence of a managed protocol for falls in the study institution, where patients under certain conditions (risk of bleeding, bone fragility, among others) are classified as high risk according to item 1 of the scale (previous defining situations of risk).A modified version of JH-FRAT evaluating 107 outpatients aged 65 years and older in nursing homes in Southern California showed sensitivity of 100.0% at cutoff point > 6 and 86.2% at cutoff point > 13 13 .A JH-FRAT study with 356 patients older than 18 years in five general school hospitals in South Korea, with cutoff point > 13 points, showed a sensitivity of 62.0% 12 .JH-FRAT showed better specificity at the cutoff point > 13, separating high-risk patients from the others.Sensitivity and specificity are inversely proportional 19,21 and sensitivity was high.Considering that the systematized actions prevented the occurrence of falls, there was an increase in the number of false positives (patients at risk, but without falls), with consequent decrease in specificity.The diagnosis of a condition and the exclusion of false positives are defined by confirmatory tests that must have high specificity, but are subsequent to screening exams 21 .The study in southern California showed specificity of 0.0% at the cutoff point > 6 and 49.3% at the cutoff point > 13 points 13 .The study in South Korea, with cutoff point > 13 points, showed specificity of 69.5% 12 .
The PPVs show that 63.8% (in the cutoff point > 6) and 58.9% (in cutoff point > 13) of patients with a positive risk test received unnecessary preventive care for falls.This interpretation must be cautious.The increased prevalence of the event determines the increase in the PPV and the decrease in the VPN, and both depend on the context in which the test is applied 16,18,20 .Considering that the effect of the "treated" population reduces the occurrence of falls among those at risk, bias can occur with decreasing PPV and increasing NPV.The study in southern California showed PPV of 39.7% and NPV of 92.9% at cutoff point > 13 points 13 .The study in South Korea, with cutoff point > 13 points, showed PPV of 33.6% and NPV of 86.0% 12 .
LR offers an odds ratio measure, combining sensitivity and specificity in the same indicator 20 .The results were greater than 1.00 and thus contribute to the identification of the risk 20 , especially the NLR of scenario 1, in which patients without falls had 4.83 times the chance of obtaining a low risk classification when compared to the others.
For a screening test, the most important feature is sensitivity, with a high rate of true-positive and a low false-negative rate 20,21 .JH-FRAT, which an instrument for risk screening, showed high sensitivity.
The analysis of the validity of concurrent criteria showed equivalences between the risk classification of the two scales ( JH-FRAT and Morse), with better performance when the moderate and high risks are grouped, suggesting that JH-FRAT has greater validity when separating the low-risk patients from the others.
Five of the eight JH-FRAT items and the overall risk score discriminated risk, with higher prevalence among patients with falls than among those without falls, as expected.
The previous situations that defined the risk (item 1 of the scale) were not able to discriminate the type of injury resulting from the fall.These results may be biased due to the institutional protocol, reducing both the occurrence and severity of falls.
The prevalence of patients classified as moderate/high risk, by both JH-FRAT and Morse, was 90.2%.A paradox in the Kappa reproducibility analyzes can be induced by a high prevalence of the analyzed condition and can be adjusted by the PABAK statistic 25 .Reproducibility between tests showed substantial agreement after adjustment for this statistic.
The interobserver test-retest measures showed substantial or near-perfect agreement after the use of the PABAK statistic on all items, except in the "prior risk situations".The results obtained by quadratic weighted Kappa showed improvement after correction by the PABAK statistic, pointing to the already mentioned paradox.The study in southern California also showed substantial or near-perfect agreement in most items and in the overall classification (cutoff point ≥ 14) 13 .
This study has some limitations.The most important is the fact that it included a "treated" population, since preventive interventions reduce the occurrence and severity of events, and may alter the outcomes and accuracy of a risk assessment scale 26,27 .Another limitation refers to the use of data collected retrospectively, a type of collection that is affected by the quality of the available data 28 .Lastly, the relatively small number of falls assessed may have generated some bias.
Falls are events that are sensitive to care practices and can be prevented through systematized approaches in institutional programs [2][3][4][5]15,26 . The bst results are obtained by combining multiple interventions, including individualized risk assessment and universal precautions 4.8 .The use of multifaceted interventions can reduce the occurrence of falls by up to 18.0%, but this reduction may vary depending on the quantity and type of actions 2 .The results showed JH-FRAT as valid and reliable in terms of accuracy, discriminant capacity and agreement on reproducibility, confirming it as a tool for risk assessment of falls and bringing contributions to professionals and institutions interested in the use of evidence-based strategies for the management of falls in hospitals.
There are few studies on the validity and reliability of JH-FRAT, and the present study is the first in Brazil.It is advisable that new studies be carried out with larger groups of hospitalized patients, preferably with a prospective design.In the absence of ethical conflicts, it is desirable to carry out studies on "untreated" populations and to compare results before and after the implementation of preventive protocols.

CONCLUSIONS
The study suggests that the Brazilian version of the JH-FRAT scale has satisfactory validity and reliability in terms of accuracy, criterion, discriminant capacity and reproducibility.It can be used in the evaluation of the fall risk in hospitalized patients, directing preventive actions.This is the first study in Brazil to evaluate JH-FRAT and has limitations, and new studies are desirable.

Table 1 .
Descriptive statistics of the study population according to the occurrence of falls, private hospital, São Paulo, 2014.

Table 2 .
Comparative analysis of the fall risk classification according to the Johns Hopkins Fall Risk Assessment Tool and Morse, private hospital, São Paulo, Brazil, 2014.

Table 3 .
Descriptive statistics between patients with and without fall event according to the presence of fall risks identified by the Johns Hopkins Fall Risk Assessment Tool, private hospital, São Paulo, Brazil, 2014. Continue...

Table 3 .
Continuation.Johns Hopkins Fall Risk Assessment Tool; *association test using χ 2 ; **multiple choice question: needs assistance/supervision and/or gait and/or visual/auditory impairment; ***multiple choice question: altered perceptions and/or impulsivity and/or lack of understanding of limitations.

Table 4 .
Test-retest agreement for each item and for the final risk classification of the Johns Hopkins Fall Risk Assessment Tool, private hospital, São Paulo, Brazil, 2014 (N = 60).