Indicators of effectiveness of nursing care in the dimension of patient safety

Objectives: to validate nursing care effectiveness indicators of patient safety dimension. Methods: quantitative survey, using the electronic Delphi sampli, with 52 participants selected by the Snowball sampling. Eight indicators were evaluated regarding the attributes: availability, reliability, simplicity, representativeness, sensitivity, comprehensiveness, objectivity, cost, utility, stability and timeliness. For validation, the minimum agreement criterion was 70%. Results: Cronbach’s alpha (0.942) evidenced the high internal consistency among the attributes. The indicators fall with damage, hip fracture, and postoperative hip fracture, incidents related to equipment, incidents due to failures in patient identification, and pressure injury were validated in all attributes, and those of medication error and hand Hygiene were not validated. Conclusions: the validated indicators allow assessment of the effectiveness of hospital nursing care. Unavailability of data is an obstacle to monitoring patient safety. Descriptors: Quality Indicators, Health Care; Effectiveness; Nursing Care; Patient Safety; Quality Management . RESUMO y los de error de medicación y la higiene de las manos no han sido validados. Conclusiones: los indicadores validados permiten evaluar la efectividad asistencial de la enfermería. La indisponibilidad de datos es un obstáculo al monitoreo de la seguridad del paciente. Descriptores: Indicadores de Calidad de la Atención de Salud; Efectividad; Atención de Enfermería; Seguridad del Paciente; Gestión de la Calidad.


INTRODUCTION
Patient safety has been part of the global health priority agenda since 2000, when an American report revealed the magnitude of errors in hospital care (1) . . This is a relevant dimension of quality and, through indicators, measures risks of patient care (2) .
Indicators are quantifiable representations of structure, processes and results, and provide information expressed by events, rates or indices (3)(4)(5) . Even relevant, they are underutilized. The systematic effectiveness measurement of in-Hospital programs is not perceived (4) , with more frequent measurements related to the structure or processes.
Indicators of results or effectiveness refer to desirable or undesirable changes attributed to health care provided (6) . Although the results indicators are influenced by many factors, they are concrete and indispensable elements to measure the impact of health care (7) . Effectiveness is the degree level to which attainable improvements in health are achieved (8) . Thus, assistance is expected to be effective and safe, providing good results and good life quality to the individual and the community, and is based on the best evidence.
Nursing is highlighted as the largest workforce in Brazilian health area, answering for the majority of hospital care, with direct involvement in the conception, operationalization and evaluation of strategies to improve patient's safety (9) . Therefore, indicators for measuring nursing care effectiveness must correlate to the main dimensions of quality, such as safety and patient-centered care; and organizations should provide mechanisms to facilitate health professionals directly involved in assistance and managers to understand the concepts related to these indicators and their use in the hospital routine.
From the perspective of contributing to the quality and safety of health care, the Sistema de Notificações à Vigilância Sanitária (NOTIVISA -notification system for sanitary surveillance) was created. According to NOTIVISA, between 2014 and 2016, there were 63,933 incidents related to assistance, 417 (0.6%) fatal (10) . All public and private services, through its patient Safety Center, are responsible for notifications of incidents in Notivisa. Nursing adherence to internal notification is emphasized, although the safety culture in hospitals still requires maturity for professionals to feel safe when informing occurrences (9) .
The content validation of indicators of effectiveness of nursing care in the patient safety dimension demonstrates the extent to which these indicators express the planned results for a given nursing care. Thus, this validation can contribute to evidence indicators sensitive to nursing care, qualify assistance planning, and explain to patients the nursing team's contribution to the results of hospital care (13) .
As criteria to assess whether the indicators measure the nursing care effectiveness, attributes recognized in the literature can be used as desirable for a good indicator, such as: availability, reliability, simplicity, representativeness, sensitivity, comprehensiveness, objectivity, low cost, usefulness, stability and timeliness (11)(12)(27)(28) .

OBJETIVES
To validate nursing care effectiveness indicators in the patient safety dimension, considering their correspondence to the attributes availability, reliability, simplicity, representativeness, sensitivity, comprehensiveness, objectivity, low cost, usefulness, stability and timeliness.

Ethical aspects
The study incorporated in its design the basic references of the resolution of National Health Council N º 466/2012, respecting the rights of participants, who agreed with the free and informed consent form (ICF). Their identities remain anonymous, and their identification is given by alphanumeric code, not causing harm or monetary and moral damage. Approval from the Research Ethics Committee of the Department of Health Sciences of the Federal University of Paraná.

Study design, location, and period
This is a cross-sectional, descriptive study, as a survey with a quantitative approach. The Delphi electronic technique was used, a method for systematic collection and aggregation of judgments informed by experts or specialists (29) , which desired the minimum consensus of 70% in answers in two rounds. Data collection was carried out from May 10 to September 19, 2017, Brazil.

Population or sample
The sample selection was intentional. All those who fulfilled the inclusion criteria were invited to participate in the research: being Brazilian and having professional experience or scientific production related to health indicators in hospitals. The exclusion criterion was the non-completeness of 90% of the answers.

Study protocol
We used snowball sampling for recruitment of participants (30) . From a list of 20 authors of articles related to health indicators and 20 individuals with experience in the use of indicators in their professional practice, we identified those that corresponded to one or more inclusion criteria, which was confirmed by analyzing their Lattes curricula. The preliminary contact occurred via electronic messages. With each contact, a possible participant recommended others, and so on. We obtained 176 possible participants.
Each one received an invite to participate. To those who accepted it, we sent an access link to the electronic platform Survey Monkey and the questionnaire, after consent to the electronic ICF. Participated in the first round 52 specialists, 43 in the second, all identified by the letter P, followed by a cardinal number, respecting the order of submission of completed instrument to the researcher. Authors (27)(28) recommend that, to verify the content validity, a panel of experts is used to assess how much the indicator satisfies certain requirements. Thus, with an approach based on objectivity and scientific neutrality, it is possible to evaluate whether the indicator or a group of indicators represent the situation or the phenomenon studied.
The survey consisted of 96 questions with five-level Likertscale answer options: "I totally disagree"; "I partially disagree"; "I do not disagree or agree"; "I partially agree"; "I totally agree". A minimum of 70% of agreement was stipulated between the answers in relation to all attributes (Chart 1) for the validation of each indicator, per round, estimated by the sum of percentages of answers "totally agree" and "partially agree".
Statistical Package for the Social Sciences (SPSS), version 22, to assess the validation after the two rounds. Cronbach's alpha coefficient was used to analyze the internal consistency and the difference of panelists' opinion in both rounds, in addition to the Spearman's C-ratio for correlation analysis between the attributes.
The Cronbach's alpha was 0.942, demonstrating that, in the set, the answers to the attributes for each indicator have high internal consistency. The Spearman's ratio, used to obtain the evaluation of the relationship intensity between the attributes of each indicator, showed a positive correlation of all attributes among each other. This result allows to infer that the validated indicators are adequate to measure the impact of nursing care in hospital assistance in patient safety dimension.

Attribute Specification
Availability Easy obtainment of data for its structuring.

Reliability
Original sources and reliable data collection and processing methods.
Simplicity Easy estimative from basic information, understanding or interpretation.

Representativeness
It faithfully represents what it proposes to measure.

Sensitivity
It distinguishes occasional variations in the trend of situation, in a given area, with reflections on its result.

Scope
It synthesizes as many conditions or factors as possible that affect the situation to be measured.

Objectivity
Clarity in measuring objective.

Low cost
Favorable relationship between costs of obtaining data and benefits arising from its use.

Utility
Provides decision-making.

Stability
The measurement series allows for consistent monitoring and comparisons.

Timeliness
Structured with current and timely information to use.
Source: Elaboration of the authors based on D'innocenzo et al. (27) , National School of Public Administration (28) ,Commitment to Hospital Quality Program (32) and Agency for Healthcare Research and Quality (33) .

Analysis of results and statistics
Data analysis was performed using uni and bivariate descriptive statistics. The Survey Monkey provided agreement percentages between first round answers. All answers were inserted in the Chart 2 presents the agreement rates between the attributes for each indicator evaluated and, in Chart 3, participants main comments and suggestions for the non-validated indicators.

DISCUSSION
The present study provides managers with information to select indicators validated by specialists, aimed at capturing how much the results of nursing care processes in the hospital meet the improvement of patient's safety.
We highlight the participants' distrust in relation to the sources of data collection since, in Brazilian reality, medical records are incomplete or omit International Classification of Diseases' diagnoses (34) .
Two out of eight indicators were not validated: "institutional adherence to the multimodal strategy of hand hygiene" and "medication errors". The non-validation resulted from consensus lower than 70% in the availability attribute, denoting that the data to structure these indicators are not easy to obtain.
In relation to the indicator "institutional adherence to the multimodal strategy of hand hygiene", hospitals often use the measurement of liquid soap and alcohol, which enables the indirect evaluation of adherence -constituting, therefore, as an indicator of process, and not of effectiveness (16) . However, the multimodal strategy is more extensive than just measuring the consumption of products, because it implies changes in the system, with infrastructure provision, professionals education, evaluation and feedback on performance and results, workplace reminders, and institutional security climate dimensioning (17) . These multiple approaches add complexity to the indicator and make data obtaining more laborious and costly, which may explain why it was not validated by the specialists. After analyzing the suggestions of panelists (Chart 3), a composite indicator was elaborated addressing these various practices to measure adherence to the multimodal strategy in the institution.
Medication errors are relevant because they result in longer patient permanence in the hospital, increased risk of death and hospital costs. Low notification and absence of incident records related to dispensation, prescription and administration steps are still common (18)(19)(20) .
Although the medication error indicator was not validated because it did not reach the minimum consensus regarding the availability attribute in the two rounds, the other attributes reached such consensus among panelists, ranging from 76.92% to 90.40%. The attributes that obtained higher agreement were objectivity (90.40%), usefulness (88.46%) and representativeness (86.55%), showing that this indicator corresponds to its objective and subsidizes a coherent decision-making with what should be measured.
Appropriately, the participants suggested reinforcement of medication error reporting systems due to low detection when there is no harm to the patient; and patient inclusion, since notifications can be originated by communication of errors detected by the patient or relatives to the health team (19) .
Incidents with medication-related damage are those that occur most frequently in hospitalized patients, and about 25% of them are preventable. Medicament therapy chain is extensive, involves logistics focused on supply, prescription, dispensing, preparation and administration, which makes the identification of errors even more complex. It is possible that errors in this chain are related to systemic latent conditions (35) .
However, the management of this indicator is closely related to nursing, since most medication errors occur in the administration phase, whose attribution is almost exclusive to this professional (18) . The absence of records in incidents related to dispensation and prescription stages is also common (19)(20) . Therefore, strategies to increase data availability for this indicator aims the reinforcement of safety culture, implementation and stimulus to adherence to protocols to prevent medication errors, and incentive to notify Incidents.
Regarding the nurses, the obstacles to report medication errors and almost incidents in hospital environments were explored in a systematic review covering the period from 1981 to 2015. Organizational barriers, such as culture, non-friendly notification system, managerial behavior and emotional factors, such as fear of consequences and burden of responsibility, were associated with low notification (36) .
As contributors of patient safety, nurses need education and qualification in error management.The organization is responsible for developing a culture of learning with errors, without blaming and punishing, as well as adopting anonymous, effective, uncomplicated and efficient reporting systems, besides providing support by managers, providing open feedback to Nurses (36) .
Additionally, "automated methods to identify incidents through electronic registers" enhance the overcoming of methodological biases related to retrospective studies in medical records, for more accurate data obtaining that favors the use of validated indicators to measure the incidence of failures, errors and omissions related to medications (37) .
Other indicators were validated. Patient's identification is fundamental to prevention of errors in the actions of nursing care, such as medication and procedures in general. This indicator aims to account for the number of incidents due to patient's identification failures. The results indicate that, although the Chart 3 -Comments and suggestions from experts on non-validated hospital nursing care effectiveness indicators attributes in patient safety dimension
Suggestions: studying different ways of collecting lost opportunities for hand cleaning (P44); Help of multimers -sectors audited with checklist (P30); Using of sampling stratified by sectors and professionals, non-identifiable observers, registering professionals behavior(P17).

Medication error.
Comments: This type of error is difficult to detect when it does not cause damage, (P16, P17, P32, P36).

of
Indicators of effectiveness of nursing care in the patient safety dimension Seiffert LS, Wolff LDG, Ferreira MMF, Cruz EDA, Silvestre AL.
patient's identification aims to safely determine the legitimacy of the procedure's receptor, in practice this receives little attention (21) -It highlights this indicator as one of the priorities for measuring patient safety. In relation to the "pressure injury indicator", the use of scales to determine the risk level for the development of this lesion is recommended. Various scales are available, such as Braden, Norton, Waterlow, and Assessment scale of risk for surgical positioning injuries (38)(39)(40) , scales that can be selected for systematic application at the discretion of each hospital.
Regarding the monitoring of hip fracture between hospitalized and postoperative patients, the simplicity in estimating (Chart 2) favors the viability of these indicators. Victims of these events are fragile, limited and physically incapable; present clinical condition worsening; they face long time of hospitalization and readmissions period; increase hospital costs; and these indicator's mortality increases in 50% after one year of fall (22)(23) . Systematic evaluation of risk by nursing using validated scales is also indispensable, implantation, monitoring and evaluation of adherence to the Protocol for the prevention of falls, the main factor associated with fractures.
The indicator of falls is recommended worldwide. In the present study, the indicator of falls with damage identification was shown to the participants, because it is sensitive to the measurement of nursing care effectiveness. Due to the low incidence, the California Nursing Outcomes Coalition accounts for the number of falls with damage for every thousand patients per day (24) . Given the importance of notification of hip fracture between hospitalized and postoperative patients, and also of falls with damage, hospitals can use an indicator consisting of the three indicators instead of a simple indicator, extending their comprehensiveness (31) .
Regarding the indicator of serious equipment-related incidents, the interaction with the clinical engineering service can intensify the technovigilance and favor the identification of causes and prevention of incidents, resulting in training programs to avoid such incidents and in opportunity for producers to make their products safer (25) . The usefulness of this indicator is backed by the frequency of these incidents and the expressive use of devices in nursing care.
Of the incidents considered critical occurring in 90 months in an intensive care unit (ICU) for adults in England, 30% were caused by equipment (25,41) . In the United Kingdom, 564 critical incidents were reported in the ICU in 13 years, involving 94 types of equipment; 12.41% were related to the use of beds, mattresses and chairs, and 12.05% to equipment and devices needed for the procedure, such as intravenous infusion and monitoring, mechanical ventilation and tracheostomy, renal therapy and bronchoscopy (22) .
Results indicate dependence of obtaining data to estimate the indicators and the need for complete and correct assistance records, as well as the use of effective systems of direct and indirect identification and notification of incidents. However, the legislation that obliges the systematic notification of events by hospitals arose only in 2013 (26) , demanding changes in safety culture, strongly linked to managerial actions, and requiring time to become routine in professional practice.
The fragility in obtaining reliable data in health is also related to absence of integration between management information systems, the poor quality of administrative records, the partially computerized records or with handwritten records, and limited scope in the coding of diagnoses related to iatrogenisis (42) .
In addition to the computational infrastructure, there is the user interface with the system. Nurses are users of hospital systems, data feeders and collectors for assistance and managerial indicators. Some factors interfere with the effectiveness of nurse-computer interaction, such as software configuration errors, inadequate workflow, lack of time, inadequate number of professionals and lack of knowledge about indicators (12) . Strategies such as use of algorithms and configuration of warnings for alarming data can minimize the unavailability of data needed to estimate indicators, allowing nurses to act in timely, performing effective and safe care for the patient.

Study limitations
The study was based on experts ' judgement on indicators attributes approached and, due to the peculiar subjectivity peculiar of survey, it is considered possible to have some bias in results. However, the minimum criterion of 70% of agreement between the specialists for indicator validation and the very high internal consistency confer reliability to results. As additional elements to results' reliability we consider the careful selection of specialists, according to their experience in studies and/or use of health indicators, and self-analysis as to their contribution to the present study.

Contributions to nursing field
The validated indicators allow the evaluation of the health effectiveness care.in hospital nursing.The adaptation of these indicators to hospital practice facilitates the monitoring of nursing efforts aimed at patient safety, with reduction of harmful incidents.

CONCLUSIONS
Validate nursing care effectiveness indicators in patient safety dimension, were evaluated considering their correspondence to the attributes availability, reliability, simplicity, representativeness, sensitivity, approach, objectivity, low cost, usefulness, stability and timeliness. Indicators were validated in relation to all attributes: "Fall with damage"; "Hip fracture"; "Postoperative hip fracture"; "Equipment-related incidents"; "Incidents due to failures in patient identification"; and "pressure injury". "Medication error" indicator and the one related to the multimodal hand hygiene strategy were not validated because of the availability of data to be structured. However, given its relevance to measure nursing care effectiveness in patient safety dimension , strategies that reinforce the safety culture, identification and reporting systems of incidents, and the computerization of records assistance, among others, are measures that should be implemented by hospitals so these indicators can be used systematically.

FUNDING
Funding Source: Leila Soares Seiffert -Coordination of Improvement of Higher Education Personnel (Capes; Scholarship of the sandwich doctoral program abroad).