Psychometric analysis of the scale for the predisposition to the occurrence of adverse events in nursing care provided in ICUS

Abstracts

OBJETIVO: apresentar os resultados do estudo de validade e fidedignidade da Escala de Predisposição à Ocorrência de Eventos Adversos (EPEA). MÉTODO: o estudo da validade de construto ocorreu pela técnica de análise de componentes principais. RESULTADOS: o exame da fidedignidade da EPEA por intermédio do alfa de Cronbach indicou boa confiabilidade (estrutura α=0,80; processo α=0,92). CONCLUSÃO: tendo por base seus indicadores psicométricos, a EPEA pode ser considerada uma medida válida para avaliar as atitudes dos enfermeiros frente aos fatores que podem predispor à ocorrência dos eventos adversos em UTI.

Qualidade da Assistência à Saúde; Doença Iatrogênica; Cuidados de Enfermagem; Psicometria; Estudos de Validação

OBJETIVO: presentar los resultados del estudio de validez y confiabilidad de la Escala de Predisposición a la Ocurrencia de Eventos Adversos (EPEA). MÉTODO: el estudio de la validez de constructo se realizó con la técnica de análisis de componentes principales. RESULTADOS: el examen de confiabilidad de la EPEA a través del Alpha de Cronbach indicó buena confiabilidad (estructura α=0,80; proceso α=0,92). CONCLUSIÓN: teniendo como base sus indicadores psicométricos, la EPEA puede ser considerada una medida válida para evaluar las actitudes de los enfermeros frente a los factores que puede predisponer a la ocurrencia de los eventos adversos en UTI.

Calidad de la Atención de Salud; Enfermedad Iatrogénica; Atención de Enfermería; Psicometría; Estudios de Validación

ORIGINAL ARTICLE

Psychometric analysis of the scale for the predisposition to the occurrence of adverse events in nursing care provided in ICUS

William Mendes Lobão^I; Igor Gomes Menezes^II

^IMSc, RN, Hospital Geral do Estado, Salvador, BA, Brazil

^IIPhD, Adjunct Professor, Instituto de Psicologia, Universidade Federal da Bahia, Salvador, BA, Brazil

^{Corresponding Author}

ABSTRACT

OBJECTIVE: to present the result of the validity and reliability studies concerning the Scale for the Predisposition to the Occurrence of Adverse Events (EPEA).

METHOD: construct validity was based on Principal Components Analysis.

RESULTS: reliability verified through Cronbach's alpha indicated good reliability (structure α=0.80; process α=0.92).

CONCLUSION: based on its psychometric indicators, the EPEA can be considered a valid measure to assess the attitudes of nurses in relation to factors that potentially lead to the occurrence of adverse events in ICUs.

Descriptors: Quality of Health Care; Iatrogenic Disease; Nursing Care; Psychometrics; Validation Studies.

Introduction

International concern with patient safety became more evident after the publication of the report "To err is human: building a safer health system"⁽¹⁾. The report was based on data from two studies verifying the occurrence of adverse events in hospitalizations in Colorado and Utah (2.9%) and also in hospitalizations that took place in New York hospitals (3.7%). When these figures are extrapolated to the 33,600,000 hospitalizations in the USA reported in 1997, it is estimated that at least 98,000 patients died in New York and 44,000 died in Colorado and Utah due to errors committed in health care delivered.

"Errors can be defined as the non-intentional use of an incorrect plan to achieve an objective, or not properly performing a planned action."⁽²⁾ International data indicate that errors in the health field affect one in every ten patients around the world and "the most important knowledge in patient safety is how to avoid harm during treatment and care delivery."⁽³⁾

Adverse events are defined as "unfavorable clinical occurrences that result in death, risk of death, hospitalization or prolongation of an existing hospitalization, or significant, persistent or permanent impairment."⁽⁴⁾ The occurrence of these adverse events during nursing care provided in ICUs is associated with significant prolongation of hospital stays and increased medical costs⁽⁵⁾.

In general, quality of care has been assessed in terms of results indicators⁽⁶⁾, e.g. assessing the success of nursing care based on the occurrence of an adverse event. However, the quality of nursing care provided in ICUs should be analyzed as one good among ideal working conditions (structure and process), which themselves derive from national and international recommendations concerning healthcare quality and patient safety in ICUs, and then the risks and benefits that arise from human fallibility while providing care.

Such fallibility can derive from different perceptions that nurses hold concerning their work environment and care protocols and that can influence their attitudes concerning conditions that potentially lead to the occurrence of adverse events. Social psychologists' discussion of attitudes is based on three important questions⁽⁷⁾: a) to what extent are internal mental attitudes related to actual behavior?; b) to what extent are individuals' internally organized views of world based on a systematic selection of points of view?; and c) why, at some point in life, do different people share the same opinions about specific issues?

In the work context, attitude consists of a disposition to either perform an action or to omit it, which directly influences the quality of care delivered by nurses in ICUs. Hence, "attitudes are a predisposition to negatively or positively react to certain objects, institutions, concepts or other people"⁽⁸⁾ and are similar both to interests and opinions. One of the most important attributes of attitudes is their subjectivity, because it reflects the way people see an object and not necessarily how this object really is⁽⁹⁾. Because attitudes are part of the individual's subjectivity, they are associated with thoughts, feelings, and actions that guide the way people live.

Due to a lack of instruments in the literature addressing the attitudes of nurses concerning aspects of the structure and process, which can compromise the quality of nursing care provided in ICUs and have adverse events as a results indicator, and also because attitude is a psychological construct not always amenable to direct observation, the Scale for the Predisposition to the Occurrence of Adverse Events (EPEA) was developed.

Psychometric scales "aim to establish a relationship function between (physical, social) environmental stimuli and the individual's behavior"⁽¹⁰⁾, to the extent they assess how much a given stimulus influences an individual's behavior, enabling, as in the case of EPEA, the measurement of nurses' attitudes concerning factors that may lead to the occurrence of adverse events.

The development of EPEA was based on three major points: a) theoretical, which refers to the theory concerning the construct and which should support the instrument's development; b) empirical, which is the stage when the procedures concerning the instrument's application and data collection are defined; and c) analytical, which establishes the procedures for statistical analysis to be employed in a psychometric instrument to test its construct validity.

The validity of a psychometric test⁽¹¹⁾ seeks to verify whether the test measures what it is supposed to measure and the extent to which such measurement is achieved. The entire validation process seeks to ensure isomorphism, i.e. equivalence among the properties of the psychological attribute and the representation of this object in the form of a measure. It can be studied from three different perspectives: content validity, criterion validity, and construct validity⁽¹²⁾.

Because the EPEA, as a psychological measure based on a theory addressing a construct, represents a latent trait (attitude), its content validity was ensured through techniques that include expert and semantic analyses⁽¹³⁾. Hence, its items are ensured to be within the theoretical approach that refers to the quality of nursing care provided in ICUs. After this stage, the instrument's final version came to be composed of 64 items grouped into two dimensions: structure (18 items) and process (46 items).

Criterion validity is defined⁽¹¹⁾ as the quality of a scale or test to function as a present or future predictor of a variable, operationally independent, called a criterion. This criterion can be a behavior displayed in an experimental or non-experimental situation, a future performance within a given field or profession, or another already validated instrument addressing the same construct. Since no instruments similar to EPEA were identified, as well as there being no directly observable behaviors that could serve as a proxy of these attitudes, we did not test criterion validity.

Construct validity takes into account the degree to which a given psychological instrument assesses the construct it was theoretically designed to assess. This procedure seeks to confer validity onto the instrument based on the analysis of its items, enabling an assessment of its psychometric quality.

Therefore, this study's objective was to present the results of the validity and reliability studies concerning the Scale for the Predisposition to the Occurrence of Adverse Events (EPEA).

Method

Methodological design focused on the development, reliability and validity tests of measurement instruments.

Study setting and participants

The study setting included six highly complex ICUs from general and university hospitals: three public and three philanthropic hospitals located in Salvador, BA, Brazil. These ICUs had 14.54 (SD=6.08) beds and 4.03 (SD=1.58) nurses per shift, on average.

Our initial proposal was to conduct a population-based study, however, given the limited number of public and philanthropic ICUs in Salvador that consented to participate in the study, and the fact that the nurses working in these units had more than one job and could not participate in the study more than once, we opted to use a convenience sample. The sample of the study implementing the validation process of the EPEA was composed of 128 nurses: 49.2% worked in public ICUs and 50.8% worked in philanthropic units.

Instruments

The EPEA was designed to assess the level of importance nurses attribute to aspects concerning the structure and process (ideal level), as well as their perception concerning the existence of these aspects in their work environment (actual level), which can influence the quality of nursing care provided in ICUs, which can be an adverse event (AE) as a results indicator.

The participants were asked to take a position in relation to these aspects in order to evaluate their attitudes and establish a relationship between stimuli and attitudes. Hence, we sought to connect attitudes and behaviors that function as predictors of AE, for each item of the instrument (Figure 1). A Likert scale containing five response intervals, ranging from "totally disagree" to "totally agree," was used. In addition to the EPEA items, we used a social-demographic questionnaire addressing information concerning the participants' professional and personal profiles, such as gender, age, number of jobs, and professional background, among others.

Data collection procedures

Data were collected through a survey, using a structured instrument that was self-administered individually in a peaceful environment, free from distractions, during working hours.

In regard to the planning of data collection, the instructions concerning the instrument's completion are provided right after the socio-demographic data and immediately before the EPEA items in order to facilitate the correct interpretation of the measurement and appropriate comprehension of the Likert scale.

Data analysis procedures

Among the different statistical techniques used for the construct validation, Principal Component Analysis (PCA) stands out. It seeks to investigate the construct's dimensionality, that is, how many factors the instrument is actually measuring. The PCA enables one to determine how many and what dimensions would be proposed for the construct's characterization. It also enables the identification of any potential pattern of correlation in order to provide an explanation for the variations in the variables analyzed in isolation for each of the dimensions (structure and process) in the evaluation of the quality of nursing care provided in ICUs.

Because the EPEA has two parallel (ideal and actual) polytomous scales (graded) for each item in this analysis, the first step was to order the items according to two different scales, so that each item could later be correlated with the total score and determine the degree in which it measures the same attitude that other items are supposed to measure. After reordering, the items that had an inverted answer were recoded so that an answer checked as "totally disagree" (score 1) was transformed into "totally agree" (score 5); "somewhat disagree" (score 2) into "somewhat agree" (score 4); "do not agree nor disagree," which indicates a neutral position, (score 3) was not modified; "somewhat agree" (score 4) into "somewhat disagree" (score 2); and the answers checked with "totally agree" (score 5) were recoded as "totally disagree" (score 1).

The next step was to build the variables dif_est_"x" (x= 1 to 18) and dif_proc_"x" (x= 1 to 46), for the items of the scales "structure" and "process," respectively, which represent the difference of results between the "ideal" and "actual" for each of the dimensions' items. Therefore, the total score of each individual could be obtained by totaling the scores of each item.

A PCA was performed for the 64 items, with varimax orthogonal rotation for each of two approaches (structure and process), seeking to maximize the variance of loads within the factors and find independent factors, confirming the theoretical assumption that these factors are not correlated, and identify the factor structure that accounts for the best variance explained by the construct. This approach is corroborated by the Classical Test Theory (CTT), which seeks to define a test's psychometric quality as behavioral stimuli, in terms of criteria such as present or future behaviors (variables)⁽¹⁰⁾. Because they are behavior-oriented (t = tau), CTT studies observable physical realities as a result of tests (criterion measure).

To measure the level of internal consistency, which is based on the correlation between the different items of the same test, we used Cronbach's alpha. This coefficient assesses whether the items designed to measure the same construct produce similar results. Cronbach's alphas above 0.70 were considered satisfactory⁽¹⁴⁾. The measure of internal consistency (item-total correlation) permits verifying the instrument's reliability, as it determines the measure's level of precision.

Ethical aspects

The study's Project was submitted to and approved by the Institutional Review Board at the Federal University of Bahia (Process No. 14/2011 - FR 412506).

Results

The results from the PCA for the dimension "structure" presented a determinant of the correlation matrix ≈ 0.003, enabling its inversion and use of the matrix of correlation for the analysis. The use of PCA was corroborated by the KMO test, which was 0.726, showing that the studied sample had a good fit. Bartlett's sphericity test presented a significant value with c²(153)=614.477 (p<0.001), indicating that the correlations among the items are sufficient for the analysis. The explained variance for one dimension was 22.85%.

PCA was the extraction method and considered the 18 items initially proposed to evaluate the dimension "structure"; 12 items were kept because they presented factor loads above 0.30 (Table 1).

The determinant of the matrix indicated the factorability of the dimension "process", presenting a non-zero value (d≈0.001). The Kaiser-Meyer-Olkin test showed sampling adequacy for the analysis (KMO=0.714) and Bartlett's sphericity test presented a significant value with c²(903)=2381.566 (p<0.001), indicating that the correlations among the items are sufficient for the analysis. The dimension's explained variance was 23.89%.

The PCA extraction method, based on the 43 items initially proposed to evaluate the dimension "process", indicated that 34 items presented factor loads above 0.30 (Table 2).

Once construct validity was ensured using the CTT, we investigated the instrument's reliability using Cronbach's alpha, which assesses the correlation among the scores obtained in each of the test's items (internal consistency). The EPEA's measure of internal consistency obtained the following coefficients: 0.80, which indicates good reliability for the 12 items of the scale "structure" and 0.92, which indicates optimum reliability for the 34 items in the scale "process". These results show the instrument has a good level of internal consistency, indicating there is coherence among the results of similar items.

After PCA, 15 items were removed from the 64 items initially proposed for the instrument because they presented factor loads below 0.30 (Figure 2): six items of the dimension "structure" and nine items from the dimension "process".

Discussion

In regard to the instrument's construct validity, even though the items that were excluded from the "structure" and "process" dimensions after PCA, were theoretically adjusted for the studied construct and dimension, they presented factor loads below 0.3, thus, low saturation in relation to the studied dimensions.

The fact that there was a political issue of great importance for nursing professionals among these items - the 30-hour workweek - drew our attention. However, as this issue is still being discussed in the National Congress, the establishment of a 30-hour workweek is not a reality experienced by the professionals who participated in the study, thus, hindering the saturation of that factor as a general factor of quality of care delivery.

The low factor load of the items dif_est_15 and dif_est_16 may be related to the fact that they refer to a working condition already common among the participants, evidenced by the average of 3.6 (SD=1.03) beds per nurse. Hence, the participants did not deem it a factor that potentially leads to the occurrence of AEs. This working condition, when not met, is related to burnout and dissatisfaction at work, resulting in a high turnover rate, increased mortality due to complications (failure to rescue), greater incidence of infections, and adverse events⁽⁵⁾.

The validation items (dif_proc20, dif_proc31, dif_proc34), were intended to assess the consistency of the responses provided by the participants and that were removed due to low saturation, possibly indicating that the nurses answered the scale in a predictable way. That is, they rejected statements that were developed to present ideas that could be considered absurd if accepted in the practice of ICU's nurses. The scale's remaining items, in turn, presented positive behaviors or behaviors that indicated actions that favored the prevention of AEs.

The low saturations of some items in the dimension "structure" and (dif_est_8, dif_est_17 e dif_est_18) in the dimension "process" (dif_proc_32) may be explained by the fact that the content of these items were linked to aspects already considered routine for all patients in ICUs. Thus, they were not considered to be characteristics, which when absent, would possibly compromise the quality of nursing care.

The final version of EPEA that resulted from the PCA containing two factors (structure and process) and 46 items (12 from the dimension "structure" and 34 from the dimension "process") proved to be reliable. The results obtained through the psychometric analysis present good correlation (internal consistency) among the instrument's different items. In relation to the data obtained from the PCA, it is also worth noting that there were no items with absolute values of factor load below 0.30. Likewise, there were no similar factor loads in two or more factors in the same item and the differences among the absolute values of the items' factor loads were above 0.10.

Conclusion

The development and validation of the Scale for the Predisposition to the Occurrence of Adverse Events (EPEA) seeks to evaluate the attitude of nurses on aspects of structure and process that may compromise the quality of nursing care in ICUs. The result of which is a result indicator of adverse event that fills a gap in the field of attitudinal measures of ICU nurses.

The Principal Components Analysis (PCA) indicated that the EPEA's items behaved according to the original model, i.e. the two dimensions (structure and process) individually possessed characteristics that ensured sufficient internal consistency to enable the measurement of nurses' attitudes concerning aspects of structure and process that may compromise the quality of nursing care provided in ICUs.

The results obtained from the instrument's construct validity confirm the hypothesis that the EPEA's items measure the construct for which they were theoretically designed and provide correct inferences and interpretations of scores obtained with its application.

A limitation of the study is the population's limited size, which led to an also limited sampling size (n=128). One of the possible reasons for not getting a larger number of nurses is the fact that some professionals work in more than a single studied ICU, which is shown by the average number of jobs (1.69; SD=0.57). In addition to that, one large philanthropic hospital that accounted for a considerable number of nurses refused to participate in the study.

The difficulty in finding hospitals in Salvador, BA, Brazil that meet the established criteria, that is, hospitals that systematically notify and monitor the occurrence of adverse events, can also be considered one of the study's limitations because it hinders comparison of the index of adverse events in nursing care provided in ICUs and the attitudes of nurses concerning factors that may lead to the occurrence of such events.

Seeking to verify the EPEA's validity beyond the sample used, we consider the need to apply it in different contexts, e.g. both public and private hospital ICUs, accredited or not, in other states, to establish a comparison among the results and enable a broader understanding of the nurses' attitudes concerning aspects of structure and process potentially leading to the occurrence of adverse events during the delivery of nursing care in ICUs.

References

1. Kohn LT, Corrigan JM, Donaldson MS, editors. To err is human: building a safer health system. [Internet]. Washington (DC): Institute of Medicine/National Academy Press; 2000. [acesso 17 fev 2011]. Disponível em: http://iom.edu/~/media/Files/Report%20Files/1999/To-Err-is-Human/To%20Err%20is%20Human%201999%20%20report%20brief.pdf.

2. Pedreira MdLG. Errar é humano: estratégias para a busca da segurança do paciente. In: Harada MdJCS, Pedreira MdLG, Peterlini MAS, Pereira SR. O erro humano e a segurança do paciente. 2ª ed. São Paulo: Atheneu; 2006. p. 2-18.

3. World Health Organization (WHO). Nine patient safety solutions: Solutions to prevent health care-related harm. 2007. [acesso 15 mar 2011]. Disponível em: http://www.who.int/mediacentre/news/releases/2007/pr22/en/index.html.

4. Consulta pública nº 21, de 27 de abril de 2006 (BR). Dispõe sobre o Regulamento Técnico para Funcionamento de Serviços de Atenção ao Paciente Crítico e Potencialmente Crítico. 2006. [acesso 2 dez 2010]. Disponível em: http://www4.anvisa.gov.br/base/visadoc/CP/CP[14558-1-0].pdf.

5. Cho SH, Ketefia S, Barkauskas VH. The effects of nurse staffing on adverse events, morbidity, mortality, and medical costs. Nurs Res. [periódico na Internet]. 2003 [acesso 23 mar 2011]; March/April: 71-79. Disponível em: http://www.pennanurses.org/pac/PDFs/Effects%20of%20RN%20Staffing%20on%20ADEs%20-%20Costs.pdf .

6. Donabedian, A. The Definition of Quality and Approaches to Its Assessment (Explorations in Quality Assessment and Monitoring, Vol 1). Ann Arbor (MI): Health Administration Press; 1980.163 p.

7. Fraser CBJ, Burchell B. Introducing Social Psychology. Cambridge: Polity; 2001.

8. Richardson RJ, Wanderley JCV. Medição de atitudes. In: Richardson RJ, Wanderley JCV. Medição de atitudes nas ciências da conduta. João Pessoa (PB): Editora Universitária/UFPb; 1985. p. 25-69.

9. Menezes IG. Escalas de Intenções Comportamentais de Comprometimento Organizacional(EICCO): concepção, desenvolvimento, validação e padronização [dissertação]. Salvador (BA): Faculdade de Filosofia e Ciências Humanas da Universidade Federal da Bahia; 2006. 335 p.

10. Pasquali L, Capovilla AGS, Alonso AOL, Alves AR, Borba ACP, Batista CG, et al. Instrumentação Psicológica: Fundamentos e Práticas. Porto Alegre: Artmed; 2010.560 p.

11. Anastasi A, Urbina S. Testagem psicológica. Porto Alegre: Artes Médicas Sul; 2000.

12. Cronbach LJ, Meehl PE. Construct Validity in Psychological Tests. Psychol Bull. 1955;52:281-302.

13. Lobão WM, Menezes IG. Construction and content validation of the scale of predisposition to the occurrence of adverse events. Rev. Latino-Am. Enfermagem. [periódico na Internet]. 2012 [acesso 19 set 2012]; July-Aug.;20(4):796-803. Disponível em: http://www.scielo.br/pdf/rlae/v20n4/21.pdf

14. Hair JF, Tatham RL, Anderson RE, Black W. Multivariate data analysis. New Jersey: Prentice Hall; 1998.

Received: May 10th 2012

Accepted: Dec. 3rd 2012

• 2. Pedreira MdLG. Errar é humano: estratégias para a busca da segurança do paciente. In: Harada MdJCS, Pedreira MdLG, Peterlini MAS, Pereira SR. O erro humano e a segurança do paciente. 2Ş ed. São Paulo: Atheneu; 2006. p. 2-18.
• 6. Donabedian, A. The Definition of Quality and Approaches to Its Assessment (Explorations in Quality Assessment and Monitoring, Vol 1). Ann Arbor (MI): Health Administration Press; 1980.163 p.
• 7. Fraser CBJ, Burchell B. Introducing Social Psychology. Cambridge: Polity; 2001.
• 8. Richardson RJ, Wanderley JCV. Medição de atitudes. In: Richardson RJ, Wanderley JCV. Medição de atitudes nas ciências da conduta. João Pessoa (PB): Editora Universitária/UFPb; 1985. p. 25-69.
• 9. Menezes IG. Escalas de Intenções Comportamentais de Comprometimento Organizacional(EICCO): concepção, desenvolvimento, validação e padronização [dissertação]. Salvador (BA): Faculdade de Filosofia e Ciências Humanas da Universidade Federal da Bahia; 2006. 335 p.
• 10. Pasquali L, Capovilla AGS, Alonso AOL, Alves AR, Borba ACP, Batista CG, et al. Instrumentação Psicológica: Fundamentos e Práticas. Porto Alegre: Artmed; 2010.560 p.
• 11. Anastasi A, Urbina S. Testagem psicológica. Porto Alegre: Artes Médicas Sul; 2000.
• 12. Cronbach LJ, Meehl PE. Construct Validity in Psychological Tests. Psychol Bull. 1955;52:281-302.
• 14. Hair JF, Tatham RL, Anderson RE, Black W. Multivariate data analysis. New Jersey: Prentice Hall; 1998.

Publication Dates

• Publication in this collection
  28 Mar 2013
• Date of issue
  Feb 2013

History

• Received
  10 May 2012
• Accepted
  03 Dec 2012