Reliability and validity of the Brazilian version of the Hospital Survey on Patient Safety Culture ( HSOPSC ) : a pilot study

The evaluation of the culture of patient safety in hospitals is nowadays considered as a management too, since it helps to identify problem areas and provide valuable information for planning improvements. This study explored the reliability and validity of the Brazilian version of the Hospital Survey on Patient Safety Culture, an instrument that evaluates characteristics of patient safety culture among hospital staff. The reliability of the instrument was evaluated by analyzing the internal consistency of each dimension. The validity of the tool was carried out by means of exploratory and confirmatory factor analysis. The sample was made up of 322 questionnaires that were collected in two Brazilian hospitals in 2012. Cronbach’s alpha ranged from 0.52 to 0.91 for the different dimensions, with the exception of two, for which it was much lower. After excluding four items, the exploratory factor analysis presented adjusted indices that were appropriate for a 10 factor model. Safety; Patient Safety; Quality of Health Care; Questionnaires Correspondence C. T. Reis Rua Manoel Ramos Trindade 76, apto. 201, Cataguases, MG 36770-014, Brasil. clautartaglia@gmail.com 1 Secretaria Municipal de Saúde de Cataguases, Cataguases, Brasil. 2 Instituto de Comunicação e Informação Científica e Tecnológica em Saúde, Fundação Oswaldo Cruz, Rio de Janeiro. Brasil. 3 Escola Nacional de Saúde Pública Sergio Arouca, Fundação Oswaldo Cruz, Rio de Janeiro, Brasil. doi:10.1590/0102-311X00115614 This article is published in Open Access under the Creative Commons Attribution license, which allows use, distribution, and reproduction in any medium, without restrictions, as long as the original work is correctly cited.


Introduction
The provision of safe, reliable health care requires that professionals participate in a robust culture of safety, particularly in hospitals.A collaborative work environment where qualified professionals treat each other with respect, leaders guide their teams efficiently and continued professional development is a priority is a core condition for reducing avoidable adverse events 1 .
In order to establish an effective safety culture in a health care organization, it is crucial to assess its current culture 2 .Information from the organization's personnel about their safety-related perceptions and behavior helps to identify problem areas for planning and implementing interventions intended to strengthen the safety culture.Sorra & Nieva 3 explain that a safety culture assessment can have multiple goals: (i) auditing the safety culture and raising employee awareness on the subject; (ii) assessing patient safety interventions implemented in the institution and tracking changes over time; (iii) making comparisons with reference data that is both internal and external to the organization; and (iv) checking compliance with regulatory requirements.
Several instruments, each with different features, are available to assess patient safety culture in health care services 4 .One of these, the Hospital Survey on Patient Safety Culture (HSOPSC) 3 , developed by the U. S. Agency for Healthcare Research and Quality (AHRQ), is considered a valid, reliable and efficient instrument.The HSOPSC was developed to assess the perception of hospital personnel with respect to different characteristics of the safety culture, and is one of the most commonly used instruments worldwide to measure patient safety culture 5,6,7 .However, in order to convert results from a cross-sectional study that uses data-collection instruments into reliable safety culture indicators, the instrument must achieve the highest possible standard of measurement 8 .To that end, after an instrument has been processed to assure semantic and conceptual equivalence for use in a culture different from that of its origin, it is essential to assess its content and measurement validity 9 .
Studies on patient safety culture in Brazil are relatively new, and a validated instrument will add potential for improving the quality of health care at Brazilian hospitals.The aim of this study was thus to examine the psychometric properties of the Brazilian version of the HSOPSC 10 .

Methods
The HSOPSC This instrument is made up of 42 questions designed to assess safety culture awareness across 12 dimensions 3 .Seven dimensions relate to the hospital department or work unit: (i) communication openness; (ii) feedback and communication about error; (iii) organizational learningcontinuous improvement; (iv) supervisor/manager expectations and actions promoting patient safety; (v) non-punitive response to errors; (vi) teamwork within units; and (vii) staffing.Three dimensions assess safety culture awareness at the hospital level: (i) management support for patient safety; (ii) handoffs & transition; and (iii) teamwork across units.Two dimensions contemplate results: (i) overall perceptions of patient safety; and (ii) frequency of events reported.The items that compose the HSOPSC questionnaire are shown in Table 1, aggregated into their respective dimensions.
The HSOPSC also contains eight questions relating to respondents' characteristics (gender, professional category, level of education, how long they have worked at the hospital, weekly working hours etc.).It takes approximately 15 minutes to complete all items.The HSOPSC measures awareness among staff who are in direct or indirect contact with patients and spend most of their working hours at the hospital.Most items are answered on a five-point (Likert-type) scale reflecting level of agreement from "I strongly disagree" (1) to "I strongly agree" (5), with a neutral category "I neither agree nor disagree" (3).Other items are answered on a five-point frequency scale from "never" (1) to "always" (5).
The original instrument was submitted at the initial stages of this study for transcultural adaptation (conceptual, item and semantic equivalences) 10 .

Setting, population and data collection
The instrument was applied to a sample of workers at two large hospitals (Hospital 1 and Hospital 2, with 150 and 499 beds, respectively) in two states in southeastern Brazil.The study population comprised staff in direct or indirect contact with inpatients at the two participating hospitals, as well as staff (leaders, managers, supervisors and administrators) whose functions did not involve direct contact with inpatients, but directly affected inpatient care.The inclusion criteria were those suggested by the developers of the HSOPSC 3 : staff with a minimum 20-hour working week at the hospital and on duty during the  Before data collection, the research team approached both hospital directors in order to explain the purpose of the study, highlight the importance of safety culture to improving care, and present the necessary procedures, and inclusion criteria for participation in the survey.Head nurses of several inpatient departments, heads of inpatient diagnostic support services, and staff from several departments in their work environments were identified as hospital leaders.
For data collection, in most instances, printed instruments were handed to participants to be answered and later returned to the lead researcher.Each participant received two copies of the Informed Consent Form; one copy was retained by the researchers and the other by the participant.On agreement, each participant received an envelope with a copy of the adapted instrument, without identifying information from the participant.Data were collected from March to May 2012.Researchers made five previously scheduled visits to each hospital to collect the completed instruments.

Data analysis
Reliability and construct validity were examined.Reliability testing evaluated the extent to which results obtained by the Brazilian version of HSOPSC could be replicated.In order to allow for comparison with other validation studies 11,12,13,14,15,16,17,18,19,20,21,22 , the reliability of HSOP-SC was assessed through its internal consistency as estimated using Cronbach's alpha 23,24 for each dimension or factor, in line with the dimensions proposed by the original model.Since some questionnaire items were worded positively and others negatively, negatively worded items were reverse coded so that in all cases a higher score would indicate a positive response 3 .The Kolmogorov Sminorv test was used to assess if the study score distribution was normal.
The construct validity was assessed by exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) 25 , and by the correlation matrix among the dimensions.These analyses were based on questionnaires with no missing data.Firstly, the dimensional structure of the original 12-dimension instrument 3 was evaluated by CFA 26 .As several dependent variables in the HSOPSC are ordinal (Likert-type, five-point response options), the weighted least squares mean and variance adjusted (WLSMV) 26 estima-tion was used.In CFA, model fit was evaluated using three indices: Root Mean Square Error of Approximation (RMSEA), Comparative Fit Index (CFI) and Tucker-Lewis Index (TLI).Absolute fit was assessed using the RMSEA index, in accordance with the use of WLSMV 25,26 .RMSEA is a poor-quality-of-fit index incorporating a penalty function to address a lack of parsimony in the model's degrees of freedom 25,26 .The lower the RMSEA value, the better the fit: less than 0.05 indicates good fit, while values greater than 0.10 indicate poor fit, and that the model should be rejected.CFI and TLI, used as measures of incremental fit 25,26 , compare the proposed model with an independence (null) model; values range from 0 to 1, with values greater than 0.90 showing proper fit 25,26 .
Based on the results of CFA, EFA was performed to investigate a plausible alternative dimensional structure fitting the study data.The factors were initially interpreted using the 42 items by considering eigenvalues greater than 13.In addition, parallel analysis (PA) was performed to identify the number of factors to be retained.The rationale underlying PA is that nontrivial components from real data with a valid underlying factor structure should have larger eigenvalues than parallel components derived from random data having the same sample size and number of variables 27 .Factor items were compared in terms of their factor loading behavior, and those that returned loads of less than 0.40 28 or cross-loading with differences of less than 0.2 between the two highest loads were excluded.This analysis used the weighted least squares mean and variance adjusted (WLSMV) estimation and the oblique rotation method (geomin), assuming that the dimensions that make up patient safety culture are interrelated.The indices of fit examined were RMSEA, CFI and TLI.
Missing data are expected when HSOPSC is used as a data collection instrument.The instructions read: "If you do not wish to answer a question, or if a question does not apply to you, you may leave your answer blank".Thereby, missing data was treated using the Listwise deletion method, which assumes that loss in collection can be characterized as "missing at random" (MAR) 24 .That choice was made given the difficulty of identifying the reasons leading to failure to complete items, since the instrument offers respondents that option.
The SPSS (SPSS Inc., Chicago, U.S.A.) version 17.0 was used to build the database, manage data, treat variables, perform descriptive statistical analyses and reliability calculations, and examine data factorability.Mplus version 5.0 (Muthén & Muthén, Los Angeles, U.S.A.) was used to apply  26 .The GAIPE package from the R statistical software (The R Foundation for Statistical Computing, Vienna, Austria; http://www.r-project.org) was applied to compute 90% confidence intervals (90%CI) for RMSEA.

Ethical considerations
This study was approved by the Ethics Research Committee of the Sergio Arouca National School of Public Health, Oswaldo Cruz Foundation (process n. 177/2011).

Results
Of the 718 workers invited to participate in the survey, 327 (149 from Hospital 1 and 178 from Hospital 2) returned the questionnaires.On reviewing all the questionnaires, two from Hospital 1 were found to have at least one section not fully completed, while one questionnaire had the same response to all items containing reverse items in one particular section.These three questionnaires were excluded from the study.Thus, Hospital 1 returned 146 valid instruments (a total response rate of 56%).At Hospital 2, meanwhile, of the 178 questionnaires completed, two did not have at least one section fully completed.These were excluded from the study.Thus, Hospital 2 returned 176 valid instruments (a 38.5% total response rate).Together, both hospitals produced 322 valid questionnaires, corresponding to a response rate of 44.8%.
The percentage of missing data per item ranged from 0.6% to 7.7%.In terms of the dimensional structure of the instrument, only the "Frequency of events reported" dimension, comprising three questions on error reporting, showed an average percentage loss greater than 5% (5.9%).Of the 322 questionnaires included in the study, 105 were missing at least one item, while 217 were fully completed.Thus, loss of statistical power due to the reduced number of cases was admitted without, however, major loss of veracity in the information.
Cronbach's alpha coefficients estimated for the twelve dimensions of the original model's dimensional structure 3 ranged from 0.52 to 0.91, except for two dimensions, "Non-punitive response to errors" and "Staffing", where internal consistency was very low, respectively, α = 0.35 and α = 0.20 (Table 2).
Kolmogorov-Smirnov tests were significant (p < 0.001) for all 42 items proposed in the original model and pointed out that the score from the Brazilian version HSOPSC does not have a normal distribution.For that reason, we applied the weighted least squares mean and variance adjusted (WLSMV) method, which provides estimates of chi-square statistics corrected for mean and variance, in addition to robust standard error estimates for the model parameters 26 .
CFA was performed to examine the fit between the data obtained in this study and the dimensional structure proposed in the original 12-factor model 3 .It showed poor fit for two indices, CFI = 0.888 and RMSEA = 0.106 (90%CI: 0.092-0.120),and a reasonably adequate fit for a third index (TLI = 0.930).Given that the original 12-factor model displayed inappropriate indices of fit, EFA was applied to explore a model that would best fit the data.Exploratory factor analysis of structures with 11 and 10 factors and the 42 original items exhibited good fit for both models among three indices.EFA of structure with 11 factors exhibited the following indices: CFI = 0.946, TLI = 0.971 and RMSEA = 0.067 (90%CI: 0.054-0.082);and the EFA with 10 factors resulted in the following indices: CFI = 0.942, TLI = 0.970 and RMSEA = 0.070 (90%CI: 0.056-0.084).
The eigenvalues obtained from EFA with the 42 items were, from factor 1 to 12, respectively, 12.38, 2.74, 2.38, 2.32, 1.98, 1.72, 1.46, 1.25, 1.19, 1.12, 1.01 and 0.95.These values supported a dimensional structure with 11 factors, although the last factor was slightly higher than 1.Although the models with 11 and 10 factors showed similar fit indices in EFA, the 10-factor model was selected based on results from parallel analysis.
After defining the number of factors to be retained, analysis of factor loading for the 10 factor model with 42 items identified problems with the following items, shown in Table 3: (i) A9 ("mistakes have led to positive changes here") and C1 ("we are given feedback about changes put into place based on event reports") had weak loadings (0.34 and 0.29, respectively); (ii) C6R ("staff are afraid to ask questions when something does not seem right") had weak and cross-loadings (0.24 on factor 3 and 0.30 on factor 9); and (iii) A16R ("staff worry that mistakes they make are kept in their personnel file") had weak loading (0.39).
With respect to the theoretical model proposed in the original instrument, some findings of the 10 factor model with 38 items were posi- tive (Table 4).Items A5R, A8R, A12R and A14R from dimensions "staffing" and "non-punitive response to errors" loaded on factor 2. Three out of four items from the "supervisor/manager expectations & actions promoting patient safety" dimension (B1, B2 and B3R) and one item from the "communication openness" dimension (C4) loaded on factor 4. Items D1, D2 and D3, which compose the "frequency of events reported" dimension in the original model, loaded strongly on factor 6 (0.91, 0.96 and 0.75, respectively).Items F3R, F5R, F7R and F11R, which make up the dimension "handoffs and transition" in the original model, loaded on factor 7, as well as items F2R, F4 and F6R, which define the dimension "teamwork across units".Three items that make up the "teamwork within units" dimension (A1, A3 and A4) loaded on factor 8 (0.59, 0.63, and 0.73, respectively) along with 2 items from the "organizational learning" and "staffing" dimensions (A6 and A7R, respectively).The other two items in the "staffing" dimension in the original model (A2 and A7R) loaded on factors 10 and 8, respectively.Item A17R, under "overall perceptions of patient safety" in the original model, loaded on factor 3, as did items A13 and B4R, under "organizational learning" and "supervisor/manager expectations", respectively.Items F1, F8 and F9R, under dimension "management support for patient safety" loaded on factor 9, together with item F10 from "team work across units".Nonetheless, a few weak points were observed in the final model: (i) items C3 and C5, under "feedback and communication about error", both loaded on factor 1, forming a factor consisting of two items; (ii) item A11, belonging to "teamwork across the units" in the original model, loaded on factor 10 (-0.69), as did item A2, under "staffing" (-0.53), forming a factor comprising only two items; and (iii) under "overall perceptions of patient safety", item A10R alone loaded on factor 5 (0.74) (Table 4).In Addition, the items A15 and A18 from "overall perceptions of patient safety" and item C2 from the "communication openness" dimension, had loadings below 0.40 (Table 4).
None of the factors presented strong correlation with each other; the correlations among the factors in the 38-item model presented values ranging from 0.007 to 0.439 (Table 5).

Discussion
Although self-administered questionnaires are widely used by researchers needing to obtain large amounts of information in a relatively short time, lengthy surveys, such as HSOPSC, can be tedious for those answering them, and in some cases respondents may simply lose interest and not answer questions accurately 11 .Despite these drawbacks, the percentage of items answered in this study was similar to those of other HSOPSC psychometric validation studies 14,15,16 , which have ranged from 47% to 55%.Regarding reliability, the Cronbach's coefficient ranged from 0.52 to 0.91, except for two dimensions, where it was less than 0.36: "staff-ing" (0.20) and "non-punitive response to errors" (0.35).Other studies have also found low Cronbach's alpha values for these two dimensions.Values estimated for the "staffing" dimension of the HSOPSC were 0.19 in the Turkish version, with 309 participants 11 ; 0.12 in the Spanish version,  20 .Likewise, values in the "non-punitive responses to errors" dimension were 0.35 in the study evaluating the Turkish version 11 and 0.55 in the Mexican study 15 .Although Cronbach's alpha has been chosen to investigate the questionnaire's reliability, this value should always be interpreted in light of the characteristics of the measurement it is associated with in the study population.Accordingly, only repeated use of the instrument with different samples will be able to provide substantial confirmation of its validity and reliability.Not until a scale has been used several times, and generated reliable data, can it safely be said, in fact, to be reliable 23 .It is therefore recommended to conduct further studies using the HSOPSC in Brazil to confirm the Cronbach's alpha values obtained for these dimensions.
Concerning validity, CFA did not confirm the 12-factor dimensional theoretical model proposed by the original questionnaire, as evidenced by the fit index values and eigenvalues greater than 1.
The EFA for all 42 items presented values that can be considered adequate fit indices for models with 11 and 10 factors, but parallel analysis indicated that a 10-factor model would be more appropriate in view of study data.However, low loadings and cross-loadings justified the exclusion of items A9, A16R, C1 and C6R and evaluation of a 38-item model.EFA results from 38 items resulted in a 10-factor model with acceptable fit indices.Hair et al. 24 considered that, for 30 or more variables and sample size less than 250, as in this study, good fit is demonstrated by RMSEA values lower than 0.08 and CFI of 0.95 or more.Although, the very low correlations obtained in the 10-factor model are considered to undermine the model's convergent validity, these values pointed to a satisfactory internal discriminant validity.As stated by Brown 25 , factor correlations equal to or above 0.85 are often taken as a criterion for problematic discriminant validity.
The purpose of EFA, besides finding good fit, is to test the dimensional model of the construct.In that respect, the factor model obtained by EFA was satisfactory, and some items loaded in a manner similar to the structure of the dimensional model of the original questionnaire.Items belonging to two dimensions in the original model, loaded on a single factor in our study.The inclusion of items from two dimensions into the same factor, as suggested by the model produced by EFA, was acceptable when the scope of the dimensions involved was related to the same level.For example, items under "handoffs and transition" were aggregated with items under "teamwork across units", since both are at hospital level.Items under "non-punitive response to errors" and "staffing" were aggregated into a single factor, as well as "supervisor/manager expectations and actions promoting patient safety" were added with items under "communication openness", taking into consideration that both dimensions relate to the hospital department or work unit.In the 10-factor model suggested by EFA, 7 factors contained 3 items or more, 2 factors contained 2 items and one factor contained a single item, which weakens the consistency of the suggested model.
From there, continuing to explore for a sound model well fitted to the study data would de-  3) HSOPSC has been used widely in diverse socio-economic and cultural contexts and at different levels of monitoring and concern with quality care and patient safety, at differing stages of development, with results that mostly confirm the original model 12,17,18,19,20 , thus suggesting that HSOPSC is an instrument which enables different levels of patient safety culture to be detected and compared.
On the other hand, some psychometric validation studies did not confirm the dimension structure of the questionnaire as proposed in the original model.Smits et al. 16 obtained a model with 11 dimensions, which together explained 57.1% of the variance.After excluding the items under "Staffing", Blegen et al. 21also obtained an 11-dimension model.Bodur & Filiz 11 identified 10 dimensions of safety culture when validating the Turkish version HSOPSC, while maintaining all 42 items of the original model proposed by AHRQ.In the validation study of the English HSOPSC (n = 1,017), Waterson et al. 22 estimated a 9-dimension model that explained 66.8% of data variance.In the Swiss study (n = 2,421), Pffeifer & Manser 14 obtained a dimensional model with 8 factors that explained 59.8% of variance in item responses.

Study limitations
One of the limitations encountered was a high rate of missing data, i.e., questionnaires where the 42 items were only partially answered, which may possibly have impaired the estimates of the instrument's reliability and construct validity.Of the 322 questionnaires included in the study, 105 lacked at least one datum, and led those instruments to be excluded from the factor analysis stage.Although at the initial stage of data analysis, use of missing-data imputation techniques was considered, this methodological option was discarded due to the impact on the estimates, and also because the directionality resulting from the techniques examined was unknown, given the small number of cases included in the study.
Another important limitation of this study was the high rate of potential respondents refusing to participate.The response rate showed that more than 50% of those eligible refused to answer the questionnaire and various reasons were given to justify that refusal.The questionnaire is lengthy, and health professionals in hospitals are usually overloaded with work, making them less readily available.Although low, the response rate was similar to those of studies in other countries.The instrument's developers in the user manual provided mention a response rate of around 50% in situations where researchers are careful enough to maintain participants' anonymity and distribute the questionnaire on paper 3 , as was the case in this study.The small sample affects data variability, impairing psychometric analyses and causing non-response bias.The items' behavior in the EFA analysis may be due to cultural and contextual effects that would compromise the comprehension of the questions as well as the level of concern and familiarity to safety patient issues.
We chose to collect data on a printed questionnaire with the aim of facilitating respondent participation, but in a scenario where increasing use is made of information technology, this could be replaced by data collection online or by email.However, this decision to collect data using hard copy questionnaires reflected issues relating to requirements from the research ethics committee, after respondents had filled out their consent forms agreeing to take part in the research.

Conclusions
Patient safety culture among healthcare professionals is gaining increasing attention.It is considered to be an aspect of organizational culture and is reflected in the behavior of individual members of an organization, unit or team, especially in the values and beliefs they share, and in what priority these professionals give to patient safety.Characteristically, in a strong patient safety culture, teamwork is valued, the organization's leaders engage with the challenges involved in providing safe health care, and those leaders take a pro-active attitude to errors, supported by open communication among staff and encouragement for people to report safety incidents and, consequently, the promotion of effective organizational learning from those incidents 29 .
Assessment of the status of safety culture in hospitals has been the subject of several studies at the global level.These have included pre- The main aim of this study was to assess the reliability and validity of the Brazilian version of the HSOPSC, a patient safety culture assessment instrument.In that respect, it has achieved its goals.Although the dimensional structure proposed by the original model of the instrument was not confirmed with the sample of this study, given the importance of the subject and scarcity of Brazilian studies that have assessed the HSOPSC's psychometric properties, we believe that the decision to reduce this instrument would be premature before testing in larger samples of health professionals working in accredited hospitals.
It is hoped that the availability of this instrument in the sphere of hospital care in Brazil will contribute to improved strategies and measures, to the benefit of patients and of professionals involved in providing healthcare that aspires towards creating the appropriate conditions for safer, more effective, timely, patient-focused, efficient and equitable health care 30 .

Contributors
C. T. Reis elaborated the project design, data analysis and interpretation, writing of the article, relevant critical revision of the content.J. Laguardia and M. Martins collaborated in the project design, data analysis and interpretation, writing of the article, relevant critical revision of the content.A. G. G. Vasconcelos collaborated in data analysis and interpretation, writing of the article.

table 1 original
version of the Hospital Survey on Patient Safety Culture questionnaire.
a9Mistakes have led to positive changes here a13 after we make changes to improve patient safety, we evaluate their effectivenessSupervisor/Manager expectations & actions promoting patient safety B1 My supervisor/manager pays a compliment when he/she sees a job done according to established patient safety procedures B2 My supervisor/manager gives serious consideration to staff suggestions for improving patient safety B3R Whenever pressure builds up, my supervisor/manager wants us to work faster, even if it means taking shortcuts B4R My supervisor/manager overlooks patient safety problems that happen repeatedly Kolmogorov-Smirnov tests and calculate correlations among the dimensions, CFA and EFA Cad. Saúde Pública 2016; 32(11):e00115614 | www.ensp.fiocruz.br/csp

table 2
Hospital Survey on Patient Safety Culture factor reliability.Brazil, 2012.

table 4 exploratory
factor analysis.eigenvalues, factor loads in each item and their residual variances for 38 items.Brazil, 2012.
Cad. Saúde Pública 2016; 32(11):e00115614 | www.ensp.fiocruz.br/csppendonfurther exploratory factor analyses and removal of problematic items until achieving a satisfactory model.The authors chose not to do so for the following reasons:(1)this study, the first to investigate the psychometric properties of the Portuguese version of the HSOPSC for use in Brazil, is exploratory in nature and, accordingly, it is advisable to test this version in other larger Brazilian samples; (2) development of the HSOPSC theoretical model involved thorough, well-documented studies by AHRQ, indicating caution and parsimony in the use of empirical results and changes to the original model after comprehensive tests, including qualitative studies, and use of larger samples; and ( Cad. Saúde Pública 2016; 32(11):e00115614 | www.ensp.fiocruz.br/cspand post-intervention study designs that showed needs perceived in a pre-intervention assessment, and gains from effective implementation of safety interventions, translated into stronger safety culture assessed post-intervention.