Quality in healthcare and patient safety: evaluation of the results of a distance learning program

Abstract

This paper aims to present an analysis of the Quality in Health and Patient Safety Training Program (CQSSP) results of the Sérgio Arouca National School of Public Health. This is a quantitative, qualitative, cross-sectional, and descriptive study based on the Kirkpatrick’s Evaluation Model. The findings provide us with a clearer picture of the success of the CQSSP pedagogical intent through a positive assessment for the Reaction, Learning, Behavioral, Results levels. From the analysis of the perception of the graduates, we observed that 82% of them started to fully consider the occurrence of an error in the provision of care, 68% incorporated the acquired knowledge into their professional practice, and 73% started to study and research themes related to Patient Safety (PS). We also conducted a survey on the number of Patient Safety Centers or equivalent bodies established or reformulated during or after the CQSSP. We also evaluated the extent of the contribution of the course in the implementation of Patient Safety protocols and actions in hospitals. The findings point to a significant contribution of the CQSSP to PS and the quality of hospital care.

Key words:
Patient safety; Health care quality; Educational evaluation; Institutional evaluation; Professional qualification

Resumo

O objetivo deste artigo é apresentar uma análise dos resultados do Programa de Formação em Qualidade em Saúde e Segurança do Paciente (CQSSP) da Escola Nacional de Saúde Pública Sergio Arouca, da Fundação Oswaldo Cruz. Trata-se de uma abordagem quantitativa e qualitativa, transversal e descritiva, baseada no Modelo de Avaliação de Kirkpatrick. Os achados dimensionaram o êxito da proposta pedagógica do CQSSP com avaliação considerada positiva nos níveis de Reação, Aprendizagem, Comportamento e Resultados. A partir de análise da percepção de egressos, observou-se que 82% destes passaram a considerar plenamente a ocorrência de erro na prática do cuidado, 68% a incorporar os conhecimentos adquiridos à sua prática profissional, e 73% passaram a estudar e pesquisar temas relacionados à segurança do paciente. Inquérito captando o número de Núcleos de Segurança do Paciente ou instâncias equivalentes constituídos ou reformulados durante ou depois do CQSSP foi realizado. Avaliou-se, ainda, o grau de contribuição do curso na implementação dos protocolos e ações de Segurança do Paciente (SP) nos hospitais, refletindo, assim, a significativa contribuição do CQSSP para a segurança do paciente e qualidade do cuidado hospitalar.

Palavras-chave:
Segurança do paciente; Qualidade da assistência à saúde; Avaliação educacional; Avaliação institucional; Formação profissional

Introduction

Recent decades have witnessed a growing debate and actions to provide care that values quality and safety. The To Err is Human report, by the Institute of Medicine (IOM) of the United States, is the milestone for patient safety (PS)¹. This report indicated that about 100,000 people die each year from adverse events². The magnitude of PS-related issues has been described in several studies in the U.S., Australia, the United Kingdom, Portugal, and Brazil³.

Several important global PS-oriented initiatives were developed, among which the World Health Organization (WHO) establishing the World Alliance for Patient Safety in 2004, which unfolded into a program⁴ whose lines of action were Education and Research initiatives, such as the creation of a specific taxonomy and campaigns called “Global Challenges” geared to curbing healthcare-related infections, improving safety in surgical care and reducing medication-related errors⁵. In 2020, celebrating the World Patient Safety Day, the WHO launched the campaign “Keep health workers safe to keep patients safe”, drawing attention to the importance of PS professionals.

In Brazil, in 2013, the Ministry of Health (MS) launched the National Patient Safety Program (PNSP)⁶. Then, the National Health Surveillance Agency (ANVISA) established actions for PS in health services⁷. The PNSP is structured into four lines of action: encouraging safe care practice; patient involved in their safety, the inclusion of the topic in Education; and increment to research⁸.

Given the proposal to include PS in Education, in partnership with the New Lisbon University, the National School of Public Health (ENSP) of the Oswaldo Cruz Foundation (FIOCRUZ) developed the International Quality Patient Health and Safety Course (CQSSP) to develop skills and scientific knowledge on care quality and PS.

As a public policy strategy, recognizing the leading role of this initiative and the relevance of its evaluation, we employed an evaluation model of training processes used in Training, Development, and Education (TD&E). TD&E is a system integrated by subsystems, where evaluation is the subsystem responsible for providing information and feedback and improving the entire system⁹.

Donald Kirkpatrick’s Four-Level Assessment Model has been widely accepted in many types of organizations since 1959. By proposing to assess the levels of Reaction, Learning, Behavior, and Results, it allows a comprehensive look at the training process and the results achieved¹⁰. Furthermore, given the theoretical-practical character of the quality of care and PS outlined in the CQSSP, Kirkpatrick’s model was presented as a potential evaluative tool suitable for dimensioning the bridge between training and practice. Shortell and Richardson (1978) argue that a program is an “intervention as an organized response to health problems, achieving goals, to improve the health of a population”¹¹. Admitting the CQSSP as a program within the PNSP, its evaluation is outlined as evaluative research, that is, a “procedure that consists of making an ex-post judgment of an intervention using scientific methods”¹², seeking to know, thus, the effects of the CQSSP in the transformation of reality and its impact on health care.

The study adds relevance as a retrospective evaluative approach, considering the context of the COVID-19 pandemic that affected the living conditions of populations and impacted health organizations. This context changed priorities and imposed the centrality of actions in the field of health. Old and new risks related to the care quality and PS emerge. Thus, courses with this theme, in the distance modality, anchored in a proposed intervention to improve the care provided, can be a strategy to achieve the SDGs, contributing, above all, to the objective 3 that proposes “To ensure access to quality health care and promote well-being for all, at all ages”¹³.

This study aimed to evaluate the quality in health and patient safety specialization course (CQSSP) offered by ENSP based on Kirkpatrick’s model, considering aspects of training, learning, practice, and production of results. The two guiding questions were: (i) As an intervention program, has the CQSSP produced results in health organizations? (ii) What characteristics were different in this intervention as a training process offered in the Distance Learning (DL) modality?

Methods

This is a cross-sectional study with a descriptive quantitative approach that involved qualitative elements. We analyzed data from an electronic questionnaire applied to CQSSP graduates and information available on the course platform in the quantitative approach. The qualitative approach was based on interviews with actors involved in the development and implementation of the course and on the review of the course documentation to support the construction of the electronic questionnaire.

According to the evaluation model applied, the training process must be evaluated at four levels: Reaction, Learning, Behavior, and Results. As it is a theoretical model classified as generic⁹, it allowed using multiple techniques and research instruments. The organization of information for each level is shown in Chart 1. The items that made up each category were measured using a five-point Likert-type scale.

Chart 1
Kirkpatrick’s Four-Level Model: aspects evaluated, evaluative categories, choice criteria, and source of information and data.

Evaluation level	Aspect evaluated	Evaluation categories	Criteria for the selection of the categories	Information and data sources
1.Reaction	The evaluation at the Reaction level evidences the opinion of CQSSP graduates from the state of Minas Gerais on various aspects of the course. The results were measured according to the Likert Scale, which underwent adaptations to better fit the achievement of the research objectives.	- Didactic Material and Program Content;	Selected from the content analysis of the document review and interviews.	- Interview and document review. - Assessment questionnaire applied to graduate students.
2. Learning	At this level, the knowledge acquired by students is evaluated, in terms of acquisition of knowledge, development of proposed skills and ability to change their attitude. Data were collected on the course platform and organized into tables and charts for analysis.	Learning units I, II, III, IV, and TCC.	The results of the learning units and the TCC of the students were compiled and analyzed, demonstrating the performance of the students throughout the course.	The results of the learning units and the TCC of the students were compiled and analyzed, demonstrating the performance of the students throughout the course.
3. Behavior	It concerns the share of knowledge, skills, and attitudes that are transferred to work. Behavioral changes were evaluated based on the self-perception of graduate students. The results were measured according to the adapted Likert scale	- Participation of CQSSP graduates in discussions about PS; - Encouraging team interaction and support in PS issues; - Incorporation of learning into professional practice; - Consideration of error as a possibility in professional practice; - Study and research on topics related to patient safety, risk management, and health quality; - Proactivity in identifying the existing risks in the hospital and the best ways to reduce them; - Proposed actions in PS to contribute to a change in the hospital’s culture; - Encouraging patient participation in matters of their safety. - Valuing error as an opportunity to improve the quality of care; - Opportunity to apply the knowledge acquired in the course.	The desired skills with CQSSP and possible behavioral changes expected with the training process were pointed out through the document review of the CQSSP and interviews with the general coordinator of the course, the pedagogical coordinator, the learning advisor and the tutor. Considering these notes, articulated with the CQSSP’s objectives, called the alumni profile, self-evaluative questions about these expected behaviors were inserted in the questionnaire sent to the students.	Interview and document review. Assessment questionnaire applied to alumni.
4. Results	It aimed to describe the actions for patient safety and quality improvement of care implemented from the students’ performance, relating them to the contributions of the CQSSP. A block of specific questions about results was included in the questionnaire applied to graduate students. The results were compiled and organized into charts and tables for analysis.	- Establishment and operation of Patient Safety Centers (NSPs); - Incident notification system; - Safety culture; - Implementation of protocols; - Continuing/permanent education actions. - Contributing factors for the implementation of PS actions: physical structure, management support, team support, external factors, communication.	The evaluative categories were selected according to the actions proposed in the PS plans, prepared by the students/teams of each hospital in the TCCs, given the legal regulations set out in ordinance MS 529/2013 and in RDC ANVISA 36/2013.	Interview and document review. Assessment questionnaire applied to alumni.

• Source: Author’s elaboration.

The CQSSP is a lato sensu specialization course offered by ENSP, in the distance/semi-classroom modality, with a workload of 408 hours, requiring a minimum weekly dedication of eight hours from the participating professionals. The first edition, the object of this analysis, was held between September 2014 and September 2015. One thousand fifty spots were offered (1,000 students from Brazil, 25 from Portugal, and 25 from African Countries of Portuguese Official Language (PALOPS) to a target audience of higher education professionals working in hospitals with 200 or more beds.

The interviews carried out with the general coordinator of the course, the pedagogical coordinator, a learning advisor, and a tutor were based on pre-established roadmaps, which sought to respect the specificity of the type of inclusion of each stakeholder.

The document review was based on CQSSP records that refer to the virtual platform, the course monitoring form (FAC), the course project, the course evaluations, the student notebook, and the course completion papers (TCC) elaborated as PS improvement plans. An extraction roadmap created for this purpose was applied for collecting documentary data.

The interviews and document review considered the following CQSSP aspects: organization and planning (background, need, objectives, and thematic content); structuring and implementation (coordination, selection of tutors, selection of participants, physical structure, and teaching resources); evaluation (institutional evaluation of learning and institutional evaluation of the course and results); and results linked to the course objectives and those in the work plan.

A questionnaire was elaborated from the previous steps and applied to a convenience sample consisting of 52 students from the course in the state of Minas Gerais, selected for the CQSSP based on the indication of the directions of the hospitals located in that state. The state of Minas Gerais was selected because of its more stable situation in the face of the economic, social, and political crises affecting the country when the study was developed, which could considerably influence the results.

The semi-structured questionnaire sent to CQSSP graduates included students’ reactions, behavioral changes, and results from the training process. It was prepared in digital form, using Google Forms’ platform, and sent via e-mail to students in 2017. An initial prototype of the questionnaire was applied to two graduate students, not included in the sample, to assess its clarity, scope, and time taken to reply. Searching the platform, we collected data on the performance of students in the different CQSSP modules and the course completion work. This information was used for the assessment at the Learning level of the model.

We used content analysis as a technique adapted to meet the proposal of this study in the qualitative approach of the research, which involved interviews and document review. We pre-explored course-related documents and proceeded with a free-floating reading of these texts and the interviews, following steps proposed by Campos¹⁴. Then, the units of analysis were selected and categorized based on the study’s theoretical framework. This theoretical foundation aimed to provide intrinsic validity to the result of the analysis. The units of analysis were grouped as categories or subcategories by frequency and implicit relevance.

In the qualitative approach, the information obtained with the application of the questionnaire and collected on the course platform was analyzed using descriptive statistics. Quantitative information was stratified by categorical variables: professional category, seniority in the profession, management position, participation in Patient Safety Centers (NSPs), and facility’s management type.

ENSP’s Research Ethics Committee approved the research project under Opinion.

Results

The CQSSP had particularities regarding the motivations and context in which it was developed, its structure, assumptions, and pedagogical instruments. It is aligned with global initiatives and was created to find solutions to PS-related issues in Brazil, Portugal, and the PALOPs, and associated with the interest of the Schools of Public Health in Brazil and Portugal in expanding their partnership.

The scientific production on PS and the experience accumulated by the two schools, an already existing discipline in the stricto sensu ENSP course, and a PS course at the Portuguese school, were facilitating elements for the establishment of the theoretical and conceptual bases of the CQSSP that formed the syllabus, culminating in the development of specific teaching material (Chart 2). The content was divided into three large blocks: (i) theoretical and historical notion of PS, (ii) care practice issues sensitive to problems in care quality and PS, and (iii) measures and actions to manage PS issues in organizations.

Chart 2
Curriculum organization of the Health Quality and Patient Safety Specialization Course.

Learning Unit (LU)	Objectives	Contents	Workload
LU I Health Quality and Patient Safety: fundamental aspects	Introduce the concepts, historical trajectory, and rationales related to patient safety.	1. Health care quality 2. The historical perspective and key developments in patient safety 3. Taxonomy in patient safety 4. Health care error and violations 5. Magnitude of the problem and factors contributing to error and adverse events 6. Patient rights and safety	80 hours
LU II Risk assessment and management in healthcare organizations	Identify the types of risks existing in a health service and the solutions already known to mitigate or avoid them.	1. Infections associated with healthcare 2. Medication-related errors 3. Safe surgery 4. Patient safety and diagnosis 5. Laboratory-related errors 6. Risk management of falls, pressure ulcers, and incidents related to blood and blood product transfusions 7. Overcrowding of emergency services 8. Nonclinical risk management 9. Safe maternity 10. Patient Safety in Primary Health Care	96 hours
LU III Patient health and safety management: cross-sectional and transdisciplinary aspects	Develop capabilities for managing patient safety in health services, including research in the area.	1. Safe and robust/reliable healthcare organizations 2. Economic and social consequences of errors and adverse health events 3. Accreditation and patient safety 4. Patient safety culture 5. Patient safety indicators 6. Occupational health, ergonomics, and patient safety 7. Communication between health professionals and patient safety 8. Patient involvement: challenges, strategies and limits 9. Most relevant aspects in patient safety investigations/research	152 hours
LU IV Scientific methodology	Use the scientific method to understand the reality of your organization and develop strategies that can contribute to improving patient safety.	1. What is the scientific method 2. Patient safety research/investigations	40 hours

• Source: Student’s Journal of the CQSSP, ENSP/Fiocruz,

The CQSSP was structured on a clear line of thought to change the PS culture in hospitals: assuming error as a possibility and a human condition and not blaming professionals as the main guidelines for PS-oriented actions. CQSSP’s methodological assumptions are built on the competence theory, involving an interactive and continuous pedagogical work, articulating theory and practice in constructing new knowledge. Thus, the reality experienced by the student was brought into the context, enhancing learning and skills development. The method included learning tools that facilitated the transfer to practice. Some methodological specificities of the CQSSP stand out:

Student/team concept/strategy as a learning unit: formed by four professionals from the same hospital, who studied and worked together, sharing learning and work.

Active methodologies: theory and practice articulation;

Construction of a PS plan by the students: it was a singularity of the CQSSP;

Pedagogical mediation: the tutor was the “mediator” who guided the student in constructing knowledge, valuing his practice and experience;

Teaching material: the course was supported by books, the theoretical basis of the course;

Procedural Assessment: assessment that concerned the student’s trajectory within the course, considering his commitment and dedication much more than a finished product of knowledge;

As a facilitator of the training process, technology is committed to enhancing learning resources and reaching the most significant number of students and health organizations. It also aimed to promote interaction between these stakeholders, responsible educational institutions, and health organizations.

Based on their perceptions and experiences in the CQSSP, the assessment made by the key actors conducting the training process was quite positive, highlighting the theme of interest to health professionals, the teaching material, the structure, teacher support, and the student/team strategy as facilitators of the training process. The political and economic crises that affected the hospitals, the difficulty of bringing the management of institutions into the course, the asymmetric training of professionals, the methodological difficulties due to teaching/learning method, as it is a new process, the criteria for appointing discretionary participants from management not always including professionals most suitable for the course, and the workload of students that could conflict with the necessary dedication were pointed out as obstacles to the training process.

The evaluation of students/alumni was based on a questionnaire built per the Kirkpatrick’s Four-Level Model and included 34 students (65%) of the total of 52 graduates from Minas Gerais, with computed losses of 35%. Forty-six students completed the course, to whom 45 questionnaires were effectively applied and from which 34 responses were obtained, corresponding to 76% of the total.

The profile of the 46 students who graduated from the CQSSP corresponded to the guidelines of the course coordination, as most were nurses (44%), pharmacists (26%), and a smaller percentage of doctors (15%). The participation of other professional categories (15%) indicated that the PS theme is essentially multidisciplinary, as disseminated by the WHO. Concerning the 34 students who responded to the questionnaire, the share of students who had never participated in the PS responsible bodies (6%) and the high proportion of students (35%) who participated but later left the NSPs or equivalent bodies (IEs) caused a stir.

The profile of respondents primarily consisted of experienced professionals with advanced seniority in the institution: 44% of participants had 11-20 years seniority, 29% had more than 20 years, and only 24% had 6-10 years. A relevant piece of information was the significant number of graduates in management positions: 71% held management positions before taking the course, and 62% continued to hold management positions at the time of the survey.

The Reaction level evaluation established evaluative categories, units of analysis included in these categories, and evaluative dimensions related to the units of analysis. In the evaluative category Teaching Material and Program Content, the reaction of respondents was quite positive for their units of analysis and evaluative dimensions (Table 1).

Table 1
CQSSP assessment at assessment levels: reaction, learning, behavior, and results.

Reaction Level	CQSSP Reaction Level Assessment of students from the state of Minas Gerais, best and worst analysis units evaluated by category Brazil, 2017.
	Assessment Categories / Units of Analysis	Totally Disagree	Partially Disagree	Neither Disagree or Agree	Partially Agree	Totally Agree	Total
	Assessment Category: Learning Material and Program Content	-	-	neutro	+	+	Total
	Best: Theoretical Content / relevant to the health area	0%	0%	0%	6%	94%	100%
	Worst: Theoretical Content/ it was adequate to my expectations	0%	3%	3%	26%	68%	100%
	Assessment Category: Activities	-	-	neutro	+	+	Total
	Best: Activities Requested in the AVA/Adequate time for its realization	0%	3%	0%	41%	56%	100%
	Worst: Forum and chats / Promoted greater interaction of the hospital group	3%	15%	6%	50%	26%	100%
	Assessment Category: Tutorship System	-	-	neutro	+	+	Total
	Best: Tutor/ he/she was committed	0%	0%	3%	18%	79%	100%
	Worst: Tutor/ He/she used good strategies to facilitate student learning	0%	3%	15%	36%	46%	100%
	Assessment Category: Learning support structure	-	-	neutro	+	+	Total
	Best: It had good learning resources (videos, links, and the like)	0%	3%	6%	21%	70%	100%
	Worst: AVA/ It had good data network quality	3%	6%	3%	32%	56%	100%
Learning Level	Individual Performance of CQSSP students in the state of Minas Gerais, Learning Unit I, II, III, and TCC Brasil, 2017.
	Concepts	A (Excellent)	B (Good)	C (Fair)	D (Insufficient)	Concepts	A (Excellent)
	Theme	Assessment Category	9.0 - 10	7.5 - 8.9	6.0 - 7.4	Theme	Assessment Category
	Quality in Patient Health and Safety: fundamental aspects Risk assessment and management in healthcare organizations Patient Health and Safety Management: Cross-sectional and Transdisciplinary Aspects Patient Safety Plan	Performance LU-I	73.91%	19.57%	6.52%	0.00%	100.00%
		Performance LU-II	63.04%	30.43%	6.52%	0.00%	100.00%
		Performance LU-III	73.91%	17.39%	8.70%	0.00%	100.00%
		Performance TCC	65.22%	28.26%	6.52%	0.00%	100.00%
Behavior Level	Assessment of Results Behavior Level of the CQSSP of graduate students from the state of Minas Gerais, Brazil, 2017.
	Protocols	Totally Disagree	Partially Disagree	Neither Disagree or Agree	Partially Agree	Totally Agree	Total
	Participation of CQSSP graduates in discussions about patient safety at the hospital where they work after the course	3%	6%	6%	41%	44%	100%
	Encouragement of team interaction, and support in patient safety issues, provided by CQSSP graduates at the hospitals where they work	3%	0%	9%	41%	47%	100%
	Incorporation of knowledge into professional practice by CQSSP graduates	0%	3%	3%	26%	68%	100%
	Percentage of CQSSP graduates who started to consider error as a possibility in professional practice after the course	0%	0%	6%	12%	82%	100%
	Percentage of CQSSP graduates who began studying and researching on topics related to patient safety, risk management, and health quality after the course	0%	3%	3%	21%	73%	100%
	Percentage of CQSSP graduates who started to identify the existing risks in the hospital and the best ways to reduce them	0%	3%	0%	35%	62%	100%
	Percentage of CQSSP graduates who started to propose patient safety actions to contribute to the change of culture in the hospital after the course	3%	3%	0%	24%	70%	100%
	Percentage of CQSSP graduates who began to encourage patient participation in issues of their safety after the course	6%	3%	24%	32%	35%	100%
	Percentage of CQSSP graduates who started to value error as an opportunity to improve the quality of care	0%	0%	3%	15%	82%	100%
	Relationship between learning the CQSSP and the implementation of patient safety protocols in the
	Assessment	Patient identification	Hand hygiene	Pressure ulcer prevention	Prevention of falls	Safety in the prescription, use, and administration of medicines	Safe surgery
		%	%	%	%	%	%
	Disagree	0%	20%	20%	20%	20%	0%
	Partially Disagree	0%	0%	0%	0%	0%	0%
	Neither Agree nor Disagree	0%	0%	0%	0%	10%	0%
	Partially Agree	50%	10%	30%	40%	30%	40%
	Totally Agree	10%	20%	0%	30%	30%	20%
	Not applicable	0%	0%	30%	0%	10%	20%
	Not applicable because the protocol was already in place before the course.	30%	50%	20%	0%	0%	10%
	Cannot answer	10%	0%	0%	0%	0%	0%
	Did not answer	0%	0%	0%	10%	0%	10%
	Total	100%	100%	100%	100%	100%	100%

• Source: Author’s elaboration.

Graduates reacted positively concerning the evaluative category “Activities”. In the Activity Requested in the Virtual Learning Environment (AVA) unit of analysis, the highest disapproval rates were in the dimensions Interest and Stimulus to search for new references. Six percent of students partially disagreed with the assertion in both dimensions. The Forums and Chats analysis unit had the lowest approval, and the highest percentage of disapproval was in the dimension that investigated the students’ opinion about this type of activity to promote interaction between the hospital staff, where 3% totally disagreed with the statement, 15% partially disagreed, and 6% neither disagreed nor agreed (Table 1).

The students’ reaction was positive in evaluating the Tutoring System analysis category in the Tutoring and Pedagogical Mediation unit of analysis. However, in the evaluative dimensions that questioned whether the tutor used good strategies to facilitate student learning, 15% maintained the neutral position and 3% partially disagreed; in the dimension in which it was asked whether the tutor helped students to build their knowledge instead of giving ready-made answers, 9% remained neutral, and 3% partially disagreed; and, if the tutor guided the construction of the TCC well, 12% remained neutral, and 3% partially disagreed (Table 1).

Regarding the Learning Support Structure category, the percentage of adverse reaction fell on the quality of the data network (internet connection), where 3% totally disagreed that it was of good quality, 6% partially disagreed, and 3% neither disagreed nor agreed with the statement (Table 1).

In the analysis category Factors That Could Impact Students’ Participation in the Course, the most relevant data was that 18% partially disagreed that they were available to fulfill the minimum course load of 8 weekly hours (Table 1).

The Learning level inferred that the CQSSP achieved excellent results. The individual performance of students in the three Learning Units (LUs), added to the individual grade of the TCC, indicated that the mean of graduate students who achieved grade A (excellent) corresponded to 69%, 23.9% achieved grade B (good), and 7.1% achieved grade C (fair); there was no record of failure (Table 1). Learning by professional category did not vary significantly, and student/staff performance did not change because of the geographic location of the hospitals where they worked.

At the Behavior level, the research indicated that the great motto of the CQSSP, graduates valuing error as an opportunity to improve the quality of care, obtained the highest percentages of total (82%) and partial (15%) agreement. Concerning the opportunity to apply what they learned, most students said they had the opportunity to apply the knowledge and skills acquired, which achieved 59% total agreement and 35% partial agreement. Furthermore, 68% of professionals totally agreed, and 26% partially agreed that they started to incorporate course learning into their professional practice. These data show changes in attitudes in professional practice. (Table 1).

The Results level addressed the contribution of the CQSSP in the implementation of PS actions in hospitals. We observed that all established NSPs or IEs, most fully operational (70%). Nurses were the predominant professional category in 100%, pharmacists in 90%, and physicians in 80% of NSPs or IEs. Aside from health professionals, the most frequent category was the administrator, found in 40% of NSPs. However, the dedication of professionals to this body was still problematic: 80% of the hospitals had only a partial dedication to the workload, and only 10% had exclusively dedicated professionals. Most of the NSPs or IEs were linked to senior management, or the general management (30%), or the clinical/technical management (50%). Half only met with such bodies, when necessary, regarding the articulation of NSPs or IEs with higher bodies. In 40% of the hospitals, these meetings were periodic, and the situation was more critical in 10%, with infrequent meetings. As for CQSSP’s contribution, we noticed that most NSPs or IEs were already established before the course (70%). However, 20% were formed during or after the course, and 10% were reformulated during or after the course.

The notification system was in operation in 90% of hospitals, mostly under NSPs or IEs. Thirty percent of hospitals implemented the system during or after taking the course, besides 20% having been improved in the same period. Overall, in 80% of hospitals, notifications followed ANVISA guidelines. The data indicated that 40% of NSPs or IEs devoted more than 50% of their time to notifications, and 20% used between 20% to 50% of their time to notifications. Concerning safety culture, essential to building safe care systems, we observed that 40% of students/teams applied a questionnaire to measure the safety culture.

The patient identification protocol was fully implemented in 50% of the hospitals; it was implemented but not monitored in 30%, and it was partially implemented in 20% (Table 2). Regarding the contribution of the CQSSP to the implementation of this protocol, 50% of the hospitals recognized a partial contribution, and 10% a total contribution (Table 2).

Table 2
Level of implementation of patient safety protocols in hospitals in the state of Minas Gerais participating in the CQSSP, Brazil, 2017.

Protocols	Not planned	Planned and not implemented	Partially implemented	Implemented and not monitored	Fully implemented	Total
Patient identification	0%	0%	20%	30%	50%	100%
Hand hygiene	0%	0%	0%	40%	60%	100%
Pressure ulcer prevention	10%	30%	30%	0%	30%	100%
Prevention of falls	0%	20%	10%	20%	50%	100%
Safety in the prescription, use, and administration of medicines	20%	20%	0%	0%	60%	100%
Safe surgery	10%	0%	30%	20%	40%	100%

• Source: Author’s elaboration.

The hand hygiene protocol was fully implemented in 60% of the hospitals and implemented and not monitored in 40% (Table 2). Regarding the relationship between learning and implementation, the total agreement rate was 20%, and 10% was partial. Half of the hospitals did not recognize a relationship between learning and the implementation of the protocol, as it was already implemented before the completion of the CQSSP, and 20% disagree with the existence of this relationship (Table 1).

The pressure ulcer prevention protocol was planned and not implemented in 30% of the hospitals and was not planned in 10% (Table 2). Thirty percent of students partially recognized the relationship between CQSSP learning and the implementation of this protocol (Table 1).

The implementation of the prevention of falls protocol was fully completed in 50% of the hospitals; it was partially implemented in 10% and was implemented and not monitored in 20%. Only 20% of hospitals planned and did not implement this protocol (Table 2). The relationship between learning the CQSSP and implementing this protocol was highly marked in the opinion of students/teams: 30% totally agreed, and 40% partially agreed, against 20% disagreement about the existence of this relationship (Table 1).

The safety protocol in the prescription, use, and administration of medications was fully implemented in 60% of the hospitals it was planned and not implemented in 20%. Only 20% of hospitals did not plan the protocol (Table 2). There was also significant recognition of the relationship between learning the CQSSP and implementing the protocol, where 30% fully agreed, and 30% partially agreed with this relationship. Disagreement was found in 20% of the total (Table 1).

The Safe Surgery protocol was fully implemented in 40% of hospitals. Thirty percent partially implemented it, and 20% implemented it and did not monitor it. This protocol was not planned in only 10% of hospitals (Table 2). In the students/teams’ opinion, the relationship between learning the CQSSP and implementing the Safe Surgery protocol was positive, and 20% agreed, and 40% partially agreed. Only 10% indicated that this item was not applicable because the protocol was already implemented before the CQSSP (Table 1).

Continuing and permanent education actions, systematic actions with formal policies and programs, covering all professional categories existed in only 30% of hospitals. Regarding the recognition of the influence of the CQSSP on continuing and permanent education actions, 70% of students/teams said that although actions already existed before the course, these were intensified during or after the CQSSP, and they were implemented by the influence of the CQSSP in 10% (Table 1).

Concerning the contributing factors that affected the implementation of PS actions, in the students’ perception of the hospital infrastructure, 9% totally disagreed, and 41% partially disagreed that this favored the provision of safe care. In evaluating the influence of external contributing factors (political, social, and economic) on the implementation of PS actions, 62% fully agreed, and 17% partially agreed with the influence of these external factors.

In the hospitals surveyed, most graduate students felt the support of management (32% partial agreement and 38% total agreement). Regarding meetings between NSPs or IEs with higher levels of management, 40% stated that they participated periodically, and 50% participated only when necessary. However, 32% of the students fully agreed, and 44% partially agreed that NSPs or IEs participated in the hospitals’ deliberations; partial disagreement was 12%, and total disagreement was 6%.

Regarding the contribution of the communication factor between NSPs or IEs and other professionals, it was noteworthy that 32% of graduates totally disagreed, and 12% partially disagreed that this factor contributed to the implementation of PS actions (Table 3).

Table 3
Contributing factors in the implementation of patient safety actions in hospitals in the state of Minas Gerais, Brazil, 2017.

Contributing factors	Totally Disagree	Partially Disagree	Neither Disagree nor Agree	Partially Agree	Totally Agree	Total
The physical infrastructure favors safe care	9%	41%	9%	26%	15%	100%
Management support in the implementation of patient safety actions	0%	24%	6%	32%	38%	100%
External factors (political decisions, economy, and regulation) influenced the implementation of patient safety patients	6%	6%	9%	17%	62%	100%
Participation of the NSP or equivalent body in the deliberations favoring the implementation of patient safety actions	6%	12%	6%	44%	32%	100%
Communication between members of the NSP or equivalent body and other professionals favored the implementation of patient safety actions	0%	12%	0%	56%	32%	100%

• Source: Author’s elaboration.

Discussion

Evaluating implies inferring a value judgment from the measurement of the properties of an object⁹. A fundamental point to be highlighted is that the evaluation of the CQSSP is related to both the training process and its results.

At the Reaction level, it can be inferred that the CQSSP opened horizons for sound learning, achieving a positive evaluation from the graduates. However, the lowest approval percentages in the Forums and Chats analysis unit caused a stir, a finding that deserves attention since the chats and forums in distance learning are essential tools promoting group interaction. Also noteworthy is the evaluation of the Strategies that the Tutor Used to Facilitate Student Learning, the Use of Resources Available in the Course to Establish the Relationship between Theory and Practice, and the Tutor’s Guidance for the Construction of the TCC. Since the percentages of disapproval and neutrality were higher, these units of analysis should also be an essential point of attention.

Regarding the assessment at the Learning level, Kirkpatrick and Kirkpatrick¹⁰ state that measuring learning must determine the knowledge acquired, skills developed, or attitudes modified, since at least one of these points must have been achieved for there to be a behavioral change. However, reaching any one of them does not essentially imply behavioral change. Associating the concepts of learning in psychology and common language, citing Michaelis (2002) and Pozo (2002), Abbad et al.⁹ highlighted that learning results from associative and constructive processes and, as such, allow the “acquisition of different types learning outcomes, from the learning of facts, behaviors, to social, verbal, conceptual, and procedural learning” that will ultimately result in “changes in knowledge structures and creation of new solutions to problems”, which in the transfer to work, learning will involve the acquisition, retention, and generalization of knowledge⁹. The CQSSP was seemingly successful in its endeavor concerning learning, an essential condition for changes in the levels of behavior and results, pointed by the good performance of students in the course modules and the TCCs.

According to Kirkpatrick and Kirkpatrick¹⁰, behavior has to do with the share of knowledge, skills, and attitudes transferred to work. It is worth emphasizing that, although it is stated that the behavior in the position does not make sense for distance courses, as they are not usually offered by an organization different from the one with which students have employment relationships, preferring to use the term transfer of training in the research, the term “behavior” was kept, seeking fidelity to the evaluation model used. However, the educational institution is not the same work organization as the graduates.

Furthermore, the Behavior level was one of the most critical points of the assessment. First, because it was drawn from what was called “Profile of graduates” and represented the success or failure of the training process and its objectives. Otherwise, it was necessary to carry out the desired changes in practice, which turned into results.

According to the students’ positive self-assessment, the learning was transferred to their work practice. It is worth noting that professionals involved in care should be deeply committed to these issues to promote PS, the quality of care, and boost the safety culture in organizations, as the safety culture is a product of values, attitudes, perceptions, competences, and individual and group behavior patterns⁴.

Kirkpatrick and Kirkpatrick¹⁰ consider the Results level the most challenging part of the assessment, given the difficulty of relating these directly to the training process. It is not always possible to obtain concrete evidence that the result stems from the formative process, but it is possible to obtain evidence. Three years into the CQSSP, we can consider a reasonable time to evaluate the implementation of the PS actions described in the hospitals’ Patient Safety Plans.

NSPs or IEs were established and fully operational, or more or less operational, in all participating hospitals, in the latter case a minority. The NSPs play a crucial role in the implementation and articulation of PS actions and could transform into learning and organizational changes the individual demands and experiences of professionals in the care process, as they analyze and act on health incidents while creating mechanisms to convert problems and solutions into organizational guidelines and policies. Some positive evidence points to a portion of CQSSP’s contribution since part of these bodies were established or reformulated during the course. Most was linked to senior management, following the recommendation by ANVISA⁷ that the NSP or IE coordinator should have a seat in the deliberative bodies. The closer to decision levels, the more strategic and widespread can become the PS actions in the organization.

However, the dedication of professionals to the NSP or IEs is still problematic. In most hospitals, the center’s professionals dedicated to only part of the workload; only a small portion had exclusively dedicated professionals, which can occur for two reasons: the low number of professionals available or the NSP is not yet seen with due importance. Regardless of the reasons, this situation can generate work overload, and the professional will not be divided between several tasks without the risk of compromising work quality.

The approach of NSPs or IEs with other sectors proved to be quite shy, which can hinder the spread of PS actions and the dissemination of the safety culture. NSPs or IEs are closer to related areas such as hospital infection, pharmacy and therapeutics, death review, and medical record review committees, but slight approximation with other hospital sectors.

The Notification System was in operation in most hospitals, mainly under the responsibility of NSPs or IEs. However, regarding the contribution of the CQSSP to the implementation of the notification system, it is worth mentioning that 40% declared not to recognize any contribution from the course. The coordination was concerned that the NSPs or IEs would occupy most of their time with the notification system when the importance of participating in internal policies was recognized, and thus promoting PS as a priority. Data analysis indicated that the desired situation did not materialize.

The findings were inconclusive regarding the contribution of the CQSSP to change the patient safety culture, given the design of this study, and most hospitals were not concerned with measuring their safety culture through the application of questionnaires.

At the Results level, the evaluation of the CQSSP for the category Implementation of PS Protocols favors the conclusion that the course collaborated, to a greater or lesser degree, in the adoption of these protocols. Given the information collected, this contribution becomes evident. Implementing the safety protocol in the prescription, use, and administration of medications showed one of the best results.

Regarding continuing and permanent education actions, which are fundamental to promote PS in hospitals, disseminating knowledge and strengthening a culture that values PS and the quality of care, the findings showed that, while these actions were in place in the hospitals studied, a greater systematization was required, with formal policies and programs that cover all professional categories. This situation exists in a small part of the surveyed hospitals.

In the perception of graduate students, half of the hospitals surveyed did not offer an infrastructure that favored the provision of safe care. Another extremely relevant factor, Support from the directors, was felt by most of the graduate students. Management committed to PS issues gives legitimacy to NSPs and reinforces PS importance. Political, economic, social, and cultural external factors also act on organizations. The political and economic crisis in the country in recent years has affected the implementation of PS actions in hospitals. The frequent meetings between NSPs or IEs and higher levels is a sensitive issue. However, low systematization does not seem to affect the participation of NSPs or IEs in the hospitals’ deliberations. A contributing factor, communication between NSPs or IEs and other professionals, significantly influenced the implementation of PS actions, a relevant factor, considering that culture change involves awareness and open dialogue with all professionals.

Some limitations of the study were the short implementation time, the distance from the geographic location of the sample, and the adaptation of the model to achieve the research objectives since it was performed after the course completion.

Final considerations

This study highlights the importance and potential of specialization courses that combine the academic axes and the perspective of transformation in practice in the distance learning modality. Based on this assessment, we can affirm that the CQSSP was successful. By analyzing it, from the moment of its conception to the results described, we could measure the reach of its objectives as a training process and program that aimed to contribute to improved quality of care and PS. Having graduated 855 students, distributed over 50 classes in the national territory, it provided the graduates with supporting elements that transformed the practice’s actions. These actions are the foundation for a cultural turning-point in organizations regarding the difficult task of promoting safe and better-quality care. However, continuity is central to the sustainability of these advances.

The adaptation and application of the model are also considered a significant contribution, which can represent the genesis of a methodological approach for evaluating the results of courses with the same essence or similarity to the CQSSP.

After the edition of the first experience, analyzed in this study, this proposal gave rise to three subsequent calls, however, with a specific focus and by demand outside the ENSP. Thus, in 2016, the Patient Safety Specialization Course was offered to Professionals from the Urgent and Emergency Care Network. The Patient Safety Improvement Course in Maternity Hospitals was developed in 2017. Both were offered on demand by the Ministry of Health. In 2019, the CQSSP was carried out in partnership with ANVISA, aimed at professionals from public and charitable hospitals, with more than 100 beds, from the State Health Secretariats, ANVISA, and the Ministry of Health. Thus, the CQSSP remains in the ENSP’s EAD catalog, but it is carried out on-demand and through partnerships with other bodies, always aimed at strengthening the PS within the scope of the Unified Health System.

Acknowledgments

We are grateful to the National Council for Scientific and Technological Development (CNPq) for their support.

Referências

• ¹
  Trindade L, Lage MJ. A perspectiva histórica e principais desenvolvimentos da segurança do paciente. In: Sousa P, Mendes W, organizadores. Segurança do Paciente: conhecendo os riscos nas organizações de saúde. Rio de Janeiro: Editora Fiocruz; 2014.
• ²
  Institute of Medicine (US) Committee on Quality of Health Care in America. Kohn LT, Corrigan JM, Donaldson MS, editors. To Err is Human: Building a Safer Health System. Washington (DC): National Academies Press (US); 2000. PMID:25077248.
• ³
  Martins M. Qualidade do cuidado em saúde. In: Sousa P, Mendes W, organizadores. Segurança do Paciente: conhecendo os riscos nas organizações de saúde. Rio de Janeiro: Fiocruz; 2014. p. 25-38.
• ⁴
  Reis CT, Martins M, Laguardia J. A segurança do paciente como dimensão da qualidade do cuidado de saúde: um olhar sobre a literatura. Cien Saude Coletiva 2013;18(7):2029-2036.
• ⁵
  Mendes WV, Noronha J. Qualidade de Serviços de Saúde no SUS - Relatório Final. Telma Ruth Pereira, coord. edit.; 2013.
• ⁶
  Brasil. Ministério da Saúde (MS). Portaria nº 529, de 1º de abril de 2013. Institui o Programa Nacional de Segurança do Paciente. Diário Oficial da União, Poder Executivo, Brasília, DF, n. 62, p. 43-44, 02 abr. 2013. Seção 1.
• ⁷
  Brasil. Agência Nacional de Vigilância Sanitária (Anvisa). Resolução da Diretoria Colegiada nº 36, de 25 de julho de 2013. Institui ações para a segurança do paciente em serviços de saúde e dá outras providências. Diário Oficial da União, Brasília, DF, n. 143, p. 32-3, 26 jul. 2013. Seção 1.
• ⁸
  Brasil. Ministério da Saúde (MS). Fundação Oswaldo Cruz. Agência Nacional de Vigilância Sanitária. Documento Referência para o Programa Nacional de Segurança do Paciente. Brasília: MS; 2014.
• ⁹
  Abbad GS, Mourão L, Meneses PPM, Zerbini T, Borges-Andrade JE, Vilas-Boas R. Medidas de avaliação em treinamento, desenvolvimento e educação: ferramentas para gestão de pessoas. Porto Alegre: Artmed; 2012.
• ¹⁰
  Kirkpatrick DL, Kirkpatrick JD. Como avaliar programas de treinamento de equipes: os quatro níveis. Tradução José Henrique Lamensdorf. Rio de Janeiro: Senac Rio; 2010.
• ¹¹
  Caldas BM. Implementação do Programa Nacional de Segurança do Paciente em hospitais públicos: uma avaliação qualitativa [tese]. Rio de Janeiro: Fundação Oswaldo Cruz / Escola Nacional de Saúde Pública Sergio Arouca; 2017.
• ¹²
  Hartz ZMA, organizador. Avaliação em Saúde: dos modelos conceituais à prática na análise da implantação de programas.Rio de Janeiro: Editora Fiocruz; 1997.
• ¹³
  Nações Unidas Brasil. Os objetivos de Desenvolvimento Sustentável no Brasil. [acessado 2020 out 28]. Disponível em: https://brasil.un.org/pt-br/sdgs
  » https://brasil.un.org/pt-br/sdgs
• ¹⁴
  Campos CJ. Método de análise de conteúdo: ferramenta para a análise de dados qualitativos no campo da saúde. Rev Bras Enf 2004; 57(5):611-614.

Edited by

Chief editors:

Romeu Gomes, Antônio Augusto Moura da Silva

Publication Dates

• Publication in this collection
  25 Oct 2021
• Date of issue
  Oct 2021

History

• Received
  20 Nov 2020
• Accepted
  22 May 2021
• Published
  24 May 2021