Challenges in the evaluation of primary care from a quality improvement program

ABSTRACT

Objective

to analyze the work processes resulting from the Program for Improvement of Access and Quality of Primary Care, according to primary care professionals working in attention and at different levels of management.

Method

a single and descriptive case study with 18 care and management professionals in the city of São Paulo, in 2017; use of thematic oral history as a technique of data collection and categorization based on Donabedian’s evaluative triad.

Results

there was a fragmentation between attention and management in the implementation of the Program, influence of organizational and institutional characteristics, in addition to normative implementation, need for performance remuneration review. The Program was a guide for Planning and induced reflection on the applicability of evaluation results and indicators.

Conclusion and implications for practice

the study revealed a predominance of perceptions about structure and process and a greater need for reflection on the impact of quality programs on the results of patient health and care.

Keywords:
Health Evaluation; Primary Health Care; Health Quality Management; Outcome and Process Assessment; Health Care; Unified Health System

RESUMO

Objetivo

analisar os processos de trabalho decorrentes do Programa de Melhoria do Acesso e da Qualidade da Atenção Básica, segundo profissionais da Atenção Básica atuantes na assistência e em diferentes níveis de gestão.

Método

estudo de caso único e descritivo, com 18 profissionais da assistência e gestão no município de São Paulo, no ano de 2017; uso da história oral temática como técnica de coleta de dados, categorização e análise a partir da tríade avaliativa de Donabedian.

Resultados

observou-se fragmentação entre assistência e gestão ao se implementar o Programa, influência de características organizacionais e institucionais, percepção de implantação normativa, necessidade de revisar a remuneração de desempenho. O Programa foi norteador para o uso no Planejamento e mobilizou a reflexão sobre a aplicabilidade dos resultados da avaliação e de indicadores.

Conclusão e implicações para a prática

o estudo revelou predomínio de percepções sobre estrutura e processo, e maior necessidade de reflexão sobre o impacto de programas de qualidade no cuidado e resultados de saúde do usuário.

Palavras-chave:
Avaliação em Saúde; Atenção Básica; Gestão da Qualidade em Saúde; Avaliação de Processos e Resultados em Cuidados de Saúde; Sistema Único de Saúde

RESUMEN

Objetivo

analizar los procesos de trabajo resultantes del Programa de Mejora del Acceso y Calidad de la Atención Primaria, según los profesionales de la Atención Primaria que trabajan en asistencia y diferentes niveles de gestión.

Método

estudio de caso único y descriptivo, con 18 profesionales de la atención y gestión en la ciudad de São Paulo, en 2017; uso de la história oral temática como técnica de recopilación de datos y análisis basada en la tríada evaluativa de Donabedian.

Resultados

hubo fragmentación entre asistencia y gestión al se implementar el Programa, influencia de características organizacionales e institucionales, además de implementación normativa, necesidad de revisar la remuneración por desempeño. El Programa fue una guía para el uso en la Planificación y movilizó la reflexión sobre la aplicabilidad de los resultados de la evaluación y de indicadores.

Conclusión e implicaciones para la práctica

el estudio reveló un predominio de percepciones sobre la estructura y el proceso y una mayor necesidad de reflexión sobre el impacto de los programas de calidad en la atención al paciente y los resultados en salud.

Palabras clave:
Evaluación en Salud; Atención Primaria; Gestión de la Calidad en Salud; Evaluación de Procesos y Resultados en Atención de Salud; Sistema Único de Salud

INTRODUCTION

The implementation of Quality Programs in different care settings, including the involvement of the actors who are part of them, are fundamental elements for achieving the expected results.

Quality in health can be conceptualized in different ways, depending on the perspective studied. From the integration of concepts from different areas, quality can be defined as:

The offer of improved patient outcomes, achieved through team and patient engagement, in the construction of a culture of safety and accountability that is committed to zero error, are efficiently planned and financed, makes use of improvement processes and measurement tools that enable operational changes and are based on a relentless commitment to continuing learning and knowledge transfer.^1:45

The history of studies on quality in health by Avedis Donabedian, starting in the 1960s, went beyond the disease, incorporating themes of prevention, rehabilitation, coordination and continuity of care, patient-health professional relationship, economic efficiency and social values. In an evaluative triad, the author defines Structure as the context in which care is offered, administrative and technical organization of the system, qualification and configuration of providers; Process as the components of care, transaction between professional and patient, and Result as the recovery of health, physiological functions and the search for patient survival, reflected in health indicators that guide the review of processes and the implementation of necessary changes.² The author presents a unidirectional relationship between these three axes, in which an adequate structure should promote effective processes and these, on the other hand, favorable results, aggregating the use of mediation and coherence between these dimensions in the evaluation of quality.³

The health work process, whose purpose is the prevention, maintenance or restoration of health, has as its object the health needs of users and is carried out by health professionals through the use of complex, material and non-material, instruments and resources. It is a reflexive process, essential to society and in which the elements of quality proposed by Donabedian are integrated, as the use of instruments and resources (structure) makes it possible to offer care (processes) and the assistance of patients’ health needs (results).⁴

There are quality initiatives in the different scenarios of Health Systems, including studies on certification for quality in Primary Health Care (PHC)^a that show positive results in relation to the rates evaluated before and after certification, mobilization of changes, improvement of integration in team care, access, resources and information and contribution to the maintenance of quality initiatives, although they also include perception of increased workload, loss of autonomy, limited gains in perceived quality, as well as the fragility of care bonds.⁵

Measuring quality in PHC is a challenge, given the multiple dimension of the concept of performance, the influence of determinants on demand and health results, as well as the complexity of the services offered that include curative, preventive and health promotion actions in the same system level.^5-7 Such measurement is also influenced by the profile of the population served, local characteristics and economic, cultural, socio-political and professional factors.⁶ In addition, quality programs, elaborated vertically, are impacted in its continuity, therefore, “without an engagement and recognition of the subjects involved in the different faces of the policies, most likely, there will be no change”.^8:298

The path of achieving quality in health actions involves evaluative actions that have reflected the trajectory of PHC structuring in Brazil, having as one of the milestones the National Policy for Monitoring and Evaluation of Primary Care in 2005. The knowledge generated has evolved in recent years, based on analysis and evaluation of results, although with few proposals for real improvements and advances in services.⁹

Evaluation corresponds to “the exercise of measuring, understanding and judging the effects of a given intervention, in order to support the choices of the political community in the decision-making process, whether at the stage of formulation or implementation of the evaluated intervention”, which constitutes a process with a political-symbolic character and not just of a technical nature.^8:290 The most recent initiative of health evaluation in PHC is Brazil’s National Program for Improving Primary Care Access and Quality (called the PMAQ-AB in Portuguese), characterized by a model of induction of performance evaluation, created by the Ministry of Health (MH) with adhesion by the city of São Paulo since its first cycle, with the last cycle completed in the External Evaluation in the year 2018.¹⁰

The use of the Donabedian’s Triad to study program results and their configurations in different scenarios, from the perspective of implementers, allows looking at the elements of care and their contexts from a matrix, supporting leaders and managers in identifying weaknesses and strengths, enabling greater assertiveness in initiatives, stimulating investment in these axes in a non-isolated way, allowing comparative data in different PHC arrangements and improving the understanding of primary care provision and its dimensions.^3,11

The use of the Donabedian triad in PHC is observed in the literature, evidencing the need to invest in the qualification of elements of structure, processes or results, as well as the potential to be explored to subsidize public management in the search for quality of services. Few studies have addressed these elements in an integrated way with professionals, managers and their respective impacts on daily work.^12-16

When dealing with evaluation and quality processes in PHC, it is sought that this level of health care plays its role better in the coordination of Health Care Networks, in the control of hospitalizations for sensitive conditions in PHC,¹⁷ and of Programs and Health Policies such as the Family Health Strategy, the National Immunization Program, Prenatal Monitoring, among others. The set of these actions is strategic for the improvement of the health conditions of the Brazilian population.

It is noteworthy that changes in practices in PHC require continuous and lasting investment, and leadership from professionals at all levels; it is therefore essential to seek the effective implementation of quality management programs in health, aiming at the breadth and consolidation of quality practices in services, factors with which the investigation proposed in this study seeks to contribute.

Thus, the study aimed to analyze the work processes resulting from the PMAQ-AB according to primary care professionals working in care and at different levels of management.

METHOD

It is a qualitative Case Study which, according to Yin, is an empirical investigation of a contemporary phenomenon, from an in-depth perspective and considering the case in its context in the real world, especially when the clarity between the limits of the phenomenon and its context cannot be so evident.¹⁸ Correspondence with the evaluative triad proposed by Avedis Donabedian was used to analyze the results presented.

Of a unique and descriptive type, the Case Study focused on the PMAQ-AB in the city of São Paulo and its implementation process in the first two evaluation cycles based on the perception of professionals categorized in the subunits of analysis – Family Health Teams (FHT), Management of Basic Health Units (BHU), Technical Health Supervision (THS), Regional Health Coordination (RHC), Municipal Health Department (MHD) and Social Health Organization (SHO). The COREQ (Consolidated criteria for reporting qualitative research) was used as a guiding instrument, which aims to ensure compliance with internationally recommended criteria for qualitative research, highlighting aspects of the research team, methods and context of the study, results, analysis and interpretations.¹⁹

The study was carried out in an Administrative District of the city of São Paulo with wide coverage of the Family Health Strategy (FHS) and of social and health resources, in the Southeast region, whose implementation of the PMAQ-AB took place through shared management and active teams’ participation, under the management of an SHO. The choice of research participants followed the criteria of having participated in all stages of the PMAQ-AB in the first and second cycles (2013 and 2015), being over 18 years of age and being available to grant an interview during the data collection period. The criteria were first applied to the BHU managers, selecting the eligible services of the territory whose managers met the inclusion criteria, then the workers from the respective BHU who also met the criteria were selected, proceeding in the same way with the responsible managers of the MHD and SHO until there was saturation of the sample. The invitation to participate in the study was carried out in person or by telephone by the researcher and first author of this article for the presentation of the project and the interviews scheduled according to the availability of the participants.

There was no refusal of the research guests. The choice of the region for the development of the study was based on the professional performance of the researcher and author of the study in the territory, where planning and management activities were developed in the Health Services network since 2008, through the partnership by SHO, including actions to implement the PMAQ-AB.

The use of these criteria in the territory studied led to the selection of six technical professionals from the FHS (E), five managers from the BHU (G), one representative from the management of the THS (S), three from the RHC (C), one from the MHD (SMS) and two from SHO (P). The acronyms in the presentation of their speeches were numbered according to the order of the interviews. The average age of respondents was 46 years. The predominant sex was female, with 16 interviewees (88%) and 77% graduated in Nursing.

Data were collected between April 2016 and July 2017. Oral history was used, in thematic modality, as a data collection technique, which allows the targeting of a theme, stimulating the community sense of collective memory from the historical and personal experience of individuals.²⁰

The questions that composed the script for the interviews were: What is your perception about the proposal of the Program and its process of implementation? How did it happen? What were the facilitating and hindering factors of this process? Did the Program influence the work process of those involved? In what way?

The concept of work process used integrated the practices developed by the professionals, each one in their field of activity. For the Teams, care practices, organization and management of the territory were considered based on health needs by and for management, practices developed with the Services to support, and guarantee of the fulfillment of the PHC’s functions. The starting point was the conception of the health work process as “the microscopic dimension of the daily work in health, that is, the practice of health workers/professionals inserted in the daily production and consumption of health services”.^4:323

Each participant was interviewed only once. The interviews lasted an average of one hour, were recorded in audio files and fully transcribed by the researcher.

The analysis of the narratives followed Yin’s proposal for a case study. A descriptive structure was elaborated starting from the set of collected and transcribed results, organized in correlation with the chosen case and with the literature review on the subject. Codes and categories were constructed in association with concepts at the same time that the emergence of patterns was evaluated.¹⁸ At the end of this process, a matrix of categories was developed from the axes of Donabedian’s triad, with fragments of the speeches that were separated and coded according to frequency with the combination of patterns found, enabling the analysis of expected and observed results. 137 Registration Units (RU) were identified, ten codes were established, which gave rise to ten subcategories, grouped by similarity and thematic relevance into two categories as shown in Chart 1 below.

Chart 1
Coding and grouping of subcategories for the elaboration of study categories.

Number of RU	Code	Subcategory	Category
17	A5	Potentialities and weaknesses of the use of the Program in local planning	Structure and Process - Fragmentation between assistance and management in the operationalization of the PMAQ-AB
18	A17	Influence of political relations
22	A19	Lack of follow-up continuity between program cycles
8	A31	Regulatory implementation - perception of obligation
24	A32	Limitations in monitoring and evaluation
8	A45	Financial return by the Program
12	A37	Fragmentation between levels of care and management
14	A42	Consolidation and use of data	Results - Use of Evaluation results to improve practices
7	A44	Forms of return of final grades
7	A43	Reliability of information and computerization

• Source: Authors, 2017.

The research project was approved by the Research Ethics Committee of the Universidade Federal de São Paulo, under opinion No. 1,402,862 and by the Municipal Health Department, under opinion No. 1,473,641. All interviewees signed the Free and Informed Consent Form.

RESULTS

Of the total number of interviewees, the predominant sex was female, with 16 interviewees (88%). The mean age was 46 years, with a minimum age of 31 years and a maximum age of 56 years. There were six respondents from 50 to 59 years old (33%), seven from 40 to 49 years old (39%) and five respondents from 30 to 39 years old (28%). The professional category, the average time since graduation, working in Primary Care and in Management of the interviewees are presented in Chart 2 below.

Chart 2
Characteristics of survey respondents.

		Age	Time working in Primary Care	Time working in Management	Time of academic education (Graduation)
Professional Category	Quantity	Average/Total	Average/Total	Average	Average/Total
Physician	1	56	10	NA	30
Nurse	14	48	12	8	18
Dentist	3	49	14	10	26
Total/Average	18	51	12	9	25

• Source: Authors, 2017.

The categories followed the triad suggested by Avedis Donabedian and the Axes of Structure and Process were unified, elements that reflected in the expected and achieved results, another category determined from the analyzed data.

Structure and Process - Fragmentation between assistance and management in the operationalization of the PMAQ-AB

There was a fragmentation between care professionals and different levels of PC (AB) Management in the implementation of the PMAQ-AB. The change in municipal management, which took place between the PMAQ-AB cycles, influenced the development and monitoring of the Program by local management, a fact superficially addressed by the team members and more directly by management professionals. The vertical implementation and turnover of professionals was also mentioned.

[...]It’s an issue that you know is political. There is no PMAQ in the world that will change that. There are no things you put into effect, if you are implementing something that is absolutely technical and is going to be used and manipulated within a political context. (E4)

As much as it was worked with the Managers, with the Supervisors, that was not how it reached the end. It was not. It was more of a “do it” thing, you have to do it then “carry it out”. So that’s what I felt. (...) Even because the way it got there I don’t know if it was the same way it was passed on here. Because things are getting lost. The “echo” decreases until it gets there. (C2)

The second cycle arrived at a time of political and management transition. So, it happened, but it happened without credibility because the management that came in didn’t realize this, this importance. (...) But, what was strong was that the teams continued what they already knew, and those teams where there was a high turnover, were lost in space. (C3)

So somehow, for example, the issue of voluntary membership that was initially given to the Teams, when it came for us to participate, we practically requested that all the Teams participate. (...) Somehow it wasn’t very voluntary in the sense that at least we asked everyone to sign up, right? (SMS)

The content of the interviews shows that the teams used the planning model suggested by the Program. The theme was little addressed in the interviews of municipal management. The SHO managers highlighted the importance of unifying the planning processes developed in the Units based on the Program Guidelines.

Because that way, the PMAQ started to be used in the headquarters, for one or two years, as a starting point for team planning. So, it was about evaluating what you had done in the previous year, with what you would change for this year. Then it changed, and other types of planning were introduced, more focused on the BHU, on the Unit as a whole, and then we ended up abandoning [the Planning based on the PMAQ]. (E4)

If we have, within the planning and within the PMAQ, the priority groups to monitor, I think we can build our planning scheme for the year already base on the PMAQ. (E5)

[adding PMAQ in Planning] We did this so they wouldn’t think that annual planning is one thing, the intervention matrix (of the PMAQ) another. Annual planning is one way and the intervention matrix is another, the language is a little different, but the objective is the same: to provide quality care to that population. I tell them to choose the themes for the PMAQ’s intervention matrix that are already in their planning. (G4)

I think the proposal of a Planning and Monitoring instrument is powerful for the Health Teams that use this tool. I think it helps to organize the work if used, it facilitates the work of the Team, it is a tool that can and has a lot of potential to trigger good actions and good movements of the Health Teams. (P1)

So, I think that unifying the processes of planning, evaluation, construction of actions and local policies, mainly, should be more articulated, should be more integrated and should always bring back all the instruments that are being used, otherwise you end up making several plans and it is not very difficult for you to see these plans even conflicting with each other, carried out by the same group to be implemented by the same work team. (P1)

Regarding monitoring and evaluation, one of the Program’s axes, the use and perception of the need for greater follow-up, feedback and monitoring of the teams’ actions stands out in the speeches brought by the Team, Management and SHO, in order to maintain the improvement process throughout the evaluation cycle.

But what happens with the PMAQ is that the implementation came, that involvement, let’s make a folder, let’s make a matrix, then we respond, an external evaluation comes and it cools down, even because of the work routine... and it ends up being remembered only in the next cycle. (G5)

[...] Because it comes in the form of a charge, I don’t think it’s educational. It brings a checklist, it kind of mobilizes you to change things, but there's a whole previous process and an after process. And I think this process before and after doesn’t happen. So it’s just that picture, let’s get ready because the PMAQ comes, everyone sets up, that beautiful thing, wonderful thing, the PMAQ passes, everyone goes through the evaluation and then... that only served for the PMAQ. (C2)

I think it [PMAQ] has this potential but it depends on the management, it even depends on the local Health Supervision itself, how they understand it. We see that this too is very fragile. Which is also a matter of demanding to implement and demanding to evaluate, but there is no demand for the process with monitoring of this, right [...] I think it is a cascading event, which when it gets there [in the Teams], it arrives as a demand and not as something to make you think about the process. (P2)

In relation to the link between the Program and the transfer of financial resources for performance, Management’s lack of knowledge about the destination of the resources was demonstrated, considering that the municipality did not choose to directly reward the Teams. The possible disarticulation of the destination of resources and the needs of the Teams and BHU was also brought up in the speech. The issue was addressed by the participants as follows:

[...] The PMAQ follows this logic a lot, you know... access improvement program that are scored and receive government incentives etc... But there is no incentive here as it comes to other municipalities for the teams themselves. It reaches the Secretariat, which will distribute this amount in the way it finds interesting, which will not necessarily get here. (G2)

And also, when I talk about transparency, I think like this: this is an incentive award, let's say, we know that money comes and everything else, but we never know where that was used, exactly what it was used for, should be used for the [Family Health] strategy; is it not for the strategy? [the Program?], so it should be... this municipality did this and this, all this money [from the PMAQ] went to this, we bought equipment for example, etc....(S1)

In the beginning, in the first cycle, we even used the resource to improve the infrastructure of the units. But then in the other cycles this did not happen, it was more in the first cycle that we had this possibility... in some way even a feedback for the teams, although we defended that the teams deserved to have something a little more concrete like equipments or some materials to be able to recognize that that was the result of their performance, but today there is no such provision. (SMS)

Results - Use of Evaluation results to improve practices

The category of results presents explanations about the lack of feedback and space for discussion of the External Evaluation with the teams and BHU in relation to their performance and work processes.

I think we send information and we don’'t have an answer [...]. So, this assessment if being from whom to me? It’s just that issue. We stop for a moment, stress the whole team. Then comes later and says: “You received an eight, this eight is great. “That's not what we want. We want an answer”. Look, your work could be improved here”. (E6)

I also feel the need, as a manager, to receive a visitor who says: “look, come here, your unit was like this, you received such a grade, what can we do to improve... where you lost points...” because sometimes all I can do as a manager I’ve already done, so I need to hear from someone. (G2)

We had some workshops in these two cycles with the Ministry of Health after the certification was released to give feedback to the teams. And even this feedback process, of giving the certification grade, maybe it wasn't as well used as it could have been, understand? [...] And the result arrived a little late, so I think this feedback to the teams was partially used. (SMS)

The underuse of the Indicators that make up the teams’ certification grade to support the assessment of the impact of care on the population’s health outcomes was also addressed in the content of the interviews, as well as the inconsistency of the information systems.

That the big problem, regardless of whether there is a PMAQ or not, whether it is implemented or not, we gather data, but the data does not come back to me. We don’t have an improvement or worsening report: You don’t have an evaluation of the result of your work. It’s impersonal. (E4)

We talked in the first cycle about how the indicators and calculations were evaluated, we showed it close to the external evaluation, but then we did not have this discussion with the teams, it was more isolated. (G5)

So, I think that the information sector keeps a lot of information, it doesn’t distribute the information. This I lack. Because then I come without much basis. So I want to understand what is happening with that unit, all the indicators of that unit... I’ll say... I have a hard time pulling these indicators. [...] And so I think that the sector, Ceinfo (Coordination of Epidemiology and Information, in free translation), not only from here but from the Secretariat, it keeps the information. (S1)

The indicators give us this view that monitoring is really important for us to do. And in the Forums with the regions, the coordinators also brought this feedback, whether or not it was possible for them to have this discussion at the end... and I realized that this was not so well discussed, that it was more used in a region that developed it, the others did not highlight it as something that the teams had incorporated. (SMS)

It is also worth mentioning that there is a period in the cycle now that is more fragile also due to the lack, the rupture of information systems. So one of the stages of the external evaluation is hampered due to the change in the system from the SIAB^bto the E-SUS^cand that the E-SUS is not fully implemented and that therefore we have not been able to provide the information that is necessary for teams to make the diagnoses of their territories and for the external evaluator to make considerations, right? (P1)

DISCUSSION

Program implementation studies identify factors that influence the results achieved and the degree of implementation of these, such as: internal, external, process, program characteristics and individual elements.²¹

The results presented showed that factors of the political context such as hierarchy and institutional identity, structure and organization of work, place of those involved in the health organization, autonomy, governability and disposition of strategies in the implementation process of the PMAQ-AB influenced its operationalization as did the change between the evaluation cycles. The management turnover itself, according to the perception of the interviewees, had an impact on the implementation of the Program, as it implies a greater or lesser emphasis on the proposals presented according to the political lines followed or with the need to establish their preferences in a space of power.²²

Considering the social game theory, the profile of the players and the game conditions of each one, these interfere with the desired results, being the government, in a broad way, considered the result of collective intentionality, based on the offer and confrontation of problems that vary according to the governability of those involved, which may imply, according to perceptions presented in the implementation of actions for quality, elements that must be the result of ethical reflections in public management.²³

The perception of the vertical implementation of the PMAQ, according to the interviewees, can show that the proposed guidelines did not start from a shared and co-responsible discussion, with the interaction between managers and health professionals, which could enable greater commitment to the implementation of innovations.¹¹ In particular in PHC, as it is a network spread throughout the national territory, it is necessary to create alternatives to overcome fragmented work processes and reduce the disconnection between formulators and executors of health actions and the fragility of communication between management and care, allowing for the creating of a sense of co-responsibility for problems and solutions.^22.24

Regarding the use of the PMAQ-AB to carry out the Planning, Monitoring and Evaluation, potential and disposition of the Teams and local management were identified for the practice of the Guidelines and instruments suggested by it. Limitations in the conduct and continuity of these practices were also mentioned, which may occur due to the absence of this routine at the other management levels, impacting as a cascade in care practices. A case study with managers showed limited scope of monitoring and evaluation of municipalities, reinforcing that the less knowledge and appropriation of the process and less participation in the construction, the less chance of using the results of the evaluation.²⁴

Reward mechanisms are configured as internal or external elements that influence the implementation process.²¹ There is a variety of studies regarding the benefits in health outcomes, in addition to the diversity of concepts and models within this payment scope. Results of this modality vary according to the level of health care, to which the remuneration program is linked, the baselines of results prior to the implementation of the program and the methodology used in studies on its impacts.²⁵

The testimonies pointed to the centralization of resources in the municipality and the lack of disclosure regarding their destination, which may have been demotivating. It is noteworthy that in the homologation ordinances published by the MH, it was possible to identify the resources related to the performance in the PMAQ evaluation process for the municipalities and the value of monthly increase per team. However, the information on the composition of the budget is generic, it was not discussed and there was no direct link to the services and teams that were committed to achieving the goals or participation in the budget planning of the variable component of PHC resources by the Teams.

There are controversies in the additional remuneration for health performance, especially in PHC, which requires medium and long term to achieve primary results or even to assess the reduction of health inequalities. There is also the risk of having a negative impact on unpaid activities, deepening health disparities and financial dependence on incentives.^26,27 The literature also raises questions about whether self-motivation is sufficient to maintain results achieved or whether this modality can generate competitive stimulus between teams. Therefore, it is suggested that the financing and planning models be merged, as well as the adequate choice of indicators by the management, avoiding the hinderring or deviation of PHC attributes in this process.²⁸

The use of evaluation results, as in the external evaluation stage of the PMAQ-AB, should be a guiding parameter for decision-making, policy formulation and changes in practices aiming at higher stages of quality. It can also be influenced by the political-organizational context, factors related to the intervention such as utility, information that justifies the intervention, form of disclosure of the findings, among other factors.²⁴ The External Evaluation was carried out by professionals hired by the Ministry of Health, a fact pointed out as fragility by professionals considering the profile of evaluators, especially in the First Cycle.²⁹ These were responsible for collecting and transmitting the data observed in loco, and the technical areas of the MH were responsible for calculating and disseminating these scores via the online system. Although the grades forms of access have been improved over the cycles, their use was limited and little publicized, also from the management perspective, which did not appropriate the results of the evaluation process for health planning.

It should be noted that the practice of performance feedback has been pointed out in the literature as an important tool to promote quality improvement activities, since measurement is fundamental for the evaluation and sustainability of an intervention, as long as it reflects a self-regulatory process and interactive with adequate frequency (real and opportune time between cycles), valid data, clarity of information and analysis of the context, ensuring its use for decision making and course correction.^7,24 Thus, limitations in the return of the results of the external evaluation and the use of the results of the indicators showed unfinished stages of the PMAQ-AB evaluation cycle, which caused the loss of continuity of the process and compromised the achievement of better outcomes, as presented in the speeches.

For the use of indicators as a result of evaluation for improvement, the availability of information systems that guarantee the quality of data is essential, a fact evidenced in the speeches. The construction of indicators for the evaluation of PHC must guarantee the participation of managers and others involved in the final use of the data and must also be carried out based on the plans and objectives for PHC, the context of each country, the structure of the health system, of the population’s needs, among others, enabling its relevance and feasibility.³⁰ It must also be associated with solid data management and allow local leadership and governance according to each reality, providing the comparison of results over time and catalyzing, thus, the improvement processes.

The weaknesses of the PMAQ-AB influenced the professionals’ perception, since the indicators and the information system used were changed over the cycles, which may have impacted on the consistency of monitoring the results. Situations of low reliability or difficulty in interpreting and analyzing indicators by professionals, lack of uniformity in the historical series and availability of data at an inappropriate time for use were identified as limiting the monitoring through quality measures, both in previous studies and in the scenario studied.^11,30,31

In general, the literature points out that quality assessment and certification programs, usually focused on hospital care, according to Donabedian’s triad, bring results more commonly associated with the structure and, to some extent, with the processes. Thus, it was observed in a previous study, results of improvement of certification standards associated with the structure and organization, without the same repercussions in the view of users and collaborators regarding the quality of work and practices in PHC.⁵ It was also noted in the present study, the lack of perceptions about the impact on direct patient care and on care relations, with the results being predominantly focused on the logic of processes of information and feedback systems.

In light of the elements of the structure and processes for improving care mentioned in the study, as well as the use of health results and indicators to identify the need for the territory, we can associate the concepts of equity, legitimacy and effectiveness with the pillars of quality proposed by Donabedian which must also integrate the care and management practice of the Unified Health System (SUS, in Portuguese) and PHC, in line with their principles and guidelines. Thus, the improvement of access and quality will have a direct effect on health indicators and can positively influence the social development of a region or even a country.³² It is worth reflecting more intensively on the importance of using these concepts, allowing greater depth on the results of care practices and the social repercussion of the care provided on individual and community health condition, intrinsic elements of primary care in the UHS. The greater this analysis at the local level, stimulated by management at its various levels, the greater possibilities should arise in the evaluation of the efficacy and effectiveness of care practices.

The results also showed reflections about the elements integrated to the concept of quality in health, such as the offer of better results to the patient, in a planned way, from processes of improvement and use of measures, reinforcing the need to promote the continuity of care learning from these processes in everyday practice.¹

The results of the study also reinforce the need to improve and invest in the development of an evaluative culture in the management bodies of the UHS, including the view of the professional who provides services to the population in the BHU, in order to qualify primary care in the country, improve the implementation of evaluation and quality programs integrating the culture of evaluation to the practice of management, strengthening the decentralization and autonomy of the different instances, based on technical and institutional support, allowing a horizontal and shared management from the perspective of managers and teams in an intervention.³³ Therefore, interventions in Permanent Education are fundamental in a broad way, integrated with investment in the training of professionals for public management, enabling more technical, autonomous, articulated and co-responsible interventions for the continuity of evaluation and quality practices in PHC as well.

FINAL CONSIDERATIONS AND IMPLICATIONS FOR PRACTICE

The presented study demonstrated the influence of the context in the implementation of actions to qualify the Services, given the complexity of multidimensional care and management in PHC, including the characteristics of the Health System where it is inserted.

Time to return PMAQ-AB results, timely use of data and indicators through information systems and qualified management are essential for structuring improvement processes. The monitoring and evaluation cycle based on a technical and continuous support subsidizes a more assertive implementation of the proposed interventions.

Using Donabedian’s triad made it possible to reflect on the investments to be developed in the different axes of Structure, Process and Results to achieve the expected objectives. Pillars of this theory such as equity, legitimacy and effectiveness are in line with the principles and guidelines of the UHS and must be present in the design of care practices.

It is necessary to invest so that evaluation and quality programs in PHC are no longer seen as bureaucratic instruments and become tools for improving the care-user relationship and the population’s health outcomes, stimulating critical and reflective thinking about the objectives of the health system in which the actors are inserted.

It is pointed out as limitations of the study the perspective of professionals who work in a specific region of the city of São Paulo, since there are heterogeneous conditions and characteristics in socioeconomic, cultural, health aspects, etc. in addition to the different distribution of resources, network of services, health needs and local management characteristics in different health regions.

The research also sought to contribute, in a problematizing perspective of the interests of health workers, users, managers and builders of public policies, presenting the repercussion of the implementation of an evaluation and quality program in the daily life of the PHC and in the reflection of the organization of work processes of assistance and management. The predominance of Nursing in this work context, whether in care work or in management, determines a relevant influence on the qualification of processes and on the health results of the population.

In this way, the knowledge produced aims to support the work of health professionals in PHC, encouraging the inclusion of monitoring agendas, shared management, institutional support, qualification and use of health information, valuing work based on the local reality and stimulating systematization of the planning and evaluation processes. It is also expected that the content that emerged will enhance actions to strengthen the implementation of other Evaluation, Quality and Continuing Education Policies and Programs, allowing for more assertive and concatenated interventions between implementers, supporters and executors in this scenario.

• a
  Although the concepts of Primary Health Care (PHC) and Primary Care are discussed in the theoretical field in their use, these terms will be used as synonyms throughout the text.
• b
  SIAB: Primary Care Information System developed by DATASUS in 1998, with the objective of processing information related to Primary Care (PC) with the Family Health Strategy (FHS) as its central strategy, being gradually replaced in 2013/2014 by the E- SUS
• c
  E-SUS: is a strategy of the Department of Family Health to restructure Primary Care information at the national level, restructuring Health Information Systems linked to the SUS (Unified Health System) computerization process.

REFERÊNCIAS

• ¹
  Heenan M. Twenty years later: do we have an agreed-upon definition of health quality management? Healthc Q. 2019;21(4):43-7. http://dx.doi.org/10.12927/hcq.2019.25741 PMid:30946654.
  » http://dx.doi.org/10.12927/hcq.2019.25741
• ²
  Ayanian JZ, Markel H. Donabedian’s lasting framework for health care quality. N Engl J Med. 2016;375(3):205-7. http://dx.doi.org/10.1056/NEJMp1605101 PMid:27468057.
  » http://dx.doi.org/10.1056/NEJMp1605101
• ³
  Ameh S, Gómez-Olivé FX, Kahn K, Tollman SM, Klipstein-Grobusch K. Relationships between structure, process and outcome to assess quality of integrated chronic disease management in a rural South African setting: applying a structural equation model. BMC Health Serv Res. 2017 mar 23;17(1):229. http://dx.doi.org/10.1186/s12913-017-2177-4 PMid:28330486.
  » http://dx.doi.org/10.1186/s12913-017-2177-4
• ⁴
  Peduzzi M, Schraiber LB. Processo de trabalho em saúde. In: Pereira IB, Lima JCF, organizadores. Dicionário da Educação profissional em saúde [Internet]. 2ª ed. Rio de Janeiro: EPSJV/FIOCRUZ-Ministério da Saúde; 2009. p. 320-8 [citado 2020 abr 2]. Disponível em: https://www.arca.fiocruz.br/bitstream/icict/25955/2/Livro%20EPSJV%20008871.pdf
  » https://www.arca.fiocruz.br/bitstream/icict/25955/2/Livro%20EPSJV%20008871.pdf
• ⁵
  Moe G, Wang K, Kousonsavath S. Accreditation: a quality improvement strategy for the comunyty based family practice. Healthc Q. 2019;21(4):13-20. http://dx.doi.org/10.12927/hcq.2019.25746 PMid:30946649.
  » http://dx.doi.org/10.12927/hcq.2019.25746
• ⁶
  Young RA, Roberts RG, Holden RJ. The challenges of measuring, improving, and reporting quality in primary care. Ann Fam Med. 2017 mar;15(2):175-82. http://dx.doi.org/10.1370/afm.2014 PMid:28289120.
  » http://dx.doi.org/10.1370/afm.2014
• ⁷
  Wagner DJ, Durbin J, Barnsley J, Ivers NM. Measurement without management: qualitative evaluation of a voluntary audit& feedback intervention for primary care teams. BMC Health Serv Res. 2019 jun 24;19(1):419. http://dx.doi.org/10.1186/s12913-019-4226-7 PMid:31234916.
  » http://dx.doi.org/10.1186/s12913-019-4226-7
• ⁸
  Sousa AN. Monitoramento e avaliação na atenção básica no Brasil: a experiência recente e desafios para a sua consolidação. Saúde Debate. 2018 set;42(spe1):289-301. http://dx.doi.org/10.1590/0103-11042018s119
  » http://dx.doi.org/10.1590/0103-11042018s119
• ⁹
  Ribeiro LA, Scatena JH. A avaliação da atenção primária à saúde no contexto brasileiro: uma análise da produção científica entre 2007 e 2017. Saude Soc. 2019 jul 1;28(2):95-110. http://dx.doi.org/10.1590/s0104-12902019180884
  » http://dx.doi.org/10.1590/s0104-12902019180884
• ¹⁰
  Portaria nº 1.645, de 2 de outubro de 2015 (BR). Dispõe sobre o Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica (PMAQ-AB). Diário Oficial da União, Brasília (DF), 5 out 2015: Seção 1: 668-9.
• ¹¹
  Rotondo F, Giovanelli L, Fadda N, Ezza A. A methodology to design a performance management system in preventive care. BMC Health Serv Res. 2018 dez 29;18(1):1002. http://dx.doi.org/10.1186/s12913-018-3837-8 PMid:30594191.
  » http://dx.doi.org/10.1186/s12913-018-3837-8
• ¹²
  Turci MA, Lima-Costa MF, Macinko J. Influência de fatores estruturais e organizacionais no desempenho da atenção primária à saúde em Belo Horizonte, Minas Gerais, Brasil, na avaliação de gestores e enfermeiros. Cad Saude Publica. 2015 Sep;31(9):1941-52. PMid:26578018.
• ¹³
  Barcelos MRB, Lima RCD, Tomasi E, Nunes BP, Duro SMS, Facchini LA. Quality of cervical cancer screening in Brazil: external assessment of the PMAQ. Rev Saude Publica. 2017;51:67. PMid:28746576.
• ¹⁴
  Tomasi E, Fernandes PAA, Fischer T, Siqueira FCV, Silveira DS, Thumé E et al. Qualidade da atenção pré-natal na rede básica de saúde do Brasil: indicadores e desigualdades sociais. Cad Saude Publica. 2017 Apr 3;33(3):e00195815. PMid:28380149.
• ¹⁵
  Facchini LA, Tomasi E, Dilélio AS. Quality of primary health care in Brazil: advances, challenges and perspectives. Saúde Debate. 2018;42(1):208-23.
• ¹⁶
  Ferreira J, Geremia DS, Geremia F, Celuppi IF, Tombini LHT, Souza JB. Assessment of the Family Health Strategy in the light of Donabedian triad. Av. Enferm. 2021 dez 10;39(1):63-73. http://dx.doi.org/10.15446/avenferm.v39n1.85939
  » http://dx.doi.org/10.15446/avenferm.v39n1.85939
• ¹⁷
  Dayanna MC. Qualidade da atenção primária à saúde e internações por condições sensíveis: um estudo a partir do Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica [dissertação]. Belo Horizonte: Universidade Federal de Minas Gerais; 2019. 116 p. [citado 2020 jun 12]. Disponível em: https://repositorio.ufmg.br/bitstream/1843/BUOS-BCDJYH/1/vers_ofinal_dissertacaodayanna.pdf
  » https://repositorio.ufmg.br/bitstream/1843/BUOS-BCDJYH/1/vers_ofinal_dissertacaodayanna.pdf
• ¹⁸
  Yin RK. Estudo de caso: planejamento e métodos. 5ª ed. Porto Alegre: Bookman; 2015. 290 p.
• ¹⁹
  Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349-57. http://dx.doi.org/10.1093/intqhc/mzm042 PMid:17872937.
  » http://dx.doi.org/10.1093/intqhc/mzm042
• ²⁰
  Meihy JCSB, Holanda F. História oral: como fazer, como pensar. 2ª ed. São Paulo: Contexto; 2019. Unidade II – Gêneros em história oral, tipos de história oral; p. 38-40.
• ²¹
  Kirk MA, Kelley C, Yankey N, Birken SA, Abadie B, Damschroder L. A systematic review of the use of the consolidated framework for implementation research. Implement Sci. 2016 maio 17;11:72. http://dx.doi.org/10.1186/s13012-016-0437-z. PMid:27189233.
  » https://doi.org/10.1186/s13012-016-0437-z
• ²²
  Dalfior ET, Lima RCD, Contarato PC, Andrade MAC. Análise do processo de implementação de políticas de saúde: um estudo de caso baseado no enfoque da política institucional. Saúde Debate. 2016;40(111):128-39. http://dx.doi.org/10.1590/0103-1104201611110
  » http://dx.doi.org/10.1590/0103-1104201611110
• ²³
  Matus C. Teoria do jogo social. São Paulo: FUNDAP; 2005.
• ²⁴
  Nickel DA, Natal S, Hartz ZMA, Calvo MCM. O uso de uma avaliação por gestores da atenção primária em saúde: um estudo de caso no Sul do Brasil. Cad Saude Publica. 2014 dez;30(12):2619-30. http://dx.doi.org/10.1590/0102-311x00022314
  » http://dx.doi.org/10.1590/0102-311x00022314
• ²⁵
  Mendelson A, Kondo K, Damberg C, Low A, Motúapuaka M, Freeman M et al. The effects of pay-for- performance programs on health, health care, use, and processes of care: a systematic review. Ann Intern Med. 2017;166(5):341-53. http://dx.doi.org/10.7326/M16-1881 PMid:28114600.
  » http://dx.doi.org/10.7326/M16-1881
• ²⁶
  Kovacs R, Maia Barreto JO, Silva EN, Borghi J, Kristensen SR, Costa DRT et al. Socioeconomic inequalities in the quality of primary care under Brazil’s national pay-for-performance programme: a longitudinal study of family health teams. Lancet Glob Health. 2021;9(3):e331-9. PMid:33607031.
• ²⁷
  Barreto JOM. Pagamento por desempenho em sistemas e serviços de saúde: uma revisão das melhores evidências disponíveis. Cien Saude Colet. 2015 maio;20(5):1497-514. http://dx.doi.org/10.1590/1413-81232015205.01652014
  » http://dx.doi.org/10.1590/1413-81232015205.01652014
• ²⁸
  Minchin M, Roland M, Richardson J, Rowark S, Guthrie B. Quality of care in the United Kingdom after removal of financial incentives. N Engl J Med. 2018 set 6;379(10):948-57. http://dx.doi.org/10.1056/NEJMsa1801495 PMid:30184445.
  » http://dx.doi.org/10.1056/NEJMsa1801495
• ²⁹
  Silva LMC, Ferreira LR, Rosa AS, Neves VR. Implementação do Programa de Melhoria do Acesso e Qualidade segundo gestores da Atenção Básica de São Paulo. Acta Paul Enferm. 2017 jul;30(4):397-403. http://dx.doi.org/10.1590/1982-0194201700059
  » http://dx.doi.org/10.1590/1982-0194201700059
• ³⁰
  Dalton AF, Lyon C, Parnes B, Fernald D, Lewis CL. Developing a quality measurement strategy in an academic primary care setting: an environmental scan. J Healthc Qual. 2018;40(6):e90-100. http://dx.doi.org/10.1097/JHQ.0000000000000155 PMid:30113366.
  » http://dx.doi.org/10.1097/JHQ.0000000000000155
• ³¹
  Kampstra NA, Zipfel N, van der Nat PB, Westert GP, van der Wees PJ, Groenewoud AS. Health outcomes measurement and organizational readiness support quality improvement: a systematic review. BMC Health Serv Res. 2018 dez;18(1):1005. http://dx.doi.org/10.1186/s12913-018-3828-9 PMid:30594193.
  » http://dx.doi.org/10.1186/s12913-018-3828-9
• ³²
  Donabedian A. An introduction to quality assurance in health care. 1^st ed. New York: Oxford University Press; 2003. 240 p. (vol. 1).
• ³³
  Flôres GMS, Weigelt LD, Rezende MS, Telles R, Krug SBR. Gestão pública no SUS: considerações acerca do PMAQ-AB. Saúde Debate. 2018 Jan;42(116):237-47.

Edited by

ASSOCIATED EDITOR

Antonio José de Almeida Filho 0000-0002-2547-9906

SCIENTIFIC EDITOR

Ivone Evangelista Cabral 0000-0002-1522-9516

Publication Dates

• Publication in this collection
  29 Apr 2022
• Date of issue
  2022

History

• Received
  04 Aug 2021
• Accepted
  12 Jan 2022