Scale for Assessment of Effectiveness of Family Health Strategy by Users

Escala de Avaliação da Eficácia da Estratégia Saúde da Família pelos Usuários

Escala de Evaluación de la Eficacia de la Estrategia Salud de la Familia por los Usuarios

Abstract

The Family Health Strategy (ESF) is the main gateway to the services of the National Health System (SUS), which needs to be constantly reviewed. This research aimed to create an instrument for assessing the effectiveness of ESF, seizing the beliefs about the effectiveness of ESF of a non-stratified random sample of 390 users in Natal, with data analyzed using descriptive, bivariate and multivariate analysis using the Statistical Package of Social Sciences software for Windows. The new instrument with 25 items and dichotomous scale has one factor, Effectiveness of care at ESF, divided into four sub-factors, level of internal consistency with Cronbach's alpha of 0.77. It is contemplated that the effectiveness of care was evaluated negatively by 74.6% of users. In conclusion, the relevance of the new scale and the need for changes to the operation of the ESF availability of quality services are clear.

Keywords:
family health; program evaluation; factor analysis

Resumo

A Estratégia Saúde da Família (ESF) é a principal porta de entrada dos serviços do Sistema Único de Saúde (SUS), necessitando ser constantemente avaliada. A presente pesquisa objetivou criar um instrumento de avaliação da eficácia da ESF, apreendendo as crenças sobre a eficácia da ESF de uma amostra não probabilística estratificada de 390 usuários em Natal, com dados analisados com estatística descritiva, bivariada e multivariada, com auxílio do software Statistical Package of Social Sciences for Windows. O novo instrumento, com 25 itens e escala dicotômica, possui um fator, Eficácia do atendimento na ESF, dividido em quatro subfatores, com índice de consistência interna alfa de Cronbach de 0,77. Contempla-se, ainda, que a eficácia do atendimento foi avaliada negativamente por 74,6% dos usuários, o que confirma a pertinência da nova escala e aponta a necessidade de alterações da operacionalização da ESF para disponibilidade de serviços de qualidade.

Palavras-chave:
saúde da família; avaliação de programa; análise fatorial

Resumen

El principal acceso a los servicios del Sistema Nacional de Salud (SUS) es la Estrategia de Salud de la Familia (ESF), y necesita ser evaluada constantemente. Esta investigación tuvo como objetivo crear un instrumento de evaluación de la eficacia de la ESF, entendiendo las creencias sobre la eficacia de la ESF en una muestra estratificada no probabilística de 390 usuarios en Natal, con datos analizados utilizando estadística descriptiva, bivariante y multivariante con auxilio del software Statistical Package of Social Sciences for Windows. El nuevo instrumento con 25 ítems y escala dicotómica posee un factor, llamado Eficacia del atendimiento en la ESF, dividido en 4 sub-factores, con índice de consistencia interna alfa de Cronbach de 0,77. Se indica, incluso, que la eficacia del atendimiento fue evaluado negativamente por 74,6% de los usuarios, lo cual confirma la relevancia de la nueva escala y señala la necesidad de cambios en el funcionamiento de la ESF para poder disponer de servicios de calidad.

Palabras clave:
salud de la familia; evaluación de programa; análisis factorial

This research aimed to develop a effectiveness measuring instrument of the Family Health Strategy (ESF), from the beliefs of its members about the effectiveness of the ESF. Building an effective tool to be applied systematically and on a large scale, contributed to the incentives for the growth of the culture of evaluation of public programs and strategies, providing feedback to managers and the community, assisting in decisions for its improvement (Melo, Alchieri, & Araújo Neto, 2012a; 2012b; Lins, Alchieri, Araújo Neto, & Melo, 2014).

A public policy is the result of a long and intricate process involving divergent interests, confrontations and negotiations between several established institutions and between the actors who are part of them (Albuquerque, Melo, Sousa Filho, & Araújo Neto, 2011). In this definition, it is necessary to point out that the term public, associated with policy, it is not an exclusive reference to the state, as many think, but the public "thing", that is of everybody. Thus, public policy expresses the convertion of private decisions in public decisions and actions that affect everyone (Pereira, 1994). These are answers links to the needs and demands of citizens (Degenszajn, 2006). Response to the fundamental Human Rights of the Social Welfare state: the right to liberty (right of movement, expression, thought), civil rights (right to vote, to justice and equality of citizens) and social rights (access to education, health and services) (Dallari, 1987).

In health, there is the Family Health Program (PSF) created in 1994 and in 2006 assumed new nomenclature, being the Family Health Strategy (ESF), configured as a structuring strategy of orientation of the Unified Health System of Primary Care (SUS) and becoming the main gateway system. It is up to the ESF provide general services and, when necessary, refer users to specialized services of greater complexity in Secondary Care and/or Tertiary Care in order to ensure the reference (forward) and counter reference (return of the reference) and guaranteed health to comprehensive care (Ministry of Health [MS], 2004; 2006). Thus, it facilitates the democratic and universal access to health.

Their services are provided by a minimum multidisciplinary team, the Family Health Team (EqSF), consisted by a doctor, a nurse, a nursing assistant (EA), a dentist, a dental assistant (ACD) and four to six community health agents (ACS), which operate mainly in the Family Health Units (USFs), also conducting home and school visits, with 40-hour work journeys. They are distributed by territorial areas in the city and follow the linked population of their area: 600 to 1000 families and must not exceed the maximum of 4,500 people (MS, 2004; 2006). It is noteworthy that this division in Natal generates five Health Districts (South, North 1, North 2, East and West), each one responsible for managing existing USFs in its territory (being 37 USFs in the city), and each unit is responsible for streets and predefined homes (Melo et al., 2012b; Lins et al., 2014).

Therefore, it is observed that the structure of the Unified Health System (SUS) is ideal and it was created with inspiration from countries like Cuba and Canada, countries considered models for their health services. Their health care systems are characterized by free and universal access of health. With an organizational structure distributed in hierarchical levels of complexity, the main access is also by the Primary Care, marked by continuous and close monitoring of family doctors (Albuquerque et al., 2011; Albuquerque & Melo, 2010). However, it is noteworthy that in these countries the effectiveness of the proposed by the ESF is possible by excellent levels of availability of doctors (For example Cuba 6.7 doctors/thousand inhabitants), better than in other countries (Campos, 2013).

However, it is shown that the Brazilian system has been criticized by users, media and academic community due to the health care reality in Brazil is very adverse to the theoretical model for universal access, marked by problems in management (Melo et al., 2012a; 2012b; Melo, 2013); in attendance, related to the physical structure and the difficulties of access caused by lack of professionals, among other reasons (1.8 doctors/thousand inhabitants) (Albuquerque et al., 2011; Albuquerque & Melo, 2010; Cotta, 2005; Frekete, 2000; Marcelino, 2010; Melo et al., 2012b; Lins et al., 2014).

In this way, the research of government strategies assessments emerge as a key tool to assist through feedback, in the decisions of managers in the implementation, process and results achieved for their services (Belloni, Magalhães, & Sousa, 2003; Tanaka & Melo, 2000; Melo et al., 2012a; 2012b; Lins et al., 2014). Therefore, through these research is possible to improve programs and mitigate the Brazilian belief that public services are poor and then, can be of low quality (Albuquerque, 2002; Lins et al., 2014).

However, most evaluation research with users focus only on the results of a single USF under qualitative perspective, deepening of the local reality without cover a large scale evaluation; even finding research using questionnaires without psychometric validation (Albuquerque & Melo, 2010; Cotta, 2005; Frekete, 2000; Lins et al., 2014). It is obsereved the lack of instruments with psychometric validity that allows the evaluation of the ESF effectiveness in large scale, being recognized only Marcelino´s (2010), organized on two factors, but with low psychometric values: accessibility/reception (α = 0.80) and service structure (α = 0.49).

Answering some of this shortage, this study sought to create a new tool to assess the effectiveness of the ESF. It is expected to promote a program evaluation culture and instigate psychologists to participate more in research to assume the task of creating national valid instruments for evaluating programs. It is still usual to conduct research with imported instruments, adapted to the Brazilian decontextualized reality (Novaes, 2000), although this is not the specific case of the ESF evaluations, as being a Brazilian strategy. It is also advocated that psychologists are qualified professionals for this practice, since it is more prepared to build constructs of assessment and skilled tools for creating instruments with psychometric validity (Albuquerque, 1999). Therefore, it is remembered that the choice of methodology, especially the instrument used is a key decision for the evaluation of research (Frekete, 2000; Furtado, 2001).

Instructed that theoretical support and technical psychology, as well as of Public Health, this research addressed the ESF members under the cognitive aspect, based on the Theory of Beliefs of Social Psychology, Rokeach´s model (1981). From this theory, it is recognized that there is a constant exchange in the man´s socialization process with people and social stimuli that lead them to collect information cognitively synthesizing internal and external stimuli. As a direct consequence of taking note of the social environment that surrounds them, impressions are formed about himself, his world, his past and his future (a process that is the object of study Social Cognition). This perception, which results in the construction of opinions, beliefs, attitudes and values shape the image of the subject on his reality, how he perceives it, not necessarily how it is (Krüger, 2004; Rokeach, 1981). These beliefs that can be purchased in direct or indirect contact (through others) with the object of belief, although remember that here were only contemplated the beliefs not derived by others (Rokeach, 1981). Variables actors, although they refer to psychological phenomena not accessible to direct observation can be inferred and integrate empirical hypotheses because they are accessible to measurement albeit indirectly (Krüger, 2004).

In dialogue with the Theory of beliefs, it was even supported by the cognitive-behavioral theory, which also states that cognitions (events) are based on activities or assumptions (schemas) developed from previous experiences (Beck, 1982; Rangé 2001), and that these beliefs acquired have an impact on their emotional state and behavior of the subjects´ standard (Cordioli, 2008; Krüger, 2004). Therefore, it understands that from the everyday experiences of search for care in USFs users, they form beliefs about the ESF effectiveness in order to evaluate the service. It also includes that the choice of these subjects is due to the fact that there are users who are at the end the production process, seeing the results of public policies and faced with their successes and their obstacles. Thus, it is clarified that the items created scale not based on users´ knowledge of theoretical aspects of the ESF, or idiosyncrasies, or the direct observation of their behavior, but the beliefs acquired in direct contact with services.

Thus, this research aimed to create and present evidence of psychometric validity of the Family Health Strategy Effectiveness Rating Scale by its users, and to evaluate the ESF effectiveness in Natal, Rio Grande do Norte. To this end, the aim was to: 1) trace a bio-demographic profile of ESF users; 2) build and validate the ESF Effectiveness Rating Scale by its users; 3) verify the assessment that users make to the factor "ESF effectiveness of care"; and 4) compare the evaluation results of the factor according to bio-demographic variables of users.

Method

Type of Study

There were two studies: 1) construction and factorial validity of the "Effectiveness Rating Scale of the Family Health Strategy by its users" and 2) evaluation of the ESF effectiveness. The number 2 is a correlational descriptive research in which antecedent variables are the services offered in the ESF and the resulting variables are the assessment that users make it.

Participants

For the realization of the two studies, there was a non-probability sampling by the ESF users quotas, distributed by gender and age (young - 14 to 18 years old; adults - 25 to 54 years old; elderly - over 55 years old) ensuring the statistical representation of the variability in the population. It was consisted of 390 representative subjects with 95% confidence level, of 251,850 ESF users in the capital.

Instrument

The "Effectiveness Rating Scale of the Family Health Strategy by its users" was developed. Because there is not a theory that supports the evaluation of the ESF effectiveness from the beliefs of its members, it was necessary to seek empirical evidence in the literature (Albuquerque & Melo, 2010; Marcelino, 2010; Oliveira & Albuquerque, 2008) to guide the construction and instrument validation stages with the main support for research with focus groups of Melo users (2013), showing 19 important attributes for users to evaluate the ESF effectiveness and serving as a theoretical model for the initial construction of the instrument now validated as supported by (Pasquali, 1999). As a result, the final instrument, whose construction and validation processes will be presented in the results, consists of 25 items with a single factor, "ESF Care Effectiveness" with level of internal consistency (Cronbach´s alpha) of 0.77. It is noteworthy that the instrument is dichotomous (YES or NO), the best option (on scales of 3 or 4 points Likert-type, or scales 0-10) for a population of low education, given that the misunderstanding on how to answer this score, shown on previous experiences with this population (Albuquerque et al., 2011; Albuquerque & Melo, 2010; Marcelino, 2010). Furthermore, the dichotomous scale was used to prevent the respondents to be concentrated in the middle of the scale, forcing them to evaluate.

It is also noteworthy that to define the profile of ESF users, as well as to perform comparisons and verification of possible relationships between profile characteristics with the results of the evaluations, bio-demographic questions were included: gender, age, marital status, education, amount of people living in the house, home income, health insurance, profession, federal government programs help.

Procedures

Considering the ethical aspects related to research involving human beings, the research was initially approved by the Municipal Health Secretary and by the Ethics Committee of the CEP/HUOL - RN under protocol number 0298, on 09/07/2008. Then, the data collection procedure began, in which users were approached on the streets, around the USFs, but never inside them (to avoid constraints of the participants or social desirability responses in the presence of professionals in the units). After explanation about objectives and procedures of the research and accepted the participants by signing the Informed Consent Form - TCLE (by the adults participants and the responsible for youth participants) and the Instrument of Consent (by the young ones), the scale was answered individually, in an interview format, with the help of netbooks, handled by applicators. It must be also highlighted that the ethical aspects required by Resolution 466/12 of the National Health Council were respected.

Data Analysis. Data analysis occurred in five steps: 1) bio-demographic profile of ESF users was defined, using descriptive statistics; 2) the scale construction process was held (described in the results); 3) the psychometric qualities of the scale were analyzed, by the method of Principal Components (PC) to check the emerging factor and adequacy of the items in it; 4) a descriptive analysis of the scale factor and its sub-factors was performed, creating the total score for each factor and sub-factor (with high and low score according to the number of household items), convertion of score for a scale from 0 to 100% standardized and interpretation of them into quartiles: 0-25% very bad, 26-50% bad, 51-75% good, 76-100% excellent (see detailed process on the results); and finally, 5) comparisons of assessments of the general factor based on gender, age, education and Health District were held.

Results

Responding to all the objectives of the two studies, the results will be presented according to the following sequence: 1) bio-demographic profile of ESF users; 2) the process of construction of the Effectiveness Rating Scale of the Family Health Strategy by its users; 3) the psychometric qualities of the scale in its factorial validity (Cassepp-Borges, Balbinotti, & Teodoro, 2010; Pasquali, 2001); 4) descriptive analysis of the scale factor and its sub-factors, and 5) comparison of the evaluations of users based on the bio-demographic data.

Bio-Demographic Profile of ESF Users

By sample stratification, 390 users were equally distributed by gender (195 men and 195 women), age range (138 youths, 138 adults and 138 elderly) and the five Health Districts -HD (78 members each) having an average age of 39.12 (± 21.68), variants values between 9 and 93 years. It was also observed that there were similar singles quantity (f = 182; 46.7%) and married (f =1 72, 44.1%), with only 22 divorced (5.6%) and 14 widowers (3.6%); and education levels were relatively low, with only 22.1% (f = 86) with school education completed.

It was observed that the average of residents per home is 3.97 (± 1.85), varying from 1 to 13 people; and the income per capita of their families was R$ 506.06/person (± 796.40), ranging from R$ 28.00 and R$ 6,800.00. Discrepancy can be explained by the high variability in the types of occupations of the participants and the high number of unemployed (f = 120, 30.76%). It is also noted that 21.3% (f = 83) had financial aid from government programs; and the majority (f = 305; 78.2%) has no health insurance, although it is important to note that 85 (21.8%) participants are ESF users despite having health insurances.

Study 1 - Construction of the Effectiveness Rating Scale of the Family Health Strategy by its Users

In the course of this study, there were presented crucial roles of assessment strategies and programs in monitoring and improvement of its actions since its elaboration, its implementation, process and results. Feedback that is of interest not only to their managers, but of all involved social subjects: makers, managers, professionals and users.

To instill a culture of evaluation of the ESF effectiveness with large samples, it has become crucial to build a valid and accurate tool for determining the beliefs of members of the ESF. Therefore, it was tried to follow the construction of instrument based model in the three poles produced by (Pasquali, 1999): 1) the theoretical pole focused on the theory about the construct being measured by the instrument; 2) the empirical or experimental polo were defined steps and pilot instrument valuation techniques to have an instrument with good psychometric qualities; and 3) the procedures were established in the analytical polo for statistical analysis of the data in verifying the validity and the instrument´s accuracy.

At first, there were 19 attributes selected that are representative for users to evaluate the ESF effectiveness, emerged in an early study with focus groups of users: 1) USF physical infrastructure; 2) USF material resources; 3) USF location and identification; 4) ESF human resources; 5) ESF services accessibility; 6) preventive care; 7) reference system; 8) understanding of the community about the ESF; 9) ESF target population; 10) ESF regionalization; 11) EqSF-community relationship; 12) Municipal Health Council (CMS); 13) need for financial resources to continue treatment; 14) awareness of their role as contributors; 15) responsible for the difficulties of the ESF; 16) ESF developments; 17) ESF importance; 18) use of private health services; 19) distinction between theory and practice (Melo, 2013).

Considering these 19 attributes as the initial theoretical model for the instrument construction, a questionnaire containing 57 items was elaborated, submitted to a panel of five judges (social psychologists with expertise in psychometrics, program evaluation and ESF connoisseurs), seeking to identify problems in the instrument, obtaining an instrument with 64 items. Then, there was the collaboration of 20 ESF users in this scale construction process, for semantic validation and construct of the items. Therefore, it was asked to these users to express what the item questioned in order to verify that it measured what was intended. It is observed that this process is not about a pilot study, and has no goal of users to answer the instrument, offering its assessment of the program, but explicating the understanding of the item in order to verify the shared clarity and significance of the items. Thus, the theoretical stage was completed, leaving the instrument to be used in the application of the scale with 64 items.

After collecting data, for the empirical/experimental stage, all positive affirmations (positive evaluation of the ESF) and negative (negative evaluation of ESF) were converted to positive through recoding responses of negative items, so then the factor analysis procedures are performed. However, it was found that the data did not confirm the original model and the first psychometric value required to allow the completion of factor analysis (FA) was lower than expected (KMO = 0.52). It was also observed that this occurred due to the heterogeneity of the sample, since KMO values were found to sometimes acceptable and sometimes underestimated, or even unfeasible rotation depending on the bio-demographic program characteristic of the sample (Health District - HD, education, or age range). It was found that most of the FA rotation when users were selected by HD were not allowed by the program, being feasible only users with the rotation of the HD. However, the south with unacceptable KMO (0.42). In the FA rotation by education, it was observed that despite some do not rotate (users with incomplete secondary education, complete high school and higher education), with illiterate rotating the FA, but with low KMO (0.46), the only sample of elementary school, besides being processed, provided acceptable values of KMO (Incomplete Elementary School, 0.70; Complete Elementary School, 0.64). Finally, it was observed that all FA rotations by age group were able to be performed with a low value KMO for young people (0.46), highlighting the acceptable values for adults (0.64) and elderly (0.70).

Thus, the reason for the difficulty of adequacy of psychometric values for a sample is understood, although they have something in common (the use of services provided by the ESF), is very heterogeneous, both by the multi-faceted presentation of the ESF, which differs over its five HD, with regard to infrastructure, material resources, availability of professionals, but also by the structure of beliefs that differ from the experiences and consequently between people of different educational backgrounds and of different ages. However, elaborating the goal for the formation of a single scale that would make it possible to evaluate the ESF effectiveness throughout its heterogeneous people, it was sought for options for factorial validation of its psychometric values. Among several attempts, the most plausible solution was: the FA was held by Principal Components (PC), making minimum requirement that in the correlation matrix between the items, the coefficients were greater than 0.30. It was observed that out the total of 64 items of the original scale, 35 items did not meet the minimum criteria, being eliminated, leaving only 29 items, treated as the only "original items" and, thereafter, new FA was held, which obeyed the psychometric requirements, obtaining new theoretical model of the attributes, with a factor (divided into four sub-factors), concluding the empirical/experimental stage.

Then, there was the analytical phase, considering the appropriateness of making a factor analysis with the remaining set of 29 items, which was fully supported by KMO=0.70 and the Bartlett Sphericity Test, x² = 1291,22, p < 0,001, being data considered relatively good, according to SPSS manual of (Pereira, 1999) and (Foster, Barkus and Yavorsky, 2006). The analysis of 390 subjects also answered the criteria of (Tabachnick and Fidell, 1989) on the minimum number of registered answers to factor analysis: at least 5 times. There was a ratio of 13.45 individuals per item.

It was then used exploratory factor analysis in order to organize the database and clustering items to form factors and identify which items were valid, and which should be discarded. With the data presented by the Principal Components (PC), the factor matrix was verified. Initially there were 10 factors found with eigenvalues greater than 1, explaining 64.14% of the total variance of the studied construct. Then, based on statistical and theoretical criteria, it was found that the best fit factor was presented in the PC of 1 factor, that best clustered items as shown in the graph of sedimentation - scree plot. It is also highlighted that this factor has eigenvalues (4.13) different from the others (1.55; 1.50; 1.42; 1.34; 1.20 ...). It was examined the correlation matrix between the items, however, there were 4 more items excluded that had lower coefficients than 0.30. In this way, the final instrument was 25 items clustered in a single factor, subdivided into four sub-factors explaining 17.19% of the total variance of the studied construct. The results of these analyzes with the items of the final instrument and their factor loadings can be observed in Table 1.

Table 1
Exploratory Factor Analysis with Principal Components Method - PC

ITEMS	FACTOR 1
	1	2	3	4
6. FHP Equipment and instruments (stethoscope, scale, pressure device, dentist's chair) work properly.	0,35
11. PHF professionals come everyday from Monday to Friday, in the morning and in the afternoon.
12. The Community Health Agents (ACS) comes to my house at least once a month.	0,40
15. I thins it is wrong when the doctor in my area/street does not come (missing or on holiday), and can not be seen by another doctor from another area.	0,32
17. Here in the neighborhood need more another team of FHP.	0,32
18. Every day, many people are left without care in the PHP, because the numbers for attendance finished.	0,42
19. Short attendance numbers are given in FHP.	0,60
20. People need to wake up at dawn to take the attendance number ensure assistance in the FHP.	0,38
21. In the FHP in my neighborhood there is a reception to find what my problem is and the urgency of it before I am assisted.	0,36
22. Some people go without assistance because the FHP only works during the day in the time when they are working.	0,52
23. I know people who had to miss the job to be assisted in FHP.	0,37
24. Failure to assist during the night hinders workers´ access to care in the FHP.	0,47
27. It's hard to get an appointment for medical specialist (cardiologist, ophthalmologist, dermatologist, etc.).		0,47
28. It takes more than one month to have an exam.		0,44
29. I've had difficulty mark the return to the specialist doctor (cardiologist, ophthalmologist, and dermatologist) who assisted me.		0,56
9. The professionals´ attendance of the FHP is good.			0,38
39. In the FHP you cannot choose the doctor who will assist me.			0,46
42. I already suspected in the consultation with what the doctor had told me.			0,34
43. I trust the FHP professionals.			0,4
44. There is often a meeting held with the community to talk about the FHP problems.			0,53
54. The FHP in my neighborhood just worsens.				0,43
57. The hospital settles better my health problem than the FHP.				0,4
61. If I could choose, instead of going to the FHP, I would go to a private clinic.				0,37
62. I think the service in private practice is faster than in the FHP.				0,34
63. There is a difference between what the law says about the care in health and how it happens in practice.				0,39

The only factor entitled "Effectiveness Care of ESF" consists of 25 items of the scale, with minimal saturation of 0.32 (items 15 and 17) and a maximum of 0.56 (item 29) with internal consistency index (alpha Cronbach) of 0.77, and reflecting the beliefs of users about the effectiveness of care of the different services provided by the ESF and is subdivided into four sub-factors: 1) access to ESF care; 2) Access to the reference system; 3) EqSF- community relationship; and 4) comparison between ESF and other services. Below, the constitutive definition of each sub-factor are described.

The first sub-factor, "Access to ESF care" has 12 items showing the beliefs of the users on the accessibility t ESF services. Reviews of the number of professionals and their working frequency, limited quantity and difficulty of getting the records for assistance, lack of late evening hours for service workers, reception for emergency and visitation of ACSs in the households are highlighted.

The "Access to the reference system" has three items that express the beliefs of users on the referral by the ESF to other levels of care for performing tests and specialist consultations, as well as counter-reference, return of patients to the first doctor in the ESF.

The third sub-factor, "EqSF-community relationship" has 5 items that include assessing that users do on the professionals´ assistance and the bond of trust they have with them. They also presented their beliefs about regionalization and the prior definition of professionals in each micro-region.

The last sub-factor, "comparison between ESF and other services" consists of 5 items that reflect the beliefs of the users about the ESF service, comparing them to the care of hospitals and private clinics. It is distinguished yet the theory and the reality of the ESF.

Study 2 - Descriptive Analysis of General Factor and its Sub-Factors of Effectiveness Rating Scale of Esf for its Users

As a dichotomous scale was used to interpret the answers given by the participants, a total score for the factor was elaborated, which is the response points of the sum of the items. Like all items in the factor were converged to positive affirmations, all statements that the participant agreed, offering "positive" assessment of the ESF, there was answered a "yes", and computed with value 1; and all statements in which the participant did not agree, characterizing a "negative" assessment of the ESF, answering "no", computed as 0.

Thus, in an extremely positive assessment the user should answer "yes" to the 25 items, scoring 25 (maximum), while for an extremely negative assessment they should answer "no" to the 25 items, getting zero score (minimum value). Thus, the minimum score for the factor is 0 points and the maximum is 25 points.

It is noteworthy that the same procedure was used in the interpretation of data from the four sub-factors (SC), adequating the total score to the amount of items of each: SC1- 12 items; SC2- 3 items; SC3- 5 items; SC4- 5 items. Then, for better visualization, it was chosen to offer in addition to the individual sum score, a standardized score scale from 0 to 100%, generated by linear transformation of the original score by simple rule of three, which was interpreted into quartiles: 0-25% very bad, 26-50% bad, 51-75% good, 76-100% excellent. Finally, there were individual results of some items in each sub-factor.

The general factor Effectiveness Care in ESF with 25 items, showed answers average of 6.35 (± 3.89), with a minimum score of 0 and maximum of 23. This means that, on average, the participants evaluated positively 6 of the 25 items. Standardizing for a scale of 0 to 100%, 25.4% approval. It can be inferred that the effectiveness of Care at ESF in Natal is considered "very bad".

The sub-factor Access to ESF care, with 12 items, showed an average of 2.85 (± 2.03), with a minimum score of 0 and maximum of 12. Therefore, it is noted that on average, participants evaluated positively 2 of 12 items. This means that 23.75 of the answers were approved. It can be inferred that access to ESF is considered "very bad" by its users.

Therefore, observing that the best evaluated items were those relating to the frequency of professionals- item 11 (52.8% approval) and visitation of ACSs at households- item 12 (55.9% approval), some other data are alarming: 87.9% of respondents complain about the need to wake up at dawn to get a number for attendance (item 20), 93.8% disapprove of the amount of numbers (item 19), 90% complain about the need for more professionals (item 17) and 95.6% speak of the need for a night shift for attendance (item 24).

The second sub-factor Access to the reference system with three items, showed an average of 0.65 (± 0.85), with a minimum 0 and maximum score 3. This means that 21.67% of respondents were of approval. It can be inferred that the forwarding and return for examinations and medical experts is considered "very bad" by its users. These data is even more troubling when considering that users of the reprove rates are 87.4% for referral to specialist (item 27), 83.1% for exams and 64.1% for returns.

The sub-factor EqSF-community relationship, with 5 items, showed an average of 2.25 (±1.30), with scores ranging from 0 to 5. Therefore, it is noted that 45% of respondents were of approval. It can be inferred that the community relationship with the professionals working in the ESF is considered "bad" by its users.

It is observed that this was the best rated construct, which was made only a negative item, the 39, with 85.1% (f = 332) of reprove the not choice for professional care, because of districting. Being positive assessment on the quality of care (f = 242; 62.1%), item 9, and the trust the user has in professional (f = 259; 66.4%).

The last sub-factor, Comparison between ESF and other services, with 5 items, showed an average of 0.60 (± 0.84), with scores ranging from 0 to 4. Thus, it was verified the average of items approved in this construct is less than 1. It is noted, therefore, that 12% of the responses were approved. It can be inferred that when comparing the ESF with other services, it is considered "very bad" by its users. It is apparent that the ESF is considered less effective than the hospital by 79.2% (f = 309) of the participants, with preference for particular clinical 93.6% (f = 365) of the participants. Even, there are 92.6% (f = 361) states the distinction between theory and practice.

It is possible to verify that the evaluations of general factor Effectiveness of ESF care is different when compared from bio-demographic data, which reinforces the hypothesis of high heterogeneity of the user population of ESF and difficulty of building a single instrument evaluation of the ESF by its users. Considering the non-normality of the data presented by Shapiro-Wilk (0.92; p=0.00), it was observed in gender, using the Mann-Whitney test, a statistically significant difference (U=16870.00, p>0.05), and a score of men (6.80±4.22) is higher than that of women (5.91±3.49), probably because women are the main frequenters of the ESF, encountering more constantly with their disability; at the age variable, through the Kruskal-Wallis, a statistically significant difference (5.89, p<0.05), in which the elderly have the best rating (6.93±4.26) than young people ( 6.30±3.48) and adults (5.83±3.85), suggesting a better care for the elderly; the variable education, with the Kruskal-Wallis test, a statistically significant difference (17.11; p < 0.05), in which users with less education have better ratings (illiterate-8.94 ± 5.07, incomplete elementary school-7.12±4.28) than the ones with best education (complete elementary school -5.59 ± 2.86; incomplete high school -5.54 ± 2.82, complete high school -5.17 ± 3.03) can infer that people with lower education levels are less demanding; in the variable Health District, through the Kruskal-Wallis test, a statistically significant difference (20.09; p < 0.05), where users with worse assessments are from the North, 2 (4.79 ± 2.56) and West (4.80 ± 3.03) and the best evaluated is the southern area (8.13 ± 5.19).

Discussion

Users are subjects that are at the end of the health care production process offered by the ESF. They are who, even without formally often know their theoretical guidelines, knowing if the service is effective (Albuquerque & Melo, 2010). People in search daily by health services acquire beliefs about the effectiveness of ESF, its positive aspects and obstacles, making them sources of excellence for assessing the effectiveness of the strategy.

Through bio-demographic profile of users of the ESF, it was found that 22.10% (f = 86) of participants completed high school, and we can infer that users of the ESF are mostly people with low educational levels. By analyzing the socioeconomic levels, it was also observed the family monthly income per capita (R$ 506.06/person, SD = 796.40), setting them to class C2 (above only Class D), according to the Brazilian Association of Research Companies (ABEP, 2013), confirming that the Brazilian culture in public services of the Family Health Units are for the lower socioeconomic levels and therefore may be of low quality (Albuquerque, 2002). This vision mitigates the participation of socio-economically more privileged classes in the use of services. More critically conscious subjects of their rights and important figures in the struggle for the improvement of public health services (Albuquerque, 2002; Lins et al., 2014).

Despite the recognized importance of users as a source of information, this study confirms the difficulty expressed in the literature for creating instruments with factorial analysis for assessing the effectiveness of the ESF (Albuquerque & Melo, 2010; Cotta, 2005; Frekete, 2000; Lins et al., 2014), especially from the beliefs of its members, due to the heterogeneity of the population, which hinders the construction of a single measuring instrument, and the low level of education, which requires the use of scores simple answer and poorer statistically (dichotomous scale). However, it was reached the goal of building a simple, brief and understandable to all users of the ESF, formed by a single factor that explains 17.19% of the total variance of the studied construct and Cronbach´s alpha index overall internal consistency scores good (α = 0.77). A scale with indices considered satisfactory, and that can be improved, but that encompasses the key attributes considered important in the literature to assess the effectiveness of the ESF by its users (Albuquerque & Melo, 2010; Marcelino, 2010; Melo, 2013) and meets needs to create reliable instruments and adapted to the reality of Brazilian programs and strategies, as advocated by (Belloni, Magalhães and Sousa, 2003) and (Novaes, 2000).

Regarding the assessment of users about ESF, it was observed that the effectiveness of care in the ESF had 75.60% reprove, data that corroborate the findings of comprehensive literature searches, mostly made with qualitative methods or from psychometric questionnaire without validation in all capital cities (Albuquerque & Melo, 2010; Cotta, 2005; Frekete, 2000; Marcelino 2010), and failed in all subfactors, with regard to accessibility to the ESF, the reference system, the EqSF-community relationship, the comparison between ESF and other services and the effectiveness of care in ESF.

In sub-factor 1, the "Access to care in ESF" was analyzed and had negative evaluation, with 76.25% reprove, supporting the extensive literature that shows the difficulties of access to ESF, represented by the frequent queues at dawn and small amount of numbers for attendance to the great demand (Albuquerque et al., 2011; Albuquerque & Melo, 2010; Frekete, 2000; Marcelino, 2010). In sub-factor 2, it was observed that "Access to the reference system", got negative assessment, with 78.33% reprove. This fact has also been explained by managers and professionals, due to the limitation of the number of experts and surveys of vacancies (Melo et al., 2012a; 2012b; Lins et al., 2014). In sub-factor 3, it was found that the "EqSF-community relationship" got negative rating, with 55.00% of reproach. Disrespectful to the ideal of co-participation between the two actors involved in the ESF (MS, 2004; 2006) can be explained by the problems presented above the high turnover of professionals, often generated by the lack of stable bond (Melo et al., 2012a; 2012b; Lins et al., 2014). Finally, the sub-factor 4, "Comparison between ESF and other services", the ESF had negative evaluation, losing to the other services in 88% of the evaluations, corroborating the study of (Albuquerque and Melo, 2010), where it was found to tendency to avoid the ESF services not only by the middle class, which refers to health insurance, as well as the lower classes, who use private services of low-cost, private polyclinics.

The comparisons of assessments of the ESF general factor, "Effectiveness care", among users depending on bio-demographic variables, it was observed that women (M = 5.91, SD = 3.49) have worse scores than men (M = 6.80, SD = 4.22). It was also found that adults (M = 5.83, SD = 3.85) assessed worse than the young (M = 6.30, SD=3.48) and elderly (M = 6.93, SD = 4.26). This is justified by the fact that these are the largest users of the ESF and that are more faced with their barriers (Albuquerque & Melo, 2010; Oliveira & Albuquerque, 2008). It was also observed that users with less education have better ratings (illiterate - M = 8.94, SD = 5.07; incomplete elementary school - M = 7.12, SD = 4.28), it can be inferred that the people with lower education levels are less demanding, content with the law of "better than nothing", as it was perceived in the study of (Albuquerque and Melo, 2010). Finally, it was found that even users with worse ratings are from HD. North 2 (M = 4.79, SD = 2.56) and HD. West (M = 4.80, SD = 3.03), considered the worst in Quality of Life Index (QLI), and the best are the HD. South (M = 8.13, SD = 5.19), the best QLI (Barroso, 2003). It is observed, a failure in SUS principle of fairness, which argues that equality is found in the unequal treatment of unequal's, bringing health to those who need it most; just being the poorest neighborhoods just the least benefit, according to the professionals and users evaluations.

Worrying data that once again denounce the distinction between theoretical guidelines of Family Health Strategy and operational reality (Albuquerque et al., 2011; Albuquerque & Melo, 2010), highlighting the growing need encouraging the culture of evaluation of public spending and collection results on programs and government strategies (Albuquerque et al., 2011; Belloni et al., 2003).

Therefore, it is proposed that the health problem is no longer wanted in the papers and the theoretical formulations of SUS, which is not perfect, ideal, but in the operation, in the daily transfer of funds, training of professionals and infrastructure the provision of services to the community sites (Lins et al., 2014). This is to mitigate the barriers that exist between the two extremes of the system, approaching theory and practice. It includes the need to instigate the population to use public service, which should not be seen as something only "to poor people" and therefore of low quality. It is time to fight for their improvement by encouraging the construction of valid instruments and adequate to the Brazilian reality, boosting the practice of supervision of public spending and evaluation of return on investment given to these services, processed, indeed, in health services efficient and effective.

References

• Albuquerque, F. J. B. (1999). Apontamentos para uma psicologia sócio-rural no Brasil. In I Congresso Norte-Nordeste de Psicologia. Anais do I Congresso Norte-Nordeste de Psicologia, Salvador, Bahia, Brasil.
• Albuquerque, F. J. B. (2002). Social psychology and rural life in Brazil. Psicologia: Teoria e Pesquisa, Brasília, 18(1), 37-42. Retrieved from http://www.scielo.br/pdf/ptp/v18n1/a05v18n1.pdf
  » http://www.scielo.br/pdf/ptp/v18n1/a05v18n1.pdf
• Albuquerque, F. J. B., & Melo, C. F. (2010). Avaliação dos serviços públicos de saúde em duas capitais nordestinas do Brasil. Psicologia: Teoria e Pesquisa, 26(2), 323-330. Retrieved from http://www.scielo.br/pdf/ptp/v26n2/a14v26n2.pdf
  » http://www.scielo.br/pdf/ptp/v26n2/a14v26n2.pdf
• Albuquerque, F. J. B., Melo, C. F., Sousa Filho, F. E., & Araújo Neto, J. L. (2011). Avaliação da estratégia saúde da família a partir das crenças dos profissionais. Estudos de Psicologia, 28(3), 363-370. Retrieved from http://www.scielo.br/pdf/estpsi/v28n3/a08v28n3
  » http://www.scielo.br/pdf/estpsi/v28n3/a08v28n3
• Associação Brasileira de Empresas de Pesquisa. (2013). Critério de classificação econômica Brasil. Brasília: Autor. Retrieved from http://www.abep.org/novo/Content.aspx?ContentID=835
  » http://www.abep.org/novo/Content.aspx?ContentID=835
• Barroso, A. V. (2003). Mapeando a qualidade de vida em natal. Natal: Secretaria Municipal de Planejamento e Gestão Estratégica. Retrieved from www.natal.rn.gov.br/sempla/paginas/File/iqv.pdf
  » www.natal.rn.gov.br/sempla/paginas/File/iqv.pdf
• Beck, A. T. (1982). Terapia cognitiva da depressão. Rio de Janeiro: Zahar.
• Belloni, I., Magalhães, H., & Sousa, L. C. (2003). Metodologia de Avaliação em Políticas Públicas (3ª ed.). São Paulo: Cortez.
• Campos, F. E. (2013). Funding, flexible management needed for Brazil's health worker gaps. Bull World Health Organ, Genebra, 91(11), 806-807. Retrieved from http://www.scielosp.org/pdf/bwho/v91n11/0042-9686-bwho-91-11-806.pdf
  » http://www.scielosp.org/pdf/bwho/v91n11/0042-9686-bwho-91-11-806.pdf
• Cassepp-Borges, V., Balbinotti, M. A. A., & Teodoro, M. L. M. (2010). Tradução e validação de conteúdo: Uma proposta para a adaptação de instrumentos. In L. Pasquali (Ed.), Instrumentação psicológica: Fundamentos e práticas (pp. 506-520). Brasília: IBAPP.
• Cordioli, A. V. (2008). Psicoterapias (3ª ed). São Paulo: Editora Artmed.
• Cotta, R. M. M. (2005). A satisfação dos usuários do Programa de Saúde da Família, avaliando o cuidado. Scientia Medica. 15(4), 227-234.
• Dallari, S. G. (1987). A Saúde do Brasileiro. São Paulo: Ed. Moderna.
• Degenszajn, R. R. (2006). Democratizar a gestão das políticas sociais: Um desafio a ser enfrentado pela sociedade civil. Retrieved from http://www.fnepas.-org.br/pdf/servico_social_saude/texto1-4.pdf
  » http://www.fnepas.-org.br/pdf/servico_social_saude/texto1-4.pdf
• Foster, J. J., Barkus, E., & Yavorsky, C. (2006). Understanding and using advanced statistics. Londres, Inglaterra: SAGE publications.
• Frekete, M.C. (2000). Estudo da acessibilidade na avaliação dos serviços de saúde (Manuscrito não publicado). Projeto GERUS.
• Furtado, J. P. (2001). Um Método construtivista para a avaliação em saúde. Ciência e Saúde Coletiva. 6(1), 165-181.
• Krunger, H. (2004). Crenças compartilhadas, preconceitos e discriminações. In: Sociedade Brasileira de Psioclogia [SBP] (Org.), XXXIV Reunião Anual da Sociedade Brasileira de Psicologia, Ribeirão Preto, SP.
• Krünger, H. (2004). Crenças compartilhadas, preconceitos e discriminações. In XXXIV Reunião Anual da Sociedade Brasileira de Psicologia. Anais do XXXIV Reunião Anual da Sociedade Brasileira de Psicologia, Ribeirão Preto, São Paulo, Brasil.
• Lins, C. F. M., Alchieri, J. C., Araújo Neto, J. L., & Melo, F. A. F. (2014) Desenvolvimento de instrumentais para avaliação da estratégia saúde da família em Natal. Psicol. Reflex. Crit, 27(2), 219-227. Retrieved from http://www.scielo.br/pdf/prc/v27n2/0102-7972-prc-27-02-00219.pdf
  » http://www.scielo.br/pdf/prc/v27n2/0102-7972-prc-27-02-00219.pdf
• Marcelino, M. Q. (2010). Avaliação do PSF pelos usuários em municípios rurais paraibanos (Tese de doutorado, Universidade Federal da Paraíba). Retrieved from http://bdtd.biblioteca.ufpb.br/tde_busca/arquivo.php?codArquivo=1005
  » http://bdtd.biblioteca.ufpb.br/tde_busca/arquivo.php?codArquivo=1005
• Melo, C. F. (2013). A realidade da estratégia saúde da família em Natal: A experiência de grupos focais de usuários. In VIII Congresso Norte-Nordeste de Psicologia. Anais do VIII Congresso Norte-Nordeste de Psicologia, Fortaleza, Ceará, Brasil.
• Melo, C. F., Alchieri, J. C., & Araújo Neto, J. L. (2012a). Assessment of the Family Health Strategy through the beliefs of its managers. Revista Enfermagem UFPE on-line, 6(2), 274-278. Retrieved from http://www.revista.ufpe.br/revistaenfermagem/index.php/revista/article/viewArticle/2107
  » http://www.revista.ufpe.br/revistaenfermagem/index.php/revista/article/viewArticle/2107
• Melo, C. F., Alchieri, J. C., & Araújo Neto, J. L. (2012b). Sistema Único de Saúde: Uma avaliação realizada em Natal, Rio Grande do Norte, Brasil. Psico-USF, 17(1), 63-72. Retrieved from http://www.scielo.br/pdf/pusf/v17n1/a08v17n1.pdf
  » http://www.scielo.br/pdf/pusf/v17n1/a08v17n1.pdf
• Melo, C. F. ; Alchieri, J. C., & Araújo Neto, J. L. (2013). Evaluation of the technical and professional profile of managers of the Brazilian Unified Health System (SUS). Revista de Enfermagem UFPE On-Line, 7(7), 4670-4680. Retrieved from http://www.revista.ufpe.br/revistaenfermagem/index.php/revista/article/viewArticle/4151
  » http://www.revista.ufpe.br/revistaenfermagem/index.php/revista/article/viewArticle/4151
• Ministério da Saúde (2004). Atenção Básica e a Saúde da Família. Brasília: Autor. Retrieved from http://dtr2004.saude.gov.br/dab/atencaobasica.php
  » http://dtr2004.saude.gov.br/dab/atencaobasica.php
• Ministério da Saúde (2006). Política Nacional de Atenção Básica. (Série Pactos pela Saúde). (vol. 4). Brasília: Autor. Retrieved from http://bvsms.saude.gov.br/bvs/publicacoes/politica_nacional_ atencao_basica_2006.pdf
  » http://bvsms.saude.gov.br/bvs/publicacoes/politica_nacional_ atencao_basica_2006.pdf
• Novaes, H. M. D. (2000). Avaliação de programas, serviços e tecnologias em saúde. Revista Saúde Pública, 34(5), 547-559. Retrieved from http://nepas.ufsc.br/files/2011/09/Novaes-Revista-de-Sa%C3%BAde-P%C3%BAblica-2000.pdf
  » http://nepas.ufsc.br/files/2011/09/Novaes-Revista-de-Sa%C3%BAde-P%C3%BAblica-2000.pdf
• Oliveira, S. F., & Albuquerque, F. J. B. (2008). Avaliação do programa saúde da família a partir dos seus prestadores de serviço. Psicologia e Sociedade (Impresso), 20(2), 76-86. Retrieved from http://www.scielo.br/pdf/psoc/v20n2/a11v20n2.pdf
  » http://www.scielo.br/pdf/psoc/v20n2/a11v20n2.pdf
• Pasquali, L. (1999). Histórico dos instrumentos psicológicos. In L. Pasquali. (Ed.), Instrumentos psicológicos: manual prático de elaboração (pp. 1-12). Brasília: IBAPP.
• Pasquali, L. (2001). Validade dos Testes Psicológicos: Será Possível Reencontrar o Caminho? Psicologia: Teoria e Pesquisa, 23 (especial), 99-107. Retrieved from http://www.scielo.br/pdf/ptp/v23nspe/18
  » http://www.scielo.br/pdf/ptp/v23nspe/18
• Pereira, A. (1999). SPSS-guia Prático de Utilização, Análise de Dados para Ciências Sociais e Psicologia (6a ed.). Lisboa, Portugal: Edições Sílabo.
• Pereira, P. A. P (1994). Concepções e propostas de políticas sociais em curso: Tendências, perspectivas e consequências. São Paulo: NEPROS/CEAM/UnB.
• Rangé, B. (2001). Psicoterapias cognitivo-comportamentais: Um diálogo com a psiquiatria. São Paulo: Editora Artmed.
• Rokeach, M. (1981) Crenças, atitudes e valores. Rio de Janeiro: Ed. Interciência.
• Tanaka, O. Y., & Melo, C. (2000). Uma proposta de abordagem transdisciplinar para avaliação em Saúde. Interface - Comunicação, Saúde, Educação, 7, 113-118. Retrieved from http://www.scielo.br/pdf/icse/v4n7/09.pdf
  » http://www.scielo.br/pdf/icse/v4n7/09.pdf
• Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6a ed). Boston, EUA: Allyn and Bacon.

Publication Dates

• Publication in this collection
  May-Aug 2016

History

• Received
  25 May 2014
• Reviewed
  28 Mar 2015
• Accepted
  02 June 2015