Evaluation of the Primary Care in leprosy control: proposal of an instrument for users^*

Evaluación de la atención primaria en el control de la hanseniasis: propuesta de una herramienta destinada a los usuarios

Abstracts

Objective

Developing an instrument to evaluate the performance of primary health care in the leprosy control actions, from the perspective of users and do the face and content validation.

Method

This is a methodological study carried out in four stages: development of instrument, face and content validation, pre-test, and analysis of test-retest reliability.

Results

The initial instrument submitted to the judgment of 15 experts was composed of 157 items. The face and content validation and pre-test of instrument were essential for the exclusion of items and adjustment of instrument to evaluate the object under study. In the analysis of test-retest reliability, the instrument proved to be reliable.

Conclusion

The instrument is considered adequate, but further studies are needed to test the psychometric properties.


Leprosy; Primary Health Care; Health Services Evaluation; Questionnaires; Validation studies

Objetivo

Desarrollar un instrumento para la evaluación del desempeño de la atención primaria a la salud en la puesta en marcha de las acciones de control de la hanseniasis bajo la perspectiva de los usuarios y realizar la validación de cara y contenido.

Método

Se trata de un estudio metodológico llevado a cabo en cuatro etapas: desarrollo del instrumento, validación de cara y de contenido, pre-prueba y análisis de la confiabilidad de prueba-reprueba.

Resultados

El instrumento inicial, sometido al juicio de 15 expertos, estaba compuesto de 157 ítems. La validación de cara, contenido y pre-prueba del instrumento fue fundamental para la exclusión de ítems y la adecuación del instrumento para evaluar el objeto de estudio. En el análisis de confiabilidad de prueba-reprueba, el instrumento se mostró fehaciente.

Conclusión

Se considera el instrumento adecuado, pero son necesarios nuevos estudios para el test de las propiedades psicométricas.


Lepra; Atención Primaria de Salud
; Evaluación de Servicios de Salud
; Cuestionarios; Estudios de validación

Objetivo

Desenvolver um instrumento para avaliação do desempenho da atenção primária à saúde na realização das ações de controle da hanseníase na perspectiva dos usuários e realizar a validação de face e conteúdo.

Método

Trata-se de um estudo metodológico realizado em quatro etapas: desenvolvimento do instrumento, validação de face e de conteúdo, pré-teste e análise da confiabilidade teste-reteste.

Resultados

O instrumento inicial, submetido ao julgamento de 15 especialistas, era composto por 157 itens. A validação de face, conteúdo e pré-teste do instrumento foi fundamental para a exclusão de itens e adequação do instrumento para avaliar o objeto em estudo. Na análise de confiabilidade teste-reteste, o instrumento mostrou-se fidedigno.

Conclusão

O instrumento é considerado adequado, mas são necessários novos estudos para o teste das propriedades psicométricas.





Hanseníase; Atenção Primária à Saúde
; Avaliação de Serviços de Saúde
; Questionários; Estudos de validação

Introduction

Leprosy is an infectious disease of slow evolution caused by the bacillus Mycobacterium leprae, which is mainly manifested by dermatological and neurological symptoms and signs(¹). Approximately 33 thousand new cases are diagnosed each year in Brazil(²), and if the leprosy is not timely diagnosed and treated, it can lead to restricted social life and physical, psychological and economic problems(³).

The National Leprosy Control Program of the Ministry of Health recommends that the model of care for the disease - based on early diagnosis, timely treatment, prevention and treatment of physical disabilities and surveillance of contacts - runs across the whole network of primary health care (PHC), with the support of secondary and tertiary care to reduce the endemic levels of disease and ensure quality of care(⁴).

According to the National Policy for Primary Care(⁵), the primary care is responsible for managing the most frequent and relevant health needs in the territory, conducting actions in health promotion, protection from diseases, diagnosis, treatment and rehabilitation at individual and collective levels. However, studies show that Brazilian municipalities still face difficulties in integrating the leprosy control actions (LCA) in the PHC(⁶-⁸), both at individual level, due to lack of trained professionals to diagnose and treat, as at the collective level, with the completion of specific educational activities for the community, search for contacts and epidemiological surveillance(⁸).

To ensure improved access and quality of health care in the Unified Health System (SUS), the Ministry of Health (MOH) implemented the National Program for Improving Access and Quality of Primary Care (Pmaq – Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica) in 2011, which contemplates leprosy in the list of full attention to the Brazilian population that should be offered by the PHC(⁹). In the self-assessment stage of the Pmaq, professionals and managers are asked to identify the problems of the work process in the PHC, and the intervention strategies. However, the inclusion of users in this evaluation process would be desirable due to understanding that all are jointly responsible for the qualification of the SUS. The Pmaq manual presupposes that teams can use other tools that better fit the raising of problems and bring a more appropriate reflection about the health needs of the population(⁹).

Accordingly, we have the PHC Assessment Tool available (Primary Care Assessment Tool - PCATool-Brasil(¹⁰)), adult users version. It assesses the presence and extent of the PHC attributes - first contact (access and use), longitudinality, coordination (of care and information system), comprehensiveness (services available and services provided), family approach and community orientation – in the experience of adult users of PHC services(¹¹).

Currently, managers and professionals use the epidemiological and operational indicators of leprosy(⁴) as tools to evaluate the disease control program. Since in the literature there is no validated instrument that assesses the attributes of PHC in the care of leprosy, the study aimed to develop an instrument for evaluating the performance of primary care services in the LCA from the perspective of users, and do the face and content validation.

Method

This is a methodological study carried out in four stages, namely: development of instrument, face and content validation, pre-test and analysis of test-retest reliability.

Stage 1: Development of instrument

The construction of the Instrument for assessing the performance of primary care in efforts for leprosy control - user version was based on the framework of PHC in Brazil(¹²) and in efforts for leprosy control performed at this level of health care(⁴,¹³). After the literature review, the instrument had 157 items divided into seven constructs: first contact (4), access (39), continued care (23), comprehensiveness of available and provided services (51), coordination (14), family orientation (14) and community orientation (12).

For the wording of questions, development of response alternatives and score calculation of the attributes, the format used in the PCATool-Brazil(¹⁰) was taken into consideration. The users answered the instrument questions according to the options: 1 (definitely not); 2 (probably not); 3 (probably); 4 (definitely); 9 (do not know/cannot remember) and 88 (not applicable - code inserted in items if the user has not had leprosy reactions, referral to a specialist and discharge due to cure). It is noteworthy that the items with dichotomous responses - yes/no – are not included in the score calculation of the attributes.

Stage 2: Face and content validation

The judgment of the adequacy of the instrument items was done by two expert committees, which included the participation of 15 professionals with experience in management, education and assistance in leprosy or in the area of primary health care. The items were analyzed for content (suitability to the theoretical concept - the PHC attributes - and to the object that is to be measured – performance of the LCA in the PHC), clarity, need for excluding and adding new items. This evaluation took place in person and by mail in March and April 2012.

In the first panel, five experts assessed the adequacy of the instrument items in relation to the object of study when answering the question: Are the contents measured by this question essential/useful/not essential to evaluate the organizational and performance characteristics of primary health care services in the care of leprosy? Items with less than 80% of positive responses (essential and useful) were excluded(¹⁴).

In the second panel, the experts judged if the content of the item was properly related to the theoretical concept of the respective PHC attribute and a qualitative assessment of each analysis of the questionnaire items was carried out.

Stage 3: Pre-test of instrument

The version of the instrument designed after the judgment of the expert committee, was subjected to a pre-test in the city of Betim (MG), in June and July 2012 to analyze the instrument adequacy: if the items are clear, understandable and relevant to reach the study objective, as well as the time required for the questionnaire application. In total, 23 users who underwent treatment of leprosy in PHC units participated in the pilot study. Participants were instructed to answer the questions according to the care received during the period of leprosy treatment, and to indicate any difficulties with understanding the questions and the range of responses. The factors facilitating and hindering the instrument application were discussed at a meeting of the research group, which resulted in the drafting of the instrument final version.

Stage 4: Test-retest reliability

For the reliability assessment, the instrument final version with 109 items was applied in two endemic municipalities of Minas Gerais - Almenara and Governador Valadares. The reliability was checked by the time stability, resulting from the questionnaire application at two different times (test-retest reliability), 30 days after the end of data collection, in 10% of the sample. The Wilcoxon test with significance level of 5% was used to assess whether there were differences in the responses obtained in the test and retest.

The convenience sample consisted of all leprosy patients reported between the years 2009 and 2012 who underwent the treatment exclusively in PHC health units and agreed to participate in the study. Patients aged under eighteen years, adults with special problems, patients in the prison and residents in rural areas were excluded. Data were collected through an interview conducted at the home of participants or at the health unit during the period between July and December 2012.

The database was built using the EPI-INFO software (version 7) and, after the data entry, the values of the items C3, C4, C5 and C8 were reversed, as they were formulated in order that the higher the value of assigned response, the fewer the guidelines for PHC. The Statistical Package for the Social Sciences (SPSS) for Windows 17 was used for statistical analysis.

Ethical aspects

The research project was approved by the Research Ethics Committee (COEP) of the Universidade Federal de Minas Gerais under number ETIC 0095.0.203.000-11. All survey participants – the experts and the leprosy patients from the municipalities of Betim, Almenara and Governador Valadares - agreed to participate in the study and signed the Free and Informed Consent Form (ICF) in duplicate.

Results

The evaluation of the questionnaire items regarding relevance, clarity, understanding and adequacy of the item to the PHC attribute was made by 15 experts in the two major areas of the instrument: leprosy and primary health care. Nine doctors, five nurses and a biologist participated in this stage, considering that 66.6% (n = 10) have stricto sensu post-graduate studies and 26.7% (n = 4) have lato sensu post-graduate studies.

The first panel of experts assessed that only five of the 157 items were not essential to evaluate the performance of PHC in leprosy control and suggested the inclusion of six items, as well as the semantic change of 17 items for better understanding. After the discussion of these findings by researchers, the following changes were made in the instrument: exclusion and semantic changes of all items suggested by the experts, and inclusion of only four items in the instrument version submitted to the judgment of the second panel. In this latest assessment, the instrument consisted of 156 items and were suggested the exclusion of 28 items, change of 25 domain items (PHC attributes) and inclusion of 11 items. After this stage, the user version of the instrument for evaluating the performance of primary care in the actions of leprosy control applied in the pilot study in the city of Betim, was composed of 139 items.

The pilot study carried out with 23 users (Table 1), was essential to understand that 32 items of the instrument were inadequate to evaluate the performance of PHC in the care of leprosy and include a new domain of the instrument called ‘degree of affiliation’ with seven questions, a question in the attribute ‘continued care’ and another in the construct ‘comprehensiveness of available services’. After evaluating the results of the pre-test, the instrument was considered appropriate to measure the object under study and was applied to 31 users who underwent the leprosy treatment in PHC units in the municipalities of Almenara and Governador Valadares (Table 1).

Table 1
Characterization of the study sample - Betim, Almenara, Governador Valadares, 2012

Variables	Betim	Almenara	Governador Valadares
	(n=23)	(n=24)	(n=7)
Gender (n):
Female	8	8	6
Male	15	16	1
Years of study (n):
No education	3	10	4
Up to 7 years	15	11	1
8 to 10 years	4	3	1
More than 11 years	1	0	1
Year of notification (n):
2012	8	4	0
2011	5	7	1
2010	7	6	0
2009	3	7	6

In the application of the instrument in this scenario, it was found that the scale of response, of the Likert type, could hold only three options, since the answers were polarized in the alternatives of definitely and certainly not. Thus, it can be inferred that the respondent did not understand the differences in degree among the options for definitely and probably.

The construct validation was done mirrored on the version for the community health agents (CHA)(¹⁵), since the sample of 380 CHA was suitable for exploratory factor analysis. At this stage, five items of the user version were excluded, therefore, the final instrument consisting of 104 items was assessed for reliability. In the reliability analysis, it was not possible to determine the internal consistency by Cronbach’s alpha due to the insufficient size of the sample (31 users). The result of the temporal stability of the instrument is shown in Table 2.

Table 2
Evaluation of the test-retest stability of the PHC attributes that make up the final version of the instrument for users - Almenara, Governador Valadares, 2012

Instrument attributes	Wilcoxon test (p value)
Degree of affiliation - PHC	0.317
First contact	1.000
Access	0.180
Continued care	0.705
Comprehensiveness of available services	1.000
Comprehensiveness of provided services	0.144
Coordination	0.180
Family orientation	0.655
Community orientation	0.581

Regarding reliability, which was partially evaluated, it was found that the scores of all the instrument attributes remained stable in the test and retest. However, further studies are required to carry out the validation of the final instrument (shown in Appendix) by the methodology of Classical Test Theory.

Discussion

The panel of experts analyzed the initial draft of the instrument, and the face and content assessment of the version for users was satisfactory, because there was a semantic change for adequacy of clarity and understanding(¹⁶), inclusion/exclusion of items and change of domains (PHC attributes) to ensure that the instrument is capable of capturing the content proposed in the research(¹⁶). The pilot study allowed checking in practice the performance of the version of instrument obtained after the experts’ judgment. Authors suggest that the pre-test should take place in a location with characteristics similar to those of the final scenario of questionnaire application(¹⁶). This step was essential to assess the adequacy of the response card, reduce the number of items in the instrument, and improve understanding of questions. Thus, the first stage of the test validation was obtained, because the items remaining in the instrument were considered relevant to represent the latent trait(¹⁷) that in this case, is the performance of PHC in carrying out leprosy control actions.

Since this is a small sample (only 31 users), it was not possible to conduct the exploratory factor analysis (which reduces the set of items in the instrument to a size that retains as much information as possible(¹⁸)) and the analysis of the internal consistency of items (which assesses the homogeneity of scales in items that can be answered in more than two alternatives(¹⁸)). A sample five times greater than the number of items to be evaluated would be necessary to evaluate the validity and accuracy of the instrument(¹⁸). However, the results of the test-retest reliability, which are also a measure of reliability of a test, showed that the instrument for the leprosy patients is precise and measured identical scores of the same respondents in two different moments in time(¹⁷-¹⁸). Thus, the instrument showed here has the characteristic of measuring without mistakes, and to evaluate this accuracy was chosen the same methodology used in the validation of the PCATool-Braszil, adult version(¹¹).

As the version of the instrument for the CHA is valid to evaluate the performance of primary care in leprosy control(¹⁵), for the user version was carried out a mirror validation. This resource was also used for presenting the professional version of the PCATool-Brazil that although still had its validation process in progress, was also recommended for mirror use in the version for adult users(¹⁰). The authors design the application of the instrument in patients undergoing treatment of leprosy in the municipalities of the metropolitan Region of Belo Horizonte to examine the validity and accuracy by the methodology of the Classical Test Theory, which aims to produce quality tests(¹⁷).

Conclusion

We conclude that the instrument is considered adequate for evaluating the performance of PHC in the development of efforts for leprosy control according to the users’ experience, after analysis by the expert committee and the preliminary test of adjustments. Future studies are suggested to refine the quality of the instrument so it can be widely used in health services as an evaluative tool to complement the Pmaq questionnaire.

• *
  Extracted from the thesis "Evaluation of primary care in leprosy control: instrument validation and performance analysis of endemic municipalities in the state of Minas Gerais", Post-Graduate Program, School of Nursing, Universidade Federal de Minas Gerais, 2014.

• Financial support

  Fundação de Amparo à Pesquisa do Estado de Minas Gerais. Processo n. CDS-APQ-01438-11.

References

• ⁰¹
  Lastoria JC, Abreu MAMM. Leprosy: review of the epidemiological, clinical, and etiopathogenic aspects - Part 1. An Bras Dermatol. 2014;89(2):205-18
• ⁰²
  World Health Organization. Global leprosy: update on the 2012 situation. Weekly Epidemiol Rec. 2013;87(34):317-28
• ⁰³
  Lana FCF, Fabri ACOC, Lopes FN, Carvalho APM, Lanza FM. Deformities due to leprosy in children under fifteen years old as an indicator of quality of the leprosy Control Programme in Brazilian Municipalities. J Trop Med [Internet]. 2013 [cited 2014 Jan 27]. Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3614053/
  » http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3614053/
• ⁰⁴
  Brasil. Ministério da Saúde. Portaria n. 3125, de 7 de outubro de 2010. Aprova as Diretrizes para Vigilância, Atenção e Controle da Hanseníase [Internet]. Brasília; 2010 [citado 2013 dez. 22]. Disponível em: http://www.anvisa.gov.br/hotsite/talidomida/legis/portaria_n_3125_hanseniase_2010.pdf
  » http://www.anvisa.gov.br/hotsite/talidomida/legis/portaria_n_3125_hanseniase_2010.pdf
• ⁰⁵
  Brasil. Ministério da Saúde; Secretaria de Atenção à Saúde, Departamento de Atenção Básica. PNAB - Política Nacional de Atenção Básica [Internet]. Brasília; 2012 [citado 2013 dez. 22]. Disponível em: http://189.28.128.100/dab/docs/publicacoes/geral/pnab.pdf
  » http://189.28.128.100/dab/docs/publicacoes/geral/pnab.pdf
• ⁰⁶
  Fuzikawa PL, Acúrcio FA, Velema JP, Cherchiglia ML. Decentralisation of leprosy control activities in the municipality of Betim, Minas Gerais State, Brazil. Lepr Rev. 2010;81(3):184-95
• ⁰⁷
  Penna MLF, Grossi MAF, Penna GO. Country profile: leprosy in Brazil. Lepr Rev. 2013;84(4):308-15.
• ⁰⁸
  Lanza FM, Lana FCF. O processo de trabalho em hanseníase: tecnologias e atuação da equipe de saúde da família. Texto Contexto Enferm. 2011;20(n.esp):238-46
• ⁰⁹
  Brasil. Ministério da Saúde; Secretaria de Atenção à Saúde, Departamento de Atenção Básica. Autoavaliação para Melhoria do Acesso e da Qualidade (AMAQ) [Internet]. Brasília; 2013 [citado 2013 dez. 22]. Disponível em: http://189.28.128.100/dab/docs/portaldab/publicacoes/AMAQ_CEO.pdf
  » http://189.28.128.100/dab/docs/portaldab/publicacoes/AMAQ_CEO.pdf
• ¹⁰
  Brasil. Ministério da Saúde. Manual do instrumento de avaliação da atenção primária à saúde: primary care assessment tool PCATOOL [Internet]. Brasília; 2010 [citado 2013 dez. 22]. Disponível em: http://189.28.128.100/dab/docs/publicacoes/geral/manual_instrumento_avaliacao.pdf
  » http://189.28.128.100/dab/docs/publicacoes/geral/manual_instrumento_avaliacao.pdf
• ¹¹
  Harzheim E, Oliveira MMC, Agostinho MR, Hauser L, Stein AT, Gonçalves MR, et al. Validação do instrumento de avaliação da atenção primária à saúde: PCATool-Brasil adultos. Rev Bras Med Fam Comunidade. 2013;8(29):274-84
• ¹²
  Starfield B. Atenção primária: equilíbrio entre necessidades de saúde, serviços e tecnologia. Brasília: UNESCO; 2002
• ¹³
  Brasil. Ministério da Saúde. Portaria n. 594, de 29 de outubro de 2010. Define os Serviços de Atenção Integral em Hanseníase [Internet]. Brasília; 2010 [citado 2013 dez. 22]. Disponível em: http://www.brasilsus.com.br/legislacoes/gm/legislacoes/sas/106061-594.html?q=
  » http://www1.saude.rs.gov.br/dados/1292591487661portaria_594_de_29-11-2010%5B1%5D.pdf
• ¹⁴
  Pasquali L. Princípios de elaboração de escalas psicológicas. Rev Psiq Clin. 1998;25(5):206-213
• ¹⁵
  Lanza FM, Vieira NF, Oliveira MMC, Lana FCF. Instrumento para avaliação das ações de controle da hanseníase na Atenção Primária. Rev Bras Enferm. 2014;67(3): 339-46
• ¹⁶
  Sánchez R, Echeverry J. Validación de escalas de medición en salud. Rev Salud Pública. 2004;6(3):302-18
• ¹⁷
  Pasquali L. Psychometrics. Rev Esc Enferm USP [Internet]. 2009 [cited 2013 Dec 22];43(n.spe):992-9. Available from: http://www.scielo.br/pdf/reeusp/v43nspe/en_a02v43ns.pdf
  » http://www.scielo.br/pdf/reeusp/v43nspe/en_a02v43ns.pdf
• ¹⁸
  Hair Júnior JF, Black WC, Babin BJ, Anderson RE, Tatham RL. Análise multivariada de dados. Porto Alegre: Bookman; 2009

Publication Dates

• Publication in this collection
  Dec 2014

History

• Received
  20 Feb 2014
• Accepted
  14 Oct 2014