Validation of an audit instrument for the Unified Health System

Abstract

Objective

To analyze the validity and reliability of an instrument used for auditing the Unified Health System (SUS).

Methods

Methodological research that used the steps of adaptation of an audit instrument, recommended by the Ministry of Health, Ordinance No. 3410/2013: expert validation of content and reliability testing of the validated instrument.

Results

Regarding the adequacy of the instrument, it originally had 100 items and 32 sub-items, and remained with 55 items, with their respective evaluation criteria, as well as the source of data collection, totaling 165 items of analysis. For validation of this instrument, the experts suggested changes in semantics and content in 77 (46.66%) items. The reliability test presented an agreement index of 0.85 among nurses one and two; 0.82 among nurses one and three; and 0.96 among nurses two and three, with a mean of 0.88, representing an almost perfect inter-rater agreement.

Conclusion

The adequacy of the instrument, based on the current legislation and expert validation, made it concise, clear, and objective, presenting statistical consistency for use in auditing in the SUS, as demonstrated by the reliability test.

Nursing audit; Unified health system; Adaptation; Validation studies

Resumo

Objetivo

Analisar a validade e a confiabilidade de um instrumento utilizado em auditoria no Sistema Único de Saúde (SUS).

Métodos

Pesquisa metodológica que empregou as etapas de adaptação de um instrumento de auditoria recomendado pelo Ministério da Saúde, Portaria nº 3410/2013, validação do conteúdo por experts e teste de confiabilidade do instrumento validado.

Resultados

Em relação a adequação do instrumento, originalmente o mesmo possuía 100 itens e 32 subitens, e passou a apresentar 55 itens, com seus respetivos critérios de avaliação, bem como a fonte de coleta da informação, totalizando 165 itens de analise. Na validação desse instrumento os experts sugeriram alterações quanto à semântica e o conteúdo em 77 (46,66%) itens. O teste de confiabilidade apresentou índice de concordância de 0,85 entre as enfermeiras auditoras 1 e 2; 0,82 entre as enfermeiras auditoras 1 e 3 e 0,96 entre as enfermeiras auditoras 2 e 3 com média de 0,88, representando uma concordância quase perfeita interavaliadoras.

Conclusão

A adequação do instrumento fundamentado na legislação vigente e a validação por experts deixou-o conciso, claro e objetivo, apresentando consistência estatística para uso em auditoria no SUS, demonstrada pelo teste de confiabilidade.

Auditoria de enfermagem; Sistema único de saúde; Adaptação; Estudos de validação

Resumen

Objetivo

evaluar la validez y confiabilidad de un instrumento utilizado en auditoría en el Sistema Único de Salud (SUS).

Métodos

investigación metodológica en la que se emplearon las siguientes etapas: adaptación de un instrumento de auditoría recomendado por el Ministerio de Salud, Resolución nro. 3410/2013, validación del contenido por experts y prueba de confiabilidad del instrumento validado.

Resultados

con relación a la adaptación del instrumento, este contenía originalmente 100 ítems y 32 subítems, y pasó a tener 55 ítems, con sus respectivos criterios de evaluación, así como la fuente de recolección de información, con un total de 165 ítems de análisis. En la validación del instrumento, los experts sugirieron cambios relacionados con la semántica y el contenido en 77 ítems (46,66%). La prueba de confiabilidad presentó un índice de concordancia de 0,85 entre las enfermeras auditoras 1 y 2; 0,82 entre las enfermeras auditoras 1 y 3, y 0,96 entre las enfermeras auditoras 2 y 3, con un promedio de 0,88, lo que representa una concordancia entre evaluadoras casi perfecta.

Conclusión

la adaptación del instrumento fundamentado en la legislación vigente y la validación por experts contribuyeron para que quedara conciso, claro y objetivo, con consistencia estadística para uso en auditoría en el SUS, demostrada por la prueba de confiabilidad.

Auditoría de enfermería; Sistema único de salud; Adaptación; Estudios de validación

Introduction

With the grown of health spending, discussions have been stimulated between researchers and policy makers, to seek innovative solutions for appropriate means to use resources, to ensure that the evolution of health care needs is met, and that health systems work effectively.¹,²

In this sense, auditing is an indispensable activity for any type of health system, both public and private. It can be performed in health facilities and service care units under federal, state, and municipal management.³

Whereas the interventions proposed by the auditor tend to promote improvements in the outcome and the quality of care provided, the audit contributes to the qualification of management.⁴

Auditing, as an assessment tool, enables monitoring of the efficiency and effectiveness of care in the Unified Health System (SUS).⁵

In this context, a study⁶ showed that, in the international scenario, the use of standardized instruments qualify and quantify compliance with signed contracts between the parts, whether public, private, or philanthropic entities. Therefore, weaknesses and potentialities are evaluated, and the quality of the care provided can be measured, which enables planning for and evaluating of the achievement of the agreed goals, such as the number of patients receiving care, the number of surgeries performed, and the offer of medical consultations, among other things.

To understand the state of the art, the Pubmed, Web of Science, PsycInfo, and Lilacs databases were systematically searched through July 31, 2018, without limitation for language or year of publication, in order to find studies related to the area of contracting health services and, specifically, the standardization of instruments. The following Health Sciences Descriptors terms were used to identify publications: nursing audit; contracts; health evaluation; management in health; and, public health. These terms were combined with Boolean operators, according to the rules of each database. The following inclusion criteria were adopted: an original study, published in a journal with an editorial board and peer review, addresses evaluation tools in the contracting of health services.

The instruments found which related to a contract with the SUS were ministerial ordinances that inform basic and general rules regarding contracting,⁷ but no model of instrument that could be implemented or followed by public managers, nor any validated instrument.

The Ministry of Health (MS) Ordinance No. 3410/2013,⁷ which “Establishes the guidelines for the contracting of hospitals within the scope of SUS in accordance with the National Policy of Hospital Attention”, in Article 4, recommends that an instrument must be used to audit service providers whose purpose is to assess the commitments entered into between the parties.⁷

Due to the autonomy that the municipalities have, there is flexibility in the strategy of evaluation of the contracted services, which can cause divergences and failures in the development of instruments in an empirical manner. As a consequence, there may be failures in public management, poor use of public resources, and poor quality of care.⁸,⁹

The instrument adopted in this study was developed at the Secretariat of Health of Londrina-PR, based on contracts and legislation. However, difficulties were encountered by the contractor and contracted services in the evaluations, because the document led to dubious interpretations, in addition to lack of clarity and objectivity. Thus, we identified the need to develop a new instrument, validated by experts with experience in contracting in the SUS, that represented the various spheres involved in this process.

In view of the above, the objective of this study was to analyze the validity and reliability of an instrument used for auditing in SUS.

Methods

This was a quantitative, methodological, applied development study, conducted in the city of Londrina, and located in the south of Brazil. The county has full health management, and is responsible for planning, execution, and evaluation of contracted services. In 2017, the population was 558,439 inhabitants⁹ and the country was a reference for 21 municipalities, covering a population of 846,708 inhabitants.¹⁰

Authorization was requested, prior the beginning of the study, from the Directorate of Health Care Regulation (DRAS), which is part of the Health Department of the study site. This sector is responsible for contracting service providers for the SUS in the municipality.¹¹

The objective of this research is the audit assessment instrument adopted in the municipality, which consists of the qualitative part of the Descriptive Document (DD), which is an annex of the contracts established between public managers and service providers, in which the following are detailed: care model, type of assessment, method of teaching, and research that will be audited.

The original auditing instrument recommended by Ordinance no. 3410/2013⁷ consisted of eight categories: Health Care, Participation in the Priority Policies of the SUS, Education Policy, Hospital Management, Actions related to the work process and local/regional integration, Ministerial Policies of care in high complexity and specific areas, Human Resources Policies, Research and Technological Health Assessment. In total, the instrument had 100 items and 32 sub-items. This study was developed in three stages, as noted in figure 1.

Figure 1
Steps of adequacy and validation of the instrument

The first one consisted of the adaptation of the original instrument by authors and nursing doctoral students of diverse specialties, based on the scientific literature and ministerial ordinances.

To conduct this stage, in-person meetings were conducted between the authors to discuss and adapt the instrument. The study was presented to the nursing doctoral students in one of the meetings of the research group; later, the material was sent by email for analysis and suggestions, which were provided electronically.

In the second step, workshops with experts were performed to validate the content of the adapted instrument. The choice of these people was intentional, as different representatives with experience in the research theme were invited. Ten professionals working in the sector responsible for contracting participated in the study, both the municipal managers and providers: two public managers, three health care providers, and five auditors. The inclusion criterions for such specialists were: working for at least three years with auditing or management of the SUS, and having experience with contracting in this System. All participants signed the Terms of Free and Informed Consent.

The selection of these experts and the validation of the instrument in workshops were done because these strategies enable the participants to conduct an analysis of the context of which they are a part, aside from being considered as a potentiating moment in the participation of the different professionals, of sharing knowledge and evaluation practices. This same strategy was adopted in other validation studies.¹²-¹⁵

Individual and personal invitations were sent to each of the experts, explaining the theme and purpose of the study. Dates, times, and places that best served the group of experts were proposed. Four days before the first workshop, a bound copy of the instrument was sent to each of the participants, and one was maintained online, so that everyone had the opportunity to read it in advance.

Due to the complexity and quantity of items and sub-items of this document, two workshops were necessary for its analysis and validation. Each of the items was analyzed, according to these guidelines: if the item was re-developed, what was the criterion for its evaluation, and the source of information collected.

The group of experts evaluated the items of the instrument for semantic equivalence (grammar and vocabulary) and content (concept, relationship of one item to another) and it was requested that, when necessary, they would recommend suggestions for possible modifications or complementation of the items. A content validity index of 100% was adopted at this step. All the experts judged each of the items of the instrument simultaneously, and only after unanimous consensus of the group was the validation process finalized.

After the workshops were completed, the reliability of the instrument was evaluated. For this step, three nurses with professional experience of at least three years in the public audit area, who did not participate in the validation step, were intentionally invited. They were asked to administer the instrument within an institution which they had never audited.

The audit nurses administered the instrument simultaneously in the same institution, but without communication between them. The objective of this step was to evaluate the same situations, in a certain time continuum, enabling analysis of the inter-rater reliability test, which indicates the degree of agreement between independent evaluations of two or more experts, estimated by the Kappa calculation.¹⁶

Kappa indices were determined for auditors one and two, one and three, and, two and three, considering all questions and items.

A Kappa value of at least 0.6 was considered as a moderate correlation. If this value was not reached, adaptations of the instrument and its readministration by the auditors would be necessary, until the minimum Kappa value adopted in the study was reached.

After the implementation phase of the instrument, the auditors responded to a test on the applicability process.¹⁷ The criteria evaluated were: understanding the instructions and the questions, and the facility to indicate the responses to the instrument.

The data were tabulated in the Microsoft Excel for Windows® program, and analyzed using the SPSS, version 22.0.

Descriptive analyses of the data were performed, with absolute and relative frequencies, and the Kappa test for reliability.

The study met the ethical norms in research involving human beings, obtaining a favorable opinion, as stated in the Plataforma Brasil nº1.702.128.

Results

The authors’ adaptation of the instrument resulted in the adequacy of content and semantics for all the items, including updates based on current legislation. The original document was composed of items and sub-items, and in this step the sub-items became items.

Ten experts participated in the content validation step. Five of them were audit nurses, one was an administrative technician, one a marketing manager, one an executive manager, one a health secretary, and one was director of public services. The experts’ time working in SUS ranged from three to 30 years, with most of them having worked between three to ten years.

The evaluation of the experts resulted in restructuring of the instrument, with grammatical changes, exclusions of categories because they were in disagreement with ministerial ordinances, insertion of 14 items based on the current legislation, subdivision of four items that evaluated different issues, and exclusion of an item that was repetitive.

Considering all the steps of adequacy of the original instrument, which had 100 items and 32 sub items, and remained with 55 items, with their respective evaluation criteria, as well as the source of data collection, totaling 165 items of analysis. During the instrument validation, experts suggested changes in semantics and content of 77 (46.66%) items.

With regard to the quantities of categories that comprised the original instrument, a reduction from eight to five was made. These exclusions were based on current legislation.

The five categories of the validated instrument were:

1. Health Care

2. Care Axis

3. Management Axis

4. Education Axis

5. Teaching and Research Axis.

The changes improved the instrument, providing clarity and objectivity; evaluation criteria were inserted by item, along with the source of information collected. The adjustments to the original and validated document were entirely integrated in the link: www.nepgese.com.

The reliability of the instrument was validated, and it was administered by three audit nurses, with a mean time of experience in SUS of 12 years, with a minimum of eight and a maximum of 20 years.

The time required for administration of the instrument was three hours and thirty minutes for all nurses, who administered it simultaneously.

Table 1 shows the Kappa indices according to the nurses’ evaluations, by category of the instrument.

Table 1
Agreement index between the nurses for the instrument used in the audit in the Unified Health System, and the general agreement index by category

Audit nurses	Instrument category	Index by category	Index for all the categories
1 & 2	Category A	0.86	0.85
	Category B	0.80
	Category C	0.63
	Category D	1.00
	Category E	1.00
1 & 3	Category A	0.86	0.82
	Category B	0.61
	Category C	0.63
	Category D	1.00
	Category E	1.00
2 & 3	Category A	1.00	0.96
	Category B	0.80
	Category C	1.00
	Category D	1.00
	Category E	1.00

The results showed that nurses one and two had almost perfect agreement (0.80 to 1.00), with a Kappa index of 0.85; one and three had substantial agreement, with a Kappa index of 0.82; and, two and three had almost perfect agreement, with a Kappa index of 0.96.

The three auditors considered the instructions on how to administer and indicate the responses to the instrument as being easily understood. Regarding the understanding of the issues, two auditors reported complete ease and partial ease.

The group of experts was proposed to determine a value for each item of the instrument; however it constituted a limitation of the study, because during the workshops, the experts suggested not doing this, as the valuation depended on the other items contracted in each municipality or sphere of public management.

Discussion

The adaptation process, initially, consisted of a comprehensive reading of the original instrument and references that addressed the theme; as the authors had experience with contracting and assessment, the process was facilitated. Much current legislation on the subject was not been identified, but three essential pieces of legislation¹⁷,¹⁸ that reference Ministerial Ordinances werefound.

In the content validation step, the profile of the experts brought substantial contributions to the adequacy of the instrument, as they had experience working in the institution and SUS. Studies that used the same diversity of professionals, with different education and functions, showed the same findings in relation to enriching the discussions, with broad and complex reflections, making it possible to look at the same subject through different aspects.¹⁵

One of the strengths of this study was the intentional selection of the experts, who represented all those involved in the contracting and assessment process, which promoted discussions between public contractors (contractors), institution management (contracted), and nurse auditors (responsible for supervising the execution of contractual items).

Difficulties for expert consensus were observed in the workshops, because participants sometimes did not realize that the intention was to make the instrument suitable for any reality, bringing debates centered only on the experiences and realities of their own institutions. It was also noted that, by involving public managers and managers of contracted institutions, sometimes the workshop became a place for discussion of situational and individual issues.

Studies that used the same methodology, with individuals integrated into different contexts and who analyzed a specific theme, evidenced times of difficulty and consensus was not always reached.¹⁹,²⁰ Such events required intervention on the part of the authors, to emphasize the purpose of the study, and to strengthen the requirements contained in legislation, so that a common understanding could be reached in an impartial and resolute manner.¹⁵,²¹

Regarding the interrater reliability analysis, the substantial agreement attributed by the audit nurses in Category B- Care Axis, was justified by the fact that the instrument is qualitative; therefore, in some items, it contained subjective questions. For example, the item “B.12- Maintain the hospital environment in accordance with current norms and regulations of the Resolutions of the Collegiate Board of Directors”, which analyzed whether lighting, noise, and environments were pleasant. Items such as this make up the mandatory legislation, although they can be evaluated as very good or excellent, because the amount of noise, for example, can be analyzed differently between people, based on individual opinion.

It should be noted that, although some items of the instrument did not reach the percentage of almost perfect agreement among the audit nurses, the authors chose not to perform a second round of administration of the instrument, because, in general, it reached 84.6% interrater agreement, which represents statistical consistency.

In relation to the time to administer the instrument (3:30 hours), as the professional administering the instrument understands the questions, uses it frequently, knows the institution well, the administration time decreases, and the service can be distributed into visits to the institution by the auditor.

Because there were no articles in the literature of the past ten years that discussed audit instruments used to evaluate the contracting of services, the lack of comparative discussions in other studies with similar findings was another limitation of this study.

A study on the development of an audit instrument was found, to assess issues related to quality of care and hospital charges. To prepare this instrument, the professionals who participated in its construction were also chosen intentionally, from professionals who worked directly with the object under study. This instrument contributes to generating audit indicators, and consequently improving the quality of the service provided.⁵

Another study on the development of audit instruments, which had some items related to contracting, was developed by physical therapy auditors, and also used analysis of the literature and legislation in its methodology to construct indicators and instrument items. This instrument also had the objective of standardizing the audit in physiotherapy services, aiming at the quality of service provision.²²

Contracting is fundamental, as a tool for modernizing and improving the quality of provided health services²³ and the effectiveness of public management, but it is a complex tool that needs to be carefully evaluated.

Conclusion

The study enabled the adaptation and validation of an assessment instrument used for the SUS audit. The method used was adequate to reach the proposed objective, and can support the accomplishment of other research related to the topic. The adaptation and validation of this instrument can be used as a basis for managers in contract assessment and processes providing services, as well as public policy makers in structuring strategies based on the identified results. The implications for the possibility of better allocation of public resources are presented. Despite the limitations of the study, as it is impossible to value the instrument as no other validation studies of audit instruments used in SUS contracting was found, it is expected that this study will contribute to the improvement of Brazilian public health.

Referencias

• ¹
  Silva KR, Lima MD, Sousa MA. Auditoria: ferramenta de enfermagem para melhoria da qualidade assistencial. Rev Gest Saúde (Brasília). 2016;7(2):793-810.
• ²
  Peruzzo HE, Liberatti VM, Rossaneis MA, Vannuchi MT, Haddad MC. Perspectivas, sentimentos e vivências de ouvidores do Sistema Único de Saúde no exercício da função. Cienc Cuid Saude. 2017; 16(1):1-7.
• ³
  Dorne J, Volpato Hungare J. Conhecimentos teóricos de auditoria em enfermagem. Rev Uningá Rev. 2017;15(1):8.
• ⁴
  Segateli TN, Castanheira N. A atuação do profissional enfermeiro na auditoria em saúde. Rev Saúde Desenvolv. 2015;7(4):43-56.
• ⁵
  Viana CD, Bragas LZ, Lazzari DD, Garcia CT, Moura GM. Implantação da auditoria concorrente de enfermagem:um relato de experiência. Texto Contexto Enferm. 2016;25(1):e3250014. .
• ⁶
  Barbosa AP, Tardivo CR, Barbosa EC. Mecanismos de controle da gestão do SUS à Luz da Lei Complementar Nº 141, de 2012. Adm Públ Gestão Social. 2016; 8(1):66-74.
• ⁷
  Brasil. Portaria nº. 3.410, de 30 de dezembro de 2013. Estabelece as diretrizes para a contratualização de hospitais no âmbito do SUS em consonância com a Política Nacional de Atenção Hospitalar [Internet]. [citado 2018 Fev 20]. Disponível em:<http://ww.bvsms.saude.gov.br/bvs/saudelegis/gm/2014/prt3410_30_12_2013.html>
  » http://ww.bvsms.saude.gov.br/bvs/saudelegis/gm/2014/prt3410_30_12_2013.html
• ⁸
  Silva GG, Cruz VL, Santos RR, Leone RJ. Contratos de gestão com organizações sociais: o papel dos sistemas de controle interno como ferramenta de acompanhamento e fiscalização. Rev Cientifica Hermes. 2018; 21:274-94.
• ⁹
  Instituto Brasileiro de Geografia e Estatística (IBGE). Brasil. Paraná. Londrina. Panorama [Internet]. Brasília (DF): IBGE; 2017. [citado 2018 Maio 6].Disponível em: https://cidades.ibge.gov.br/brasil/pr/londrina/panorama
  » https://cidades.ibge.gov.br/brasil/pr/londrina/panorama
• ¹⁰
  Paraná. Regionais de Saúde. Regionais SESA - 17ª RS – Londrina [Internet]. Londrina: Secretaria da Saúde; sd. [citado 2018 Maio 10]. Disponível em: http://www.saude.pr.gov.br/modules/conteudo/conteudo.php?conteudo=2769 .
  » http://www.saude.pr.gov.br/modules/conteudo/conteudo.php?conteudo=2769
• ¹¹
  Londrina. Prefeitura de Londrina. Secretaria de Saúde de Londrina. Secretarias. Saúde. DRAS. Histórico [Internet]. [citado 2018 Jan 10]. Disponível em: http://www.londrina.pr.gov.br/index.php?option=com_content&view=article&id=559&Itemid=603
  » http://www.londrina.pr.gov.br/index.php?option=com_content&view=article&id=559&Itemid=603
• ¹²
  Assis MN, Andrade AC, Rogenski KE, Castilho V, Fugulin FMT. Intervenções de enfermagem em pediatria: contribuição para a mensuração da carga de trabalho. Rev Esc Enferm USP · 2015; 49(Esp2):83-89.
• ¹³
  Cruz CW, Gaidzinski RR. Tempo de enfermagem em centro de diagnóstico por imagem: desenvolvimento de instrumento. Acta Paul Enferm. 2013; 26(1):79-85.
• ¹⁴
  Bonfim D, Pereira MJ, Pierantoni CR, Haddad AE, Gaidzinski RR. Instrumento de medida de carga de trabalho dos profissionais de Saúde na Atenção Primária: desenvolvimento e validação. Rev Esc Enferm USP. 2015; 49(Esp 2):25-34.
• ¹⁵
  Onocko Campos RT, Miranda L, Gama CA, Ferrer AL, Diaz AR, Gonçalves L, Trapé TL. Oficinas de construção de indicadores e dispositivos de avaliação: uma nova técnica de consenso. Estud Pesqui Psicol. 2010;10(1):221-41.
• ¹⁶
  Perroca MG. The new version of a patient classification instrument: assessment of psychometric properties. J Adv Nurs. 2013;69(8):1862-8.
• ¹⁷
  Coluci MZ, Alexandre NM. Development of a questionnaire to evaluate the usability of assessment instruments. Rev Enferm UERJ. 2009;17(3):378-82.
• ¹⁸
  Brasil. Portaria nº. 2.925, de 01 de novembro de 2017. Revoga as legislações que instituem o Incentivo de Qualificação da Gestão Hospitalar (IGH) [Internet]. [citado 2018 Set 20]. Disponível em http://bvsms.saude.gov.br/bvs/saudelegis/gm/2017/prt2925_03_11_2017.html
  » http://bvsms.saude.gov.br/bvs/saudelegis/gm/2017/prt2925_03_11_2017.html
• ¹⁹
  Martins EP, Rutizat NS, Costa JC, Fernandes ES. Variação linguística em aulas de língua portuguesa: uma experiência com a leitura em oficinas do PIBID. In: II CONEDU – Congresso Nacional de Educação. 2015. [citado 2018 Mar 20] Disponível em: http://www.editorarealize.com.br/revistas/conedu/trabalhos/TRABALHO_EV045_MD1_SA15_ID7733_03092015104823.pdf
  » http://www.editorarealize.com.br/revistas/conedu/trabalhos/TRABALHO_EV045_MD1_SA15_ID7733_03092015104823.pdf
• ²⁰
  Ibiapina AR, Monteiro CF, Alencar DC, Fernandes MA, Costa Filho AA. Oficinas Terapêuticas e as mudanças sociais em pacientes com transtorno mental. Escola Anna Nery. 2017;21(3):e20160375
• ²¹
  Luiz GM, Prá RM, Azevedo RC. Intervenção psicossocial por meio de oficina de dinâmica de grupo em uma instituição: relato de experiência. Psic Rev São Paulo. 2014;2(2):245-60.
• ²²
  Aleluia IR, Santos FC. Auditoria em fisioterapia no Sistema Único de Saúde: proposta de um protocolo específico. Fisioter Mov. 2013;26(4):725-41.
• ²³
  Heyer G. New public management: A strategy for democratic police reform in transitioning and developing countries. Policing: Int J Police Strategies Manag. 2011; 34(3):419-33, 2011.

Publication Dates

• Publication in this collection
  10 Oct 2019
• Date of issue
  Sep-Oct 2019

History

• Received
  8 Oct 2018
• Accepted
  2 May 2019