Primary Health Care Planning workshops: construction and validation of an assessment instrument

Talleres de planificación de atención primaria de salud: construcción y validación de instrumentos de evaluación

ABSTRACT

Objective:

to describe the stages of construction and content validation of an instrument to assess Primary Health Care Planning workshops.

Methods:

this methodological study focused on validating the instrument’s content. The instrument developed was assessed by a committee of experts using the Delphi Technique, in two rounds. For the degree of agreement, percentage agreement and Content Validity Index (CVI) were used.

Results:

in the first round, six experts participated, and the degree of agreement was 87% for clarity and 94% for representativeness. In the second round, five experts participated, the CVI was 0.95 for clarity, 0.97 for representativeness and 0.96 total CVI. The final instrument had 42 items divided into three chunks.

Conclusion:

the instrument has content validity to assess Primary Health Care Planning workshops, being a tool for the use of state and municipal administrations.

Descriptors:
Primary Health Care; Health Care Evaluation Mechanisms; Validation Studies; Program Evaluation; Health Planning Support

RESUMEN

Objetivo:

describir las etapas de construcción y validación de contenido del instrumento para evaluar los talleres de Planificación de la Atención Primaria de Salud.

Métodos:

estudio metodológico, enfocado en validar el contenido del instrumento. El instrumento desarrollado fue evaluado por un comité de expertos utilizando la técnica Delphi, en dos rondas. Para el grado de acuerdo, se utilizaron el Porcentaje de Acuerdo y el Índice de Validez del Contenido (IVC).

Resultados:

en la primera ronda, participaron seis expertos, y la tasa de acuerdo fue del 87% para mayor claridad y del 94% para la representatividad. En la segunda ronda, participaron cinco expertos, el IVC fue de 0.95 por claridad, 0.97 por representatividad y 0.96 IVC total. El instrumento final tiene 42 ítems divididos en tres bloques.

Conclusión:

el instrumento de evaluación incorporado tiene validez de contenido para evaluar los talleres de Planificación de Atención Primaria de Salud, siendo una herramienta para el uso de las administraciones estatales y municipales.

Descriptores:
Atención Primaria de Salud; Evaluación en Salud; Estudios de Validación; Evaluación de Programas y Proyectos de Salud; Apoyo a la Planificación en Salud

RESUMO

Objetivo:

descrever as etapas de construção e validação de conteúdo do instrumento para avaliação das oficinas de Planificação da Atenção Primária à Saúde.

Métodos:

estudo metodológico, com foco na validação de conteúdo do instrumento. O instrumento elaborado foi avaliado por um comitê de especialistas utilizando a Técnica Delphi, em duas rodadas. Para o grau de concordância, foram utilizados o Percentual de Concordância e o Índice de Validade de Conteúdo (IVC).

Resultados:

na primeira rodada, participaram seis especialistas, e a taxa de concordância foi 87% para clareza e 94% para representatividade. Na segunda rodada, participaram cinco especialistas, o IVC foi 0,95 para clareza, 0,97 para representatividade e 0,96 IVC total. O instrumento final possui 42 itens divididos em três blocos.

Conclusão:

o instrumento de avaliação construído possui validade de conteúdo para avaliar as oficinas de Planificação da Atenção Primária à Saúde, sendo uma ferramenta para utilização das gestões estadual e municipal.

Descritores:
Atenção Primária à Saúde; Avaliação em Saúde; Estudos de Validação; Avaliação de Programas e Projetos de Saúde; Apoio ao Planejamento em Saúde

INTRODUCTION

A health system that aims to meet the needs of users must seek to organize the offer of its services and practices to achieve this objective. However, after more than 20 years of the implementation of the Brazilian Unified Health System (Sistema Único de Saúde, abbreviated SUS), sector analysis reveals a low effectiveness in the provision of services and other limitations resulting from scarce public financing, persistence of segmentation in the system, organizational barriers to access, incipient integration and coordination between levels of care^(1-2).

Primary Health Care (PHC), the first level of care, the communication center of the system must order the network and coordinate care. A health system based on PHC is a comprehensive organization strategy, ensuring that services are adjusted to the population’s health needs and coordinated care for other points of care, seeking to strengthen Health Care Networks (Redes de Atenção à Saúde, abbreviated RAS)^(1,3).

Access to health services in relation to the needs of the population has been widely debated. However, insufficient coherence and coordination in health care are still considered the main causes of lack of responses to the population⁽⁴⁾. Seeking institutional responses to the health needs to be faced in PHC, the Brazilian National Council of Health Officers (Conselho Nacional de Secretários de Saúde, abbreviated CONASS) has been developing Health Care Planning since 2007. The methodological proposal of CONASS, which manages and monitors the initiative, goes beyond professional training, as it contributes to the organization of services in RAS. Planning is being developed in 25 regions of eleven states in Brazil, which joined from the institutional partnership with the entity⁽⁵⁾.

Planning is a health care process that aims to problematize and reflect on the role of PHC as the organizer of the network. It seeks to provide technical support to municipal management teams and workers in the field to qualify the organization of the RAS according to SUS principles and comprehensive care between the primary, secondary, and tertiary care levels. It consists of six Primary Health Care Planning workshops, tutorials from PHC, and tutorials from Specialized Outpatient Care.

PHC Planning workshops methodology, the first stage of the process, provides for monthly meetings, with the participation of health team workers, managers, and state and municipal technicians⁽⁶⁾. Such theoretical workshops aim at the conceptual alignment of themes relevant to PHC and RAS, which are continued in on-site tutoring/supervision⁽⁵⁾. CONASS facilitators participate in the meetings by providing technical and operational support, aiming to unite theory to practice, with a view to reorganizing the work processes in health services, defining flows in the RAS and making agreements for greater resolution, based on the reality of participants.

Health assessment is necessary for RAS through tools that identify the critical nodes of the system. In Brazil, PHC assessment initiatives that contribute to the involvement of health teams in the process of improving the quality of services have been gaining prominence^(6-7). Despite the relevance of the assessment to measure the performance of the health system, there was a lack of instruments for assessing practices similar to Planning. We intend to present the construction and validation of an instrument to assess PHC Planning workshops, with a view to contributing to a systematic assessment of processes and development of tools that support the conduction of changes in health care practices.

OBJECTIVE

This study aims to describe the stages of construction and content validation of an instrument to assess Primary Health Care Planning workshops.

METHODS

Ethical aspects

The Research Ethics Committee (REC) of the Universidade Federal de Ciências da Saúde de Porto Alegre (UFCSPA) approved the research, is in compliance with the ethical and legal precepts that govern Resolution 466/12, which deals with research involving human beings⁽⁸⁾.

Study design, period, and place

This methodological study aimed at the construction and content validation of an instrument to assess PHC Planning workshops. It is a strategy that systematically uses the existing knowledge to develop a reliable, accurate, and usable instrument that can be used by other researchers⁽⁹⁾. For content validation, the Delphi Technique was used, which consists of judging the instrument by experts with extensive experience in the subject in question, throughout rounds, aiming to ascertain trends in the object under study⁽¹⁰⁾.

The design used was a methodological study using the Delphi Technique. The instrument was composed of construction, pilot test, content validation and assessment of the degree of agreement by experts. The two Delphi rounds were held between December 2017 and March 2018, in order to reach consensus on the instrument’s items.

Population and sample; inclusion and exclusion criteria

For convenience, five professionals were selected for the pilot test, two representatives from state management, two representatives from universities and one representative from Primary Care assistance. The research population was composed of a group of experts, formed by health professionals with experience in the field of PHC and/or Planning, from different states of Brazil and chosen for convenience, after analyzing the Currículo Lattes platform. The expected expert committee was 14 people, not being invited the same professionals who performed the pilot test.

The criteria to select participants were: being a health professional and having experience in PHC and/or Planning; being a professor in the field of PHC or RAS; being a professional in the state and/or municipal management of SUS, Primary Care and/or Specialized Care; being a CONASS server/consultant and who accepted to participate in the study. Moreover, expertise (training, professional experience, knowledge production) in the requested field and, if possible, qualification (specialization/master’s/doctorate) in the fields of PHC, public health, public health or health management were observed in the Currículo Lattes platform.

Study protocol

To construct the instrument’s first version, the documents made available by CONASS were considered, such as workshop booklets and experience of researchers. Fourty-three items were formulated, according to interrelation structure, process and result, proposed by Donabedian (1988) for health assessment⁽¹¹⁾. In relation to the score of the instrument items, a five-position Likert scale was used: “excellent” (5), “good” (4), “regular” (3), “bad” (2) and “terrible” (1) for questions in general and “very satisfied” (5), “satisfied” (4), “indifferent” (3), “dissatisfied” (2) and “very dissatisfied” (1) for questions that assessed degree of satisfaction.

After construction, a self-administered, individual and electronic form was sent to five professionals for the pilot test. This aimed to observe the pertinence of the items, the understanding of the instrument and the feasibility of the electronic form. At this stage, the Delphi Technique methodology was not used. Three professionals responded, two from state management and one from PHC assistance. Two university representatives did not return to the invitation for the pilot test. As for the items, no suggestions were made for inclusions and 35 items (81%) were restructured as to the form and/or the score of the Likert scale (> 90% with an opinion in favor of the modification). One item (2%) was completely rewritten and one item (2%) was deleted after being unified with another. The instrument totaled 42 items for content validation.

After the pilot test, the instrument was submitted to the selected experts via email, along with the link to the electronic form for content validation. The self-applicable form, individual and without nominal identification of the participants, was made available through the Google Forms tool, in order to judge its content. The form had, in its first section, the Informed Consent Term and clarifications in relation to ethical and methodological aspects, being an essential condition for completing the questionnaire.

The experts received specific instructions on content validation, in which they were instructed to assess the instrument in two different stages, based on the models by Coluci et al (2015)⁽¹²⁾. In the first round, the experts assessed each of the chunks and items of the instrument, determining the scope, that is, whether each chunk was adequately represented by the set of items and whether all dimensions were included. They also checked whether content was appropriate, whether structure was adequate and whether content was representative⁽¹²⁾. It was only in this round to suggest inclusion and/or elimination of items, in addition to comments and suggestions.

In the second round, the experts assessed each item individually, in addition to the format, title and scores of the instrument, considering the clarity and/or relevance of each aspect. Regarding clarity, they assessed the wording of the items, if they were understandable and adequately expressed what is expected to be measured. As for pertinence or representativeness, they verified whether the items really reflected the concepts involved, whether they were relevant and adequate to achieve the proposed objectives⁽¹²⁾.

Analysis of results, and statistics

In the first round, the degree of agreement was verified using the Degree of agreement, using the formula: % agreement = number of participants who agreed x 100/total number of participants. Items with an degree of agreement of 90% among experts were considered validated⁽¹³⁾. Items with agreement below this percentage were reworked or rewritten, according to the assessment, giving rise to the second version of the instrument.

In the second round, degree of agreement was verified through the Content Validity Index (CVI), which measures the proportion of experts in agreement, in relation to the items and the general aspects of the instrument, in a quantitative way. The CVI of each item, each chunk and instrument was calculated using a Likert scale with four positions. To assess the item’s relevance/representativeness, experts could choose between: 1 = not relevant or not representative; 2 = item needs major revision to be representative; 3 = item needs minor revision to be representative; 4 = relevant item or representative. Scope, clarity, and relevance were assessed using the same type of scale: 1 = not clear; 2 = not very clear; 3 = very clear; 4 = very clear. Calculation was performed from the sum of the answers “3” and “4” of each expert in each of the items and divided by the total number of answers. The formula used was: CVI = number of responses “3” or “4”/total number of responses. Items with a score of “1” or “2” have been revised or eliminated. Items that obtained CVI above 0.78 were considered validated⁽¹⁴⁾.

RESULTS

In the first round, six experts responded to the form, most of whom were women (66.7%) with a predominant age range of 30 to 39 years old (50%). Most had 5 to 9 years of training (50%). The professional category that most appeared was psychology (33.3%). Most participants had a specialization (50%) in the health field and the majority were state/municipal managers (33.3%) and/or professors (33.3%). The majority (83.3%) reported experience with Health Care Planning. In this round, no suggestions were made for the inclusion of new items. However, 25 items (59.5%) were maintained according to the first version, six items (14.3%) were restructured regarding writing and 11 items (26.2 %) were completely rewritten as to the form and/or punctuation of the Likert scale. In terms of clarity, the instrument presented an 87% degree of agreement among items (<90%). As for representativeness, degree of agreement was 94% between items (> 90%).

In the second round, five experts responded to the form, most of the participants were also women (80%), with predominant age groups from 30 to 39 years old (40%) and 40 to 49 years old (40%). Most had 5 to 9 years of training (60%). The predominant professional category was nursing (40%). Of the participants, most had a doctorate (60%) and were professors (60%). Most (60%) reported experience with Health Care Planning. In this round, no suggestions were made for inclusion or exclusion of items, all 42 being maintained. Of these, 40 items (95.2%) were maintained according to the second version and two (4.8%) were restructured as to writing. Moreover, two items were relocated to another chunk. The instrument reached CVI 0.95 for clarity and 0.97 for representativeness. This resulted in a total CVI of the instrument of 0.96 (> 0.78), characterizing the validation of its content.

As for the title, the CVI reached 0.80, with respect to layout and scores, the CVI was 1.00 for both. Table 1 shows in detail the degree of agreement in relation to clarity and representativeness of items, chunks, and the instrument in general, related to the two rounds of the Delphi Technique for content validation.

Table 1
Agreement of the Delphi Technique rounds for content validation. Degree of agreement (%) and Content Validity Index (CVI), Porto Alegre, Rio Grande do Sul, Brazil

Agreement	First round - degree of agreement (%)		Second round - Content Validity Index (CVI)
Item	Clarity	Representativeness	Clarity	Representativeness
Chunk 1 items - Structure
1.1	100	83.3	1.00	1.00
1.2	100	100	1.00	1.00
1.3	100	100	1.00	1.00
1.4	83.3	100	1.00	1.00
1.5	100	100	1.00	1.00
1.6	83.3	100	1.00	1.00
1.7	100	100	0.80	1.00
1.8	100	83.3	1.00	1.00
1.9	100	100	1.00	1.00
Chunk 2 items - Content
2.1	100	100	1.00	1.00
2.2	100	83.3	1.00	1.00
2.3	33.3	83.3	1.00	1.00
2.4	33.3	83.3	1.00	1.00
2.5	83.3	100	1.00	1.00
Workshop I
2.6	100	100	1.00	1.00
2.7	50	100	1.00	1.00
2.8	100	83.3	1.00	1.00
Workshop II
2.9	100	100	1.00	1.00
2.10	50	100	1.00	1.00
2.11	100	83.3	1.00	1.00
Workshop III
2.12	100	100	1.00	1.00
2.13	50	100	1.00	1.00
2.14	100	83.3	1.00	1.00
Workshop IV
2.15	100	100	1.00	1.00
2.16	100	100	1.00	1.00
2.17	100	100	1.00	1.00
2.18	50	100	1.00	1.00
2.19	100	83.3	1.00	1.00
Workshop V
2.20	100	100	1.00	1.00
2.21	100	100	1.00	1.00
2.22	50	100	1.00	1.00
2.23	100	83.3	1.00	1.00
Workshop VI
2.24	100	100	1.00	1.00
2.25	50	100	1.00	1.00
2.26	100	83.3	1.00	1.00
Chunk 3 items - Applicability
3.1	83.3	66.6	1.00	1.00
3.2	83.3	100	1.00	1.00
3.3	83.3	100	1.00	1.00
3.4	100	100	0.80	0.80
3.5	100	83.3	1.00	1.00
3.6	100	100	1.00	1.00
3.7	100	100	1.00	1.00
Scope of Chunks	Coverage	Does the item remain in the chunk?	Coverage	Is the item removed or inserted?
1 Structure	100	100	1.00	Não
2 Content	83.3	100	1.00	Não
3 Applicability	83.3	100	1.00	Não
Scope of the instrument			1.00	Não
Instrument	Clarity		Representativeness
Title	0.80		-
Layout	1.00		-
Score classification	1.00		1.00
Final agreement	Degree of agreement (%)		Content Validity Index (CVI)
Total chunks	Clarity	Representativeness	Clarity	Representativeness
	87	94	0.95	0.97
Total instrument CVI			0.96

Based on the results obtained, the content validation process was completed, giving rise to the third version of the instrument, considered the final validated version (Chart 1).

Chart 1
Instrument to Assess Primary Health Care Planning. Porto Alegre, Rio Grande do Sul, Brazil, 2018

INSTRUMENT TO ASSESS RIMARY HEALTH CARE PLANNING WORKSHOPS
City Professional category: Place of performance:
CHUNK 1. STRUCTURE
1.1 What is your general satisfaction with the structure of PHC Planning workshops? ( ) Very satisfied ( ) Satisfied ( ) Indifferent ( ) Dissatisfied ( ) Very Dissatisfied
1.2 How do you assess the organization/logistics of PHC Planning workshops (location, rooms, schedule, punctuality)? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
1.3 How do you assess the didactic material used in the six PHC Planning workshops? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
1.4 How do you assess the activities adopted during the six PHC Planning workshops (individual and group readings, group work/dynamics, dialogues, videos) ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
1.5 How do you assess the workload and number of days in PHC Planning workshops? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible Why? _________
1.6 How do you assess the number of workshops held (six)? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
1.7 How do you assess the dispersion tasks regarding the ease and operability of contents of workshops in professional practice? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
1.8 How do you assess the performance of the group of facilitators regarding mediation, dynamics, and ethics in PHC Planning workshops? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
1.9 In your opinion, could some themes be addressed through distance education? Yes ( ) No ( ) Which?
1.10 How do you assess the participation of MANAGERS (such as Health Officers) in your city in PHC Planning workshops? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
CHUNK 2 CONTENT
2.1 How satisfied are you with the content of PHC Planning w orkshops? ( ) Very satisfied ( ) Satisfied ( ) Indifferent ( ) Dissatisfied ( ) Very Dissatisfied
2.2 You consider that your understanding of the objectives of each workshop was: ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
2.3 Classify, in order of importance, the subjects of each Planning WORKSHOP, using a scale of 1 to 6, 1 being the one that LESS contributed and 6 that contributed MORE to the change and/or qualification of your work process and his team: ( ) Workshop I - Health Care Networks ( ) Workshop II - Primary Health Care ( ) Workshop III - Territorialization and Health Surveillance ( ) Workshop IV - Organization of care for acute events and chronic conditions in Primary Health Care ( ) Workshop V - Pharmaceutical Assistance and Diagnostic Support ( ) Workshop VI - Monitoring and Assessment in Primary Health Care
2.4 Classify, in order of importance, the DISPERSION activities of each Planning Workshop, using a scale from 1 to 6, 1 being the one that LESS contributed to and 6 that contributed MORE to the change and/or qualification of your process and your team: ( ) Workshop I - Health Care Networks ( ) Workshop II - Primary Health Care ( ) Workshop III - Territorialization and Health Surveillance ( ) Workshop IV - Organization of care for acute events and chronic conditions in Primary Health Care ( ) Workshop V - Pharmaceutical Assistance and Diagnostic Support ( ) Workshop VI - Monitoring and Assessment in Primary Health Care
2.5 Of the subjects covered in the six Workshops, do you think that any could be suppressed? What? ( ) Health Care Networs ( ) Primary Health Care ( ) Territorialization and Health Surveillance ( ) Organization of care for acute events and chronic conditions in Primary Health Care ( ) Pharmaceutical Assistance and Diagnostic Support ( ) Monitoring and Assessment in Primary Health Care ( ) All items need to be addressed For what reason do you think the subject could be suppressed? ______________
Workshop I - Health Care Networks
2.6 How do you assess your understanding of the functioning of the health system in Care Networks after the completion of I Planning Workshop? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible 2.7 Classify, in order of importance, each subject addressed in Workshop I (Health Care Networks), using a scale of 1 to 6, 1 being the one that LESS contributed and 6 that contributed MORE to your learning: ( ) Reasons for the contemporary crisis in health care systems ( ) Changes in the health situation in the country ( ) Ways of structuring health care systems ( ) Fundamentals of Health Care Networks ( ) Operational structure of Health Care Networks ( ) Modeling Health Care Networks 2.8 How do you assess the applicability of the subjects covered in Workshop I in your work process? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
Workshop II - Primary Health Care
2.9 How do you assess your understanding of access to Primary Health Care, based on the health situation in the city after the conclusion of II Planning Workshop? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
2.10 Classify, in order of importance, each subject addressed in Workshop II (Primary Health Care), using a scale of 1 to 5, with 1 being the one that LESS contributed and 5 that contributed MORE to your learning: ( ) Concept and constituent elements of Primary Health Care ( ) Primary Health Care crisis at the clinic's micro level ( ) Main demands of Primary Health Care ( ) Proposal for the social construction of Primary Health Care ( ) Main foundations for the organization of access in Primary Health Care. 2.11 How do you assess the applicability of the subjects covered in Workshop II in your work process? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
Workshop III - Territorialization and Health Surveillance
2.12 How do you assess your understanding of the integration of Primary Care with Surveillance in the territory after the conclusion of III Planning Workshop III? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
2.13 Classify, in order of importance, each subject addressed in Workshop III (Territorialization and Health Surveillance), using a scale from 1 to 4, 1 being the one that LESS contributed and 4 that contributed MORE to your learning: ( ) Concept of territory and territorialization, as well as its applicability in health planning ( ) Concepts of risk and vulnerability in the territories assigned to teams ( ) Situational analysis tools for planning, monitoring, and assessing actions ( ) Relationships between Health Surveillance and Primary Health Care focusing on the territory 2.14 How do you assess the applicability of the subjects covered in Workshop III in your work process? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
Workshop IV - Organization of care for acute events and chronic conditions in Primary Health Care
2.15 How do you assess your understanding of the organization of care for acute events and chronic conditions in your Primary Care work process after the conclusion of IV Planning Workshop? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
2.16 How do you assess your qualification in relation to the organization of care for acute events (reception with risk classification) in the Basic Health Unit after the completion of Planning workshops? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
2.17 How do you assess your qualification in relation to the organization of care for chronic conditions (programmed demand and risk stratification) in the Basic Health Unit after the completion of the Planning workshops? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
2.18 Classify, in order of importance, each subject addressed in Workshop IV (Organization of care for acute events and chronic conditions in Primary Health Care), using a scale of 1 to 5, 1 being the one that LESS contributed to and 5 for what MORE contributed to your learning: ( ) Current PHC organization to meet acute events and chronic conditions ( ) Model of Care for Acute Events ( ) Chronic Conditions Care Model (Modelo de Atenção às Condições Crônicas, abbreviated MACC) proposed for the Brazilian Unified Health System ( ) Organization of care for acute events in PHC (reception, risk classification) ( ) Organization of care for chronic conditions in PHC (risk stratification, schedule, agenda)
2.19 How do you assess the applicability of the subjects covered in Workshop IV in your work process? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
Workshop V - Pharmaceutical Assistance and Diagnostic Support
2.20 How do you assess your understanding of the organization and integration of pharmaceutical care in Primary Care after the completion of V Planning Workshop? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
2.21 How do you assess your understanding of the RAS Diagnostic Support System presented in Workshop V? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
2.22 Classify, in order of importance, each subject addressed in Workshop V (Pharmaceutical Assistance and Diagnostic Support), using a scale of 1 to 4, 1 being the one that LESS contributed and 4 that contributed MORE to your learning: ( ) Importance of pharmaceutical assistance as a support system for Health Care Networks ( ) The role of PHC in Pharmaceutical Assistance, especially in the execution of the stages of the logistical cycle ( ) Strategies for implementing pharmaceutical care in Primary Health Care ( ) Support systems in Health Care Networks
2.23 How do you assess the applicability of the subjects covered in Workshop V in your work process? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
Workshop VI - Monitoring and Assessment in Primary Health Care
2.24 How do you assess your understanding of monitoring and assessment to improve Primary Health Care actions and services after the conclusion of VI Planning Workshop? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
2.25 Classify, in order of importance, each subject addressed in Workshop VI (Monitoring and Assessment in Primary Health Care), using a scale from 1 to 5, with 1 being the one that LESS contributed and 5 that contributed MORE to your learning: ( ) Fundamentals of monitoring and assessment ( ) Importance of monitoring and assessing health actions in Primary Care ( ) Indicators and targets related to chronic conditions in Primary Health Care ( ) Matrix for monitoring the goals agreed in Primary Health Care ( ) Plan for operationalization of agreed goals
2.26 How do you assess the applicability of the subjects covered in Workshop VI in your work process? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
CHUNK 3 APPLICABILITY
3.1 How do you assess the contribution of PHC Planning Workshops to YOUR professional qualification? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
3.2 After completing Planning Workshops, how do you assess the achievement of the overall objective of Planning? General objective: "Supporting the induction of the organization of Primary Health Care (PHC), providing tools and strategies for programming and organizing the work process of the teams, as well as integration between the various points and levels of care". ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
3.3 How satisfied are you with the qualification of YOUR TEAM work process at the end of the Planning process? ( ) Very satisfied ( ) Satisfied ( ) Indifferent ( ) Dissatisfied ( ) Very Dissatisfied
3.4 How do you assess YOUR ability to put into practice the issues covered during the PHC Planning process? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible
3.5 How do you assess the capacity of YOUR TEAM to develop a PHC intervention plan, after the dispersion activities of Planning? ( ) Great ( ) Good ( ) Regular ( ) Bad ( ) Terrible

DISCUSSION

Analyzing the assessment instrument in relation to structure, corresponding to Chunk 1 with nine items, four showed below-expected agreement (<90%). Difficulties were observed in the interpretations related to the writing of items; thus, they were rewritten to provide better clarity and understanding to the respondents. According to the literature, “clarity” must be intelligible to all strata of the target population because understanding phrases is more important than their artistic elegance⁽¹⁵⁾.

In the last round, all nine related items were validated above the expected agreement (CVI> 0.78). The literature points to the need to value the structural components in the assessment of health services and to discuss their relationship with the quality of work processes and the achievement of results, with respect to the health of the population. Findings reiterate the need for investments in the structure dimension aimed at the needs of professionals working in teams^(16-17).

As for content (process), relating to Chunk 2 with 26 items, 16 presented below-expected agreement among experts (<90%). Most of the items were rewritten, based on the suggestions of the experts who contributed qualitatively to their writing; however, some items were kept due to insufficient suggestions. A similar finding was found in another study, in which the process dimension was the one with the highest number of rectifications⁽¹⁵⁾.

It is noteworthy the fact that all 26 items related to content were validated with 100% agreement in the last round (CVI 1.00), which demonstrates stability in their clarity and representativeness. In another validation study, some items also showed full consensus of relevance among the participants, with maximum CVI, making it possible to verify credibility and transparency in these items of the instrument⁽¹⁸⁾.

The need to assess the understanding of contents of Planning workshops seeks to provide changes in the health care process offered by PHC teams. With regard to the workshops, it was considered relevant to assess, based on the proposed instrument, the level of understanding, professional training and integration among team members, because they are strategies that influence health care. Efforts must be made to improve the quality of care offered, based on assessment processes that allow monitoring the services’ capacity to respond to health needs. It is necessary to expand the coverage of programs, motivate and train professionals, encourage teamwork, organize communication between levels of care and systematically assess the results obtained^(15,19).

Nevertheless, monitoring and assessment in PHC are relevant to understand the assessment processes as essential in guiding teams’ health practices. The incorporation of assessment in the routine of health organizations can be carried out through instruments that integrate the planning process and the management of policies and programs. There is a long way to the institutionalization of the assessment, result of a joint work of parties and conquest of political space for autonomy of the necessary resources and strengthening of the evaluative capacity^(20-22).

Finally, with regard to the operational applicability of Planning (result), relating to Chunk 3 with seven items, four showed below-expected agreement among experts (<90%), being restructured according to suggestions. In the last round, all seven items related to applicability were validated above the expected agreement (CVI> 0.78). Assessment of the result dimension consists of verifying whether they correspond to what was expected, i.e., to the objectives that the intervention proposed to achieve, which can influence the decision-making of managers and teams⁽²³⁾.

The items in the applicability chunk listed issues that address the ability to put into practice what was apprehended. An instrument must become dynamic from the monitoring and assessment of the results obtained and the need for the service, as well as to compose other indicators. When assessing the degree of integration of the RAS, it also makes it possible to assess the capacity of Primary Care, aiming to subsidize new strategies regarding the structuring and organization of the system⁽²⁰⁾. Therefore, it is considered that the assessment of health services from instruments acts to build a new perspective of care, since a valid judgment on the results of an intervention helps in the daily life of health services and management^(18,23).

Therefore, the study provides information on the construction, validation, representativeness and clarity of the proposed instrument, based on the collaboration of experts in the field. In this way, the instrument’s final version allows the knowledge of the local reality, aiming to subsidize the assessment of health care processes, as well as serving as a basis for assessing future interventions.

Study limitations

As a limitation of the study, the sample universe of participants is mentioned, in view of refusal of some selected experts, delay in returning the material sent, and loss of participants between rounds.

Contributions to nursing, health, and public policies

Construction and validation of the instrument seeks to contribute to assess the Primary Health Care Planning process and to use instruments as tools that lead to changes in health care. Even though the final instrument was assessed by judges from various professional categories with expertise in the subject, as it is a tool that assesses structure, process and results from the perspective of RAS, it can be applied by nurses to measure the effect Planning workshops in training and in the work process. Concerning health assessment in PHC, institutionalization of this practice can support managers in decision-making, teams in health care planning and constant monitoring of results. The methodology applied in the study used parameters that showed the instrument’s content validity, which can be replicated in other studies and similar initiatives.

CONCLUSION

The critical assessment carried out by the expert committee in the two content validation rounds allowed the improvement of the instrument to assess PHC Planning workshops, with reformulated questions to better suit its target audience. The use of the Delphi Technique, associated with methodologies to verify degree of agreement, was very effective in the rounds and the priorities in the approach of the workshops.

Other advantages of the Delphi Technique were observed such as anonymity among participants, provision of equal conditions of participation, application of the form to distant people and composition quality of the expert committee. Despite the fact that electronic platforms are well accepted by study participants, some challenges were observed, such as difficulty in attracting and complying with experts and loss of participants between rounds. Such challenges did not affect the validity or quality of the study.

The results contribute to the expansion of knowledge on the subject and to highlight the relevance of health assessment processes, through the use of instruments that measure the performance of practices. The instrument has content validity to assess Primary Health Care Planning workshops, being a tool for the use of state and municipal administrations.

REFERENCES

• ¹
  Almeida PF, Medina MG, Fausto MCR, Giovanella L, Bousquat A, Mendonça MHM. Coordenação do cuidado e Atenção Primária à Saúde no Sistema Único de Saúde. Saúde Debate. 2018;42(spe1):244-260. doi: 10.1590/0103-11042018s116
  » https://doi.org/10.1590/0103-11042018s116
• ²
  Facchini LA, Garcia LP. Evolução e avanços da Saúde da Família e os 20 anos do Sistema Único de Saúde no Brasil. In: Brasil. Ministério da Saúde. Secretaria de Vigilância em Saúde. Departamento de Análise de Situação de Saúde. Saúde Brasil 2008: 20 anos de Sistema Único de Saúde (SUS) no Brasil [Internet]. Brasília: Ministério da Saúde; 2009 [cited 2018 Mar 22]. 416 p. Available from: http://bvsms.saude.gov.br/bvs/publicacoes/saude_brasil_2008.pdf
  » http://bvsms.saude.gov.br/bvs/publicacoes/saude_brasil_2008.pdf
• ³
  Starfield B. Atenção primária: equilíbrio entre necessidade de saúde, serviços e tecnologia [Internet]. Brasília: UNESCO, Ministério da Saúde; 2002 [cited 2018 Mar 22]. 726p. Available from: http://bvsms.saude.gov.br/bvs/publicacoes/atencao_primaria_p1.pdf
  » http://bvsms.saude.gov.br/bvs/publicacoes/atencao_primaria_p1.pdf
• ⁴
  Saltman RB, Rico A, Boerma WGW, editors. Atenção Primária conduzindo as redes de atenção à saúde: reforma organizacional na atenção primária europeia [Internet]. Berkshire: Open University Press/McGraw-Hill Education; 2006 [cited 2018 Mar 22]. 342p. Available from: http://bvsms.saude.gov.br/bvs/publicacoes/atencao_primaria_conduzindo_redes.pdf
  » http://bvsms.saude.gov.br/bvs/publicacoes/atencao_primaria_conduzindo_redes.pdf
• ⁵
  Evangelista MJO, Guimarães AMDAN, Dourado EMR, Vale FLB, Lins MZS, Matos MAB, et al. O Planejamento e a construção das Redes de Atenção à Saúde no DF, Brasil. Ciênc Saúde Colet. 2019; 24(suppl 6):2115-24. doi: 10.1590/1413-81232018246.08882019
  » https://doi.org/10.1590/1413-81232018246.08882019
• ⁶
  Conselho Nacional de Secretários de Saúde (BR). Planificação da Atenção Primária à Saúde nos Estados [Internet]. Brasília: CONASS; 2011 [cited 2018 Mar 22]. 436 p. Available from: https://www.conass.org.br/conassdocumenta/cd_23.pdf
  » https://www.conass.org.br/conassdocumenta/cd_23.pdf
• ⁷
  Novaes HMD, Soárez PC. Health technology assessment (HTA) organizations: dimensions of the institutional and political framework. Cad Saúde Pública. 2016;32(suppl-2):e00022315. doi: 10.1590/0102-311X00022315
  » https://doi.org/10.1590/0102-311X00022315
• ⁸
  Conselho Nacional de Saúde (BR). Resolução Nº466, de 12 de dezembro de 2012. Diretrizes e Normas Regulamentadoras de Pesquisas Envolvendo Seres Humanos. Diário Oficial União: República Federativa do Brasil; 2013. Jun 13, Seção 71: p.59.
• ⁹
  Polit DF, Beck CT. Fundamentos de Pesquisa em enfermagem: avaliação de evidências para as práticas da enfermagem. Porto Alegre (RS): Artmed; 2018. 864 p.
• ¹⁰
  Revorêdo LS, Maia RS, Torres GV, Maia EMC. O uso da técnica delphi em saúde: uma revisão integrativa de estudos brasileiros. Arq Ciênc Saúde. 2015;22(suppl-2):16-21. doi: 10.17696/2318-3691.22.2.2015.136
  » https://doi.org/10.17696/2318-3691.22.2.2015.136
• ¹¹
  Donabedian A. Assessment of technology and quality: A Comparative Study of Certainties and Ambiguities. Int J Technol Assess Health Care. 1988;4(suppl-4):487-96. doi: 10.1017/S0266462300007571
  » https://doi.org/10.1017/S0266462300007571
• ¹²
  Coluci MZO, Alexandre NMC, Milani D. Construção de instrumentos de medida na área da saúde. Ciênc Saúde Colet. 2015; 20(suppl 3):925-36. doi: 10.1590/1413-81232015203.04332013
  » https://doi.org/10.1590/1413-81232015203.04332013
• ¹³
  Hulley SB. Delineando a pesquisa clínica. Porto Alegre: Artmed; 2015. 400 p.
• ¹⁴
  Alexandre NMC, Coluci MZO. Validade de conteúdo nos processos de construção e adaptação de instrumentos de medidas. Ciênc Saúde Colet. 2011;16(suppl7):3061-8. doi: 10.1590/S1413-81232011000800006
  » https://doi.org/10.1590/S1413-81232011000800006
• ¹⁵
  Bellucci JJA, Matsuda LM. Construction and validation of an instrument to assess the Reception with Risk Rating. Rev Bras Enferm. 2012;65(suppl5):751-7. doi: 10.1590/S0034-71672012000500006
  » https://doi.org/10.1590/S0034-71672012000500006
• ¹⁶
  Poças KC, Freitas LRS, Duarte EC. Census of the Primary Health Care structure in Brazil (2012): potential coverage estimates. Epidemiol Serv Saúde. 2017;26(suppl2):275-84. doi: 10.5123/S1679-49742017000200005
  » https://doi.org/10.5123/S1679-49742017000200005
• ¹⁷
  Moura BLA, Cunha RC, Fonseca ACF, Aquino R, Medina MG, Vilasbôas ALQ, et al. Atenção primária à saúde: estrutura das unidades como componente da atenção à saúde. Rev Bras Saúde Matern. Infant. 2010;10(suppl1):s69-s81. doi: 10.1590/S1519-38292010000500007
  » https://doi.org/10.1590/S1519-38292010000500007
• ¹⁸
  Cubas MR, Faoro NT, Moysés ST, Carvalho DR. Evaluation of Primary Health Care: validation of an instrument to analyze the performance of services. Saúde Debate. 2017;41(suppl113):471-85. doi: 10.1590/0103-1104201711310
  » https://doi.org/10.1590/0103-1104201711310
• ¹⁹
  Ferreira SRS, Périco LAD, Dias VRGF. The complexity of the work of nurses in Primary Health Care. Rev Bras Enferm. 2018; 71(suppl 1):704-9. doi: 10.1590/0034-7167-2017-0471
  » https://doi.org/10.1590/0034-7167-2017-0471
• ²⁰
  Mendes EV. As Redes de Atenção à Saúde [Internet]. Brasília: Organização Pan-Americana da Saúde; 2011 [cited 2018 Mar 22]. 549 p. Available from: http://bvsms.saude.gov.br/bvs/publicacoes/redes_de_atencao_saude.pdf
  » http://bvsms.saude.gov.br/bvs/publicacoes/redes_de_atencao_saude.pdf
• ²¹
  Souza GS, Costa EA, Barros RD, Pereira MT, Barreto JL, Guerra JAA, et al. Characterization of the institutionalization of pharmaceutical services in Brazilian primary health care. Rev Saúde Pública. 2017;51(suppl2):7s. doi: 10.11606/s1518-8787.2017051007135
  » https://doi.org/10.11606/s1518-8787.2017051007135
• ²²
  Nickel DA, Natal S, Hartz ZMA, Calvo MCM. O uso de uma avaliação por gestores da atenção primária em saúde: um estudo de caso no Sul do Brasil. Cad Saúde Pública. 2014; 30(suppl 12):2619-30. doi: 10.1590/0102-311x00022314
  » https://doi.org/10.1590/0102-311x00022314
• ²³
  Hartz ZMA (org.) Avaliação em Saúde: dos modelos conceituais à prática na análise da implantação de programas [Internet]. Rio de Janeiro: Fiocruz; 1997 [cited 2018 Mar 22]. 132 p. doi: 10.7476/9788575414033
  » https://doi.org/10.7476/9788575414033

Edited by

EDITOR IN CHIEF: Dulce Barbosa

ASSOCIATE EDITOR: Hugo Fernandes

Publication Dates

• Publication in this collection
  07 Sept 2020
• Date of issue
  2020

History

• Received
  10 July 2019
• Accepted
  17 Apr 2020