Validation of indicators for monitoring the quality of prenatal care^a

Abstract

Objective

to validate indicators for monitoring the quality of prenatal care.

Method

methodological study conducted in 2020 with 11 specialists of the maternal and child health care line of Paraná. The indicators were organized in domains of a logical model and in the triad structure, process, and result. The calculated agreement rate, content validity ratio, content validity index, and Cronbach’s alpha reliability were analyzed.

Results

35 indicators were elaborated, which after the validation procedures were readjusted for clarity, two were excluded. It showed excellent reliability for clarity and relevance of the structure (0.94), process (0.98), and result (0.94), as well as in relation to the domains of the logical model of inputs (0.96), activities (0.86), outputs (0.98), results (0.86), and impact (0.96).

Conclusion

the indicators are valid and reliable for evaluating the quality of prenatal care, reflecting the impact of this care on quality management.

Implications for Practice

The construct presents flexibility of application for several territorial dimensions such as municipalities, regional health, and state.

Keywords:
Evidence-Informed Policy; Health Evaluation; Maternal and Child Health; Prenatal Care; Quality Indicators, Health Care

Resumo

Objetivo

validar indicadores para o monitoramento da qualidade da assistência pré-natal.

Método

estudo metodológico, com 11 especialistas da Linha de Cuidado à Saúde Materna e Infantil do Paraná, realizado em 2020. Os indicadores foram organizados em domínios de um modelo lógico e na tríade estrutura, processo e resultado. Analisado Taxa de Concordância, Razão de Validade de Conteúdo, Índice de Validade de Conteúdo e confiabilidade pelo Alfa de Cronbach.

Resultados

elaboração de 35 indicadores e, após os procedimentos de validação foram readequados quanto a clareza, dois foram excluídos. Apresentaram confiabilidade excelente para clareza e relevância da estrutura (0,94), do processo (0,98) e do resultado (0,94); bem como, em relação aos domínios do modelo lógico de entradas (0,96), atividades (0,86), saídas (0,98), resultados (0,86) e impacto (0,96).

Conclusão

os indicadores apresentam validade e confiabilidade para da qualidade do pré-natal, sob a ótica do monitoramento e da qualidade em saúde.

Implicações para a Prática

o constructo apresenta flexibilidade de aplicação para diversas dimensões territoriais como municípios, regionais de saúde e estado.

Palavras-chave:
Avaliação em Saúde; Cuidado Pré-Natal; Indicadores de Qualidade em Assistência à Saúde; Política Informada por Evidências; Saúde Materno-Infantil

Resumen

Objetivo

validar indicadores para el seguimiento de la calidad de la atención prenatal.

Método

estudio metodológico, con 11 especialistas de la Línea de Atención Materno infantil de Paraná, indicadores organizados en dominios de un modelo lógico y en la organización de la tríada estructura, proceso y resultado, realizado en 2020. Tasa de Concordancia Calculada, Razón de Validez de Contenido, Índice de Validez de Contenido; y confiabilidad por Alfa de Cronbach.

Resultados

Se elaboraron 35 indicadores, que luego de reajustar los procedimientos de validación para mayor claridad, se excluyeron dos. Mostró una excelente confiabilidad para la claridad y relevancia de la estructura (0.94), el proceso (0.98) y el resultado (0.94); así como en relación con los dominios del modelo lógico de insumos (0,96), actividades (0,86), productos (0,98), resultados (0,86) e impacto (0,96).

Conclusión

los indicadores son válidos y confiables para evaluar la calidad de la atención prenatal, reflejando el impacto de esta atención en la gestión de la calidad.

Implicaciones para la práctica

El constructo presenta flexibilidad de aplicación para varias dimensiones territoriales como municipios, salud regional y estadual.

Palabras clave:
Atención Prenatal; Evaluación en Salud; Indicadores de Calidad de la Atención de Salud; Política Informada por la Evidencia; Salud Materno-Infantil

Introduction

Prenatal care is of substantial importance for promoting maternal and child health since its actions seek to promote the prevention, diagnosis, and treatment of diseases or deficiencies. In addition, it offers information, especially on health and aspects related to the social, cultural, and psychological support of pregnant women, reducing preventable deaths and improving the quality of life of the population involved by increasing health care quality^1,2.

To strengthen prenatal care in Brazil, the Prenatal and Birth Humanization Program³ was instituted in 2000, which guarantees access to care, the right to be accompanied, and linking of the pregnant woman to the maternity hospital at the time of delivery. Furthermore, the first medical consultation must take place before the 16^th week of pregnancy, totaling a minimum of seven consultations, one of them during the puerperal period. These consultations must include the main medical exams, such as blood typing, anemia tests, fasting glycemia, and syphilis and HIV tests, as well as provide the opportunity for recommended vaccination, establish the risk classification of the pregnant woman and, when necessary, make referrals to high-risk outpatient care or hospital follow-up. These strategies must be accompanied by significant health education for pregnant women^2,3.

In 2011, the Stork Network⁴ was established, which organized Brazil’s Unified Health System (SUS) similarly to the Care Networks in order to improve access, coverage, and quality of prenatal care, childbirth, puerperium, and care for the newborn. Prenatal care is understood as one of its components, strengthening the recommended service and promptly, primarily through risk stratification, support for pregnant women, and availability of transportation for access^2,4, thereby reducing maternal and infant mortality.

In 2014, a national study (Nascer no Brasil) evaluated the quality of prenatal care provided by SUS and reported that 75.8% of pregnant women started follow-up before the 16^th week of gestation, and 73.1% underwent six recommended consultations. The late initiation of prenatal care was related to difficulty in diagnosing pregnancy (4.6%), reports of personal problems (30.1%), and difficulty in access (23.2%). The difficulty in access was identified mainly in indigenous women and in northern Brazil. Moreover, the prenatal care booklet’s data on tests performed and recorded were lower in the northern and northeastern regions, both for fasting glucose tests and ultrasound examinations⁵.

In 2016, a study conducted in Paraná State evaluated the implementation of the Stork Network in three health regions and identified a positive impact with its implementation, showing improved maternal and child health indicators. Moreover, it identified gaps that require qualification strategies of professionals for records in the health information systems⁶. Another study conducted in the health region of Foz do Iguaçu reported that the quality of prenatal care is intermediate, although 81.2% (p < 0.001) of pregnant women had six consultations or more, and 79.4% (p < 0.001) started the follow-up assessments before 16 weeks of gestation. Quality failures occur due to the absence of dental appointments (72.9%; p < 0.001), oncotic cytology collection (62.7%; p < 0.001), and clinical breast exams (42.8%; p < 0.001). There are failures in the health education due to the lack of information explaining the gestational risk (59.7%; p < 0.001), non-participation in a group for pregnant women (88.4%; p < 0.001), and lack of visits to the maternity hospital before delivery (61.6%; p < 0.001)⁷.

To institutionalize health care quality monitoring, it is necessary to evaluate triad of structure, process, and result. This need can be met by measuring indicators, which prioritize identifying health status changes caused by interventions proposed in public policies. Hence, quantifying the reduction of preventable maternal and infant deaths at the local level and increasing quality of life corroborates the Sustainable Development Goals and reinforces the 2016-2030 global agenda. With this, this study sought to validate indicators to monitor the quality of prenatal care, considering the experience of Paraná State that follows the recommendations proposed by the Stork Network.

Method

This is an excerpt from a methodological study that developed and validated a logic model for management under the governance model of the Maternal and Child Health Care Line, from prenatal care (as shown herein) to newborn care. The study was developed in five stages: bibliographic research to select scientific evidence; constructing the first version of the instrument; data collection for content validation; data analysis of the content validation; and redesign of the instrument⁸. To develop this study, we followed the Standards for Quality Improvement Reporting Excellence (SQUIRE)⁹ recommendations, which guide the structure for reporting new knowledge on improving health care.

In the first stage, a structured literature review¹⁰ was conducted based on the question: “What are the criteria for assessing the quality of women’s and children’s health care proposed by Brazilian public health policies?” Literature identification was collated and supported by legal documents such as the Constitution, laws, resolutions, ordinances, normative instructions, and documents from the Ministry of Health, which guide public health policies. These documents were located on official websites of the Federal Government, analyzed, and interpreted in the dimensions of care while considering the actions in force in the national policy of prenatal care.

The second stage consisted of constructing the indicators; the logical model¹¹ was related to the triad structure, process, and result¹². The logical model was composed of five domains: inputs, activities, outputs, results, and impact, in which the domains inputs and activities represent the structure, the process is represented by the outputs, and the outcomes and impact represent the results^11,12. In order to select the indicators, we considered those presented in the legislation and others developed by authors, being distributed according to the domains of the logic model.

This study was conducted in Paraná State, which is located in southern Brazil and composed of 399 municipalities. To establish the ordering of maternal and child health services, as recommended by the Stork Network implemented in 2012, there is a Line of Care for Maternal and Child Health in the logic of the health care networks, determining assistance to the prenatal care, childbirth, puerperium and newborn¹³.

The third stage consisted of data collection for content validation. Professionals with experience in assistance, management, teaching, or maternal and child health research were included without limiting them to their field of training and with a minimum of two years of professional experience in this line of care. The exclusion criterion was not filling out the content validation questionnaire within 60 days. Twenty health professionals were invited, seven refused to participate, and two were excluded according to the applicable criteria, totaling 11 participants who comprised the expert judges committee.

The expert judges were contacted by e-mail through an invitation letter, which explained the objectives, justification, and validation procedures. After acceptance, the participants received the informed consent form that was answered individually. Data collection for content validation took place online between June and December 2020 through a link to access the data collection form.

The responses generated a database that underwent pre-processing to correct typing inadequacies when filling out the identification fields; two participants were excluded for not completing the instrument. Finally, the amounts of responses were checked to avoid duplicate data or missing information. The database was imported into the IBM SPSS software that supported the data analysis with the pre-processing completed.

The data analysis followed the methodological reference of psychometrics¹⁴ that guides the validation of instruments. Data analysis was carried out in seven steps: defining the content and its explanation considering the care dimensions that involve the theme; organizing the logic model relating the triad and the domains; representativeness of the content addressed by the indicators that composed the logic model; preparing a matrix addressing the specifications of the triad, the domains, and the descriptions of the indicators; preparing definitions regarding the calculation of these indicators presented in the matrix; content validity analysis regarding clarity, relevance, and comprehensiveness considering the opinion of the committee of expert judges; and finally, empirical analysis of the determination of the levels and discrimination of the indicators.

The data collection instrument was structured into two phases¹⁵: the first presented closed questions applying the two-point Thurstone scale (agree and disagree). It made it possible to carry out the fourth stage of the study, which included data analysis for content validation by calculating the agreement rate (AR), which corresponds to the number of participants who agreed with each other multiplied by 100 and divided by the total number of participants. When values were 90% or higher, the content was considered adequate, while in lower values, the content was readjusted according to the suggestions of the committee of expert judges¹⁵.

Next, the content validity ratio (CVR) was calculated considering the minimum value of 0.59 for the indicators, representing the unlikely agreement between the responses that occurred by chance. When the CVR was below 0.59, it was understood that there was a greater chance of 5% agreeing at random, suggesting the exclusion of the indicator¹⁵. Both the AR and CVR were analyzed for the clarity and relevance of the indicators.

The second phase of the data collection instrument used a four-point Likert scale: 1) not at all; 2) a little; 3) quite a lot; 4) a lot. These judgments were relative to the clarity and pertinence of the indicators and provided the calculation of the content validity index (CVI), which summed up “quite a lot” and “a lot” responses that were then divided by the total responses. The value of 0.78 was considered acceptable agreement, 0.80 for minimum agreement, and preferential agreement as of 0.90¹⁵. Next, the reliability analysis of the internal consistency of the indicators was performed in relation to the coverage according to the triad and the domains by calculating Cronbach’s alpha coefficient. It was considered minimum reliability of 0.70, measuring the correlation between the expert judges regarding the agreement in relation to the indicators^15,16. The fifth stage counted on the remodeling of the indicators following the adjustments, inclusion, and exclusion suggested by the committee of expert judges. This study was approved by the Research Ethics Committee of the Federal University of Paraná (Opinion No. 3.912.101).

Results

To monitor the quality of prenatal care, 35 indicators were established (Chart 1).

Chart 1
Agreement rate, content validity ratio, and content validity index of the indicators to monitor the quality of prenatal care. Paraná, 2021.

Indicators	AR of domain clarity	CVR	AR of domain relevance	CVR	CVI of item clarity	CVI of item relevance
B.1 Health professionals	72.70	0.45	72.70	0.45	0.82	0.73
B.2 Health professionals: physicians	81.80	0.64	90.90	0.82	0.91	0.82
B.3 Health professionals: obstetric nurses	90.90	0.82	90.90	0.82	1.00	0.91
B.4 Health professionals: pediatricians	81.80	0.64	100.00	1.00	1.00	0.91
B.5 Health professionals: pediatric or neonatal nurses	90.90	0.82	100.00	1.00	1.00	0.91
B.6 Health establishment: healthcare unit	63.60	0.27	90.90	0.82	0.91	0.91
B.7 Health establishment: mobile emergency care service	72.70	0.45	81.80	0.64	0.82	0.82
B.8 Expenditures	54.50	0.09	81.80	0.64	0.73	0.82
B.9 Expenses	81.80	0.64	81.80	0.64	0.82	0.82
B.10. Investments	81.80	0.64	90.90	0.82	0.82	0.82
B.11 Cost	72.70	0.45	81.80	0.64	0.82	0.82
C.1 Coverage of the health information system for primary care.	100.00	1.00	100.00	1.00	0.91	0.91
C.2 Training in the health information system for primary care.	81.80	0.64	90.90	0.82	0.82	0.91
C.3 Health information system completeness for primary care.	90.90	0.82	90.90	0.82	0.82	0.91
D.1 Pregnant woman with prenatal care	90.90	0.82	90.90	0.82	1.00	1.00
D.2 Early initiation of prenatal care	90.90	0.82	90.90	0.82	1.00	1.00
D.3 Pregnant woman with stratified risk	90.90	0.82	81.80	0.64	1.00	1.00
D.4 Number of prenatal visits	90.90	0.82	81.80	0.64	1.00	1.00
D.5 Number of prenatal + puerperium visits	81.80	0.64	90.90	0.82	1.00	1.00
D.6 Number of prenatal visits + all exams	81.80	0.64	90.90	0.82	1.00	1.00
D.7 Number of prenatal + puerperium visits + all exams	81.80	0.64	90.90	0.82	1.00	1.00
D.8 Number of prenatal + puerperium visits + all exams + diphtheria and tetanus vaccination + HIV test	81.80	0.64	90.90	0.82	1.00	1.00
D.9 Pregnant women immunized with diphtheria and tetanus vaccine.	90.90	0.82	81.80	0.64	1.00	1.00
E.1 Congenital syphilis	81.80	0.64	90.90	0.82	0.82	1.00
E.2 Neonatal tetanus	90.90	0.82	90.90	0.82	1.00	1.00
E.3 Vertical HIV transmission	90.90	0.82	90.90	0.82	1.00	1.00
E.4 Pregnant women accompanied by the supplementary health system (health plans)	81.80	0.64	81.80	0.64	0.73	0.91
E.5 Abortion treatment	72.70	0.45	90.90	0.82	0.82	0.91
E.6 Teenage pregnancy	81.80	0.64	81.80	0.64	0.91	1.00
E.7 Early diagnosis of congenital anomalies	90.90	0.82	90.90	0.82	0.91	1.00
F.1 Maternal mortality ratio	100.00	1.00	100.00	1.00	1.00	1.00
F.2 Early neonatal mortality rate	100.00	1.00	100.00	1.00	1.00	1.00
F.3 Late neonatal mortality rate	100.00	1.00	100.00	1.00	1.00	1.00
F.4 Neonatal mortality rate	100.00	1.00	100.00	1.00	1.00	1.00
F.5 Perinatal mortality rate	90.90	0.82	100.00	1.00	0.91	1.00
F.6 Fetal mortality rate	100.00	1.00	100.00	1.00	0.91	1.00
F.7 Infant mortality rate	100.00	1.00	100.00	1.00	1.00	1.00
F.8 Abortion rate	100.00	1.00	100.00	1.00	0.91	1.00

• Source: the authors. Abbreviations: agreement rate (AR); content validity ratio (CVR); content validity index (CVI); health information system for primary care (HIS); human immunodeficiency virus (HIV); diphtheria and tetanus (DT).

Of these, 11 indicators composed the input domain and 9 (B.1, B.2, B.4, B.6, B.7, B.8, B.9, B.10, and B.11) showed an AR below 90.0%. They were re-evaluated for clarity and relevance and readjusted as listed in Chart 2. The CVR on the indicator’s clarity and relevance was calculated, which expresses whether they are clear and considered essential by the committee of expert judges. When the CVR is more significant than 0.59, it is unlikely that the agreement happened by chance. Item B.1 had a CVR of 0.45 for both clarity and relevance and was excluded (Chart 2).

Chart 2
Adjustments made in the prenatal monitoring indicators according to the suggestions proposed by the expert judge committee. Paraná, 2021.

Indicators	Judges’ suggestions in terms of clarity and relevance of the item to the domain	Adaptations
B.1	- The item label does not refer to its contents for calculation.	Item deleted.
	- Work individually by quantifying gynecologist and obstetric nurse.
	- In primary care, the doctor and general nurse guarantee care, but this item does not contemplate the other levels of care.
	- Most pregnant women are at habitual or intermediate risk and are not seen by specialists.
B.2	- The most accurate information would be to supplement the name of the indicator for gynecologists and obstetricians.	Adjusted the name of the indicator to gynecologist and obstetrician.
	- The indicator does not include the three levels of care.
B.4	- In neonatal units, the physician needs to be a neonatologist.	Adjusted the name and definition of the indicator by including the neonatologist.
	- The indicator does not include the three levels of care.
B.6	- Include the family health strategies in the evaluation.	Adjusted the name to primary health care to include all available types of units and the definition of the indicator by including the word number.
	- It is not clear why the line of care measures the number of primary health care units.
	- Indicator is unclear.
B.7	- Describe better what you need and if you want to find it.	Adjusted the name to health establishment - emergency care network, and the definition of the indicator according to the network premises.
	- Use the definition from the State Health Plan 2020-2023.
B.8	- Text is not clear, it could be replaced by the total effective cost.	Adjusted the indicator’s name to revenues collected for financing the Maternal and child health care line and its definition specifying the disbursement by each sphere of government.
	- Clarify what is referred to as expense acquisition.
	- Specify the spheres of government, considering that the funding is tripartite, as defined in the Constitution and the State Health Plan 2020-2023.
B.9	- Separate funding figures for each sphere of government.	Adjusted definition considering hospital and outpatient care specific to obstetric and neonatal care.
	- The financing of medium and high complexity is a tripartite responsibility. Assistance is funded by the National Health Fund, State Health Fund, and Municipal Health Funds. Use the definition of the State Health Plan 2020-2023.
	- Consider that the transference occurs through contracts to philanthropic entities, consortiums, city halls, and municipal funds.
	- Complex to include in an evaluation process.
B.10	- The financing of medium and high complexity is a tripartite responsibility. Use the definition of the State Health Plan 2020-2023.	Adjusted definition considering investments for the expansion of obstetric and neonatal care.
	- Specify what is invested in obstetric and neonatal care.
B.11	- The evaluation was very broad with this item.	Item deleted after having considered the changes in items B.8 to B.10.
	- The financing of medium and high complexity is a tripartite responsibility. Use the definition of the State Health Plan 2020-2023.
C.2	- Specify the number of professionals who have taken the training or would need to take it.	Adjusted the item’s definition to the number of professionals who received training.
D.5	- Make the variables involved in the calculation clearer.	Adequate the name and definition of the indicator considering only puerperium.
	- Since prenatal care was quantified in the previous item, keep this indicator to puerperal consultations only.
D.6	- Make the variables involved in the calculation clearer.	Adequate the definition, including the types of examinations.
	- It was not clear what type of exams, include the laboratory and/or imaging exams.
D.7	- Make the variables involved in the calculation clearer.	Adequate the definition, including the types of examinations.
	- It was not clear what type of exams, include the laboratory and/or imaging exams.
D.8	- Make the variables involved in the calculation clearer.	Adequate the definition, including the types of examinations.
	- Make it clear they are laboratory and imaging exams.
D.9	- Make the variables involved in the calculation clearer.	Suitable name and definition, including all recommended vaccines.
	- Include immunization with influenza, diphtheria and tetanus, and hepatitis B, which are also vaccines recommended for pregnant women.
E.1	- Separate syphilis diagnosis from syphilis deaths.	Adequate the definition for syphilis diagnosis, considering only Congenital Syphilis.
E.4	- Make the variables involved in the calculation clearer.	Adjusted the definition, changed the calculation base to the numerator and CNES source, and the denominator by live births from the SINASC source.
	- Denominator adjusted for live births, with SINASC source.
E.5	- Relate to the number of pregnant women.	Adequate to the title and definition of the item.
	- Review clarity in the definition regarding the calculation.
E.6	- Make the variables involved in the calculation clearer.	Adequate definition and calculation basis per thousand live births.

• Source: the authors.

Items B.6, B.7, and B.8 presented CVRs suggesting exclusion as to clarity, but we decided to adjust the indicator according to the suggestions of some expert judges. Indicator B.8 had a CVR for clarity of 0.09 and thus underwent an important change in its title and definition. These three items had an adequate CVR of >0.59 when assessing the indicator’s relevance regarding the domain, thereby not being excluded (Chart 2).

Indicator B.11 had a CVR of 0.45 for clarity and 0.64 for relevance. Despite being relevant, it was excluded because the adjustments made to items B.8, B.9, and B.10 (Chart 2) made it redundant, and there were suggestions from some expert judges regarding its exclusion because it would extend the monitoring related to funding.

The CVI calculation was also performed concerning the clarity and relevance of the indicator, and only 2 were identified with a value below 0.78, suggesting it is unacceptable. Indicator B.1 had a CVI for the relevance of 0.73, which justified its exclusion; B. 8 had a CVI for clarity of 0.73, substantiating the need for a major readjustment.

Of the three indicators that assess the activities domain, only indicator C.2 had an AR lower than 90.0%, being readjusted (Chart 2). The CVR was calculated on the item clarity in relation to the domain, with no values lower than 0.59. No CVI was identified for clarity and relevance with values less than 0.78. These two domains, inputs and activity, make up indicators that can measure the structure of the Maternal and Child Health Care Line.

Of the nine indicators that make up the domain outputs and refer to the monitoring of the process, four of them (D.5, D.6, D.7, D.8) presented AR below 90.0% for clarity and one (D.9) for relevance (Chart 1), which were readjusted as suggested by the expert judges (Chart 2). There were no CVR values below 0.59. As for the CVI, for clarity and relevance, all indicators presented preferential agreement.

Of the 7 indicators that make up the results domain, four of them (E.1, E.4, E.5, E.6) showed AR lower than 90.0% for clarity and two of them (E.4, E.6) for relevance, hence being readjusted (Table 2). A CVR below 0.59 was identified only for indicator E.5 for clarity (0.45), and as its CVR for relevance was 0.82 (Chart 1); thus, we decided to adjust the definition of the indicator regarding its form of calculation (Chart 2) to be maintained. As for the CVI for clarity, only item E.4 had a value of 0.73, although its CVI for relevance was high (0.92) (Chart 1), so the indicator underwent significant adjustment and was not deleted (Chart 2).

Table 2
Reliability of prenatal monitoring indicators according to the domains inputs, activities, outputs, outcomes, and impacts. Paraná, 2021.

Scale statistics	Entries B.2 to B.10	Activities C.1 to C.3	Outputs D.1 to D.9	Results E.1 to E.7	Impact F.1 to F.8
Cronbach’s alpha coefficient	0.957	0.863	0.980	0.862	0.957
Average	63.09	21.09	68.36	51.45	61.64
Variance	125.491	10.891	41.255	28.673	26.255
Standard deviation	11.202	3.300	6.423	5.355	5.124

• Source: the authors.

The eight indicators that make up the impact domain presented AR above 90.0%, not requiring adjustments. Therefore, to calculate the CVR, all indicators presented values above 0.59, not indicating the exclusion of indicators. The CVI for clarity and relevance were all higher than 0.78, showing preferential agreement. Both the outcomes and impact domains comprise indicators that assess results.

To estimate the reliability of the internal consistency of the indicators proposed for prenatal monitoring, Cronbach’s alpha coefficient test was applied, which estimates the correlation between the participants’ answers and the average correlation between the questions. The values indicated excellent internal reliability for clarity and relevance of the indicators organized for monitoring the structure (0.94), process (0.98), and outcome (0.94) (Table 1).

Table 1
Reliability of prenatal care monitoring indicators according to the triad structure, process, and outcome. Paraná, 2021.

Scale statistics	Structure	Process	Results
Cronbach’s alpha coefficient	0.936	0.980	0.943
Average	84.18	68.36	113.09
Variance	147.164	41.255	95.291
Standard deviation	12.131	6.423	9.762

• Source: the authors.

The internal consistency reliability was also analyzed for clarity and relevance across domains, where three of them showed excellent reliability: inputs (0.96), outputs (0.98), and impact (0.96). The other two showed good reliability: activities (0.86) and results (0.86), meeting the minimum required internal consistency (Table 2).

Thus, the indicators were readjusted according to the suggestion of the expert judges committee, with no inclusion of new indicators and only the exclusion of two indicators. Chart 3 presents the indicators with the redefinition of their codes according to their title, definition, and calculation basis, developed in the domains of a logic model and aligned to the triad of structure, process, and result of quality in health.

Chart 3
Matrix containing the indicators developed in the triad structure, process, and results according to the domains, name, definition, and calculation basis. Paraná, 2021.

STRUCTURE
Domains	Name	Definition	Numerator	Source	Denominator	Source
B. ENTRY	B.2 Health professionals - gynecologists and obstetricians	Number of active gynecologists and obstetricians in a given region divided by the total population and multiplied by 100.	N	CNES	Inhabitants	IBGE
	B.3 Health professionals - obstetric Nurses	Number of active obstetric nurses in a given region divided by the total population and multiplied by 100.	N	CNES	Inhabitants	IBGE
	B.4 Health professionals - pediatricians and neonatologists	Number of active pediatricians and neonatologists in a given region divided by the total population and multiplied by 100.	N	CNES	Inhabitants	IBGE
	B.5 Health professionals - pediatric and neonatologist nurses	Number of active pediatric nurses or neonatologists in a given region divided by the total population and multiplied by 100.	N	CNES	Inhabitants	IBGE
	B.6 Health care establishment - primary health care	Number of units that provide basic and integral care to a population, in a scheduled or unscheduled manner, in basic specialties, and may offer dental care and other higher-level professionals. The assistance must be permanent and provided by a general practitioner or specialist in these areas. It may or may not offer: auxiliary services of diagnosis and therapy (SADT) and 24-hour emergency care. Divided by the total population of the region and multiplied by 100.	N	CNES	Inhabitants	IBGE
	B.7. Health care establishment - emergency care network	Number of land, air, or waterway vehicles used to receive the user, classify the assistance risk according to the degree of urgency, establish the definitive diagnosis, apply the therapeutic measures of the assistance point, and refer the user for therapeutic continuity. Divided by the total number of inhabitants in the region and multiplied by 100.	N	CNES	Inhabitants	IBGE
	B.8. Revenues collected for financing the maternal and child health care line	Sum of financial resources (in reais) disbursed by the municipalities, state, and Union for the payment of expenses, costs, and investments of the Maternal and Child Health Care Line. Considering for municipalities and states, 15% and 12% of tax collection, and for the Union the value of the previous year plus the variation of the Gross Domestic Product respecting the Annual Budget Law (consider the EC 95 of 2016 that freezes spending for 20 years).	R$	Contractualization	Inhabitants	IBGE
	B.9. Expenses	Financial resource disbursed (in Brazilian Reais) by the Health Funds (municipal, state, and federal) for the maintenance of outpatient and hospital care services (contracts with philanthropic entities, consortiums, and municipalities) aimed at providing obstetric and neonatal care.	R$	Contractualization	Inhabitants	IBGE
	B.10. Investments	Financial resource disbursed (in Reais) by Health Funds (municipal, state, and federal) to expand the provision of outpatient and hospital care services for obstetric and neonatal care. Example: purchase of equipment, expansion of physical structure, acquisition of real estate, etc.	R$	Contractualization	Inhabitants	IBGE
C. ACTIVITIES	C.1. SISAB Coverage	Number of pregnant women registered in the information system for prenatal care, SISAB, by the municipality, divided by the number of live births and multiplied by 100.	N	SISAB	Live Births	SINASC
	C.2 SISAB Training	Number of health professionals who perform prenatal consultations and who received training in the format of training to raise awareness of the importance of using the information system to record standardized information according to the protocol established for the collection of data on the care provided. Divided by the total number of professionals who perform this service and multiplied by 100.	N	Contractualization	Total professionals	CNES
	C.3. SISAB Completeness	Completeness of the variables entered into the information system according to the variables collected that reflect the behaviors performed during prenatal care, divided by the total number of pregnant women and multiplied by 100.	N (filled in)	SISAB	Total pregnant women	SISAB
PROCESS
D. EXIT	D.1 Pregnant women with prenatal care	Proportion of pregnant women who were attended for prenatal care, regardless of the number of consultations, divided by the total number of pregnant women and multiplied by 100.	N	SISAB	Total pregnant women	SISAB
	Early start of prenatal care	Proportion of pregnant women who had their first prenatal visit before the 12th week of gestational age, divided by the total number of pregnant women and multiplied by 100.	N	SISAB	Total pregnant women	SISAB
	D.3. Pregnant woman with stratified risk	Proportion of pregnant women who were assessed during prenatal visits and had their risk-stratified appropriately as usual risk, intermediate risk, or high risk by the Basic Health Unit, divided by the total number of pregnant women and multiplied by 100.	N	SISAB	Total pregnant women	SISAB
	D.4 Number of prenatal consultations	Proportion of pregnant women in relation to the number of prenatal consultations performed during the gestational period, according to the adequacy: 2 consultations in the first trimester: 2 consultations in the second; 3 consultations in the third; and one consultation in the puerperium. Divided by the total number of pregnant women and multiplied by 100.	N	SISAB	Live births	SINASC
	D.5 Number of puerperal consultations	Proportion of women who had a postpartum puerperal visit, divided by the total number of pregnant women and multiplied by 100.	N	SISAB	Total pregnant women	SISAB
	D.6 Number of prenatal appointments + All Tests	Proportion of pregnant women with the appropriate number of prenatal visits and performance of all laboratory and imaging tests, divided by the total number of pregnant women and multiplied by 100.	N	SISAB	Total pregnant women	SISAB
	D.7 Number of Prenatal + Puerperium visits + All Exams	Proportion of pregnant women according to the appropriate number of prenatal and puerperium visits and the performance of all laboratory and imaging exams.	N	SISAB	Total pregnant women	SISAB
	D.8. Number of prenatal + puerperium + all exams + diphtheria and tetanus vaccination + hiv test	Proportion of pregnant women who had the appropriate number of prenatal visits, puerperium visits, had all laboratory and imaging exams, had antitetanic vaccination, and underwent HIV testing. Divided by the total number of pregnant women and multiplied by 100.	N	SISAB	Total pregnant women	SISAB
	D.9 Immunized pregnant women	Proportion of pregnant women who have updated their vaccination schedule with all the recommended vaccines such as anti-tetanus, influenza, and hepatitis B. Divided by the total number of pregnant women and multiplied by 100.	N	SISAB	Total pregnant women	SISAB
RESULTS
E. RESULTS	E.1 Congenital syphilis rate	Number of live births with a diagnosis of congenital syphilis, divided by total live births, multiplied by 100,000 live births.	N	HIS	Live births	SINASC
	E.2 Neonatal tetanus	Proportion of live births with a diagnosis of neonatal tetanus, divided by the total number of live births and multiplied by 100.	N	HIS	Live births	SINASC
	E.3. HIV vertical transmission	Proportion of live births diagnosed with HIV by vertical transmission, divided by the total number of live births and multiplied by 100.	N	HIS	Live births	SINASC
	E.4 Pregnant women followed by the supplementary health system	Proportion of pregnant women who receive prenatal care through the Supplementary Health System (health plan), divided by the total number of live births and multiplied by 1,000 live births.	N	CNES	Live births	SINASC
	E.5 Treatment of early fetal loss	Rate of hospital admissions for treatment of pregnant women in early fetal loss condition, with gestational age less than 22 weeks, divided by the number of live births plus fetal deaths and multiplied by 1,000 live births.	N	HIS	Live births + fetal deaths	SINASC SIM
	E.6 Teenage pregnancy	Proportion of adolescents, aged under 20, who had a confirmed pregnancy diagnosis, multiplied by the total number of live births and multiplied by 1,000 live births.	N	SINASC	Live births	SINASC
	E.7 Early diagnosis of congenital anomalies	Proportion of early diagnosis, during the gestational period, of fetuses with presence of congenital anomalies confirmed by imaging examinations, divided by the total number of live births and multiplied by 100.	N	SISAB	Live births and fetal deaths	SINASC SIM
F. IMPACT	F.1 Maternal mortality ratio	Rate that expresses the maternal deaths related or aggravated by the gravidic-puerperal process, pregnancy, childbirth and up to the 42nd day postpartum. Regardless of the duration of pregnancy or the place where it occurred, per 100,000 live births.	N	YES	Live births	SINASC
	F.2. Early neonatal mortality rate	Rate that expresses the death of live births up to the 6th full day of life, born with a gestational age of 22 weeks or more per 1,000 live births.	N	YES	Live births	SINASC
	F.3 Late neonatal mortality Rate	Rate that expresses the death of live births, between the 7th and 28th day of life, born with a gestational age of 22 weeks or more per 1,000 live births.	N	YES	Live births	SINASC
	F.4. Neonatal mortality rate	Rate that expresses the death of live births until the 28th day of life, born with gestational age from 22 weeks per 1,000 live births.	N	YES	Live births	SINASC
	F.5. Perinatal mortality rate	Rate that expresses fetal death from the 22nd week of gestation, up to the 6th full day of life of births with gestational age 22 weeks and older per 1,000 live births.	N	YES	Live births	SINASC
	F.6. Fetal mortality rate	Rate that expresses fetal deaths from the 22nd week of gestation onwards, occurring intrauterine per 1,000 live births.	N	YES	Live births and fetal deaths	SINASC SIM
	F.7. Infant mortality rate	Rate that expresses the death of children up to 1 year old, or 365 days, per 1,000 live births.	N	YES	Live births	SINASC
	F.8. Abortion rate	Rate that expresses the mortality of fetuses with gestational age less than 22 weeks, or with weight less than 500g, or height less than 25 cm per 1,000 live births.	N	YES	Live births and fetal deaths	SINASC SIM

• Source: the authors. Abbreviations: N: number of cases; R$: total value of revenue; CNES: National Registry of Health Establishments; IBGE: Brazilian Institute of Geography and Statistics; SISAB: Information System for Primary Care; SINASC: Live Births Information System; HIS: Hospital Information System; SIM: Mortality Information System.

Discussion

The monitoring procedures proposed herein seek to contribute to the management of the thematic care network for women’s and children’s health, which is known as the Stork Network. The definition of indicators that can measure prenatal care structure, process, and results will enable the various health actors to develop the planning and monitoring of services and health systems¹⁷. The monitoring process seeks to overcome the weaknesses of quality in health due to the low interaction between the services linked to a thematic network so that it can improve health promotion actions and care practices¹⁸.

The management model applied to a care network involves governance, a new organization with a relationship between the State and its municipal, state, and federal institutions, whether private or non-profit, and the entire civil society. In this way, the articulation among these various services contributes to improving access and integrality, for more resoluteness and humanization to improve the quality¹⁷ of health focused on prenatal care. Thus, the monitoring procedures proposed in this study can be applied at the institutional, municipal, regional, or state levels, contributing to the institutionalization of prenatal quality monitoring.

Therefore, this management model consists of managerial governance organized at the territory level¹⁷, where relationships are established following the laws that guide the thematic networks and their health priorities, with primary care as the care coordinator¹⁸. Prenatal care in Paraná State follows the guidelines proposed by the Stork Network applied to the Guide Line¹³, which contains definitions of actions performed in the Maternal and Child Health Care Line, evidencing its legitimacy and social conformity¹⁹. These monitoring procedures will enable the production of scientific evidence, from the local to the state level, contributing to the increase in quality concerning the efficacy, effectiveness, efficiency, and optimization¹⁹.

Nevertheless, managerial governance aims to develop committees that are conformed to a federal structure to coordinate governmental commitments and priorities, favoring the dialogue among teams, managers, and users¹⁸. In this dialogue, the articulation between these various social actors involved in developing a policy informed by scientific evidence corroborates the advance of quality in health, considering accessibility¹⁹, besides appreciating the wishes and expectations of the population assisted in the professional management practice.

The governance of financial resources (i.e., funding governance) is carried out by defining programs that meet the request of legislation and contractualizations between establishments of several municipalities expanding access¹⁷ in order to overcome the low management and care capacity of small-sized municipalities¹⁸. This articulation strengthens prenatal care actions, with greater access and completeness, to offer the necessary care on time to reduce maternal and infant morbidity and mortality. Thus, it respects the equity component¹⁹ by the fair distribution of care in a democratically free society.

The scientific evidence produced by the monitoring procedures proposed herein has as its main implication the formulation of continuing education actions that will support changes in the care provided. This is an essential tool that cooperates in strengthening the performance of health professionals who work in the assistance and management¹⁸ of the Stork Network. Additionally, it allows the comparison between the results achieved according to the availability of structural and process elements, considering their local context.

A cross-sectional study conducted in Sergipe State evaluated the quality of prenatal care according to the Prenatal and Birth Humanization Program recommendations. It identified that the coverage was 99.3%, and 43.0% started after the 16^th week and 74.7% had six consultations, as recommended by the Ministry of Health, with statistical significance (p < 0.005) regarding schooling have having completed high school, living with a partner, presenting the desire to get pregnant, and being satisfied with the pregnancy. The quality of the exam records in the prenatal card was low, making it difficult to analyze the routine of exams and the guidelines to be performed. About 64.9% of the consultations occurred in primary care, of which nurses performed in 48.0%, and 16.6% were classified as high risk, where only half of the women were referred to the outpatient follow-up of reference²⁰.

This data corroborates the monitoring procedures proposed by this study according to the need for further analysis by applying indicators that monitor the process performed during the prenatal period. Notably, the recommended tests refer to the local protocol of each program in execution, suggesting the need for a HIS that shows these data applicable to an analysis for quality management. However, the analysis of structure and process indicators allows the in-depth monitoring of health outcomes, a fact not related to the cross-sectional study presented. This condition emphasizes the need for complete monitoring that points out the elements of structure, process, and result.

Conclusion

It is concluded that the indicators organized and proposed for monitoring the quality of prenatal care present content validity and reliability established in accordance with public health policies. The format of this construct allows flexibility in the breadth of its application considering the various territorial dimensions (i.e., it can be applied in spaces in which the process of prenatal care occurs), whether they are restricted to health services, health districts, municipalities, regions, and macro-regions of health. Thus, in a movement of increasing expansion and potentiality, its analysis may embrace the state sphere, converging with the recommendations of the Stork Network and reaching the federal sphere.

The application of the procedures for monitoring favors the institutionalization of the evaluation, making it a practice absorbed in the daily activities of the services and health system as a whole. This exercise is powerful to raise the need to expand the indicators and extend its scope to the other dimensions of care, such as childbirth, puerperium, and care to the newborn.

This study presents as a limitation the absence of specialists in the area of monitoring the quality of care, which was compensated by the participation of specialists in public, child, and women’s health. Nonetheless, each participant collaborated from the standpoint of his or her area of knowledge and professional competence. This study addresses a complex activity that can be accomplished with the engagement of the various social actors involved in managing the health services and system. Another limitation arising from the health scenario of the COVID-19 pandemic is the incipient prenatal care guidelines for affected pregnant women, thus justifying the non-inclusion of specific indicators in the monitoring.

• a
  Article extracted from the Doctoral Thesis entitled Logical Model of Health Quality Assessment in the Maternal and Child Care Line: Project for Management under Governance, authored by Michelle Thais Migoto, under the guidance of Dr. Márcia Helena de Souza Freire, developed at Graduate Program in Nursing at the Federal University of Paraná. Year of thesis defense: 2021.
• FINANCIAL SUPPORT Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brazil (CAPES) - Funding Code 001, PhD scholarship awarded to the author: Michelle Thais Migoto.

Referências

• ¹
  World Health Organization. Recommendations on antenatal care for a positive pregnancy experience. Geneva: WHO; 2016.
• ²
  Leal MD, Szwarcwald CL, Almeida PVB, Aquino EML, Barreto ML, Barros F et al. Saúde reprodutiva, materna, neonatal e infantil nos 30 anos do Sistema Único de Saúde (SUS). Cien Saude Colet. 2018;23(6):1915-28. http://dx.doi.org/10.1590/1413-81232018236.03942018 PMid:29972499.
  » http://dx.doi.org/10.1590/1413-81232018236.03942018
• ³
  Portaria n° 569 de 2000, de 1 de junho de 2000 (BR). Diário Oficial da União [periódico na internet], Brasília (DF), 8 jun 2000 [citado 7 jul 2021]. Disponível em: http://bvsms.saude.gov.br/bvs/saudelegis/gm/2000/prt0569_01_06_2000_rep.html
  » http://bvsms.saude.gov.br/bvs/saudelegis/gm/2000/prt0569_01_06_2000_rep.html
• ⁴
  Portaria n° 1.459 de 2011, de 24 de junho de 2011 (BR). Institui, no âmbito do Sistema Único de Saúde - SUS - a Rede Cegonha. Diário Oficial da União [periódico na internet], Brasília (DF), 24 jun 2011 [citado 7 jul 2021]. Disponível em: http://bvsms.saude.gov.br/bvs/saudelegis/gm/2011/prt1459_24_06_2011.html
  » http://bvsms.saude.gov.br/bvs/saudelegis/gm/2011/prt1459_24_06_2011.html
• ⁵
  Viellas EF, Domingues RMSM, Dias MAB, Gama SGN, Theme-Filha MM, Costa JV et al. Prenatal care in Brazil. Cad Saude Publica. 2014;30(Supl. 1):S85-100. http://dx.doi.org/10.1590/0102-311X00126013 PMid:25167194.
  » http://dx.doi.org/10.1590/0102-311X00126013
• ⁶
  Frank BRB, Toso BRGO, Viera CS, Guimarães ATB, Caldeira S. Avaliação da implementação da Rede Mãe Paranaense em três Regionais de Saúde do Paraná. Saúde Debate. 2016;40(109):163-74. http://dx.doi.org/10.1590/0103-1104201610913
  » http://dx.doi.org/10.1590/0103-1104201610913
• ⁷
  Machado AFC, Arcoverde MAM, Caldeira S, Silva-Sobrinho RA, Silva RMM, Zilly A. Prenatal care in the perspective of the Mãe Paranaense network. Rev Norte Mineira de Enferm. 2020;1(2):78-89. http://dx.doi.org/10.46551/rnm23173092202090208
  » http://dx.doi.org/10.46551/rnm23173092202090208
• ⁸
  Ferreira SCM, Silvino ZR, Souza DF. Pesquisa metodológica. In: Zenith RS, editor. Gestão baseada em evidência: recursos inteligentes para a solução de problemas da prática em saúde. Curitiba: Editora CRV; 2018.
• ⁹
  Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0: Standards for Quality Improvement Reporting Excellence: revised publication guidelines from a detailed consensus process. BMJ Qual Saf. 2016;25(12):986-92. http://dx.doi.org/10.1136/bmjqs-2015-004411 PMid:26369893.
  » http://dx.doi.org/10.1136/bmjqs-2015-004411
• ¹⁰
  Markoni MA, Lakatos EM. Fundamentos de metodologia científica. 8. ed. São Paulo: Atlas, 2017.
• ¹¹
  Kellogg-Foundation. Logic model development guide. 1st ed. Michigan: Kellogg; 2004.
• ¹²
  Aggarwal A, Aeran H, Rathee M. Quality management in healthcare: The pivotal desideratum. J Oral Biol Craniofac Res. 2019;9(2):180-2. http://dx.doi.org/10.1016/j.jobcr.2018.06.006 PMid:31211031.
  » http://dx.doi.org/10.1016/j.jobcr.2018.06.006
• ¹³
  Secretaria de Estado da Saúde do Paraná (PR). Plano Estadual de Saúde do Paraná 2016-2019 [Internet]. Curitiba: SESA; 2018 [citado 2021 jul 7]. Disponível em: http://www.saude.pr.gov.br/arquivos/File/PlanoEstadualSaude2016MioloAlt.pdf
  » http://www.saude.pr.gov.br/arquivos/File/PlanoEstadualSaude2016MioloAlt.pdf
• ¹⁴
  Medeiros RKS, Ferreira-Júnior MA, Pinto DPSR, Vitor AF, Santos VEP, Barichello E. Pasquali’s model of content validation in the Nursing research. Rev Enf Ref. 2015;IV série(4):127-35. http://dx.doi.org/10.12707/RIV14009
  » http://dx.doi.org/10.12707/RIV14009
• ¹⁵
  Coluci MZO, Alexandre NMC, Milani D. Construção de instrumentos de medida na área da saúde. Cien Saude Colet. 2015;20(3):925-36. http://dx.doi.org/10.1590/1413-81232015203.04332013
  » http://dx.doi.org/10.1590/1413-81232015203.04332013
• ¹⁶
  Zanon C, Filho-Hauck N. Fidedignidade. In: Hutz CS, Baneira DR, Trentini CM, editores. Psicometria. Porto Alegre: Artmed; 2015.
• ¹⁷
  Ministério da Saúde (BR). Implantação das Redes de Atenção à Saúde e outras estratégias da SAS. Brasília: Ministério da Saúde; 2014.
• ¹⁸
  Peiter CC, Santos JLG, Lanzoni GMM, Mello ALSF, Costa MFBNA, Andrade SR. Healthcare networks: trends of knowledge development in Brazil. Esc Anna Nery. 2019;23(1):e20180214. http://dx.doi.org/10.1590/2177-9465-ean-2018-0214
  » http://dx.doi.org/10.1590/2177-9465-ean-2018-0214
• ¹⁹
  Bond A, Retief F, Cave B, Fundingsland M, Duinker PN, Verheem R et al. A contribution to the conceptualisation of quality in impact assessment. Environ Impact Assess Rev. 2018;68:49-58. http://dx.doi.org/10.1016/j.eiar.2017.10.006
  » http://dx.doi.org/10.1016/j.eiar.2017.10.006
• ²⁰
  Mendes RB, Santos JMJ, Prado DS, Gurgel RQ, Bezerra FD, Gurgel RQ. Evaluation of the quality of prenatal care based on the recommendations Prenatal and Birth Humanization Program. Cien Saude Colet. 2020;25(3):793-804. http://dx.doi.org/10.1590/1413-81232020253.13182018 PMid:32159650.
  » http://dx.doi.org/10.1590/1413-81232020253.13182018

Edited by

ASSOCIATED EDITOR

SCIENTIFIC EDITOR

Publication Dates

• Publication in this collection
  15 Nov 2021
• Date of issue
  2022

History

• Received
  07 July 2021
• Accepted
  13 Sept 2021