Internal Quality Monitoring and Interobserver Agreement of Cytopathological Tests Conducted in the Municipal Laboratory of Manaus, Amazonas, Brazil

ABSTRACT

Introduction:  Internal quality monitoring aims to minimize failures and provide corrections, ensuring laboratory effectiveness using pap smears in the cervical cancer screening program.

Objective:  To evaluate the quality indicators of the municipal laboratory of Manaus-AM and the interobserver agreement rates from 2019 to 2022.

Method:  Descriptive, retrospective, and quantitative analysis of pap smears from the municipal laboratory of Manaus-AM conducted from January 2019 to December 2022. Six quality indicators recommended by the Ministry of Health were applied, as well as the Kappa concordance index to assess laboratory and individual cytologist performance, measuring the consistency between analysts’ and reviewers’ reports using data from the national systems Siscolo, Siscan, and internal laboratory records.

Results:  The laboratory remained within the recommended parameters for most quality indicators, except for an increase in the percentage of atypical squamous cells (ASC) among altered exams, detected in 2019 (61.27%) and 2020 (60.89%), with a progressive annual decrease. The Kappa agreement index for the laboratory was considered "good," ranging from 0.73 to 0.77 over the four years. However, the individual Kappa agreement indicators showed that 2 of the 15 analysts had results classified as "poor" to "moderate," their personal quality indicators differing from those of the other cytologists.

Conclusion:  The study demonstrated the importance of integrating quality indicators with interobserver agreement, suggesting new methodologies for investigating quality in cytopathology laboratories.

Key words:
Quality Control; Public Health Laboratory Services; Papanicolaou Test; Quality Indicators, Health Care

RESUMO

Introdução:  Um monitoramento interno da qualidade tem como finalidade minimizar falhas e fornecer correções, assegurando a eficácia laboratorial no programa de rastreamento do câncer cervical pelo exame de Papanicolau.

Objetivo:  Avaliar os indicadores de qualidade do laboratório municipal de Manaus-AM e os índices de concordância interobservador durante o período de 2019 a 2022.

Método:  Análise descritiva, retrospectiva e quantitativa dos exames de Papanicolau do laboratório municipal de Manaus-AM, de janeiro de 2019 a dezembro de 2022. Foram aplicados seis indicadores de qualidade recomendados pelo Ministério da Saúde, assim como o índice de concordância Kappa para avaliar o desempenho laboratorial e individual dos citologistas, medindo a consistência entre os laudos dos analistas e revisores, utilizando dados dos sistemas nacionais Siscolo, Siscan e registros internos do laboratório.

Resultados:  O laboratório manteve-se dentro dos parâmetros recomendados para a maioria dos indicadores de qualidade, com exceção do aumento do percentual de células escamosas atípicas (ASC) entre exames alterados, detectado em 2019 (61,27%) e 2020 (60,89%), com diminuição anual progressiva. O índice de concordância Kappa laboratorial foi considerado "boa", apresentando-se na faixa de 0,73 a 0,77 durante o quadriênio. Entretanto, os indicadores de concordância Kappa individuais mostraram que dois dos 15 analistas apresentaram resultados classificados como "pobre" a "moderada", e seus indicadores individuais de qualidade divergiram dos valores próximos aos demais citologistas.

Conclusão:  O estudo demonstrou a importância de integrar os indicadores de qualidade com a concordância interobservador, sugerindo novas metodologias para investigação da qualidade em laboratórios de citopatologia.

Palavras-chave:
Controle de Qualidade; Serviços Laboratoriais de Saúde Pública; Teste de Papanicolaou; Indicadores de Qualidade em Assistência à Saúde

RESUMEN

Introducción:  El monitoreo interno de calidad tiene como objetivo minimizar las fallas y proporcionar correcciones, garantizando la eficacia del laboratorio en el programa de tamizaje de Papanicolaou para el cáncer de cuello uterino.

Objetivo:  Evaluar los indicadores de calidad del laboratorio municipal de Manaos-AM y las tasas de concordancia interobservador durante el período de 2019 a 2022.

Método:  Análisis descriptivo, retrospectivo y cuantitativo de las pruebas de Papanicolaou en el laboratorio municipal de Manaos entre enero de 2019 y diciembre de 2022. Se aplicaron seis indicadores de calidad recomendados por el Ministerio de Salud, así como el índice de concordancia Kappa para evaluar el desempeño del laboratorio y del citólogo individual, midiendo la consistencia entre los informes de los analistas y los revisores, utilizando datos de los sistemas nacionales Siscolo y Siscan, y los registros internos del laboratorio.

Resultados:  El laboratorio se mantuvo dentro de los parámetros recomendados para la mayoría de los indicadores de calidad, con excepción del aumento del porcentaje de células escamosas atípicas (ASC) entre los exámenes alterados, detectado en 2019 (61,27%) y 2020 (60,89%), con una disminución anual progresiva. El índice de concordancia Kappa del laboratorio se consideró "bueno", oscilando entre 0,73 y 0,77 durante el cuatrienio. Sin embargo, los indicadores individuales de concordancia Kappa mostraron que 2 de los 15 analistas obtuvieron resultados clasificados como "pobres" o "moderados", y sus indicadores individuales de calidad divergieron de los valores de los demás citólogos.

Conclusión:  El estudio demostró la importancia de integrar los indicadores de calidad con la concordancia entre observadores, sugiriendo nuevas metodologías para investigar la calidad en los laboratorios de citopatología.

Palabras clave:
Control de Calidad; Servicios Laboratoriales de Salud Publica; Prueba de Papanicolaou; Indicadores de Calidad de la Atención de Salud

INTRODUCTION

The Pap smear test is the main preventive tool for detecting cervical cancer precursor lesions. Due to its simplicity, specificity, low cost and being generally painless during collection, it constitutes an effective cytological exfoliative screening to detect uterine cervical lesions. This screening method offers good assistance coverage, promoting early diagnosis before lesions become invasive and may lead to tumor progression¹,².

With manual sampling procedures, false-positive and false-negative results may occur if there is no proper monitoring. Errors may be related to the collection, preparation, and storage of slides (pre-analytical phase), subjective interpretation for diagnosis (analytical phase) or recording and quality control of results in the post-analytical phase³,⁴.

One of the challenges for the effectiveness of preventive tests is related to sensitivity oscillations. These oscillations are associated with variables in the routine of professionals, including intersectoral turnover, the need for training and updates, as well as interventions and technical failures, which may compromise laboratory performance⁵.

The Ministry of Health, by Ordinance GM/MS No. 3,388⁶, of December 30, 2013, conducted the quality control actions redefining the National Qualification in Cytopathology, and established criteria for the quality parameters and performance evaluation of the service provider laboratories through the execution of internal quality monitoring (IQM) and external quality monitoring (EQM) in order to ensure the quality of the results obtained, and statistically identify the possible causes⁷.

In the post-analytical phase, considering the IQM, it is possible to perform quality control based on effective calculations of the results issued with the use of statistical indicators, comparing them with values recommended by the Ministry of Health to evaluate the laboratory performance and indicate the need for professional training if the values diverge from the recommended limits⁸.

A publication focused exclusively on the Amazonas State scenario was carried out by Nobre and Lopes Neto⁹, with indicators between 2001 and 2005, highlighting improvements related to the annual decrease in the results of cellular atypia and dissatisfaction. In addition, there are few studies in Brazil focused on the performance analysis of cytologists, because the vast majority are focused on the results disclosed and recorded by the laboratory.

It is essential that there be more research and updated discussions on this topic in Amazonas, to disseminate the current scenario of the Region with the highest national incidence rates of cervical cancer, with 31.71 cases per 100 thousand women¹⁰.

Given the above, this study aimed to perform the IQM of the cytopathological tests released by the municipal laboratory of Manaus-AM in the 2019-2022 quadrennium, applying six quality indicators established by the Ministry of Health for the data contained in the Cervical Cancer Information System (Siscolo)¹¹ and the Cancer Information System (Siscan)¹², as well as to analyze the individual performance of laboratory cytologists and target corrective measures in the face of the problems encountered.

METHOD

Considering that this work is an observational, descriptive, retrospective, and quantitative study, its design, methodology, presentation of results and discussion were conducted based on the STROBE checklists¹³ (supplementary material). The results of preventive tests performed between January 2019 to December 2022 by the Specialties Laboratory "Prof. Sebastian Ferreira Marinho" of the Municipal Health Department (SEMSA) in Manaus-AM were evaluated. The analysis was based on information from the Siscolo¹¹ system database (for the period 2019 to 2020), with the last update collected on August 25, 2022, and Siscan¹² (2021 to 2022), grouped annually. A total of 362,461 results were obtained in the databases, being used for IQM analysis.

For the performance analysis of cytologists, 19,063 valid data from internal records of slides with changes selected for review between 2019 and 2021 were consulted and computed, disregarding slides that did not have a registered clinical outcome or no completion in the internal laboratory record. Figure 1 presents the flowchart of the steps followed in the construction of this study's methodology.

Figure 1
Flowchart of the steps of the study

The Ministry of Health, through the National Cancer Institute (INCA), developed the Manual of Quality Management for Cytopathology Laboratories (QualiCito)¹⁴ in 2012, with a most recent edition in 2016 and subsequent annual updates. Six indicators for IQM established in this manual were used in this study, based on its limits and recommendations:

Positivity Index (PI) – expresses the prevalence of cellular alterations in the tests and characterizes the sensitivity of the screening process for the detection of atypia or cervical cancer precursor lesions. The formula is presented as: (number of altered tests at a given location and year x 100)/total satisfactory tests.

PI results are classified as: Very low (below 2%); low (between 2% and 2.9%); expected (between 3% and 10%); above expected (above 10%, considering that such providers may serve secondary reference services in cervical pathology). Based on QualiCito¹⁴, a PI with a value equal to 3% or more is recommended.

Percentage of tests compatible with atypical squamous cells (ASC) among satisfactory tests – ASC is a terminology applied when there are diagnostic doubts to fit in a characteristic intraepithelial lesion, encompassing two distinct groups: The possibly non-neoplastic (ASC-US) and those that do not discard high-grade intraepithelial lesion (ASC-H). According to QualiCito¹⁴, these values in relation to satisfactory values cannot exceed the 4-5% mark. The formula corresponds to the (number of tests with ASC-US and ASC-H x 100)/total of satisfactory tests.

Percentage of ASC among the modified tests – This evaluation is performed along with the PI value, because a high value is associated with a high percentage of tests released as ASC, suggestive of many false-negatives or false-positives, following the formula: (number of tests with ASC-US and ASC-H x 100)/total altered tests. Based on QualiCito¹⁴, the recommended percentage of ASC among altered tests should be between 37.3% and 57.2%, not exceeding 60%.

Ratio between ASC cells/squamous intraepithelial lesion (SIL) – This value contributes to the identification of technical difficulties for the diagnosis of low SIL (LSIL) and high grade (HSIL), avoiding excessive use of ASC terminology resulting in uncertain diagnoses. QualiCito¹⁴ recommends a value not exceeding 3 using the formula: number of tests with ASC-US and ASC-H/number of tests with LSIL and HSIL.

Percentage of HSIL-compatible tests – The result should be greater than or equal to 0.4%¹⁴ for successful screening and decrease in incidence and mortality rates from cervical cancer from the precursor findings: (number of HSIL tests x 100)/Total satisfactory tests¹⁴.

Percentage of unsatisfactory – a parameter that includes estimating the number of unsatisfactory samples per month and facilitating the identification of failures in the internal quality control of the service provider unit, following the calculation: (number of unsatisfactory tests in month x 100)/total tests performed in the month. Based on QualiCito¹⁴, the results should be less than 5% every month.

The individual performance of the cytologists was assessed from the productivity index of the laboratory professionals, comparing the results issued by the first analyst (object of study) with the reports by the slide reviewers (considered standard). Among the 21 cytologists working in the laboratory, 15 were selected by the triennial yield cut-off point (n>2,800 slides) for having higher productivity in number of slides read per year.

The Kappa¹⁵ agreement index was applied, which measures the agreement between ASC and SIL reports issued and individual quality indicators, adapted from the IQM methodology for laboratories. In addition, a performance analysis of cytologists was included, based on the same IQM model of the laboratory, but focused on the individual production numbers, in which the released quantitative and reports were computed as results for the calculations.

Kappa statistics for the degree of agreement were performed from an adaptation of the spreadsheet provided by INCA¹⁵. In the interpretation of the Kappa coefficient of the Landis and Kock model of 1977, agreement intervals were considered for the interobserver measure, in which a value equal to zero indicates absence of agreement; values above zero to 0.39 indicate "poor" agreement; values between 0.40 and 0.59 suggest "moderate" agreement; values between 0.60 and 0.79 suggest "good" agreement; values between 0.80 and 0.99 correspond to an "excellent" (or substantial) agreement; and a maximum value of 1 suggests a "perfect" agreement (or complete agreement).

For data analysis, the Tabnet^®16 (DATASUS) software and Microsoft Excel ^® were used to quantify the results and for statistical assembly.

This research project was approved by SEMSA, authorization letter no. 40/2020, and by the Research Ethics Committee of the Universidade Federal do Amazonas (UFAM), report number 4.261.386 (CAAE (submission for ethical review): 35582620.4.0000.5020), according to Resolution of the National Health Council (CNS) no. 466/2012¹⁷.

RESULTS

From the data provided by Siscolo¹¹ (2019 to 2020) and Siscan¹² (2021 to 2022) it was possible to quantify the results of the tests performed during the quadrennium. Table 1 corresponds to the reports issued by the service provider unit in the official systems of the Ministry of Health.

Table 1
Cervical cytopathological tests performed by the municipal laboratory Prof. Sebastian Marinho (Manaus-AM) in the 2019-2022 triennium as found in the information systems

Year	Database	Tests Satisfactory		Tests Unsatisfactory	NM tests and benign findings	Tests with alterations							Total tests with alterations	Total tests
						Squamous				Glandular		Carcinomas
						LSIL	HSIL	HSIL Micro	ASC	AGC-NOS	AGC-NEO
2019	Siscolo	n	95,960	1,541	82,656	1,092	643	53	3,022	94	20	8	4,932	97,501
		%	98.42	1.58	84.77	1.12	0.66	0.05	3.10	0.10	0.02	0.01	5.06	100.00
2020	Siscolo	n	62,372	879	53,939	751	452	30	2,024	62	11	4	3,324	63,251
		%	98.61	1.39	85.28	1.19	0.71	0.05	3.20	0.10	0.02	0.01	5.26	100.00
2021	Siscan	n	87,585	1,573	83,942	1,011	585	35	2,543	126	17	13	4,330	89,190
		%	98.20	1.76	93.50	1.13	0.65	0.04	2.85	0.14	0.02	0.01	4.85	100.00
2022	Siscan	n	110,129	2,226	106,526	1,154	632	46	2,147	132	24	14	4,144	112,519
		%	97.88	1.98	94.67	1.03	0.56	0.04	1.91	0.12	0.02	0.01	3.68	100.00

• Source: Elaborated by the authors based on DATASUS data: Siscolo¹¹ – 2019, 2020; Siscan¹² – 2021, 2022.
• Captions: NM: negative for malignancy; LSIL: low-grade squamous intraepithelial lesion; HSIL: high-grade intraepithelial lesion; HSIL micro: high-grade intraepithelial lesion cannot exclude microinvasion; ASC: atypical squamous cells of undetermined significance; AGC-NOS: atypical glandular cells, not otherwise specified (possibly non-neoplastic); AGC-NEO: atypical glandular cells of undetermined significance, possibly neoplastic (cannot discard high-grade lesion); carcinomas: include findings of invasive squamous cell carcinoma, adenocarcinoma in situ, invasive adenocarcinoma and other neoplasms.

According to the annual report of the Department of Cancer Prevention and Control (DCPC) of the Fundação Centro de Controle de Oncologia do Estado do Amazonas – FCECON¹⁸, the Manaus SEMSA only implemented Siscan¹² as an information system from 2020 onwards. However, most of the information from 2019 and 2020 was not complete or in accordance with the official figures disclosed, still present in the previous system (Siscolo/Sismama¹¹), generating discrepancies in the annual reporting systems. Therefore, the two systems were consulted to achieve the objective of the study.

Within the ASC values, Table 1 shows individual results of ASC-US: 1,939 in 2019 (1.99%), 1,273 (2.01%) in 2020, 1,357 in 2021 (1.52%), and 1,168 in 2022 (1.04%). For ASC-H, the values were: 1,083 (1.11%) in 2019, 751 (1.19%) in 2020, 1,186 (1.33%) in 2021, and 979 (0.87%) results in 2022. In 2020, the proportion of issued ASC was higher compared to the other years of the study, with 3.20% (2,024 tests).

Table 2 shows the five IQM indicators. The recommended values of each applied indicator were included as a comparative basis for assessing the calculated statistical results. Values above 60% for the percentage of ASC among the altered tests were observed in 2019 (61.27%) and 2020 (60.89%).

Table 2
Assessment of the quality indicators of the service provider laboratory from 2019 to 2022

No.	IQM parameters	RV	2019	2020	2021	2022
			Siscolo	Siscolo	Siscan	Siscan
1)	PI (%)	3 to 10	5.14	5.33	4.94	3.76
2)	ASC among satisfactory tests (%)	< 4	3.15	3.25	2.90	1.95
3)	ASC among altered tests (%)	< 60	61.27	60.89	58.73	51.81
4)	ASC/SIL ratio	< 3	1.69	1.64	1.56	1.17
5)	Tests compatible with HSIL(%)	≥ 0.4	0.73	0.77	0.71	0.62
6)	Annual mean of unsatisfactory tests (%)	< 5	1.77	1.47	1.70	1.97

• Source: Elaborated by the authors based on DATASUS data: Siscolo¹¹ - 2019, 2020; Siscan¹² - 2021, 2022; RV based on the Manual of Quality Management for Cytopathology Laboratories (QualiCito)¹².
• Captions: ASC: Atypical squamous cells of undetermined significance; PI: Positivity index; RV: Recommended quality indicator values; SIL: Squamous intraepithelial lesion; HSIL: High-grade squamous intraepithelial lesion.

The percentage of unsatisfactory data were adapted representing an annual average, because no monthly value during the quadrennium analyzed exceeds the limit of 5%, according to the recommendation by QualiCito¹⁴. Monthly data obtained higher values in 2019, ranging from 2% to 2.78% from October to December; in 2020 from 1.67 to 2.78% from March to May; in 2021, presenting values of 2.04 in July and 2.16 in August; and in 2022 2.10% to 2.70% from March to June.

Table 3 presents data regarding the analysis of the 15 cytologists to compare the agreement in the reports issued between the first analyst and reviewers, by applying the Kappa¹⁵ index to determine the agreement, the agreement of the issued ASC and SIL reports and the individual quality indicators of the cytologists.

Table 3
Performance assessment of cytologists by productivity, agreement, and quality indicators during the 2019-2021 triennium

Analyst identification	Shift	Mean triennium productivity	Agreement with reviewers					Mean quality indicators in the triennium per cytologist
			Kappa Agreement			Mean triennium results(%)
			2019	2020	2021	ASC	SIL	PI (%)	ASC/Satisf.	ASC/ALT	ASC/SIL	HSIL%
Analyst 1	A	3337	0.81	0.74	0.85	66.75 *	84.29	3.92	1.35%	34.38%	0.90	0.87%
Analyst 2	M	4853	0.81	0.86	0.67	78.33	92.04	10.51 **	5.83% **	55.51%	2.40	0.94%
Analyst 3	M/A	4899	0.73	0.78	0.68	70.57	82.81	14.17 **	7.70% **	54.16%	3.09 **	1.22%
Analyst 4	A	4685	0.75	0.64	0.77	62.96 *	86.23	4.37	2.18%	53.00%	3.77 **	0.21% *
Analyst 5	A	4278	0.62	0.64	0.73	56.24 *	75.86 *	8.15	3.51%	42.58%	3.37 **	0.33% *
Analyst 6	M	3926	0.37 *	0.41 *	0.48 *	35.24 *	73.57 *	4.77	3.18%	66.91% **	7.73**	0.22% *
Analyst 7	M/A	4479	0.9	0.91	0.90	84.22	93.66	14.36 **	4.75%	33.13%	1.42	1.56%
Analyst 8	A	4615	0.78	0.81	0.80	74.36	90.76	9.46	4.23%	45.08%	1.90	0.91%
Analyst 9	A	2811	0.5 *	0.62	0.51 *	51.23*	73.28 *	14.98 **	8.22% **	54.82%	3.12 **	0.85%
Analyst 10	M	2913	0.7	0.77	0.66	73.08	82.76	6.11	3.27%	52.82%	1.66	0.56%
Analyst 11	M	4474	0.73	0.68	0.72	65.36 *	79.34 *	5.74	2.31%	40.24%	1.53	0.42%
Analyst 12	A	3426	0.72	0.8	0.66	66.74 *	87.97	6.37	3.00%	46.62%	1.30	0.84%
Analyst 13	A	2984	0.67	0.8	0.76	63.59 *	79.63 *	7.71	2.08%	27.22%	0.92	0.76%
Analyst 14	M	4830	0.78	0.84	0.62	71.94	85.11	10.26 **	5.16% **	50.54%	2.27	0.55%
Analyst 15	M	4038	0.85	0.89	0.90	82.48	91.34	6.45	3.28%	50.58%	1.72	0.63%
Laboratory mean			0.74	0.77	0.73	69.51%	85.89%	8.15%	3.94%	48.14%	2.31	0.71%
Recommended Values		κ > 0.6 Good 0.4 < κ >0.6 Moderate						3 - 10	< 4%	< 60%	< 3	≥ 0.4%

• Source: Internal Laboratory Registration Data.
• Captions: ASC: Atypical squamous cells of undetermined significance; PI: Positivity index; Shift: Morning (M) and Afternoon (A); RV: quality indicator recommended values; SIL: Squamous intraepithelial lesion; HSIL: High-grade squamous intraepithelial lesion.
• ^*
  Values below the laboratory mean/recommended values established;
• ^**
  values above the laboratory mean/recommended values established.

DISCUSSION

The application of IQM in cervical cytology is a decisive step to ensure technical rigor in laboratory performance. The assessment of quality indicators facilitates the identification of failures in care coverage, aiming to correct them through strategic actions of revalidation of the steps and in the continuing education of professionals involved in the pre-analytical and analytical phases¹⁹.

According to the data in Table 1, the period of 2020 shows a decrease in the number of tests performed, resulting from the COVID-19 pandemic, impacting the estimated opportunistic screening of 30 thousand women when compared to the other years. However, the percentage of altered tests this year was the highest (5.26% compared to the total) among the assessed years, as well as the one that presented the lowest value regarding the number of unsatisfactory tests (1.39%).

Due to the lower opportunistic demand of the public in 2020, a greater proportional number of alterations was identified in relation to the total of tests, about 5.26%. From the findings, the hypotheses are similar to those of Ribeiro et al.²⁰, corroborating the hypothesis that a higher proportion of symptomatic women, in follow-up or requiring repetition of tests due to alterations, sought to perform the preventive test compared to those who sought healthcare at the right time.

The results of this study showed that, in general, the laboratory presented indicators compatible with the recommendations of the Ministry of Health. The PI, HSIL percentage and unsatisfactory parameter remained within the recommended values during the assessed period. However, values above the recommended limit for ASC were observed among altered tests in some years, with a progressive recovery in subsequent years.

The laboratory PI, presented in Table 2, shows values within the recommended range, between 3% and 10%, during the whole quadrennium. In a study conducted in the State of Bahia, during the 2015-2019 period, PI was below 3% (between 1.85% and 2.36%), not reaching the range recommended by the Ministry of Health²¹. In a municipality of Paraná²², the PI of 2012 and 2013 was far below the recommended (1.92% and 0.46%), but from 2014 to 2018 it stayed in the range of 4.77% to 6.44%.

Low PI would indicate an excess of false-negative results, in which truly positive samples would not be detected in screening, which would require a better assessment and intensification of the analytical and pre-analytical phases ⁸.

As for the HSIL percentage (Table 2), all the values of the analyzed years were as expected, above 0.4%. SIL results are the main objective of screening, representing lesions that can effectively progress to cervical cancer, with HSIL (or classified as CIN II and III) being the true precursors; its detection is paramount for the prevention of cervical cancer. Results below the recommended may mean false-negative, indicating diagnostic failures in the referral of patients to adequate treatment²³,²⁴.

In a study conducted in Anápolis, Goiás, it was indicated that the percentage of HSIL by the annual rate was lower than that recommended by the Ministry of Health, with 0.13% between 2013 and 2014²⁵. Tobias et al.²⁶ reported that, out of a total of 20 laboratories in the Minas Gerais Region, only three presented this rate within the recommended value in 2012.

Turkiewicz et al.²⁷ verified PI rates and HSIL percentage below the recommended between 2014 and 2018 in Foz do Iguaçu, Paraná. The authors reported an increase in rates from 2019 to 2021, mentioning the possibility that this may come from a higher proportion of symptomatic women, in follow-up or requiring repeated tests due to altered findings, who continued to seek the healthcare service more than those with negative results.

The main highlight in the results shown in Table 2 was the value for the "percentage of ASC among altered tests", which was 61.27% in 2019 and 60.89% in 2020, based on Siscolo data, indicating a quantitatively higher proportion of ASC-US results. The Ministry of Health recommends an acceptable range of 37.3% to 57.2%, not exceeding 60% when the PI is within the recommended use of this terminology in the total number of reports issued in relation to the total number of tests with alterations¹⁴.

ASCs are considered cytological ambiguities due to the interpretative difficulty of the cytologist, representing a group of dubious diagnostic findings, which do not have cellular characteristics consistent with reactive SIL processes. These atypical alterations may be related to the presence of repair, inflammation, or technical problems in the preparation of the sample, making the findings insufficient for a definitive diagnosis²⁸,²⁹.

Laboratories with a value of "ASC among altered tests" higher than recommended need to re-assess the collection steps, the quality of sample reception and the training of cytologists. This is because atypia are the main terminologies (after unsatisfactory results) responsible for the need to repeat tests, due to the increase of false positives and false negatives in findings with alterations²⁴.

These factors reflect the greater possibility of follow-up loss in screening, contributing to higher rates of positive results in more advanced degrees of lesions. These referral errors lead to misguided conduct and decision-making, generating higher expenses with unnecessary repetitions of tests and consequences for the health of users³⁰,³¹.

Although the years 2019 and 2020 presented values, as previously mentioned, above the recommended, values were reduced and adjusted in 2021 and 2022, with 58.73% and 51.81%, respectively. In studies conducted in other locations, the laboratories analyzed presented failures in this parameter. In Caruaru, Pernambuco, 2019 values were identified above the recommended values for "ASC among satisfactory tests" (6.5%) and "ASC among alterations" (68.9%). In the year 2013, in Goiás, a range of 7% was observed in the values of "ASC among satisfactory results" in the PUC-GO Clinical Laboratory³²,³³.

As for the ASC/SIL ratio values, these were below the recommended maximum value. This analysis is an important control tool since the diagnosis of ASC has less differential diagnostic significance than that of SIL. Laboratories with high values need performance improvements and reassessment of the classification criteria³⁴.

When analyzing Table 2 on the annual indicator of unsatisfactory status, each month presented values within the recommended maximum limit (up to 5%). There were higher percentages of unsatisfactory results from October to December 2019 and a growing increase between April and June in 2020 and 2022. A higher demand for tests collected during the periods of annual campaigns, such as "Outubro Rosa" (Pink October) and "Março lilás" (Lilac March), intensifies the number of samples to be collected and analyzed up to 30 days after collection, thus producing a greater number of unsatisfactory results.

An unsatisfactory result is a slide reading in which a report is not determined due to problems occurring in the pre-analytical (collection, preparation, production, recording) and analytical (insufficient number of cells for analysis, presence of blood or other contaminants, inadequate fixation or artifacts by dissecting) phases, which impair the interpretation by the analyst, requiring repetition of sample collection³⁵. According to Mori et al.³⁶, pre-analytical errors correspond to 20%-39%, and the collection stage is considered one of the main compromising factors for obtaining satisfactory samples for reading.

The interobservational analysis was applied to investigate internal failures related to the analytical process of the laboratory. The laboratory presented a Kappa agreement level classified as "good," being in the range between 0.73 and 0.77 with an average agreement level (Po) of 82.68% and an average agreement of terminologies in ASC of 69.51% and SIL of 85.89%.

Studies show a variability in Kappa interobserver agreement in gynecological cytology; however, the present study presented satisfactory results. A study conducted at the Universidade Federal do Paraná from 2007 to 2015 obtained a reviewer Kappa agreement of 0.46, with the largest agreement range for ASC-H and HSIL corresponding to 68.6%³⁷.

The study by Ázara et al.³⁸ conducted in 14 laboratories in Goiânia between 2007 and 2008 assessed 10,053 cases, with an average Kappa agreement result of 0.81. Siddegowda et al.³⁹ analyzed the variability of the results, presenting a final Kappa value of 0.61, in which 200 slides were prepared and read by a first and a second analyst.

In Table 3 results, the lower performance of two of the 15 main analysts of the laboratory stood out. Analysts 6 and 9 presented the lowest Kappa concordance values, classified as poor to moderate, and whose individual values of quality parameters differed from the values close to the other cytologists.

The ASC and SIL agreement between analysts and reviewers was a valuable tool to highlight discrepancies in the results issued by the laboratory. Table 3 data indicate that eight of the 15 cytologists showed moderate agreement with the reviewers in their ASC reports, with results below 70%. For SIL, five of the cytologists showed levels below 80%, including analysts 5, 6 and 9.

Despite the limitations of the Kappa method for the present study, a complementary methodology using statistical calculations of IQM, according to Table 3, allowed to elucidate the technical failures of each cytologist based on their individual indicators mean.

In the analysis of Table 3, cytologists 2, 3, 9 and 14 presented PI above 10% and ASC/satisfactory values above 4%, indicating that these professionals may be producing reports with excess positive findings for abnormalities, exceeding the limit of the ASC nomenclature, and presenting a value of ASC/altered above 50%, near the maximum limit. These analysts have a clearly "good" to "optimal" agreement, which indicates the need for a more in-depth analysis to better understand their influence on the increase of laboratory PI.

When analyzing the Kappa index, it is necessary to consider its limitations and variables, since its simple calculation has a dichotomous nature, and is not suitable for cases with multiple diagnostic criteria⁴⁰. In addition, the absence of standardized methods, such as the use of double-blind in the analytical process, can introduce bias, especially by allowing cytologists to review previously assessed slides. Factors such as lack of homogeneity in laboratory activities, turnover between sectors, reading time per slide and behavioral bias (a significant subjective element) can also impact the results.

This study presents limitations that should be considered when interpreting the results. First, the reduction in the number of tests performed during 2020, due to the COVID-19 pandemic, may have influenced the findings, generating a greater proportion of alterations due to the differentiated profile of patients who sought cytology in this period.

Finally, the data used was extracted from laboratory records and may be subject to errors or inconsistencies in the information collection and recording. Despite this, the tools and methodologies used in the study provided a broad view on the quality and performance indicators of the laboratory, contributing to the identification of failures and potential improvement of processes.

Given the results of this study, it is essential that laboratories maintain technical rigor, seeking to overcome individual failures by professionals, conducting continuous training, and observing aspects in addition to those demonstrated in this work. Implementing an improvement training focused on the differentiation between SIL to reduce the emission of doubtful results (ASC) is recommended.

CONCLUSION

Given the results of this study, it is essential that laboratories maintain technical rigor, seeking to overcome individual failures by professionals through continuous training and compliance with aspects in addition to those demonstrated in this work.

The interobserver analysis revealed good general agreement, but also pointed out specific discrepancies among some cytologists, evidencing the need for actions directed at training and standardization to ensure the quality of the emitted results. In this sense, implementing specific training aimed at improving the work of cytologists, with emphasis on the differentiation between SIL to reduce the emission of doubtful results, such as ASC, is recommended.

• FUNDING SOURCES
  Carlos Henrique Lamego Guimarães Thomaz Branco received a grant from UFAM through the Institutional Scientific Initiation Scholarship Program (PIBIC) (PIBIC-S/0017/2021), funded by the National Council for Scientific and Technological Development (CNPq). Financial aid from the funding agency Fundação de Amparo à Pesquisa do Estado do Amazonas (FAPEAM).

ACKNOWLEDGMENTS

To the Escola de Saúde Pública de Manaus, for the consent granted, and to the employees of the Specialties Laboratory "Professor Sebastian Ferreira Marinho" who contributed directly or indirectly to the execution of this work.

References

• ¹ Santos JC. A importância da prevenção do câncer de colo do útero: uma breve revisão [monografia]. [Araçuaí]: Universidade Federal de Minas Gerais; 2010.
• ² Ribeiro Filho MA, Quirino GS, Lima MAG, et al. Estratégias utilizadas para a prevenção do câncer de colo uterino na atenção primária em saúde: revisão da literatura. Pesq Ens Ciênc Exatas Nat. 2021;5(1):e1643. doi: https://doi.org/10.29215/pecen.v5i0.1643
  » https://doi.org/10.29215/pecen.v5i0.1643
• ³ Costa MCO, Melo CMS, Lima ES, et al. Fatores que provocam resultados falso-negativos nos exames de citologia oncótica: uma revisão integrativa. RSD. 2021;10(10):e361101019079. doi: https://doi.org/10.33448/rsd-v10i10.19079
  » https://doi.org/10.33448/rsd-v10i10.19079
• ⁴ Gullo CE, Dami ALT, Barbosa AP, et al. Results of a control quality strategy in cervical cytology. Einstein (São Paulo). 2012;10(1):86-91. doi: https://doi.org/10.1590/s1679-45082012000100018
  » https://doi.org/10.1590/s1679-45082012000100018
• ⁵ Barros ALS, Lima DNO, Azevedo MD, et al., organizadores. Caderno de referência 1: citopatologia ginecológica [Internet]. Rio de Janeiro: Ministério da Saúde; 2012. [acesso 2023 fev 4]. v. 1. Disponível em: https://bvsms.saude.gov.br/bvs/publicacoes/tecnico_citopatologia_caderno_referencia_1.pdf
  » https://bvsms.saude.gov.br/bvs/publicacoes/tecnico_citopatologia_caderno_referencia_1.pdf
• ⁶ Ministério da Saúde (BR). Portaria n° 3.388, de 30 de dezembro de 2013. Redefine a Qualificação Nacional em Citopatologia na prevenção do câncer do colo do útero (QualiCito) [Internet]. Diário Oficial da União, Brasília, DF. 31 dez 2013 [acesso 2023 fev 4]; Seção 1:42-5. Disponível em: https://bvsms.saude.gov.br/bvs/saudelegis/gm/2013/prt3388_30_12_2013.html
  » https://bvsms.saude.gov.br/bvs/saudelegis/gm/2013/prt3388_30_12_2013.html
• ⁷ Araujo Junior MLC, Santana DA, Almeida LB, et al. Quality in cytopathology: an analysis of the internal quality monitoring indicators of the Instituto Nacional de Câncer. J Bras Patol Med Lab. 2015;51(2):102-7. doi: https://www.doi.org/10.5935/1676-2444.20150018
  » https://www.doi.org/10.5935/1676-2444.20150018
• ⁸ Bortolon PC, Silva MAF, Corrêa FM, et al. Avaliação da qualidade dos laboratórios de citopatologia do colo do útero no brasil. Rev Bras Cancerol. 2012;58(3):435-44. doi: https://www.doi.org/10.32635/2176-9745.rbc.2012v58n3.600
  » https://www.doi.org/10.32635/2176-9745.rbc.2012v58n3.600
• ⁹ Nobre JCAA, Lopes Neto D. Avaliação de indicadores de rastreamento do câncer do colo do útero no Amazonas, norte do Brasil, de 2001 a 2005. Rev Bras Cancerol. 2009;55(3): 1609. doi: https://www.doi.org/10.32635/2176-9745.rbc.2009v55n3.1609
  » https://www.doi.org/10.32635/2176-9745.rbc.2009v55n3.1609
• ¹⁰ Instituto Nacional de Câncer José Alencar Gomes da Silva [Internet]. Rio de Janeiro: INCA; 2022. Estatísticas de câncer, 2022 jun 23, atualizado em 2023 jul 18. [acesso 2023 fev 3]. Disponível em: https://www.gov.br/inca/pt-br/assuntos/cancer/numeros/
  » https://www.gov.br/inca/pt-br/assuntos/cancer/numeros/
• ¹¹ SISCOLO: Sistema de Informação do Câncer do Colo do Útero [Internet] Brasília (DF): DATASUS. [data desconhecida] – [acesso 2024 ago 25]. Disponível em: https://datasus.saude.gov.br/acesso-a-informacao/cancer-de-colo-de-utero-e-de-mama-siscolo-sismama/
  » https://datasus.saude.gov.br/acesso-a-informacao/cancer-de-colo-de-utero-e-de-mama-siscolo-sismama/
• ¹² SISCAN: Sistema de Informação do Câncer [Internet] Brasília (DF): DATASUS. [data desconhecida] – [acesso 2024 ago 25]. Disponível em: https://datasus.saude.gov.br/acesso-a-informacao/sistema-de-informacao-do-cancer-siscan-colo-do-utero-e-mama/
  » https://datasus.saude.gov.br/acesso-a-informacao/sistema-de-informacao-do-cancer-siscan-colo-do-utero-e-mama/
• ¹³ von Elm E, Altman DG, Egger M, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. 2008;61(4):344-9. doi: https://www.doi.org/10.1016/j.jclinepi.2007.11.008
  » https://www.doi.org/10.1016/j.jclinepi.2007.11.008
• ¹⁴ Instituto Nacional de Câncer José Alencar Gomes da Silva. Manual de gestão da qualidade para laboratório de citopatologia [Internet]. 2. ed. rev. ampl. Rio de Janeiro: Inca; 2016. [acesso 2023 fev 2]. 160 p. Disponível em: https://www.inca.gov.br/publicacoes/manuais/manual-de-gestao-da-qualidade-para-laboratorio-de-citopatologia
  » https://www.inca.gov.br/publicacoes/manuais/manual-de-gestao-da-qualidade-para-laboratorio-de-citopatologia
• ¹⁵ Instituto Nacional de Câncer José Alencar Gomes da Silva. Planilha Kappa. In: Instituto Nacional de Câncer José Alencar Gomes da Silva. Manual de gestão da qualidade para laboratório de citopatologia [Internet]. 2. ed rev ampl. Rio de Janeiro: Inca; 2016. [acesso 2023 fev 2]. 160 p. Disponível em: https://www.inca.gov.br/publicacoes/manuais/manual-de-gestao-da-qualidade-para-laboratorio-de-citopatologia
  » https://www.inca.gov.br/publicacoes/manuais/manual-de-gestao-da-qualidade-para-laboratorio-de-citopatologia
• ¹⁶ TABNET [Internet]. Brasília (DF): DATASUS. c2008 – [acesso 2024 ago 28]. Disponível em: https://datasus.saude.gov.br/informacoes-de-saude-tabnet/
  » https://datasus.saude.gov.br/informacoes-de-saude-tabnet/
• ¹⁷ Conselho Nacional de Saúde (BR). Resolução n° 466, de 12 de dezembro de 2012. Aprova as diretrizes e normas regulamentadoras de pesquisas envolvendo seres humanos. Diário Oficial da União, Brasília, DF. 2013 jun 13; Seção I:59
• ¹⁸ Fundação Centro de Controle de Oncologia do Amazonas. Relatório ano de 2020 [Internet]. Amazonas: DPCC/FCECON; 2020. [acesso 2023 fev 2]. 78 p. Disponível em: http://www.fcecon.am.gov.br/wp-content/uploads/2021/06/Relatorio-Anual-DPCC-2020.-FINAL.-28.06.21.-PDF.pdf
  » http://www.fcecon.am.gov.br/wp-content/uploads/2021/06/Relatorio-Anual-DPCC-2020.-FINAL.-28.06.21.-PDF.pdf
• ¹⁹ Silva Santos MJ, Alves Ribeiro A. Estratégias utilizadas para melhorar a qualidade dos exames citopatológicos. Rev Bras Cancerol. 2020;66(1):e-05104. doi: https://www.doi.org/10.32635/2176-9745.rbc.2020v66n1.104
  » https://www.doi.org/10.32635/2176-9745.rbc.2020v66n1.104
• ²⁰ Ribeiro CM, Miranda Correa F, Migowski A. Short-term effects of the COVID-19 pandemic on cancer screening, diagnosis and treatment procedures in Brazil: A descriptive study, 2019-2020. Epidemiol Serv Saúde. 2022;31(1):e2021405. doi: https://www.doi.org/10.1590/S1679-49742022000100010
  » https://www.doi.org/10.1590/S1679-49742022000100010
• ²¹ Santos BL, Costa ÉSBO, Moreira GD, et al. Análise de citopatologias cérvico-vaginal realizadas na Bahia entre 2015-2019: indicadores técnicos. RSD. 2021;10(12):e404101220609. doi: https://www.doi.org/10.33448/rsd-v10i12.20609
  » https://www.doi.org/10.33448/rsd-v10i12.20609
• ²² Magalhães JC, Morais LSF, Plewka J, et al. Avaliação dos indicadores de qualidade dos exames citopatológicos do colo do útero realizados em um município do Paraná, Brasil. J Bras Patol Med Lab. 2020;56. doi: https://www.doi.org/10.5935/1676-2444.20200041
  » https://www.doi.org/10.5935/1676-2444.20200041
• ²³ Alrajjal A, Pansare V, Choudhury MSR, et al. Squamous intraepithelial lesions (SIL: LSIL, HSIL, ASCUS, ASC-H, LSIL-H) of uterine cervix and bethesda system. CytoJournal. 2021;18:16. doi: https://www.doi.org/10.25259/CYTOJOURNAL_24_2021
  » https://www.doi.org/10.25259/CYTOJOURNAL_24_2021
• ²⁴ Cardoso Filho LI, Tavares SBN, Siqueira MLB, et al. Internal quality control indicators in cervical cytopathology of a university laboratory. Cytopathology. 2018;29(4):355-60. doi: https://www.doi.org/10.1111/cyt.12571
  » https://www.doi.org/10.1111/cyt.12571
• ²⁵ Lemos ARM, Silva MS, Segati KD. Lesão de alto grau e carcinoma escamoso: um estudo de prevalências em pacientes atendidas pelo Sistema Único de Saúde (SUS) na cidade de Anápolis, GO, Brasil. Rev Bras Anál Clín. 2017;49(2):1. doi: https://www.doi.org/10.21877/2448-3877.201600494
  » https://www.doi.org/10.21877/2448-3877.201600494
• ²⁶ Tobias A, Amaral R, Diniz E, et al. Quality indicators of cervical cytopathology tests in the public service in Minas Gerais, Brazil. Rev Bras Ginecol Obstet. 2016;38(2):65-70. doi: https://www.doi.org/10.1055/s-0035-1571175
  » https://www.doi.org/10.1055/s-0035-1571175
• ²⁷ Turkiewicz M, Plewka J, Santos MA, et al., et al. Os impactos da qualidade nos exames citopatológicos do colo do útero, numa cidade de tríplice fronteira, na pandemia de COVID-19. RSD. 2022;11(6):e52411629428. doi: https://www.doi.org/10.33448/rsd-v11i6.29428
  » https://www.doi.org/10.33448/rsd-v11i6.29428
• ²⁸ Lima CS, Barroso SPC, Boettger LEM, et al. Controle de qualidade da fase pós-analítica do setor de citologia de um laboratório privado de Macapá, Amapá, Brasil. ABMN [Internet]. 2018 [acesso 2023 fev 2];79(1):5. Disponível em: https://portaldeperiodicos.marinha.mil.br/index.php/abmn/issue/view/53
  » https://portaldeperiodicos.marinha.mil.br/index.php/abmn/issue/view/53
• ²⁹ Machado EP, Alves MBM, Irie MMT, et al. Controle interno de qualidade em citopatologia: o dilema da subjetividade. Rev Bras Anál Clín [Internet]. 2018;50(3). Disponível em: doi: https://www.doi.org/10.21877/2448-3877.201800662
  » https://www.doi.org/10.21877/2448-3877.201800662
• ³⁰ Araújo ES, Barbosa FM, Ázara CZS, et al. Avaliação do seguimento de mulheres com exames citopatológicos alterados de acordo com as condutas preconizadas pelo ministério da saúde do Brasil em Goiânia, Goiás. Rev Bras Cancerol. 2014;60(1):7-13. doi: https://www.doi.org/10.32635/2176-9745.RBC.2014v60n1.486
  » https://www.doi.org/10.32635/2176-9745.RBC.2014v60n1.486
• ³¹ Cavalcante GHO, Reis GJ. Avaliação do seguimento de lesões precursoras de câncer do colo do útero – uma revisão bibliográfica. Pesq Ens Ciênc Exatas Nat. 2021;5:e1741. doi: https://doi.org/10.29215/pecen.v5i0.1741
  » https://doi.org/10.29215/pecen.v5i0.1741
• ³² Santos MA, Etlinger-Colonelli D. Análise dos indicadores de qualidade dos exames de citologia cervical do município de Caruaru/PE: uma estratégia para incentivar o monitoramento da qualidade. Recis. 2022;3(3):25-31. doi: https://www.doi.org/10.51909/recis.v3i3.237
  » https://www.doi.org/10.51909/recis.v3i3.237
• ³³ Paula AC, Souza NG, Prado TC, et al. Indicadores do monitoramento interno da qualidade dos exames citopatológicos do Laboratório Clínico da Pontifícia Universidade Católica de Goiás (PUC-GO). Rev Bras Análises Clínic. 2017;49(2):1. doi: https://www.doi.org/10.21877/2448-3877.201700534
  » https://www.doi.org/10.21877/2448-3877.201700534
• ³⁴ Tavares LCC, Coelho KMPA, França PHC, et al. Avaliação da utilidade da razão ASC/SIL como indicador de qualidade em citopatologia. Rev Bras Cancerol. 2015;61(1):31-6. doi: https://www.doi.org/10.32635/2176-9745.rbc.2015v61n1.758
  » https://www.doi.org/10.32635/2176-9745.rbc.2015v61n1.758
• ³⁵ Galvão EFB, Silva MJM, Esteves FAM, et al. Frequência de amostras insatisfatórias dos exames preventivos do câncer de colo uterino na rede pública de saúde, em município do agreste Pernambucano. Rev para med. 2015;29(2):51-6.
• ³⁶ Mori MA, Ribeiro CNM. Falhas no diagnóstico do câncer de colo uterino. Rev Eletr Biociênc Biotecnol Saúde [Internet]. 2015 [acesso 2023 fev 2];2(11):37-50. Disponível em: https://interin.utp.br/index.php/GR1/article/view/1827/1561
  » https://interin.utp.br/index.php/GR1/article/view/1827/1561
• ³⁷ Pergo LS, Maffini CF, Zanine RM, et al. Concordância interobservador no diagnóstico citológico de atipia escamosa de significado indeterminado favorecendo lesão de alto grau e de lesão intraepitelial escamosa de alto grau nas lesões do colo uterino. Rev Bras Cancerol. 2022;68(1):e-041338. doi: https://www.doi.org/10.32635/2176-9745.rbc.2022v68n1.1338
  » https://www.doi.org/10.32635/2176-9745.rbc.2022v68n1.1338
• ³⁸ Ázara CZS, Manrique EJC, Souza NLA, et al. External quality control of cervical cytopathology: interlaboratory variability. Acta Cytol. 2013;57(6):585-90. doi: https://www.doi.org/10.1159/000353843
  » https://www.doi.org/10.1159/000353843
• ³⁹ Siddegowda RB, DivyaRani M, Natarajan M, et al. Inter-Observer variation in reporting of pap smears. Natl J Lab Med. 2016;5(3):22-5. doi: https://www.doi.org/10.7860/NJLM/2016/20002:2135
  » https://www.doi.org/10.7860/NJLM/2016/20002:2135
• ⁴⁰ Silva AF, Velo MMAC, Pereira AC. Importância da reprodutibilidade dos métodos para diagnóstico em odontologia. RFO-UPF. 2016;21(1):115-20. doi: https://www.doi.org/10.5335/rfo.v21i1.4433
  » https://www.doi.org/10.5335/rfo.v21i1.4433

Edited by

• Scientific-editor:
  Anke Bergmann. Orcid iD: https://orcid.org/0000-0002-1972-8777

Data availability

Publication Dates

• Publication in this collection
  23 May 2025
• Date of issue
  2025

History

• Received
  16 Oct 2024
• Accepted
  08 Jan 2025