Evaluation of the clinical protocol quality for family planning services of people living with HIV/AIDS^*

ABSTRACT

Objective:

To evaluate the quality of a clinical protocol for family planning care for people living with HIV/AIDS.

Method:

An evaluative study based on the six domains of the Appraisal of Guidelines for Research & Evaluation II and on Pearson's Coefficient of Variation.

Results:

The protocol reached between 88.8% and 100.0% quality in the domains of the Appraisal of Guidelines for Research & Evaluation II and 93.3% in the overall evaluation. The obtained Pearson's coefficient of variation was between zero and 18.6. Considering that a minimum percentage of 70.0% was adopted for the quality attributed by the evaluators, quality has been achieved for all domains of the Appraisal of Guidelines for Research & Evaluation II. As a coefficient for all domains was less than 25%, we can infer that the scores attributed by the evaluators were linear or homogeneous, meaning high agreement between them.

Conclusion:

The protocol was evaluated as a quality instrument, recommended for use by health professionals who deal with family planning for people living with HIV/AIDS.

DESCRIPTORS
Protocols; Acquired Immunodeficiency Syndrome; Family Planning; HIV Seropositivity; Public Health Nursing

RESUMO

Objetivo:

Avaliar a qualidade de protocolo clínico para atendimento em planejamento familiar de pessoas vivendo com HIV/aids.

Método:

Pesquisa avaliativa realizada com base nos seis domínios do Appraisal of Guideline for Research & Evaluation II e no Coeficiente de Variação de Pearson.

Resultados:

O protocolo alcançou entre 88,8% e 100,0% de qualidade nos domínios do Appraisal of Guideline for Research & Evaluation II e 93,3% na avaliação global. Obteve-se coeficiente de variação de Pearson entre zero e 18,6. Uma vez que se adotou percentual mínimo de 70,0% à qualidade atribuída pelos avaliadores, conferiu-se qualidade em todos os domínios do Appraisal of Guideline for Research & Evaluation II. Com o coeficiente em todos os domínios inferior a 25%, infere-se que as pontuações atribuídas pelos avaliadores foram lineares ou homogêneas, significando elevada concordância entre eles.

Conclusão:

O protocolo foi avaliado como instrumento de qualidade, recomendado para uso por profissionais de saúde que lidam com o planejamento familiar de pessoas vivendo com HIV/aids.

DESCRITORES
Protocolos; Síndrome de Imunodeficiência Adquirida; Planejamento Familiar; Soropositividade para HIV; Enfermagem em Saúde Pública

RESUMEN

Objetivo:

Evaluar la calidad de protocolo clínico para atención en planificación familiar de personas viviendo con VIH/SIDA.

Método:

Investigación evaluativa llevada a cabo con base en los seis dominios del Appraisal of Guideline for Research & Evaluation II y en el Coeficiente de Variación de Pearson.

Resultados:

El protocolo alcanzó entre el 88,8% y el 100,0% de calidad en los dominios del Appraisal of Guideline for Research & Evaluation II y el 93,3% en la evaluación global. Se logró el coeficiente de variación de Pearson entre cero y 18,6. Toda vez que se adoptó porcentual mínimo del 70,0% a la calidad atribuida por los evaluadores, se verificó calidad en todos los dominios del Appraisal of Guideline for Research & Evaluation II. Con el coeficiente inferior al 25% en todos los dominios, se infiere que los puntajes atribuidos por los evaluadores fueron lineales y homogéneos, significando elevada concordancia entre ellos.

Conclusión:

El protocolo fue evaluado como instrumento de calidad, recomendado para empleo por profesionales sanitarios que manejan la planificación familiar de personas viviendo con VIH/SIDA.

DESCRIPTORES
Protocolos; Síndrome de Inmunodeficiencia Adquirida; Planificación Familiar; Seropositividad para VIH; Enfermería en Salud Pública

INTRODUCTION

Protocols are recommendations that must be developed in an organized way based on the best scientific information, thereby aiding in the clinical management of a problem or health condition. Thus, they become tools to be used in the health area, reducing inappropriate variations in clinical practice⁽¹⁾.

As they are instruments aimed at the health of their users, clinical protocols can present characteristics aimed at health prevention, promotion and education. Their use contributes to: qualification of provided care; organization and optimization of care; time saving, avoiding task repetition; rationalization of care, procedures, conducts and materials; adaptation of scientific evidence to local realities and user satisfaction⁽¹⁾.

Despite the relevance of implementing protocols in clinical practice and its positive impact on the improvement of the outcomes among patients, the development of these instruments continues to lack clearly articulated objectives, coherent structures, reliable evaluation and implementation mechanisms, adding good quality and the best cost-effectiveness to health care⁽²⁾.

In this sense, a structured review of the scientific literature was carried out in order to identify the prospect of effective instruments that guarantee the quality of clinical protocols, listing several instruments that enable evaluating the quality of a clinical protocol, where the most reliable, tested and recommended were the Appraisal of Guidelines for Research and Evaluation (AGREE-II), a checklist drawn up by the World Health Organization (WHO); and another one elaborated by the Agency for Healthcare Research and Quality (AHRQ), the National Guideline Clearinghouse (NGC)⁽³⁾.

In view of the above, we chose to adopt the AGREE-II as an instrument for assessing the quality of a clinical protocol for offering family planning care to people living with HIV/AIDS (PAPHA) due to its ease of access and practical application⁽³⁾. Moreover, this instrument presents the advantage of the possibility of obtaining a checklist, thereby facilitating the process of planning, elaboration and implementation of guidelines and allowing its use by different health professionals⁽⁴⁾.

The PAPHA has been identified as a pressing need for health professionals who deal with this area of care, since the reproductive and/or contraceptive choice of this target population requires specific care based on up-to-date knowledge. Authors recommend that this service should be specialized, safe, standardized and effective, so that conception and contraception can be approached in a current and scientific manner, thus guaranteeing the least (possible) risk to the woman, the partner and their future child⁽⁵⁾.

A study conducted in Lusaka, Zambia, for example, reaffirms the discontinuation of contraceptive methods for reasons that go beyond professional guidance, being directly related to signs and symptoms such as severe menstrual bleeding and dysuria⁽⁶⁾. In addition, a study addressing the health strategy adopted in the United States to assist in the family planning of women living with HIV/AIDS highlighted the importance of providing quality services based on consistent scientific evidence⁽⁷⁾.

Thus, the need to use an effective clinical protocol in professional practice that addresses the family planning of people with HIV/AIDS is recurrent, and requires the availability of a concise evaluation instrument that guarantees its quality. According to the literature, the AGREE-II⁽⁸⁾ is an evaluation system that groups these characteristics, which justifies the development of the present study.

Thus, the following research question was outlined: Is the protocol presented herein satisfactory for use in clinical practice, offering up-to-date and objective information? To answer this question, the present study was developed with the objective of evaluating the quality of a clinical protocol for family planning care of people living with HIV/AIDS (PAPHA).

METHOD

An evaluative study carried out from January to March 2016 based on the six domains proposed by the Appraisal of Guidelines for Research & Evaluation II (AGREE-II): scope and purpose, stakeholder involvement, rigour of development, clarity of presentation, applicability and editorial independence. This set of domains are standardized into 23 evaluation items⁽⁸⁾. Questions regarding the characterization of the evaluators were added to the AGREE-II “Quality Assessment Instrument”.

The AGREE-II instrument has been developed in order to address quality, methodological rigour and the transparency with which the protocol is developed. This instrument is comprehensive, allowing its application to protocols directed toward any diseases and health situations, including aspects related to both health promotion and collective health, as well as to diagnoses, treatments or interventions⁽⁸⁾. It was possible to establish a methodological strategy for the evaluation of the protocol's quality with the use of this instrument.

The study was carried out with the purpose of describing the cross-cultural adaptation process of AGREE-II and it emphasizes the importance of using this instrument, which allows a structured and rigorous methodology for protocol development, resulting in a more effective tool when inserted into clinical practice⁽⁴⁾. Five evaluators (E1, E2, E3, E4 and E5) participated who were intentionally chosen by the “convenience or network” search tool. Evaluators from the two care areas of family planning and HIV/AIDS were selected; and an evaluator of materials who were related to teaching, research and service. These evaluators had to have a master's and/or doctorate degree, in addition to direct performance and clinical experience in these knowledge areas.

A scale of scores varying from one to seven was applied for each evaluated item, in which a score of one (Strongly Disagree) is applied when there is no relevant information, or if the concept is poorly reported; and a score of seven (Strongly Agree) when the quality of the information is excellent, meaning that all proposed criteria and considerations were promptly met. Scores from two to six are assigned when the information item does not totally meet all criteria or considerations. The score should be attributed according to the quality of the report and of the complete exposition of information, meaning that it should be directly proportional to the addressed criteria and expositions, increasing as they are contemplated⁽⁸⁾.

The judges’ evaluation was calculated “individually according to the sum of all the item scores of each domain, scaling the total with a percentage of the maximum possible score per domain”⁽⁸⁾. This is possible by using the formula:

$$\frac{\text{Score achived}–\text{Minimum score}\mathrm{x}100}{\text{Minimum score}-\text{Minimum score}}$$

The obtained score is given by the sum of the total values of each item from the five evaluators; the minimum score is given by 1 (Strongly Disagree) x amount of items evaluated in the domain x 5 (number of evaluators); the maximum score is given by 5 (Strongly Agree) x amount of items evaluated in the domain x 5 (number of evaluators)⁽⁴⁾.

Although the scores obtained in the domains allow for a more reliable comparison between guidelines and offer elements for a protocol to be recommended or not, AGREE-II does not define minimum domain scores in order to differentiate between high and low quality protocols. This decision must be made by the author and guided by the context regarding the protocol use⁽⁸⁾. Thus, the authors established a domain that reached a score equal to or greater than 70% as being as “satisfactory quality”; a percentage that generally means acceptable minimum performance in the evaluations.

A space was provided for the evaluators in each domain to describe their recommendations, which were critically analyzed by the researchers to make the necessary adaptations in order to increase the quality of the PAPHA.

In addition to this evaluation parameter proposed by AGREE-II, the Pearson's Coefficient of Variation (PCV) was calculated, which provides the variation of the obtained data in relation to the mean. The lower its value, the more homogenous are the opinions, and consequently the scores attributed by the evaluators. The coefficient is considered low when it is less than or equal to 25%, pointing to a set of homogeneous data. PCV calculations are given by the standard deviation divided by the mean of the scores.

In the case of a domain with a result below 70% and/or PCV greater than 25%, this domain would have the modifications sent to the evaluators as many times as necessary until the established parameters were reached, according to the Delphi technique. This technique is intended for the deduction and refinement of opinions of a group of people specialized in a certain subject, seeking to reach consensus on this judgment, thus contributing to the quality and excellence of the evaluated material⁽⁹⁾.

The evaluators received the free and informed consent form (ICF) and the “Quality Assessment Instrument” via e-mail. A 15-day deadline was given for the evaluators to return the evaluation result along with the signed ICF. The study met the requirements of the Guidelines and Norms of Research on Human Beings presented in Resolution 466/12 of the National Health Council, on the ethical questions of research involving human beings. The research project was submitted to the Plataforma Brasil – Ethics Committee of the Universidade Federal do Ceará – UFC, obtaining a favorable opinion according to protocol number 1.086.282.

RESULTS

The evaluators were all nurses with training time ranging from 6 to 31 years ($\overline{\chi }=12.8)$. Four developed or had developed activities in the area of family planning (E1, E2, E3 and E5), two integrated care to people living with HIV/AIDS (E3 and E4), and two had experience with protocol elaboration and evaluations (E1 and E5).

The evaluation results were presented according to the six domains proposed by the AGREE-II, grouped two by two (Tables 1 to 3 and Chart 1). Next, the overall evaluation of the PAPHA and the Pearson's Coefficient of Variation (PCV) are presented (Tabela 4).

Table 1
Evaluation of the clinical protocol according to scope and purpose (domain 1) and stakeholder involvement (domain 2) (AGREE-II) – Fortaleza, CE, Brazil, 2016.

*Domain 1 − Scope and purpose	Evaluators’ Score
	E1	E2	E3	E4	E5	Total
1. The overall objective(s) of the guideline is (are) specifically described.	6	7	7	7	7	34
2. The health question(s) covered by the guideline is (are) specifically described.	7	6	7	7	7	34
3. The population (patients, public, etc.) to whom the guideline is meant to apply is specifically described.	7	7	7	7	6	34
Total	20	20	21	21	20	102
*Domain 2 − Stakeholder involvement
4. The guideline development group includes individuals from all the relevant professional groups.	7	6	7	6	4	30
5. The views and preferences of the target population (patients, public, etc.) have been sought.	7	7	7	7	4	32
6. The target users of the guideline are clearly defined.	7	7	6	7	6	33
Total	21	20	20	20	14	95

• ^*
  Domain items that make up the study⁽8⁾, cited by the authors.

Table 2
Evaluation of the clinical protocol according to the rigour of development (domain 3) and clarity of the presentation (domain 4) (AGREE-II) – Fortaleza, CE, Brazil, 2016.

*Domain 3 – Rigour of development	Evaluators’ Score
	E1	E2	E3	E4	E5	Total
7. Systematic methods were used to search for evidence.	7	7	7	7	7	35
8. The criteria for selecting the evidence are clearly described.	7	7	7	7	7	35
9. The strengths and limitations of the body of evidence are clearly described.	7	7	7	7	7	35
10. The methods for formulating the recommendations are clearly described.	7	7	6	7	3	30
11. The health benefits, side effects, and risks have been considered in formulating the recommendations.	6	5	7	7	7	32
12. There is an explicit link between the recommendations and the supporting evidence.	6	7	7	7	7	34
13. The guideline has been externally reviewed by experts prior to its publication.	7	7	7	7	7	35
14. A procedure for updating the guideline is provided.	7	7	7	7	7	35
Total	54	54	55	56	52	271
*Domain 4 – Clarity of the presentation
15. The recommendations are specific and unambiguous.	6	6	7	7	7	33
16. The different options for management of the condition or health issue are clearly presented.	6	7	7	7	7	34
17. Key recommendations are easily identifiable.	7	6	7	7	7
Total	19	19	21	21	21

• ^*
  Domain items that make up the study⁽8⁾, cited by the authors.

Table 3
Evaluation of the clinical protocol according to applicability (domain 5) and editorial independence (domain 6) (AGREE-II) – Fortaleza, CE, Brazil, 2016.

*Domain 5 – Applicability	Evaluators’ Score
	E1	E2	E3	E4	E5	Total
18. The guideline describes facilitators and barriers to its application.	7	6	6	7	7	33
19. The guideline provides advice and/or tools on how the recommendations can be put into practice.	7	7	7	7	7	35
20. The potential resource implications of applying the recommendations have been considered.	2	7	7	7	6	29
21. The guideline presents monitoring and/or auditing criteria.	2	7	7	7	7	30
Total	18	27	27	28	27	127
*Domain 6 – Editorial independence
22. The views of the funding body have not influenced the content of the guideline.	7	7	7	7	7	35
23. Competing interests of guideline development group members have been recorded and addressed.	7	7	7	7	7	35
Total	14	14	14	14	14	70

• ^*
  Domain items that make up the study⁽8⁾, cited by the authors.

Chart 1
Evaluators’ suggestions in accordance with the AGREE-II domains – Fortaleza, CE, Brazil, 2016.

Domain	Evaluators’ suggestions
Domain 1	Changes in the second objective of the protocol (E1) and defining of the age group to be attended (E5). E2 did not present any recommendations despite assigning a score below 7 on an item.
Domain 2	E2 and E5 suggested adding the highest qualifications of the professionals participating in its development to the PAPHA. E3 brought up the need to specify to which healthcare professionals the protocol is intended. E4 did not present any recommendations despite assigning a score below 7 on an item.
Domain 3	E5 inquired to what content each participant contributed in the development of the protocol and how the PAPHA contributions were inserted. E1, E2 and E3 did not present any recommendations despite assigning a score below 7 on an item.
Domain 4	E2 suggested the inclusion of a text addressing vaccination of seropositive pregnant women. E1 did not present any recommendations despite assigning a score below 7 on an item.
Domain 5	E5 suggested adding a list of the health services that provide care for people living with HIV/AIDS. E2 and E3 did not present any recommendations despite assigning a score below 7 on an item.
Domain 6	No suggestions.

Table 4
Values for the standard deviation (S), mean ($\overline{\chi }$) and Pearson's Coefficient of Variation (PCV) for each domain of the AGREE-II – Fortaleza, CE, Brazil, 2016.

Domain	S	$\overline{\chi }$	PCV (%)
Scope and purpose	0.54	20.4	2.6
Stakeholder involvement	3.2	17.6	18.6
Rigour of development	1.4	54.2	2.7
Clarity of the presentation	1.09	20.2	5.4
Applicability	4.1	25.4	16.3
Editorial independence	0.0	14	0.0

Domain 1 was mostly evaluated at the maximum levels (6 and 7). However, evaluator E1 recommended a change in the second objective of the protocol, while evaluator E5 suggested defining the age range of the target population to receive care based on the protocol (Chart 1). After a critical reading of these recommendations, it was decided to keep the description of the objective in its original format, considering that it was adequate and that it encompassed the purpose of the PAPHA, which is to serve people with HIV/AIDS in the area of family planning, focusing on the reproductive phase.

Evaluators E2 and E5 suggested adding their higher titles to domain 2, in addition to the training areas of the professionals who participated in the protocol development. The recommendation was accepted, and the respective information was included since it was considered to be an aspect which would generate greater credibility to the protocol users.

Evaluator E3 brought up the need to specify to which healthcare professionals the protocol was intended for. However, it is explicit in the protocol that it is intended for nurses and other health professionals who deal with people living with HIV/AIDS and family planning.

Therefore, domain 1 achieved a 96.6% quality percentage, and domain 2 scored 88.8%.

Regarding domain 3, evaluator E5 questioned the work dynamics of the professionals who participated in developing the protocol (which content each of them contributed and how the organization and the insertion of that material occurred). After a critical analysis of this observation, it was understood that the evaluator referred to the process of surveying topics and the organization of a preliminary “summary”, meaning which topics the protocol would address. In this regard, it was concluded that such information is complementary and is not required by the AGREE-II to compose PAPHA.

In domain 4, evaluator E2 suggested adding information on the vaccines used by seropositive pregnant women during prenatal care, and evaluator E5 suggested adding a list of health services that provide care for people living with HIV/AIDS, in the municipality where the protocol was developed.

Thus, domain 3 scores reached a quality percentage of 96.2%, and domain 4 reached 95.5%.

Thus, a 95.5% quality percentage was obtained for domain 5, and domain 6 achieved the maximum score on all addressed issues, resulting in 100%.

Chart 1 presents the recommendations proposed by the evaluators according to each domain. We emphasize that even though some evaluators did not assign a maximum score to a certain item that composed the domain, they did not suggest modifications to the protocol.

Regarding the overall PAPHA evaluation, three evaluators attributed a maximum score (7), and two scored it 6, generating an overall PAPHA quality assessment of 93.3%.

Regarding the recommendation for using the protocol, four reviewers answered “I would recommend it without restrictions”, while one answered “I would recommend it with some restrictions”, emphasizing that the respective restrictions were solved, such as the importance of mentioning their higher degrees/titles in addition to the training areas of the professionals who participated in developing the protocol.

Table 4 specifies the values of standard deviation, mean and PCV for each domain.

The PCV found in all domains was below 25%, which means high homogeneity in the scores assigned to the domains. This finding suggests similar points of view among the evaluators regarding the protocol.

The PCV showed a greater degree of dispersion among the evaluation percentages for domains 2 (18.6%) and 5 (16.3%) due to the greater variation of the scores attributed by the evaluators. Domain 2 obtained scores ranging from 4 to 7, and domain 5 received scores ranging from 2 to 7. Domains 6, 1 and 3 presented the lowest PCV values of zero, 2.6% and 2.7%, respectively. Domain 4 had a PCV equal to 5.4%.

DISCUSSION

In domain 1 (scope and purpose), item 1 infers the possible impact of the guideline on the health of the population or individuals involved, it analyzes whether the general objectives of the guideline are explicit, and whether the health benefits acquired from the guideline address the clinical problem or the exposed health situation. Item 2 assesses whether the key recommendations described are in accordance with the addressed health situations. Item 3 observes whether the description of the population covered by the guideline is clear. The contents of the items include: purpose; expected benefit or outcome; target population; interventions; comparisons where appropriate; health care environment or context⁽⁸⁾.

Although it was suggested by evaluator E5, it was not possible to define the age range of the reproductive phase from 12 to 49 years, since it seemed inappropriate in the sense of incurring an exclusion of people due the variation in the ages at the beginning and the end of the reproductive phase. An outpatient study of Infectious Diseases in Atlanta, United States, for example, reaffirmed the desire of seropositive women to become pregnant; however, they did not determine the period when they wished to get pregnant. Thus, restricting the age for protocol application could lead to the future exclusion of these women, who despite being at an advanced age for reproduction, could express the desire to conceive and should have their rights fulfilled⁽¹⁰⁾.

In domain 2 (stakeholder involvement), item 4 refers to professionals who participated in developing the protocol, excluding external reviewers (evaluators). For each member of the technical development group, the following should be presented: name, discipline or content of the expertise, institution, geographical location and their role description. In item 5, it is understood that the guideline should be based on the experiences of the target population and their expectations of health care, meaning that there must be evidence in order to consider the views of the interested parties. The strategy used to capture the opinions and preferences of the population should be reported. Item 6 asserts that the target users must be expressed in the guideline. In addition, it should encompass how the guideline should be used by them, thus, it is possible for the reader to determine its relevance⁽⁸⁾.

Although evaluator E3 suggested specifying for which professionals the protocol is intended, it is understood that this clinical protocol was a product of research in Nursing and led by a nurse, whose technical development group was basically composed of nurses. However, these factors do not limit the use of the protocol by other health professionals with an interest in knowledge in the areas of care for people with HIV/AIDS. A review study which included 20 articles representing 19 different studies located in Sub-Saharan Africa (15), Haiti (1), United Kingdom (1), United States (1) and Ukraine (1), found that the integration of services performed by various health professionals such as physicians, nurses and nutritionists contributed to positive results both in relation to care related to HIV infection and to those related to reproductive matters⁽¹¹⁾. Thus, although the protocol gives emphasis to nursing practice, its content reaches other professionals who can positively contribute to the family planning of people living with HIV/AIDS.

Regarding domain 3 (rigour of development), item 7 describes whether details regarding the search strategy for evidence is provided, including the descriptors, names of the consulted databases, and time defined for the search. Items 8 and 9 request that the inclusion/exclusion criteria for the evidence are identified, the description of bias and how the interpretation of evidence was performed. Item 10 examines the importance of mentioning the methods used to formulate the recommendations and how the final decisions were reached, in addition to highlighting issues of disagreement and how these were elucidated.

Item 11 assesses whether the benefits, side effects and health risks are in the recommendations. Item 12 considers that there should be an explicit link between the recommendations and the evidence, and this should be included in the guideline so that the user can identify it. Item 13 verifies whether the protocol was submitted for external review prior to publication and advises that evaluators should be experts, and that representatives of the target population may be included. To classify this item, it should be noted whether there is purpose and intention of external review, the methods used to conduct the review, the external reviewers’ description, the outcomes and the information obtained in the external review. Item 14 foresees the need to update the protocol, with an explicit time interval and the methodology for its performance⁽⁸⁾.

In domain 4 (clarity of the presentation), item 15 supports that a recommendation should contain correct and accurate description that the information contained in the guideline is appropriate for the target population. In order to classify the item, it is necessary to observe the objective and the purpose of the recommended action, the identification of the relevant population and the exceptions when pertinent. Item 16 ponders whether the guideline addresses the different ways of screening, prevention or treatment of the addressed clinical condition. Item 17 considers that users of the guideline should be able to easily identify the most relevant recommendations through information summarized into tables, by parts highlighted in bold or underscored, and flowcharts/algorithms. It also advises that specific recommendations should be grouped together⁽⁸⁾.

In spite of the suggestion by evaluator E2 for adding a text on the vaccines used by seropositive pregnant women, we understood that this issue is not the focal theme of the PAPHA, and that it can be treated in a different protocol aimed at the prenatal care of women living with HIV/AIDS.

Regarding domain 5 (applicability), item 18 highlights the facilitating factors and barriers that have an impact on implementing the recommendations. Item 19 deals with the effectiveness of the guideline and the need for its dissemination with additional materials such as: documents that summarize it, instruments to check, algorithms and tools to benefit from its facilitating factors. Item 20 assesses the resources needed to implement the recommendations, and their potential impacts on the costs considered in developing the guideline and in implementing the recommendations should be presented. Item 21 refers to measuring the applicability of the guideline, with identification of criteria for evaluating its implementation or adherence to the recommendations, analyzing positive or negative impacts of implementing the recommendations and advising on the periodicity of measurement⁽⁴⁾.

Evaluator E1 assigned score 2 to items 20 and 21 of this domain, since there was no information about the resources needed to implement the recommendations of the protocol, nor on the criteria for monitoring and/or auditing. This aspect was the main limitation of this evaluation. However, due to the complexity of these criteria, we suggest that they be included in further studies on the cost-effectiveness and applicability of PAPHA in clinical practice.

A quasi-experimental study on the applicability of a clinical protocol was performed in a Spanish hospital with the objective of evaluating the compliance degree of the interventions of a clinical protocol aimed at patients with severe sepsis and septic shock, and analyzing their impact on the clinical prognosis and duration of hospital admissions. The authors emphasized that the results were significant, as they improved the knowledge of clinical management of sepsis in practice, suggesting that the clinical protocol can be a useful tool in planning, contributing to the excellence of the highly complex care process⁽¹²⁾.

Although evaluator E5 suggested adding the list of health services that provide care for people living with HIV/AIDS, we understood that this care should be provided by the Basic Health Units and Outpatient Centers for HIV/AIDS, therefore the creation of this list in the middle of a decentralization process in the care of these people was not feasible. Decentralization is one of the greatest challenges for actions against HIV and AIDS to contemplate, in addition to cultural, regional and social diversity, egalitarian principles and the implementation of comprehensive and inclusive state and municipal strategies for people living with HIV/AIDS⁽¹³⁾.

Regarding domain 6 (editorial independence), items 22 and 23 deal with external financing and conflicts of interest in designing the guidelines. This information must be clearly stated whether they exist or not⁽⁸⁾.

The PAPHA did not have any external funding, so the development costs were the responsibility of the authors. In relation to conflicts of interest, the technical development group and evaluators formally declared that there were no conflicts of interest with the authors, nor among themselves. Thus, the domain reached 100.0% in the evaluation.

We can observe that part of the evaluators’ recommendations was promptly incorporated into the PAPHA, and those that were refuted received the necessary justifications without compromising the protocol's quality.

The PCV showed a greater degree of dispersion of the evaluation percentages in domains 2 (18.6%) and 5 (16.3%) due to the greater variation in the scores attributed by the evaluators. Domain 2 received scores between 4 and 7, and domain 5 received scores between 2 and 7. Domains 6, 1 and 3 presented the lowest PCV values of zero, 2.6% and 2.7%, respectively. Domain 4 had a PCV equal to 5.4%.

All domains were assessed as having quality above 88.80%, in which domain 6 (editorial independence) reached a maximum evaluation from all the evaluators (7), generating a percentage of quality of 100.0%; three domains obtained 95.50% or more in the evaluations, and two domains reached 88.80% and 89.10%. Considering that the cut-off point adopted by the authors to consider the quality of the PAPHA as satisfactory was 70.0%, the PCV for all domains remained below 25.0%, thus demonstrating homogeneity in the evaluators’ opinions (consensus) and that all domains were evaluated above 88.80%. The protocol did not return for one or more evaluation, as recommended by the Delphi technique⁽⁹⁾.

In the overall evaluation, the evaluator makes a judgment about the final quality of the protocol, considering the 23 previously evaluated items and scoring from 1 (lowest possible quality) to 7 (highest possible quality). They subsequently issue an opinion recommending or not the use of the guideline in practice⁽⁸⁾. In this sense, the PAPHA was recommended for use in clinical practice by four evaluators, without restriction. One evaluator indicated its use with restrictions; however, these did not compromise the quality and safety of the content of the instrument and were resolved.

A study conducted with the objective of evaluating the quality of Clinical Protocols and Therapeutic Guidelines (PCDT – Protocolos Clínicos e Diretrizes Terapêuticas) elaborated between 2009 and 2012 by the Ministry of Health using AGREE-II identified 59 PCDTs, of which eight were selected and evaluated by three independent evaluators. “For the item Recommendation for guideline use, two evaluators recommended the use of all guidelines, however with modifications, and one did not recommend any of the guidelines”⁽¹⁴⁾.

These results add up to strengthen the need for adaptations of the clinical protocols in relation to AGREE-II domains.

CONCLUSION

The PAPHA presented satisfactory quality in the evaluation, since the percentages of quality attributed by the evaluators were higher than the cut-off point adopted by the authors (70.0%), reaching 96.6% in domain 1 – scope and purpose; 88.8% in domain 2 – stakeholder involvement; 96.2% in domain 3 – rigour of development; 95.5% in domain 4 – clarity of presentation; 89.1% in domain 5 – applicability; and 100.0% in domain 6 – editorial independence.

The PCV was less than 25% for all of the AGREE-II domains, meaning that the scores attributed by the evaluators were linear or homogeneous and represented high agreement between them. With the quality assessment percentages and the satisfactory PCV, we can infer that the protocol is of satisfactory quality.

The benefits of AGREE-II in the process of evaluating the quality of clinical protocols are irrefutable, such as the systematization of the main characteristics of the protocol and the easy application and consolidation of the scores attributed by the judges.

We suggest that further studies are carried out on the cost-effectiveness of implementing PAPHA in the SUS and on monitoring and auditing a pilot implementation experience.

• *
  Extracted from the dissertation: “Desenvolvimento de protocolo clínico para o planejamento familiar de pessoas vivendo com HIV/aids”, Universidade Federal do Ceará, 2016.

REFERÊNCIAS

• ¹
  Werneck MAF, Faria HP, Campos KFC. Protocolos de cuidado à saúde e de organização do serviço [Internet]. Belo Horizonte: Universidade Federal de Minas Gerais/Núcleo de Educação em Saúde Coletiva; 2009 [citado 2016 set. 12]. Disponível em: https://www.nescon.medicina.ufmg.br/biblioteca/imagem/1750.pdf
  » https://www.nescon.medicina.ufmg.br/biblioteca/imagem/1750.pdf
• ²
  Manchikanti L, Singh V, Helm S 2nd, Trescot AM, Hirsch JA. A critical appraisal of 2007 American College of Occupational and Environmental Medi­cine (ACOEM) Practice Guidelines for Interventional Pain Man­agement: an independent review utilizing AGREE, AMA, IOM, and other criteria. Pain Physician. 2008;11(3):291-310.
• ³
  Ribeiro RC. Clinical guidelines: how to evaluate its quality? Rev Bras Clin Med. 2010;8(4):350-5.
• ⁴
  Khan GSC, Stein AT. Adaptação transcultural do instrumento Appraisal of Guidelines For Research & Evaluation II (AGREE II) para avaliação de diretrizes clínicas. Cad Saúde Pública [Internet]. 2014 [citado 2016 set. 12];30(5):1111-4. Disponível em: http://www.scielo.br/pdf/csp/v30n5/0102-311X-csp-30-5-1111.pdf
  » http://www.scielo.br/pdf/csp/v30n5/0102-311X-csp-30-5-1111.pdf
• ⁵
  Gonçalves TR, Carvalho FT, Faria ER, Goldim JR, Piccinini CA. Vida reprodutiva de pessoas vivendo com HIV/aids: revisando a literatura. Psicol Soc. 2009;21(2):223-32.
• ⁶
  Haddad L, Wall KM, Vwalika B, Khu NH, Brill I, Kilembe W, et al. Contraceptive discontinuation and switching among couples receiving integrated HIV and family planning services in Lusaka, Zambia. AIDS. 2013;27 (1):93-103.
• ⁷
  Johston B, Ligiero D, Silva S, Medley A, Nightingale V, Sripibatana T, et al. Meeting the family planning needs of women living with HIV in US government global health programs. AIDS. 2013;Suppl 1:S121-5.
• ⁸
  Agree Next Steps Consortium. Appraisal of Guidelines for Research & Evaluation (AGREE II). Instrumento para Avaliação de Diretrizes Clínicas [Internet]. Ontário; 2009 [citado 2015 dez. 15]. Disponível em: http://www.agreetrust.org/wpcontent/uploads/2013/06/AGREE_II_Brazilian_Portuguese.pdf
  » http://www.agreetrust.org/wpcontent/uploads/2013/06/AGREE_II_Brazilian_Portuguese.pdf
• ⁹
  Scarparo AF, Laus ALCS, Freitas MRI, Gabriel CS, Chaves LDP. Reflections on the use of Delphi technique in a reserach in nursing. Rev RENE. 2012;13(1):242-51.
• ¹⁰
  Cranksshaw TL, Matthews LT, Giddy J, Kaida A, Ware NC, Smit JA, et al. A conceptual framework for understanding HIV risk behavior in the context of supporting fertility goals among HIV-serodiscordantcouples. Reprod Health Matters. 2012;30(39):50-60.
• ¹¹
  Lisy K. Integration of HIV/AIDS Services with Maternal, Neonatal and Child Health, Nutrition, and Family Planning Services. Pub Health Nurs. 2013;30(5):451-3.
• ¹²
  Laguna-Pérez A, Chilet-Rosell E, Lacosta MD, Alvarez-Dardet C, Selles JU, Muñoz-Mendoza CL. Observância e efetividade das intervenções de um protocolo clínico utilizado para pacientes com sepse grave e choque séptico de uma Unidade de Cuidados Intensivos da Espanha. Rev Latino Am Enfermagem [Internet]. 2012 [cited 2016 sept 12]; 20(4):. Available from: http://www.scielo.br/pdf/rlae/v20n4/pt_02.pdf
  » http://www.scielo.br/pdf/rlae/v20n4/pt_02.pdf
• ¹³
  Villarinho MV, Padilha MI, Berardinelli LMM, Borenstein MS, Meirelles BHS, Andrade SR. Public health policies facing the epidemic of AIDS and the assistance for people with the disease. Rev Bras Enferm [Internet]. 2013 [citado 2016 set. 22];66(2):271-7. Available from: http://www.scielo.br/pdf/reben/v66n2/18.pdf
  » http://www.scielo.br/pdf/reben/v66n2/18.pdf
• ¹⁴
  Ronsoni D’M, Pereira CCA, Stein AT, Osanai MH, Machado CJ. Avaliação de oito Protocolos Clínicos e Diretrizes Terapêuticas (PCDT) do Ministério da Saúde por meio do instrumento AGREE II: um estudo piloto. Cad Saúde Pública [Internet]. 2015 [citado 2016 set. 23];31(6):1157-62 Disponível em: http://www.scielosp.org/pdf/csp/v31n6/0102-311X-csp-31-6-1157.pdf
  » http://www.scielosp.org/pdf/csp/v31n6/0102-311X-csp-31-6-1157.pdf

Publication Dates

• Publication in this collection
  2018

History

• Received
  06 Mar 2017
• Accepted
  29 Jan 2018