Evaluation of the clinical protocol quality for family planning services of people living with HIV / AIDS *

Objective: To evaluate the quality of a clinical protocol for family planning care for people living with HIV/AIDS. Method: An evaluative study based on the six domains of the Appraisal of Guidelines for Research & Evaluation II and on Pearson’s Coefficient of Variation. Results: The protocol reached between 88.8% and 100.0% quality in the domains of the Appraisal of Guidelines for Research & Evaluation II and 93.3% in the overall evaluation. The obtained Pearson’s coefficient of variation was between zero and 18.6. Considering that a minimum percentage of 70.0% was adopted for the quality attributed by the evaluators, quality has been achieved for all domains of the Appraisal of Guidelines for Research & Evaluation II. As a coefficient for all domains was less than 25%, we can infer that the scores attributed by the evaluators were linear or homogeneous, meaning high agreement between them. Conclusion: The protocol was evaluated as a quality instrument, recommended for use by health professionals who deal with family planning for people living with HIV/AIDS. DESCRIPTORS Protocols; Acquired Immunodeficiency Syndrome; Family Planning; HIV Seropositivity; Public Health Nursing.


INTRODUCTION
Protocols are recommendations that must be developed in an organized way based on the best scientific information, thereby aiding in the clinical management of a problem or health condition.Thus, they become tools to be used in the health area, reducing inappropriate variations in clinical practice (1) .
As they are instruments aimed at the health of their users, clinical protocols can present characteristics aimed at health prevention, promotion and education.Their use contributes to: qualification of provided care; organization and optimization of care; time saving, avoiding task repetition; rationalization of care, procedures, conducts and materials; adaptation of scientific evidence to local realities and user satisfaction (1) .
Despite the relevance of implementing protocols in clinical practice and its positive impact on the improvement of the outcomes among patients, the development of these instruments continues to lack clearly articulated objectives, coherent structures, reliable evaluation and implementation mechanisms, adding good quality and the best cost-effectiveness to health care (2) .
In this sense, a structured review of the scientific literature was carried out in order to identify the prospect of effective instruments that guarantee the quality of clinical protocols, listing several instruments that enable evaluating the quality of a clinical protocol, where the most reliable, tested and recommended were the Appraisal of Guidelines for Research and Evaluation (AGREE-II), a checklist drawn up by the World Health Organization (WHO); and another one elaborated by the Agency for Healthcare Research and Quality (AHRQ), the National Guideline Clearinghouse (NGC) (3) .
In view of the above, we chose to adopt the AGREE-II as an instrument for assessing the quality of a clinical protocol for offering family planning care to people living with HIV/AIDS (PAPHA) due to its ease of access and practical application (3) .Moreover, this instrument presents the advantage of the possibility of obtaining a checklist, thereby facilitating the process of planning, elaboration and implementation of guidelines and allowing its use by different health professionals (4) .
The PAPHA has been identified as a pressing need for health professionals who deal with this area of care, since the reproductive and/or contraceptive choice of this target population requires specific care based on up-todate knowledge.Authors recommend that this service should be specialized, safe, standardized and effective, so that conception and contraception can be approached in a current and scientific manner, thus guaranteeing the least (possible) risk to the woman, the partner and their future child (5) .
A study conducted in Lusaka, Zambia, for example, reaffirms the discontinuation of contraceptive methods for reasons that go beyond professional guidance, being directly related to signs and symptoms such as severe menstrual bleeding and dysuria (6) .In addition, a study addressing the health strategy adopted in the United States to assist in the family planning of women living with HIV/AIDS highlighted the importance of providing quality services based on consistent scientific evidence (7) .
Thus, the need to use an effective clinical protocol in professional practice that addresses the family planning of people with HIV/AIDS is recurrent, and requires the availability of a concise evaluation instrument that guarantees its quality.According to the literature, the AGREE-II (8) is an evaluation system that groups these characteristics, which justifies the development of the present study.
Thus, the following research question was outlined: Is the protocol presented herein satisfactory for use in clinical practice, offering up-to-date and objective information?To answer this question, the present study was developed with the objective of evaluating the quality of a clinical protocol for family planning care of people living with HIV/AIDS (PAPHA).

METHOD
An evaluative study carried out from January to March 2016 based on the six domains proposed by the Appraisal of Guidelines for Research & Evaluation II (AGREE-II): scope and purpose, stakeholder involvement, rigour of development, clarity of presentation, applicability and editorial independence.This set of domains are standardized into 23 evaluation items (8) .Questions regarding the characterization of the evaluators were added to the AGREE-II "Quality Assessment Instrument".
The AGREE-II instrument has been developed in order to address quality, methodological rigour and the transparency with which the protocol is developed.This instrument is comprehensive, allowing its application to protocols directed toward any diseases and health situations, including aspects related to both health promotion and collective health, as well as to diagnoses, treatments or interventions (8) .It was possible to establish a methodological strategy for the evaluation of the protocol's quality with the use of this instrument.
The study was carried out with the purpose of describing the cross-cultural adaptation process of AGREE-II and it emphasizes the importance of using this instrument, which allows a structured and rigorous methodology for protocol development, resulting in a more effective tool when inserted into clinical practice (4) .Five evaluators (E1, E2, E3, E4 and E5) participated who were intentionally chosen by the "convenience or network" search tool.Evaluators from the two care areas of family planning and HIV/AIDS were selected; and an evaluator of materials who were related to teaching, research and service.These evaluators had to have a master's and/or doctorate degree, in addition to direct performance and clinical experience in these knowledge areas.
A scale of scores varying from one to seven was applied for each evaluated item, in which a score of one (Strongly Disagree) is applied when there is no relevant information, or if the concept is poorly reported; and a score of seven (Strongly Agree) when the quality of the information is excellent, meaning that all proposed criteria and considerations were promptly met.Scores from two to six are assigned when the information item does not totally meet all criteria or considerations.The score should be attributed according to the quality of the report and of the complete exposition of information, meaning that it should be directly proportional to the addressed criteria and expositions, increasing as they are contemplated (8) .
The judges' evaluation was calculated "individually according to the sum of all the item scores of each domain, scaling the total with a percentage of the maximum possible score per domain" (8) .This is possible by using the formula: The obtained score is given by the sum of the total values of each item from the five evaluators; the minimum score is given by 1 (Strongly Disagree) x amount of items evaluated in the domain x 5 (number of evaluators); the maximum score is given by 5 (Strongly Agree) x amount of items evaluated in the domain x 5 (number of evaluators) (4) .
Although the scores obtained in the domains allow for a more reliable comparison between guidelines and offer elements for a protocol to be recommended or not, AGREE-II does not define minimum domain scores in order to differentiate between high and low quality protocols.This decision must be made by the author and guided by the context regarding the protocol use (8) .Thus, the authors established a domain that reached a score equal to or greater than 70% as being as "satisfactory quality"; a percentage that generally means acceptable minimum performance in the evaluations.
A space was provided for the evaluators in each domain to describe their recommendations, which were critically analyzed by the researchers to make the necessary adaptations in order to increase the quality of the PAPHA.
In addition to this evaluation parameter proposed by AGREE-II, the Pearson's Coefficient of Variation (PCV ) was calculated, which provides the variation of the obtained data in relation to the mean.The lower its value, the more homogenous are the opinions, and consequently the scores attributed by the evaluators.The coefficient is considered low when it is less than or equal to 25%, pointing to a set of homogeneous data.PCV calculations are given by the standard deviation divided by the mean of the scores.
In the case of a domain with a result below 70% and/or PCV greater than 25%, this domain would have the modifications sent to the evaluators as many times as necessary until the established parameters were reached, according to the Delphi technique.This technique is intended for the deduction and refinement of opinions of a group of people specialized in a certain subject, seeking to reach consensus on this judgment, thus contributing to the quality and excellence of the evaluated material (9) .
The evaluators received the free and informed consent form (ICF) and the "Quality Assessment Instrument" via e-mail.A 15-day deadline was given for the evaluators to return the evaluation result along with the signed ICF.The study met the requirements of the Guidelines and Norms of Research on Human Beings presented in Resolution 466/12 of the National Health Council, on the ethical questions of research involving human beings.The research project was submitted to the Plataforma Brasil -Ethics Committee of the Universidade Federal do Ceará -UFC, obtaining a favorable opinion according to protocol number 1.086.282.

RESULTS
The evaluators were all nurses with training time ranging from 6 to 31 years (χ ¯ = 12.8).Four developed or had developed activities in the area of family planning (E1, E2, E3 and E5), two integrated care to people living with HIV/ AIDS (E3 and E4), and two had experience with protocol elaboration and evaluations (E1 and E5).
The evaluation results were presented according to the six domains proposed by the AGREE-II, grouped two by two (Tables 1 to 3 and Chart 1).Next, the overall evaluation of the PAPHA and the Pearson's Coefficient of Variation (PCV) are presented (Tabela 4).* Domain items that make up the study (8) , cited by the authors.
Domain 1 was mostly evaluated at the maximum levels (6 and 7).However, evaluator E1 recommended a change in the second objective of the protocol, while evaluator E5 suggested defining the age range of the target population to receive care based on the protocol (Chart 1).After a critical reading of these recommendations, it was decided to keep the description of the objective in its original format, considering that it was adequate and that it encompassed the purpose of the PAPHA, which is to serve people with HIV/AIDS in the area of family planning, focusing on the reproductive phase.
Evaluators E2 and E5 suggested adding their higher titles to domain 2, in addition to the training areas of the professionals who participated in the protocol development.The recommendation was accepted, and the respective information was included since it was considered to be an aspect which would generate greater credibility to the protocol users.
Evaluator E3 brought up the need to specify to which healthcare professionals the protocol was intended for.However, it is explicit in the protocol that it is intended for nurses and other health professionals who deal with people living with HIV/AIDS and family planning.
Therefore, domain 1 achieved a 96.6% quality percentage, and domain 2 scored 88.8%.* Domain items that make up the study (8) , cited by the authors.
Regarding domain 3, evaluator E5 questioned the work dynamics of the professionals who participated in developing the protocol (which content each of them contributed and how the organization and the insertion of that material occurred).After a critical analysis of this observation, it was understood that the evaluator referred to the process of surveying topics and the organization of a preliminary "summary", meaning which topics the protocol would address.In this regard, it was concluded that such information is complementary and is not required by the AGREE-II to compose PAPHA.
In domain 4, evaluator E2 suggested adding information on the vaccines used by seropositive pregnant women during prenatal care, and evaluator E5 suggested adding a list of health services that provide care for people living with HIV/ AIDS, in the municipality where the protocol was developed.
Thus, domain 3 scores reached a quality percentage of 96.2%, and domain 4 reached 95.5%.* Domain items that make up the study (8) , cited by the authors.
Thus, a 95.5% quality percentage was obtained for domain 5, and domain 6 achieved the maximum score on all addressed issues, resulting in 100%.
Chart 1 presents the recommendations proposed by the evaluators according to each domain.We emphasize that even though some evaluators did not assign a maximum score to a certain item that composed the domain, they did not suggest modifications to the protocol.Regarding the overall PAPHA evaluation, three evaluators attributed a maximum score (7), and two scored it 6, generating an overall PAPHA quality assessment of 93.3%.
Regarding the recommendation for using the protocol, four reviewers answered "I would recommend it without restrictions", while one answered "I would recommend it with some restrictions", emphasizing that the respective restrictions were solved, such as the importance of mentioning their higher degrees/titles in addition to the training areas of the professionals who participated in developing the protocol.
Table 4 specifies the values of standard deviation, mean and PCV for each domain.The PCV found in all domains was below 25%, which means high homogeneity in the scores assigned to the domains.This finding suggests similar points of view among the evaluators regarding the protocol.
The PCV showed a greater degree of dispersion among the evaluation percentages for domains 2 (18.6%) and 5 (16.3%) due to the greater variation of the scores attributed by the evaluators.Domain 2 obtained scores ranging from 4 to 7, and domain 5 received scores ranging from 2 to 7. Domains 6, 1 and 3 presented the lowest PCV values of zero, 2.6% and 2.7%, respectively.Domain 4 had a PCV equal to 5.4%.

DISCUSSION
In domain 1 (scope and purpose), item 1 infers the possible impact of the guideline on the health of the population or individuals involved, it analyzes whether the general objectives of the guideline are explicit, and whether the health benefits acquired from the guideline address the clinical problem or the exposed health situation.Item 2 assesses whether the key recommendations described are in accordance with the addressed health situations.Item 3 observes whether the description of the population covered by the guideline is clear.The contents of the items include: purpose; expected benefit or outcome; target population; interventions; comparisons where appropriate; health care environment or context (8) .
Although it was suggested by evaluator E5, it was not possible to define the age range of the reproductive phase from 12 to 49 years, since it seemed inappropriate in the sense of incurring an exclusion of people due the variation in the ages at the beginning and the end of the reproductive phase.An outpatient study of Infectious Diseases in Atlanta, United States, for example, reaffirmed the desire of seropositive women to become pregnant; however, they did not determine the period when they wished to get pregnant.Thus, restricting the age for protocol application could lead to the future exclusion of these women, who despite being at an advanced age for reproduction, could express the desire to conceive and should have their rights fulfilled (10) .
In domain 2 (stakeholder involvement), item 4 refers to professionals who participated in developing the protocol, excluding external reviewers (evaluators).For each member of the technical development group, the following should be presented: name, discipline or content of the expertise, institution, geographical location and their role description.In item 5, it is understood that the guideline should be based on the experiences of the target population and their expectations of health care, meaning that there must be evidence in order to consider the views of the interested parties.The strategy used to capture the opinions and preferences of the population should be reported.Item 6 asserts that the target users must be expressed in the guideline.In addition, it should encompass how the guideline should be used by them, thus, it is possible for the reader to determine its relevance (8) .
Although evaluator E3 suggested specifying for which professionals the protocol is intended, it is understood that this clinical protocol was a product of research in Nursing and led by a nurse, whose technical development group was basically composed of nurses.However, these factors do not limit the use of the protocol by other health professionals with an interest in knowledge in the areas of care for people with HIV/AIDS.A review study which included 20 articles representing 19 different studies located in Sub-Saharan Africa (15), Haiti (1), United Kingdom (1), United States (1) and Ukraine (1), found that the integration of services performed by various health professionals such as physicians, nurses and nutritionists contributed to positive results both in relation to care related to HIV infection and to those related to reproductive matters (11) .Thus, although the protocol gives emphasis to nursing practice, its content reaches other professionals who can positively contribute to the family planning of people living with HIV/AIDS.
Regarding domain 3 (rigour of development), item 7 describes whether details regarding the search strategy for evidence is provided, including the descriptors, names of the consulted databases, and time defined for the search.Items 8 and 9 request that the inclusion/exclusion criteria for the evidence are identified, the description of bias and how the interpretation of evidence was performed.Item 10 examines the importance of mentioning the methods used to formulate the recommendations and how the final decisions were reached, in addition to highlighting issues of disagreement and how these were elucidated.
Item 11 assesses whether the benefits, side effects and health risks are in the recommendations.Item 12 considers that there should be an explicit link between the recommendations and the evidence, and this should be included in the guideline so that the user can identify it.Item 13 verifies whether the protocol was submitted for external review prior to publication and advises that evaluators should be experts, and that representatives of the target population may be included.To classify this item, it should be noted whether there is purpose and intention of external review, the methods used to conduct the review, the external reviewers' description, the outcomes and the information obtained in the external review.Item 14 foresees the need to update the protocol, with an explicit time interval and the methodology for its performance (8) .
In domain 4 (clarity of the presentation), item 15 supports that a recommendation should contain correct and accurate description that the information contained in the guideline is appropriate for the target population.In order to classify the item, it is necessary to observe the objective and the purpose of the recommended action, the identification of the relevant population and the exceptions when pertinent.Item 16 ponders whether the guideline addresses the different ways of screening, prevention or treatment of the addressed clinical condition.Item 17 considers that users of the guideline should be able to easily identify the most relevant recommendations through information summarized into tables, by parts highlighted in bold or underscored, and flowcharts/algorithms.It also advises that specific recommendations should be grouped together (8) .
In spite of the suggestion by evaluator E2 for adding a text on the vaccines used by seropositive pregnant women, we understood that this issue is not the focal theme of the PAPHA, and that it can be treated in a different protocol aimed at the prenatal care of women living with HIV/AIDS.
Regarding domain 5 (applicability), item 18 highlights the facilitating factors and barriers that have an impact on implementing the recommendations.Item 19 deals with the effectiveness of the guideline and the need for its dissemination with additional materials such as: documents that summarize it, instruments to check, algorithms and tools to benefit from its facilitating factors.Item 20 assesses the resources needed to implement the recommendations, and their potential impacts on the costs considered in developing the guideline and in implementing the recommendations should be presented.Item 21 refers to measuring the applicability of the guideline, with identification of criteria for evaluating its implementation or adherence to the recommendations, analyzing positive or negative impacts of implementing the recommendations and advising on the periodicity of measurement (4) .
Evaluator E1 assigned score 2 to items 20 and 21 of this domain, since there was no information about the resources needed to implement the recommendations of the protocol, nor on the criteria for monitoring and/ or auditing.This aspect was the main limitation of this evaluation.However, due to the complexity of these criteria, we suggest that they be included in further studies on the cost-effectiveness and applicability of PAPHA in clinical practice.
A quasi-experimental study on the applicability of a clinical protocol was performed in a Spanish hospital with the objective of evaluating the compliance degree of the interventions of a clinical protocol aimed at patients with severe sepsis and septic shock, and analyzing their impact on the clinical prognosis and duration of hospital admissions.The authors emphasized that the results were significant, as they improved the knowledge of clinical management of sepsis in practice, suggesting that the clinical protocol can be a useful tool in planning, contributing to the excellence of the highly complex care process (12) .
Although evaluator E5 suggested adding the list of health services that provide care for people living with HIV/AIDS, we understood that this care should be provided by the Basic Health Units and Outpatient Centers for HIV/AIDS, therefore the creation of this list in the middle of a decentralization process in the care of these people was not feasible.Decentralization is one of the greatest challenges for actions against HIV and AIDS to contemplate, in addition to cultural, regional and social diversity, egalitarian principles and the implementation of comprehensive and inclusive state and municipal strategies for people living with HIV/AIDS (13) .
Regarding domain 6 (editorial independence), items 22 and 23 deal with external financing and conflicts of interest in designing the guidelines.This information must be clearly stated whether they exist or not (8) .
www.ee.usp.br/reeuspRev Esc Enferm USP • 2018;52:e03335 The PAPHA did not have any external funding, so the development costs were the responsibility of the authors.In relation to conflicts of interest, the technical development group and evaluators formally declared that there were no conflicts of interest with the authors, nor among themselves.Thus, the domain reached 100.0% in the evaluation.
We can observe that part of the evaluators' recommendations was promptly incorporated into the PAPHA, and those that were refuted received the necessary justifications without compromising the protocol's quality.
The PCV showed a greater degree of dispersion of the evaluation percentages in domains 2 (18.6%) and 5 (16.3%) due to the greater variation in the scores attributed by the evaluators.Domain 2 received scores between 4 and 7, and domain 5 received scores between 2 and 7. Domains 6, 1 and 3 presented the lowest PCV values of zero, 2.6% and 2.7%, respectively.Domain 4 had a PCV equal to 5.4%.
All domains were assessed as having quality above 88.80%, in which domain 6 (editorial independence) reached a maximum evaluation from all the evaluators (7), generating a percentage of quality of 100.0%; three domains obtained 95.50% or more in the evaluations, and two domains reached 88.80% and 89.10%.Considering that the cut-off point adopted by the authors to consider the quality of the PAPHA as satisfactory was 70.0%, the PCV for all domains remained below 25.0%, thus demonstrating homogeneity in the evaluators' opinions (consensus) and that all domains were evaluated above 88.80%.The protocol did not return for one or more evaluation, as recommended by the Delphi technique (9) .
In the overall evaluation, the evaluator makes a judgment about the final quality of the protocol, considering the 23 previously evaluated items and scoring from 1 (lowest possible quality) to 7 (highest possible quality).They subsequently issue an opinion recommending or not the use of the guideline in practice (8) .In this sense, the PAPHA was recommended for use in clinical practice by four evaluators, without restriction.One evaluator indicated its use with restrictions; however, these did not compromise the quality and safety of the content of the instrument and were resolved.
A study conducted with the objective of evaluating the quality of Clinical Protocols and Therapeutic Guidelines (PCDT -Protocolos Clínicos e Diretrizes Terapêuticas) elaborated between 2009 and 2012 by the Ministry of Health using AGREE-II identified 59 PCDTs, of which eight were selected and evaluated by three independent evaluators."For the item Recommendation for guideline use, two evaluators recommended the use of all guidelines, however with modifications, and one did not recommend any of the guidelines" (14) .
These results add up to strengthen the need for adaptations of the clinical protocols in relation to AGREE-II domains.

CONCLUSION
The PAPHA presented satisfactory quality in the evaluation, since the percentages of quality attributed by the evaluators were higher than the cut-off point adopted by the authors (70.0%), reaching 96.6% in domain 1scope and purpose; 88.8% in domain 2 -stakeholder involvement; 96.2% in domain 3 -rigour of development; 95.5% in domain 4 -clarity of presentation; 89.1% in domain 5 -applicability; and 100.0% in domain 6 -editorial independence.
The PCV was less than 25% for all of the AGREE-II domains, meaning that the scores attributed by the evaluators were linear or homogeneous and represented high agreement between them.With the quality assessment percentages and the satisfactory PCV, we can infer that the protocol is of satisfactory quality.
The benefits of AGREE-II in the process of evaluating the quality of clinical protocols are irrefutable, such as the systematization of the main characteristics of the protocol and the easy application and consolidation of the scores attributed by the judges.
We suggest that further studies are carried out on the cost-effectiveness of implementing PAPHA in the SUS and on monitoring and auditing a pilot implementation experience.Appraisal of Guideline for Research & Evaluation II y el 93,3% en la evaluación global.Se logró el coeficiente de variación de Pearson entre cero y 18,6.Toda vez que se adoptó porcentual mínimo del 70,0% a la calidad atribuida por los evaluadores, se verificó calidad en todos los dominios del Appraisal of Guideline for Research & Evaluation II.Con el coeficiente inferior al 25% en todos los dominios, se infiere que los puntajes atribuidos por los evaluadores fueron lineales y homogéneos, significando elevada concordancia entre ellos.Conclusión: El protocolo fue evaluado como instrumento de calidad, recomendado para empleo por profesionales sanitarios que manejan la planificación familiar de personas viviendo con VIH/SIDA.

Table 2 -
Evaluation of the clinical protocol according to the rigour of development (domain 3) and clarity of the presentation (domain 4) (AGREE-II) -Fortaleza, CE, Brazil, 2016.

Total 18 27 27 28 27 127 *Domain 6 -Editorial independence
-Evaluators' suggestions in accordance with the AGREE-II domains -Fortaleza, CE, Brazil, 2016.Changes in the second objective of the protocol (E1) and defining of the age group to be attended (E5).E2 did not present any recommendations despite assigning a score below 7 on an item.

Table 4 -
Values for the standard deviation (S), mean (χ ¯) and Pearson's Coefficient of Variation (PCV) for each domain of the AGREE-II -Fortaleza, CE, Brazil, 2016.