Validation methods of nursing protocols: an integrative review

Métodos de validación de los protocolos de atención de enfermería: revisión integrativa

ABSTRACT

Objective:

to identify scientific production about validation methods of nursing care protocols.

Method:

an integrative review with search at Scielo, Pubmed/MEDLINE, Virtual Health Library, Web of Science, Scopus, and EBSCOhost. The descriptors “validation studies”, “validation studies as topic”, “protocols”, “clinical protocols”, “practice guidelines as topic”, “nursing” and “nursing assessment” and the uncontrolled descriptor “validation” were used.

Results:

thirty-two articles were selected, most of them Brazilian. Content validation by experts was the most frequent method, with no consensus on the number of participants for the process. The collection instruments were mostly created by the authors. Data analysis was performed using descriptive statistics and Content Validity Index, with a variable consensus rate in the analyzed articles.

Conclusion:

protocols validated by experts are robust tools for use in clinical practice, with methodological rigor in development essential for its quality.

Descriptors:
Validation Studies; Consensus; Practice Guidelines; Methods; Nursing

RESUMEN

Objetivo:

identificar la producción científica sobre los métodos de validación de los protocolos de atención de enfermería.

Método:

revisión integradora, búsqueda en las bases de datos Scielo, PubMed/MEDLINE, Biblioteca Virtual en Salud, Web of Science, Scopus y EBSCOhost, utilizando los descriptores validation studies, validation studies as topic, protocols, clinical protocols, practice guidelines as topic, nursing e nursing assessment y el descriptor no controlado validation.

Resultados:

se seleccionaron 32 artículos, la mayoría de ellos brasileños. La validación del contenido por expertos fue el método más frecuente, sin consenso sobre el número de participantes para el proceso. Los instrumentos de colección fueron elaborados principalmente por los autores. El análisis de datos se realizó utilizando estadísticas descriptivas e índice de validez de contenido, con una tasa de acuerdo variable en los artículos analizados.

Conclusión:

los protocolos validados por especialistas son herramientas robustas para su uso en la práctica clínica, y el rigor metodológico en el desarrollo es esencial para su calidad.

Descriptores:
Estudios de Validación; Consenso; Protocolos; Métodos; Enfermería

RESUMO

Objetivo:

identificar a produção científica acerca dos métodos de validação de protocolos assistenciais de enfermagem.

Método:

revisão integrativa, com busca nas bases de dados Scielo, PubMed/MEDLINE, Biblioteca Virtual em Saúde, Web of Science, Scopus e EBSCOhost, utilizando os descritores validation studies, validation studies as topic, protocols, clinical protocols, practice guidelines as topic, nursing e nursing assessment e o descritor não controlado validation.

Resultados:

foram selecionados 32 artigos, sendo a maioria brasileiros. A validação de conteúdo por especialistas foi o método mais frequente, sem consenso sobre o número de participantes para o processo. Os instrumentos de coleta foram majoritariamente elaborados pelos autores. A análise dos dados foi realizada por estatística descritiva e Índice de Validade de Conteúdo, com taxa de concordância variável nos artigos analisados.

Conclusão:

protocolos validados por especialistas constituem ferramentas robustas para o uso na prática clínica, sendo o rigor metodológico no desenvolvimento imprescindível para a qualidade do mesmo.

Descritores:
Estudos de Validação; Consenso; Protocolos; Métodos; Enfermagem

INTRODUCTION

Evidence-based practice (EBP), from the medical field, has become a multidisciplinary approach to decision making in clinical practice worldwide and for all health professions with a view to identifying knowledge gaps and best available practice. The classic definition of EBP refers to the conscious, explicit and judicious use of the best current evidence to make decisions about the individual care of each patient⁽¹⁾.

Nursing, in this context, has been working to incorporate the evidence into its praxis, evolving from empirical work to a care centered on the needs of patients, combined with data available in nursing literature. Florence Nightingale was considered by many authors to be the pioneer of evidence-based nursing, given that the observations available in her studies at the time were based on epidemiological data and her experiments⁽²⁾.

Nursing is globally recognized as central to the process of implementing EBP, considering that it consists of the largest health workforce⁽¹⁾. In Brazil, data obtained from the Federal Nursing Council (Conselho Federal de Enfermagem) 2020 page indicate 2,222,056 nursing professionals, including nurses, technicians, assistants and midwives, working in different spheres of health care.

Currently, nursing research has sought ways to enable the incorporation of evidence into practice, mainly through instruments that facilitate the work of nurses and their staff. Assistance protocols are examples of such instruments. They are built based on the best available evidence, according to the local reality, the experience of professionals and the preference of customers⁽³⁾.

Protocols aim to provide adequate care efficiently, ensuring effectiveness for the clinical situation, and causing more benefits than harm⁽¹⁾. Furthermore, assistance protocols aim to facilitate decision making, describing a specific situation of care with details and operational specifications. In this way, they bring greater security to the team; reducing stock variability; facilitating the incorporation of new technologies and rational use of resources; allowing the monitoring of process and results indicators; and contributing to maintenance of services and assessment of the quality and safety of care^(3-5).

Considering that protocols constitute a foundation for professional conduct, their construction must be carried out with methodological rigor in order to guarantee its credibility and legitimacy. To this end, validation is recommended, which assesses whether the content of protocols achieves the proposed objectives^(6-8). Validation is widely used in nursing literature; however, there is no consensus on the best method to be used⁽⁷⁾ and how it should be structured. Content validation occurs through instrument creation and assessment by experts (or judges), so that consensus methods among experts vary among themselves, as will be elucidated in this research.

Therefore, there was an interest in investigating which methods are used to validate nursing care protocols in national and international literature in the last five years and also how they are conducted. The objective is to present the scientific production on the subject to assist in decision making in choosing the most appropriate validation method for the creation of nursing care protocols.

OBJECTIVE

To identify scientific production about validation methods of nursing care protocols.

METHOD

Ethical aspects

This is a literature review using materials available in online databases, in line with ethical research principles.

Study design

It is an integrative review, a method whose main objective is the gathering and synthesis of results on the topic, presenting the state of the art and enhancing the construction of knowledge to enable incorporating evidence in clinical practice^(9-11). Integrative review differs from meta-analysis and systematic review, as it allows the inclusion of research of different methods, seeking a broader knowledge of nursing literature to know and understand the phenomenon studied⁽⁹⁾.

The research was structured in six stages: 1) identification of the theme and definition of the research question; 2) searching or sampling in the literature; 3) data collection; 4) critical analysis of the included studies; 5) discussion of results, and 6) presentation of the integrative review⁽¹¹⁾. The names assigned by the authors to each phase of the integrative review vary, but the steps included in each phase are common to each other^(9-11).

The guiding question was: what are the methods used to validate nursing care protocols?

Sample, and inclusion and exclusion criteria

Scientific articles published in full with free online access in Brazilian Portuguese, English, and Spanish, addressing validation of nursing care protocols from 2014 to 2019 were included. Studies classified as editorial, letters, dissertations and theses, articles not from nursing and whose validation objects were not care protocols, clinical guidelines or checklists, and which did not deal with validation itself were excluded.

The databases used were Medical Literature Analysis and Retrieval System Online (MEDLINE/PubMed); Virtual Health Library (VHL); Scientific Electronic Library Online (SciELO); EBSCOhost; Scopus; and Isi Web of Science. Identifying articles in databases occurred from August 21, 2019 to September 6, 2019 using the controlled descriptors obtained from the Health Sciences Descriptors (Descritores em Ciências da Saúde, abbreviated DeCS) and Medical Subject Headlines (MeSH) such as “validation studies”, “validation studies as topic”, “protocols”, “clinical protocols”, “practice guidelines as topic”, “nursing”, and “nursing assessment” . The uncontrolled descriptor validation was also used. They were combined by the Boolean operators “AND” and “OR” using combinations according to the databases searched, using language filters (English, Brazilian Portuguese, Spanish) and in the last five years.

Study protocol

The research question was defined and the controlled and uncontrolled descriptors were selected. We proceeded with the search for articles in the databases using the aforementioned search strategies, according to the search tool for each database, and language filters and year of publication were applied. Search was carried out by a researcher. To select the articles, the titles and abstracts were read first, so that those whose study objects were not care protocols or were not from nursing were excluded. The remaining articles were read in full, so that those whose validation process was not described and did not deal with protocols related to nursing care were excluded.

The remaining articles were read in full by two researchers. The data were stored in a Microsoft Excel 2010 analysis matrix containing the variables: method of protocol creation; protocol validation method; number of participants; sampling method; collection instrument used for validation; and data analysis. From this reading, articles were excluded in which the protocol was not related to nursing and whose adopted validation method was not described. At the end, the articles were critically analyzed, with the results discussed and presented in this study.

Analysis of data

Descriptive statistics were used to describe the profile of the studies. The data were grouped according to the type of method used to validate the protocol; the number of participants involved; the type of sampling; the types of instruments used; and methods of data analysis.

RESULTS

Searching the databases totaled 827 studies, with titles and abstracts being read and the inclusion and exclusion criteria applied. Seven hundred and fifty-five articles were excluded as they were not studies whose object was protocols or were not from nursing. The remaining 72 articles were assessed, 15 of which were repeated in two or more databases, and 50 were read in full. Of these, 14 did not deal with validation of protocols; 3 were not from the nursing field and did not even deal with validation; and 1 was not available in full online, totaling 18 excluded articles and a final sample of 32 articles, as shown in Figure 1.

Figure 1
Strategy for selecting articles on validation methods of nursing protocols

Regarding the characteristics of the articles identified and included in this review (n=32), most were published in Brazil (n=24), followed by Canada, Spain, and Portugal, with two studies each; and the United States and China, both with a publication. As for year, 2017 stood out with 11 studies, followed by 2014 (n=7), 2018 (n=5) and 2019 (n=5), 2015 and 2016, with two studies each.

All articles were published in journals of strata equal to or greater than B1, according to the classification of journals in the QUALIS System of CAPES (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Coordination for the Improvement of Higher Education Personnel). Most were classified as A1 (n=17), followed by A2 (n=8) and B1 (n=7). Most were Brazilian journals (n=25) and seven international studies.

From reading the articles, data emerged concerning protocol creation. All articles mention the literature review as part of protocol creation, be it a simple or structured literature review, such as integrative^(12-15) and systematic^(16-17) reviews, with emphasis on scoping reviews^(4,6). In some articles, literature review is associated with another type of data collection, such as interviews^(18-21) and group discussions^(22-24).

As for content validation method of protocols, assessment by experts was the most frequent (n=17), followed by expert assessment using Delphi Technique (n=12), and other methods such as convergent care research (Pesquisa Convergente Assistencial, abbreviated PCA)⁽²⁴⁾.

The number of participants in validation with experts ranged from 3 to 42 participants, with a median of 9 participants. In Delphi Technique, the number varied from 5 to 43 participants, with a median of 14 participants, without considering the losses between one round and another.

Selection of participants mostly took into account criteria for title, experience and knowledge on the subject under study, in addition to studies in the field. Six studies made a selection based on the search for curricula in the Lattes Platform^{(4,6,13,25-27)}. three used scoring systems for curriculum assessment^(4,6,13); three used snowball sampling^(16,28-29), in which one participant indicates the next; and the others did not detail the form of selection, only the inclusion and exclusion criteria.

Concerning data collection instruments used for validation, only two studies used a standardized tool^(14,21), and the others created their own instrument according to different validation criteria according to the adopted framework. The aforementioned studies used AGREE II (Appraisal of Guidelines for Research & Evaluation). AGREE II is an international tool developed for assessment of protocols, clinical therapeutic guidelines and guidelines, already translated and adapted for use in Brazil⁽³⁰⁾.

As for the method of data analysis, the articles mostly used consensus rate and Content Validity Index (CVI). The Kappa coefficient is used in two studies^(29,31), isolated or associated with CVI. The studies that used AGREE II as a tool follow analysis according to the instrument’s guidelines. Thematic content analysis was used in PCA. Three studies do not describe how to analyze the data.

Chart 1 shows the studies selected in this review organized in descending order of publication over the past five years.

Chart 1
Characterization of the articles identified in the review that deal with validation of nursing care protocols

Title	Year Country	Protocol topic	Validation method used	Number of experts
Avaliação da segurança no cuidado com vacinas: construção e validação de protocolo ( 6 )	2019 Brazil	Vaccine care	Delphi Technique	1st round: 12 2nd round: 8
Validação de protocolo assistencial ao paciente séptico na Unidade de Terapia Intensiva ( 25 )	2019 Brazil	Sepsis in intensive care	Delphi Technique	1st round: 34 2nd round: 26
Validação de protocolos gráficos para avaliação da segurança do paciente politraumatizado ( 4 )	2019 Brazil	Polytraumatized patient safety	Delphi Technique	1st round: 15 2nd round: 13
Nursing care after coronary transluminal angioplasty: Protocol validation(32)	2019 Brazil	Care after percutaneous transluminal coronary angioplasty	Delphi Technique	12
Elaboração e validação de checklist para administração de medicamentos para pacientes em protocolos de pesquisa(33)	2019 Brazil	Checklist for administering medication to patients in clinical research	Validation with experts	9
Balanço hídrico na nefrologia pediátrica: construção de um Procedimento Operacional Padrão ( 15 )	2018 Brazil	Water balance in pediatrics	Validation with experts	9
Desenvolvimento de protocolo clínico para detecção de lesões precursoras do câncer de colo uterino ( 14 )	2018 Brazil	Protocol for the detection of precursor lesions of cervical cancer	Validation with experts	4
Managing the environment to prevent falls in the institutionalized elderly: Protocol validation(20)	2018 Portugal	Management of the environment to prevent falls in institutionalized elderly	Delphi Technique	1st round: 14 2nd round: 10
Development, validation and application of a ventilator-associated pneumonia prevention checklist in a single cardiac surgery centre(21)	2018 China	Checklist for preventing pneumonia associated with mechanical ventilation	Delphi Technique	20
Adaptação e validação de checklist de segurança cirúrgica na cesárea(27)	2018 Brazil	Safe surgery checklist: cesarean section	Delphi Technique	43
Construção e validação de um protocolo assistencial de enfermagem em anestesia ( 12 )	2017 Brazil	Anesthesia Nursing	Validation with experts	5
Protocolo de cuidados de enfermagem a pacientes com dispositivo de assistência ventricular ( 29 )	2017 Spain	Nursing care for patients with ventricular assist devices	Delphi Technique	1st round: 42 2nd round: 34
Protocol for the management of psychiatric patients with psychomotor agitation(16)	2017c Spain	Management of patients in psychomotor agitation	Delphi Technique	Undefined
Queda nas Instituições de Longa Permanência para Idosos: validação de protocolo ( 18 )	2017 Portugal	Prevention of falls of institutionalized elderly	Delphi Technique	14
Construção e validação de um algoritmo para aplicação de laser no tratamento de ferida ( 34 )	2017 Brazil	Algorithm for using laser to treat wounds	Validation with experts	24
Adaptação transcultural do protocolo Filial Responsibility para uso no Brasil(35)	2017 Brazil	Affiliate Responsibility Protocol	Validation with experts	9
Protocolo de acolhimento com classificação de risco em pediatria: confiabilidade interobservadores ( 28 )	2017 Brazil	Welcoming protocol with risk stratification in pediatrics	Validation with experts	9
Ações do protocolo prevenção de quedas: mapeamento com a classificação de intervenções de enfermagem ( 36 )	2017 Brazil	Mapping of fall prevention interventions	Delphi Technique	5
Validação de instrumento para intervenção de enfermagem ao paciente em terapia vasoativa ( 19 )	2017 Brazil	Nursing care with vasoactive therapy	Validation with experts	16
The Development and Content Validation of a Multidisciplinary, Evidence-based Wound Infection Prevention and Treatment Guideline(17)	2017 Canada	Protocol for prevention and treatment of wound infection	Validation with experts	42
Percurso metodológico em pesquisas de enfermagem para construção e validação de protocolos ( 7 )	2017 Brazil	Integrative review about methods of construction and validation of protocols used in nursing research (theses and dissertations)
Validação de protocolo assistencial de enfermagem para pacientes em cuidados paliativos ( 13 )	2016 Brazil	Palliative care	Validation with experts	11
Construção, validação e adequação cultural do protocolo COMPASSO: Adesão ao autocuidado em diabetes ( 37 )	2016 Brazil	Diabetes self-care adherence protocol	Validation with experts	9
Validação de protocolo de posicionamento de recém-nascido em Unidade de Terapia Intensiva ( 38 )	2015 Brazil	Positioning of newborns in the Intensive Care Unit	Delphi Technique	5
Classificação de risco em pediatria: construção e validação de um guia para enfermeiros ( 26 )	2015 Brazil	Guide to risk stratification in pediatrics	Validation with experts	9
Protocolo de cuidados ao cliente com distúrbio respiratório: ferramenta para tomada de decisão aplicada à enfermagem ( 5 )	2014 Brazil	Nursing care for patients with respiratory disorder	Validation with experts	3
Validação das diretrizes gerais de comunicação do enfermeiro com o cego ( 39 )	2014 Brazil	Communication guidelines for the blind	Validation with experts	3
The Association for the Advancement of Wound Care (AAWC) venous and pressure ulcer guidelines(40)	2014 Canada	Prevention and management of pressure injuries and venous injuries	Validation with experts	32
Protocolo de acolhimento e atenção para usuários submetidos à endoscopia digestiva alta e seus acompanhantes ( 22 )	2014 Brazil	Reception and guidance for patients undergoing upper digestive endoscopy	Validation with experts	3
Clinical practice guidelines-Nursing management for pediatric patients with small bowel or multivisceral transplant(23)	2014 United States	Nursing care for pediatric patients with short bowel and multiple organ transplant	Validation with experts	4
Protocolo de cuidados para punção aspirativa por agulha fina de mama e tireoide ( 24 )	2014 Brazil	Care for fine needle aspiration of breast and thyroid	PCA	88 patientes 20 professionals
Protocolo de avaliação e classificação de risco de pacientes em unidade de emergência ( 31 )	2014 Brazil	Emergency risk stratification	Validation with experts	7

DISCUSSION

Validation of protocols is essential to ensure safety in using instruments in clinical practice when verifying that they achieve the proposed objectives and measure adequately and reliably what they propose to measure^(8,41). Validity can be assessed using different methods, according to the research objective and can be content, construct and related to a criterion⁽⁴¹⁾. Construct validity aims to identify the extent to which the measure reflects the theoretical construction of the phenomenon under analysis⁽⁸⁾, while criterion validity reflects the degree of prediction of measures in relation to the studied phenomenon⁽⁸⁾. Content validity associates abstract concepts and measurable indicators, i.e., how relevant each element of the instrument is and represents a particular construct^(8,41).

In nursing research, mainly content validity has been used, and this type of validation is based on instrument creation and assessment by experts (or judges) to decide if content is adequate and correct to what is proposed^(8,41).

The first stage of preparing the instrument is due to a bibliographic review. The instruments are often prepared by the authors following the theoretical framework of choice, as discussed in the following paragraphs. Literature review provides a common starting point and strengthens participants’ perception of an evidence-based process⁽⁴²⁾.

The second stage, on the other hand, consists of assessment of judges, which can be conducted in a simplified way, as in most of the articles analyzed in this review, or following different formal methods of consensus⁽⁴²⁾. Consensus is defined as the general consensus among group members, and does not need to be unanimous, as it allows the inclusion of different ideas and puts them into discussion⁽⁴³⁾. The methods for consensus among experts are also diverse in the literature, highlighting Delphi Technique, nominal groups, Consensus Development Conference, RAND/UCLA Appropriateness Method, among others^(42-43).

Delphi Technique aims to reach consensus among a group of experts in a reliable and systematic way, based on a series of well-outlined questionnaires and feedbacks. Its operation takes place through structured or semi-structured questionnaires, which can be sent by email to participants or applied in person in successive rounds. In the first round, the questionnaire contains instructions for completing and returning it, the objectives of the study, and the research question itself, being general and comprehensive. In the second round, the questionnaires are prepared based on the responses from the previous one (including the results obtained in the first round); they are added to the points where group consensus is still sought so that experts can reassess their responses based on the data provided^(44-45).

Among the advantages of the technique when applied at a distance are anonymity among participants, avoiding direct influence between information; access to geographically distant people; decreased transport and material costs; the possibility of individual and collective reflection on the topic; high number of participants possible; easy implementation applicable to a wide range of topics. The main disadvantages are creating the questionnaire; the possibility of bias that influences the group’s responses, abstaining from participants between rounds; and long performance time^(42,45).

An integrative review about using Delphi Technique in Brazilian studies⁽⁴⁴⁾ showed its use in the process of building and validating instruments in 60% of the analyzed articles. Another reflective study⁽⁴⁵⁾ highlights the great diversity of applications of the technique and different themes in Brazilian nursing research, while, abroad, the method is already widely disseminated and used.

However, it is noteworthy that such articles bring Delphi Technique as a method of validating instruments in general. Care protocols are rarely mentioned as assessment products. An integrative Brazilian review⁽⁷⁾ on the methodological path for construction and validation of nursing protocols, published in 2017, which focused on theses and dissertations dated from 2002 to 2014, does not bring Delphi Technique in its results. For comparison, in the articles analyzed in this review, Delphi Technique appears in articles published from 2015.

The same integrative review⁽⁷⁾ highlights PCA as the most widely used theoretical framework in the analyzed studies. PCA is a qualitative research approach that is characterized by the improvement of care practices from the introduction of innovations in such a context⁽⁴⁶⁾. PCA differs from other similar methods precisely because it includes professionals with expertise in care knowledge, as illustrated in the article included in this review⁽²⁴⁾, in which the protocol construction and validation process included 20 professionals.

It is important to note that all articles included in this review went through a validation process with experts, varying only the technique used for this - traditional, Delphi Technique and PCA, in this order of frequency.

The variability in the number of experts included in the analyzed studies is consistent with the results reported in scientific literature. Some authors indicate a minimum of five to six participants^(8,41), while others defend groups with a maximum number of participants, ranging from 12 to 20^(7,40-41). There is a dichotomy between the benefits of larger groups, due to the diversity of information and robustness of results or if above a certain number the reliability of the results decreases or remains the same. Delphi Technique allows including a larger number of participants. It can vary from 10 to hundreds, as long as the sample is representative^(41-42,45), and there must be a forecast of abstention rate between rounds, which can vary from 20 to 50%^(44-45).

The form of inclusion of participants is essential to guarantee reliability of results. The expert panel quality is more important than the size of the group. The heterogeneity of the group brings advantages and disadvantages to the consensus process. A diverse group achieves greater variability of ideas in relation to the topic. But there may also be more lack of consensus in relation to the same item⁽⁴²⁾, so that researchers should consider such aspects when defining the selection and inclusion criteria.

The definition of “expert” and its criteria is still divergent in scientific literature. Thus, selecting participants in the analyzed articles was also diverse. Although the concept is still not well established, criteria such as knowledge, experience and critical thinking skills in relation to the theme are points in common among different authors^(8,47-48). Fehring⁽⁴⁷⁾ proposed a scoring model for the criteria for defining experts to validate nursing diagnoses, being cited by two authors in this review^(6,28). Two authors cited Jasper^(13-14), who published an article in 1994 seeking to define the concept and criteria for an “expert” in nursing.

In Brazil, the Lattes Platform integrates the databases of curricula, research groups and institutions in a single information system. It is a tool aimed at several public and private institutions in the search for information about professionals and researchers from different fields. Thus, it was used as a search strategy for experts to validate the protocols proposed in six studies analyzed, based on keywords. Searching for participants from a national and integrated curriculum platform allows including highly qualified participants from different locations, guaranteeing heterogeneity of the sample and a validation process that includes the country’s regional differences.

Snowball sampling takes place from the indication of people who have characteristics common to the research interest⁽⁴⁵⁾, which can also be used as a method for selecting participants.

It should also be noted that the protocol scope can also influence with the definition of the number and profile of experts. National protocols must include national experts on the other hand, regional and local protocols must consider experts with knowledge of the local reality.

In relation to data collection instruments, the vast majority of articles used questionnaires formulated by the authors themselves, with variation between the criteria used. Two authors used Pasquali’s framework, which highlights behavioral criteria, objectivity, simplicity, clarity, relevance, precision, variety, modality, typicality, credibility, breadth and balance⁽⁸⁾. The others use criteria mainly for clarity, comprehensiveness and relevance, and variations, without indicating the source of the framework that guided the questionnaire creation.

Using standardized questionnaires is not a requirement for different validation methods. At Delphi Technique, for instance, the questionnaire is formulated based on the research question, preferably with closed questions of the Likert type, and space for judges’ considerations. Each round the questionnaire is changed, incorporating the results of the previous round, and discussing the points that did not reach consensus⁽⁴³⁾. The questionnaire must be focused, clear and precise, must contain instructions on how to fill it out and the content validation criteria that follow the methodological framework adopted by the author⁽⁴¹⁾.

As for standardized instruments, AGREE II stands out. AGREE II is a tool developed by international collaboration that aims to address the variability in the quality of clinical protocols and guidelines. It assesses methodological rigor and transparency in its development⁽³⁰⁾, being divided into six domains that assess scope and purpose, stakeholder involvement, development rigor, clarity of presentation, applicability and editorial independence. Each domain includes assessment items that use a Likert scale ranging from 1 (I strongly disagree) to 7 (I strongly agree), and considers assessment of 4 evaluators. The instrument is accompanied by a user manual with all instructions for completing and subsequent data analysis. It is a useful tool for validation studies, considering that it is already adapted for use in Brazil.

The fact that AGREE II has been available since 2003 in international literature and 2009 in Brazilian Portuguese raises the authors’ interest with regard to its small use in nursing research in the studied sample. There is a tendency to use instruments created by the authors themselves, with criteria little addressed in the articles studied.

The CVI is a formula widely used for data analysis to validate the content of instruments in health, and especially in nursing⁽⁴⁹⁾. CVI can be defined as the proportion or percentage of judges who are in consensus on relevance of the item in relation to the construct^(41,49). CVI assesses the items individually (I-CVI) or the instrument as a whole (S-CVI), on a Likert scale preferably of 4 points, to avoid the occurrence of neutral assessments. There is often variation of 1=not relevant, 2=not very relevant, 3=something relevant, 4=very relevant⁽⁴⁹⁾. CVI is calculated from the formula below (Figure 2):

Figure 2
Content Validity Index formula

CVI ranges from 0 to 1.0, with some authors recommending that if the sample consists of five experts or less, everyone should agree on the item (CVI=1.0), and if the sample is larger, a CVI ≥ 0.78 is acceptable^(41,49). There was variance in consensus rates used in the analyzed articles (60 to 100%), with a tendency to adopt values between 70 to 80%. This shows, despite its frequent use, there is still no consensus about the ideal values to be adopted, and the description of data analysis is not very detailed in the studies assessed.

Of the advantages of using CVI, we have ease of calculating and understanding, well acceptance and dissemination in scientific literature for validation of instruments. Concerning disadvantages, the main one mentioned in the articles is in relation to Odds Ratio, not provided for in the formula⁽⁴⁹⁾. Thus, some authors have suggested new formulas and alternatives for such.

Several other formulas have been proposed, including the Kappa Coefficient, which can be used alone or with CVI. The formula consists of the ratio of the number of times judges agree with the maximum times judges could agree (both corrected by consensus due to chance). This formula is useful when data is categorical and on a nominal scale, ranging from -1 (total absence of consensus) to 1 (total consensus)⁽⁴¹⁾.

It was observed in the selection of articles that it is not common in international literature to publish articles about protocol validation process, so that this phase is described in reading the protocol in full. This fact can be a justification for the composition of a sample mainly composed of Brazilian studies in this study, limiting, therefore, comparison with other realities.

Therefore, the methodological path for validation of nursing protocols is still variable in the literature, so that the data presented here do not allow us to infer which would be the ideal method. However, it brings information about what else has been used in the current literature, providing support for the authors’ decision before starting a validation process.

Study limitations

A limitation is related to the focus of nursing-only articles, causing loss of data about protocol validation and/or comparison with protocol validation methods from other fields of health knowledge. Search and selection of articles was performed by a researcher, increasing the risk of selection bias.

Contributions to nursing

This study contributes to nursing knowledge in order to guide the decision process in choosing the method of content validation of care protocols. Such a process has become increasingly common in nursing research in recent years. Articles with subsidies for the discussion of rigorous methods for this aim to facilitate their conduct and, consequently, the implementation of these technologies in clinical practice, aiming to ensure safe and evidence-based care.

Moreover, this integrative review highlights that expert assessment has been conducted by nursing, mostly, in a simplified way, with limitations in terms of describing the methodological rigor adopted. However, more consolidated methods of consensus, such as Delphi Technique, have become more common, and can be a promising strategy in validating care protocols.

EBP, as previously mentioned, has several advantages, at individual and collective levels, with nurses and their team central to their implementation process.

Dissemination of results of nursing research worldwide, especially with regard to implementation of new care technologies (among them, protocols) aimed at improving care practice, bring professional visibility and strengthening the role of nurses in multidisciplinary teams, encouraging their insertion in the different health care settings.

CONCLUSION

It was evidenced that content validation of the nursing protocols assessed in this review, in the last 5 years, was carried out by expert assessment, in a simplified way or using consensus methods such as Delphi Technique or PCA. The instruments used were variable, and most of them were created by the authors, following some theoretical framework or not, using standardized instruments, such as AGREE II, only in two articles. The criteria for choosing experts were also variable, including selection (search for resumes on the Lattes Platform (Curriculum Lattes is a resume developed according to Platforma Lattes (Lattes Platform) standards), even snowball sampling), and inclusion and exclusion criteria (with curriculum scoring systems, or not). In relation to analysis of results, CVI was the most used formula, despite its advantages and disadvantages, and can be added to other forms of analysis.

The articles were assessed in relation to their level of evidence according to Melnik and Fineoult-Overholt, keeping the lowest level because they are based on the opinion of experts in the field. The methods of creating protocols emerged in analysis of results. Literature review was highlighted in all articles, ranging from simple, integrative and systematic review, with emphasis on scoping reviews; and the combination of literature review and other forms of data collection such as interviews and groups.

Protocols created from robust evidence and assessed by experts on the subject are valuable and strong tools for use in clinical practice, considering that they combine science and the experience of those who use them on a daily basis. It is one of the EBP principles, when adapting the data found in nursing literature to the local reality, facilitating its implementation. Concluding, more studies related to validation of protocols in other fields of health besides nursing are suggested, in order to allow more complete comparisons and conclusions to standardize the methods to be used.

REFERENCES

• ¹
  Jylhä V, Oikarainen A, Perälä M-L, Holopainen A. Facilitating evidence-based practice in nursing and midwifery in the WHO European Region [Internet]. Denmark; 2017 [cited 2019 Dec 3]. Available from: http://www.euro.who.int/pubrequest
  » http://www.euro.who.int/pubrequest
• ²
  Mackey A, Bassendowski S. The history of evidence-based practice in nursing education and practice. J Prof Nurs. 2017;33(1):51-5. doi: 10.1016/j.profnurs.2016.05.009
  » https://doi.org/10.1016/j.profnurs.2016.05.009
• ³
  Pimenta CAM, Pastana ICASS, Sichieri K, Solha RKT, Souza W. Guia para construção de protocolos assistenciais de enfermagem [Internet]. São Paulo: COREN-SP; 2015 [cited 2019 May 14]. Available from: https://portal.coren-sp.gov.br/sites/default/files/Protocolo-web.pdf
  » https://portal.coren-sp.gov.br/sites/default/files/Protocolo-web.pdf
• ⁴
  Gomes ATL, Alves KYA, Bezerril MS, Rodrigues CCFM, Ferreira Jr MA, Santos VEP. Validation of graphic protocols to evaluate the safety of polytrauma patients. Acta Paul Enferm. 2018;31(5):504-17. doi: 10.1590/1982-0194201800071
  » https://doi.org/10.1590/1982-0194201800071
• ⁵
  Paes GO, Mello ECP, Leite JL, Mesquita MGR, Oliveira FT, Carvalho SM. Care protocol for clients with respiratory disorder: tool for decision making in nursing. Esc Anna Nery. 2014;18(2):303-10. doi: 10.5935/1414-8145.20140044
  » https://doi.org/10.5935/1414-8145.20140044
• ⁶
  Medeiros SG, Lima Neto AV, Saraiva COPO, Barbosa ML, Santos VEP. Safety evaluation in vaccine care: elaborating and validating a protocol. Acta Paul Enferm. 2019;32(1):53-64. doi: 10.1590/1982-0194201900008
  » https://doi.org/10.1590/1982-0194201900008
• ⁷
  Catunda HLO, Bernardo EBR, Vasconcelos CTM, Moura ERF, Pinheiro AKB, Aquino PS. Methodological approach in nursing research for constructing and validating protocols. Texto Contexto Enferm. 2017;26(2). doi: 10.1590/0104-07072017000650016
  » https://doi.org/10.1590/0104-07072017000650016
• ⁸
  Medeiros RKS, Ferreira Junior MA, Pinto DPS, Vitor AF, Santos VEP, Barrichello E. Modelo de validação de conteúdo de Pasquali nas pesquisas em Enfermagem. Rev Enferm Ref. 2015;(4):127-35. doi: 10.12707/RIV14009
  » https://doi.org/10.12707/RIV14009
• ⁹
  Whittemore R, Knafl K. The integrative review: updated methodology. J Adv Nurs. 2005;52(5):546-53. doi: 10.1111/j.1365-2648.2005.03621.x
  » https://doi.org/10.1111/j.1365-2648.2005.03621.x
• ¹⁰
  Mendes KDS, Silveira RCCP, Galvão CM. Revisão integrativa: método de pesquisa para a incorporação de evidências na saúde e na enfermagem. Texto Contexto Enferm. 2008;17(4):758-64. doi: 10.1590/S0104-07072008000400018
  » https://doi.org/10.1590/S0104-07072008000400018
• ¹¹
  Souza MT, Silva MD, Carvalho R. Revisão integrativa: o que é e como fazer. Einstein. 2010;8(1):102-6. doi: 10.1590/S0104-07072008000400018
  » https://doi.org/10.1590/S0104-07072008000400018
• ¹²
  Lemos CS, Poveda VB, Peniche ACG. Construction and validation of a nursing care protocol in anesthesia. Rev Latino-Am Enfermagem. 2017;25:e2952. doi: 10.1590/1518-8345.2143.2952
  » https://doi.org/10.1590/1518-8345.2143.2952
• ¹³
  Santos EC, Oliveira IC, Feijão AR. Validation of a nursing care protocol for patients undergoing palliative care. ACTA Paul Enferm. 2016;29(4):363-73. doi: 10.1590/1982-0194201600051
  » https://doi.org/10.1590/1982-0194201600051
• ¹⁴
  Sousa DMN, Chagas ACMA, Vasconcelos CTM, Stein AT, Oria MOB. Development of a clinical protocol for detection of cervical cancer precursor lesions. Rev Latino-Am Enfermagem. 2018;26. doi: 10.1590/1518-8345.2340.2999
  » https://doi.org/10.1590/1518-8345.2340.2999
• ¹⁵
  Gomes PPS, Lima FET, Araujo PR, Oliveira MR, Brito EGFM, Carneiro JL, et al. Water balance in pediatric nephrology: construction of a Standard Operating Procedure. Rev Bras Enferm. 2018;71:1404-11. doi: 10.1590/0034-7167-2017-0045
  » https://doi.org/10.1590/0034-7167-2017-0045
• ¹⁶
  Vieta E, Garriga M, Cardete L, Bernardo M, Lombrana M, Blanch J, et al. Protocol for the management of psychiatric patients with psychomotor agitation. BMC Psychiatry. 2017;17. Available from: doi: 10.1186/s12888-017-1490-0
  » https://doi.org/10.1186/s12888-017-1490-0
• ¹⁷
  Zakhary SA, Davey C, Bari R, Bean J, Reber T, Gallagher K, et al. The development and content validation of a multidisciplinary, evidencebased wound infection prevention and treatment guideline. Ostomy Wound Manage [Internet]. 2017 [cited 2019 Dec 9];63(11):18-29. Available from: https://pubmed.ncbi.nlm.nih.gov/29166260/
  » https://pubmed.ncbi.nlm.nih.gov/29166260/
• ¹⁸
  Baixinho CRSL, Dixe MACR, Henriques MAP. Falls in long-term care institutions for elderly people: protocol validation. Rev Bras Enferm. 2017;70(4):773-9. doi: 10.1590/0034-7167-2017-0109
  » https://doi.org/10.1590/0034-7167-2017-0109
• ¹⁹
  Paim AE, Nascimento ERPD, Bertoncello KCG, Sifroni KG, Salum NC, Nascimento KCD. Validation of an instrument regarding nursing intervention in patients in vasoactive therapy. Rev Bras Enferm. 2017;70(3):453-60. doi: 10.1590/0034-7167-2016-0254
  » https://doi.org/10.1590/0034-7167-2016-0254
• ²⁰
  Baixinho CRL, Dixe MDA, Henriques MA. Managing the environment to prevent falls in the institutionalized elderly: protocol validation. Index Enferm [Internet]. 2018 [cited 2019 Dec 9];27(1-2):72-6.. Available from: http://scielo.isciii.es/scielo.php?script=sci_arttext&pid=S1132-12962018000100015&lng=es&nrm=iso&tlng=es
  » http://scielo.isciii.es/scielo.php?script=sci_arttext&pid=S1132-12962018000100015&lng=es&nrm=iso&tlng=es
• ²¹
  Li L, Wang Q, Wang J, Liu K, Wang P, Li X, et al. Development, validation and application of a ventilator-associated pneumonia prevention checklist in a single cardiac surgery centre. Intensive Crit Care Nurs [Internet]. 2018;49:58-64. doi: 10.1016/j.iccn.2017.10.002
  » https://doi.org/10.1016/j.iccn.2017.10.002
• ²²
  Selhorst IS, Bub MB, Girondi JB. Protocol for embracement and attention to users that underwent upper gastrointestinal endoscopy and persons accompanying them. Rev Bras Enferm. 2014;67(4):575-80. doi: 10.1590/0034-7167.2014670412
  » https://doi.org/10.1590/0034-7167.2014670412
• ²³
  Bodeur C, Aucoin J, Johnson R, Garrison K, Summers A, Schutz K, et al. Clinical practice guidelines-Nursing management for pediatric patients with small bowel or multivisceral transplant. J Spec Pediatr Nurs. 2014;19(1):90-100. doi: 10.1111/jspn.12056
  » https://doi.org/10.1111/jspn.12056
• ²⁴
  Rosini I, Salum NC. Care protocol for fine-needle aspiration biopsy of breast and thyroid. Texto Contexto Enferm. 2014;23(4):1059-67. doi: 10.1590/0104-07072014003520012
  » https://doi.org/10.1590/0104-07072014003520012
• ²⁵
  Pedrosa KKA, Oliveira SA, Machado RC. Validation of a care protocol for the septic patient in the Intensive Care Unit. Rev Bras Enferm. 2018;71(3):1106-14. doi: 10.1590/0034-7167-2017-0312
  » https://doi.org/10.1590/0034-7167-2017-0312
• ²⁶
  Veras JEGLF, Joventino ES, Coutinho JFV, Lima FET, Rodrigues AP, Ximenes LB. Risk classification in pediatrics: development and validation of a guide for nurses. Rev Bras Enferm. 2015;68(5):630-9. doi: 10.1590/0034-7167.2015680521p
  » https://doi.org/10.1590/0034-7167.2015680521p
• ²⁷
  Boeckmann LMM, Rodrigues MCS. Adaptation and validation of a surgical safety checklist in the cesarean delivery. Texto Contexto Enferm. 2018;27(3). doi: 10.1590/0104-070720180002780017
  » https://doi.org/10.1590/0104-070720180002780017
• ²⁸
  Magalhaes FJ, Teixeira Lima FE, Almeida PC, Ximenes LB, Pessoa Chaves CM. Care protocols with risk classification in pediatrics: interobserver reliability. Acta Paul Enferm. 2017;30(3):262-70. doi: 10.1590/1982-0194201700040
  » https://doi.org/10.1590/1982-0194201700040
• ²⁹
  Machado RC, Girones P, Souza AR, Moreira RSL, Jakitsch CB von, Branco JNR. Nursing care protocol for patients with a ventricular assist device. Rev Bras Enferm. 2017;70(2):335-41. doi: 10.1590/0034-7167-2016-0363
  » https://doi.org/10.1590/0034-7167-2016-0363
• ³⁰
  AGREE Next Steps Consortium. The AGREE II Instrument [Internet]. 2009 [cited 2017 Aug 7]. Available from: http://www.agreetrust.org/wpcontent/uploads/2013/06/AGREE_II_Brazilian_Portuguese.pdf
  » http://www.agreetrust.org/wpcontent/uploads/2013/06/AGREE_II_Brazilian_Portuguese.pdf
• ³¹
  Silva MFN, Oliveira GN, Pergola-Marconato AM, Marconato RS, Bargas EB, Araujo IEM. Assessment and risk classification protocol for patients in emergency units. Rev Latino-Am Enfermagem. 2014;22(2):218-25. doi: 10.1590/0104-1169.3172.2405
  » https://doi.org/10.1590/0104-1169.3172.2405
• ³²
  Lima VCGS, Queluci GC, Brandao ES, Silva CPG, Oliveira FT. Nursing care after coronary transluminal angioplasty: Protocol validation. Enferm Glob. 2019;18(2):374-409. doi: 10.6018/eglobal.18.2.338831
  » https://doi.org/10.6018/eglobal.18.2.338831
• ³³
  Cardoso ASF, Muller S, Echer IC, Rabelo-Silva ER, Boni FG, Ribeiro AS. Elaboration and validation of a drug administration checklist for patients in research protocols. Rev Gaucha Enferm. 2019;40(spe). doi: 10.1590/1983-1447.2019.20180311
  » https://doi.org/10.1590/1983-1447.2019.20180311
• ³⁴
  Cunha DR, Salome GM, Massahud Jr MR, Mendes B, Ferreira LM. Development and validation of an algorithm for laser application in wound treatment. Rev Latino-Am Enfermagem. 2017;25(e2955). doi: 10.1590/1518-8345.1998.2955
  » https://doi.org/10.1590/1518-8345.1998.2955
• ³⁵
  Aires M, Pizzol FLFD, Mocellin D, Rosset I, Morais EP de, Paskulin LMG. Cross-cultural adaptation of the Filial Responsibility protocol for use in Brazil. Rev Bras Enferm. 2017;70(6):1268-76. doi: 10.1590/0034-7167-2016-0479
  » https://doi.org/10.1590/0034-7167-2016-0479
• ³⁶
  Alves VC, Freitas WCJ, Ramos JS, Chagas SRG, Azevedo C, Mata LRF. Actions of the fall prevention protocol: mapping with the classification of nursing interventions. Rev Latino-Am Enfermagem. 2017;25(e2986). doi: 10.1590/1518-8345.2394.2986
  » https://doi.org/10.1590/1518-8345.2394.2986
• ³⁷
  Fernandes BSM, Reis IA, Pagano AS, Cecilio SG, Torres HDC. Development, validation and cultural adaptation of the COMPASSO protocol: adherence to self-care in diabetes. Acta Paul Enferm. 2016;29(4):421-9. doi: 10.1590/1982-0194201600058
  » https://doi.org/10.1590/1982-0194201600058
• ³⁸
  Toso BRGO, Viera CS, Valter JM, Delatore S, Barreto GMS. Validation of newborn positioning protocol in Intensive Care Unit. Rev Bras Enferm [Internet]. 2015;68(6):1147-53. doi: 10.1590/0034-7167.2015680621i
  » https://doi.org/10.1590/0034-7167.2015680621i
• ³⁹
  Pagliuca LM, Macedo-Costa KN, Reboucas CB, Almeida PC, Sampaio AF. Validation of the general guidelines of communication between the nurse and the blind. Rev Bras Enferm. 2014;67(5):715-21. doi: 10.1590/0034-7167.2014670507
  » https://doi.org/10.1590/0034-7167.2014670507
• ⁴⁰
  Bolton LL, Girolami S, Corbett L, Van Rijswijk L. The Association for the Advancement of Wound Care (AAWC) venous and pressure ulcer guidelines. Ostomy Wound Manag [Internet]. 2014 [cited 2019 Dec 10];60(11):24-66. Available from: https://pubmed.ncbi.nlm.nih.gov/25380098/
  » https://pubmed.ncbi.nlm.nih.gov/25380098/
• ⁴¹
  Alexandre NMC, Coluci MZO. Validade de conteúdo nos processos de construção de medidas. Cienc Saude Coletiva. 2011;16(7):3061-8. doi: 10.1590/S1413-81232011000800006
  » https://doi.org/10.1590/S1413-81232011000800006
• ⁴²
  Nair R, Aggarwal R, Khanna D. Methods of Formal Consensus in Classification/Diagnostic Criteria and Guideline Development. Semin Arthritis Rheum. 2011;41:95-195. doi: 10.1016/j.semarthrit.2010.12.001
  » https://doi.org/10.1016/j.semarthrit.2010.12.001
• ⁴³
  Bourree F, Michel P, Salmi LR. Consensus methods: review of original methods and their main alternatives used in public health. Rev Epidemiol Sante Publique. 2008;56(6):e13-21. doi: 10.1016/j.respe.2008.10.005
  » https://doi.org/10.1016/j.respe.2008.10.005
• ⁴⁴
  Revoredo LS, Maia RS, Torres GV, Maia EMC. O uso da técnica delphi em saúde: uma revisão integrativa de estudos brasileiros [Internet]. 2015 [cited 2019 Nov 11];22(2):16-21. Available from: http://www.cienciasdasaude.famerp.br/index.php/racs/article/view/136/61
  » http://www.cienciasdasaude.famerp.br/index.php/racs/article/view/136/61
• ⁴⁵
  Scarparo AF, Laus AM, Azevedo ALCS, Freitas MRI, Gabriel CS, Chaves LDP. Reflexões sobre o uso da Técnica Delphi em pesquisas na enfermagem. Rev Rene [Internet]. 2012 [cited 2019 Nov 11];13(1):242-51. Available from: http://www.periodicos.ufc.br/rene/article/view/3803
  » http://www.periodicos.ufc.br/rene/article/view/3803
• ⁴⁶
  Trentini M, Paim L, Silva DMGV da. O método da pesquisa convergente assistencial e sua aplicação na prática de enfermagem. Texto Contexto Enferm. 2017;26(4):e1450017. doi: 10.1590/0104-07072017001450017
  » https://doi.org/10.1590/0104-07072017001450017
• ⁴⁷
  Melo R, Moreira R, Fontenele F, Aguiar A, Joventino E, Carvalho E. Criteria for selection of experts for validation studies of nursing phenomena. Rev Rene. 2011;12(2):424-31. doi: 10.15253/revrene.v12i2.174
  » https://doi.org/10.15253/revrene.v12i2.174
• ⁴⁸
  Jasper MA. Expert: a discussion of the implications of the concept as used in nursing. J Adv Nurs. 1994;20(4):769-76. doi: 10.1046/j.1365-2648.1994.20040769.x
  » https://doi.org/10.1046/j.1365-2648.1994.20040769.x
• ⁴⁹
  Polit DF, Beck CT. The content validity index: are you sure you know what’s being reported? critique and recommendations. Res Nurs Health. 2006;29:489-97. doi: 10.1002/nur.20147
  » https://doi.org/10.1002/nur.20147

Edited by

EDITOR IN CHIEF: Antonio José De Almeida Filho

ASSOCIATE EDITOR: Priscilla Broca

Publication Dates

• Publication in this collection
  19 Oct 2020
• Date of issue
  2020

History

• Received
  03 Apr 2020
• Accepted
  12 June 2020