Validation of criteria for nosocomial use of amikacin in Brazil with the Delphi technique

ORIGINAL PAPERS

Validation of criteria for nosocomial use of amikacin in Brazil with the Delphi technique

Noblat L.^I; Oliveira M.G.G.^I; Santos-Jesus R.^I; Noblat A.C.B.^I; Badaró R.^{I, II}

^IFederal University of Bahia; Salvador, BA, Brazil

^IIUniversity of California San Diego; San Diego, CA, USA

^{Address for correspondence}

ABSTRACT

The Delphi technique has been used since the 1950s to collect the opinions of experts; to gauge their indications, and in some instances, to develop a consensus. This systematic collection and aggregation of informed judgments from a group of experts on specific questions or issues is a highly efficient and cost-effective means to establish guidelines and policies, when compared to other strategies, such as committee meetings or personal interviews.

OBJECTIVE: Examine the content validation process of the proposed criteria of the American Society of Health System Pharmacists (ASHP) for amikacin use in hospital settings.

MATERIAL AND METHODS: The Delphi technique was applied using the proposed ASHP criteria questionnaire containing 102 specific questions related to the nosocomial use of amikacin by individual patients. The questionnaire contained six groups of questions: 1) Identification and basic demographic data, 2) Relevant data for the use of amikacin, 3) Justification of its usage, 4) Critical parameters of amikacin use, 5) Complications, 6) Measurement of results. Eight hospital specialist medical doctors were selected, including five in the area of infectious diseases, one surgeon, one nephrologist and one in critical care medicine. The questionnaire was e-mailed to the doctors and they were asked for their opinion about the appropriateness of the questions. They were to say whether the general concept seemed totally or partially adequate to the proposed process, what grade (0 to 10) they would give to each section, and if there were any perceived deficiencies, they could add, omit or modify individual questions. A second questionnaire containing the questions for which there had been no consensus based on the answers to the previous one was re-sent to the participants for consolidation.

RESULTS: Feedback revealed an agreement of 75% concerning the utility and appropriateness of sections 1 and 2. The section about the justification of amikacin usage was agreed on by 50%. There was a total agreement of 62% for the critical parameters of amikacin use, and a partial agreement of 37%. The complication of usage of the questionnaire was agreed upon by 50% of the participants, and positive measurement of the results was totally agreed on by 62%, and partially by 37%. The overall score for the questionnaire was 8.77 ± 0.25.

CONCLUSION: The usage criteria for amikacin recommended by ASHP were validated by the Delphi technique for utilization in Brazilian hospital settings. The Delphi technique applied to validate a questionnaire instrument for monitoring the correct use of a specific strategic antibiotic indicated for the treatment and prophylaxis of serious antibiotic-resistant Gram-negative bacteria, proved to be a reliable and simple tool for designing guidelines and a consensus document for hospital use of antibiotics.

Key Words: Delphi technique, content validation, drug use utilization, amikacin, consensus technique.

Increased prevalence of multiresistant Enterobacteriaceae (MRE) is often due to the indiscriminate use of antibiotics; though dissemination of these organisms by inappropriate hygienic measures is also a contributing factor [1-4]. Temporary restrictive antibiotic policy, TRAP, uses an antibiotic control form containing a list of freely-available and of controlled antibiotics [1-5]. Controlled antibiotics are available for use only with prior approval by a consulting microbiologist or an infectious-diseases specialist. Antibiotics not on this list are not available in most settings to prevent dissemination of MRE. After implementation of TRAP, the incidence of MRE decreased to <50% of the level before intervention. Even prior to implementation of TRAP, no new cases were detected from clinical samples. This was most likely a result of the screening procedures that were implemented, since intestinal colonization with MRE precedes detection of the same strains in clinical samples by about 10 days.

The Council for Appropriate and Rational Antibiotic Therapy (CARAT) is an independent, multidisciplinary panel of healthcare professionals, clinicians and scientists, established to advocate the appropriate and accurate use of antibiotics. The CARAT has developed seven criteria to assist healthcare providers in selecting the most appropriate and accurate treatment regimens: Evidence-based results, Therapeutic benefits, Safety, Cost-effectiveness, Optimal drug dose and duration, and Shorter-course, more aggressive therapy [6].

The Delphi technique has gained extensive popularity across many scientific disciplines as a method of inquiry [7-9]. There are numerous guidelines for using the Delphi technique, as well as many offshoots of this method [10-12].

Measurement Validity: Concepts and Classification

Validation is a procedure for estimating the level of compliance of a model or measurement with reality [13]. There are several different forms and methods for evaluating validity. The classification proposed by Champagne et al. [14], Polit & Hungler [15] and Contandriopoulos et al. [16] includes three different strategies for establishing measurement validity: content validity, criteria validity and construct validity. We prioritized this classification, especially content validity, because it has been widely applied in social welfare and healthcare programs.

Content validity is a judgment concerning how well the items selected to measure a theoretical construction represent all the dimensions of the concept being measured. [14-16]. The content validity of a variable can be finely tuned by breaking down its concept into as many dimensions as possible. The Specialist Consensus Technique can be used in this way [14]. The two main techniques for determining consensus are: The Delphi Technique and the Nominal Group Technique (NGT). We opted for the Delphi Technique.

Material and Method

Criteria. The instrument for validation of amikacin usage was constructed utilizing the criteria for use of amikacin established by the ASHP in 1989 [17-19]. A questionnaire with 102 variables, shown in Figure 1, was prepared. Its content validation process utilized the Delphi technique, which consisted of: an initial questionnaire (round one), which served as an idea-generation strategy to uncover issues pertaining to the topic under study. To do this, the respondents, referred to as panelists, were asked to put forward as many relevant issues as possible in round one. Once analyzed, these responses served as a springboard for the rest of the Delphi process [19]. The feedback obtained in round one resulted in the formulation of a second questionnaire and again opinions were requested. Normally, in subsequent rounds, panel members are provided with their own responses as well as those of the other panelists, and he or she is asked to reconsider and (if he or she wishes) change the response in the light of other panelists' responses. This continues in subsequent rounds until consensus is reached [20]. This process is best described as multi-stage, where each stage builds on the results of the previous one. We used two rounds of questionnaires that were sent by e-mail to eight selected medical specialists who are involved daily with hospital antibiotic prescriptions. The following specialists were included: five in the area of infectious diseases, one surgeon, one nephrologist and one critical care medicine doctor. The questionnaire was e-mailed to these doctors and we initially asked them to give their opinion about the appropriateness or not of the questions, grade their degree of adequacy using a score system ranging from 0 to 10, and give specific suggestions for exclusion, addition or modification of the items in the questionnaire. A second questionnaire containing the questions for which no consensus was reached resulting from the answers to the previous one was re-sent to the participants for consolidation. All participants signed the inform consent prior to receiving the questionnaire. Each criterion of amikacin usage resulted in one or more questions that composed six sections of data. Section 1 included identification and demographic data, section 2 included data related to the usage of amikacin, and sections 3, 4, 5 and 6 were specific for each criterion of amikacin usage: reason for use, critical indicators of the process, complications and measurements of the results. In order to validate our questionnaire, we designed an additional one for specific evaluation (Figure 2). In this additional questionnaire, the participants had to indicate if the section was totally adequate, partially adequate or inadequate. Scoring ranged from 0 to 10. At the end of the first round, a consolidated result of the items for which there was no consensus was re-mailed to the participants. Consensus was indicated whenever the section obtained an evaluation score greater than seven, and the specialist did not consider it inadequate. This study was conducted from July to December 2004.

Results

All eight specialists agreed to take part in the study and completed the two rounds of the questionnaire. Consensus was obtained after the second round of evaluations. None of the specialists disagreed with the overall concept of the instrument. As noted in Figure 3, in sections 1 and 2, an agreement was achieved by 6 out of 8 (75%) of the specialists. A reason to use amikacin had the concordance of half of the specialists. Critical indicators for use of amikacin were considered absolutely adequate by five of them (62%). Also, the result measurements of uses of amikacin were absolutely agreed on by 62% of the specialists and 37% considered them partially adequate. Complications were agreed on by 50%. Table 1 presents the average score attributed by each specialist.

Discussion

Overuse and misuse of antibiotics has contributed to an increase in bacterial resistance patterns, which may differ by locality [2;21-24]. The Delphi technique provides an opportunity for experts (panelists) to communicate their opinions and knowledge anonymously about a complex problem, to see how their evaluation of the issue aligns with those of others, and to change their opinion, if desired, after reconsideration of the team's findings. The work continues over a series of interactive rounds, until consensus or stability is reached about the problem at hand [7]. This technique was utilized to validate the criteria to use amikacin in a university hospital in Bahia, Brazil

These criteria are designed to help guide healthcare practitioners in the use of antibiotics whenever they are appropriately indicated. We addressed issues of appropriate criteria for amikacin use in hospitalized patients. The identification section's purpose is to retrieve data that could later be correlated with gender, locality or specific patient group. The doctors in our study suggested to include an open question about initial diagnosis, final diagnosis, patient's weight and information about surgical procedure. The variable race was not replaced by pregnancy and breastfeeding, as suggested, because it would not make any sense.

In section 2, several suggestions were made, such as revision of the definition and criteria of nosocomial infection, inclusion of topographic localization of the infection, specification of the reason for the therapeutic choice, and if it was based on the susceptibility pattern of the bacteria isolated in the hospital. In section 3, the main suggestion concerned the quality control issues of the microbiology laboratory. Also, information about the source of the culture and its relation to the topographic area of infection was suggested. Length of hospitalization was initially thought to be important information requested as justification for the use of amikacin by one of the specialists. However, it was discarded by all panelists in the second round. In section 4, there was an agreement that the duration of therapy should be greater than 7 days instead of 14 days. The reason for that was the knowledge that 90% of infections treated with amikacin resolve within 7 days. The suggestion to replace the question about hemodialysis for dialysis was accepted because a patient can take amikacin in situations other than hemodialysis procedures. The requested inclusion of a question about anti-pyretic usage was incorporated as an observation in the question related to fever reduction, as it could influence the results. In section 5, there was a consensus about the inclusion of the amikacin dosage schedule, informing if there was a single daily dosing or with intervals of 12 hours. This was incorporated into section 2 as an open question.

In section 6, substitution of the term `source of infection' for `topography of the infection' was requested; definitions of clinical cure and bacteriological cure were also requested. The average grade given by the specialists for each section fulfilled the consensus criteria established by the investigator. Only two changes in the criteria were proposed by the specialists: altering the duration of the treatment from 14 to 7 days, plus inclusion of a single daily dosing of amikacin. These proposals reflect the updating of the criteria indicated by the ASHP; they also follow recommendations that the criteria for drug use must be up to date and should reflect current medical practice standards [25].

This study had some limitations. Although the database search was extensive, we may have overlooked one or more relevant questions due to a lack of clinical data from the hospital to justify specific needs for particular information. Another limitation is that the Delphi group consisted of experts (academic and practitioner) of various professional specialties who are routinely prescribing antibiotics. Their familiarity, or not, with current opinion, as expressed in the published literature, could have influenced the agreement between the determinants identified in such literature and their own opinion, producing more apparent agreement than there was in reality. This is a particular problem in the Delphi technique, because it selects specialists to construct guidelines and specific recommendations. There is a paucity of research that provides follow-up data to see if Delphi findings are substantiated in real situations, regardless of the area under investigation. In Bowles's (1999) review of the use of the Delphi technique in nursing and allied health literature, only four such studies were identified [19]. This corroborates the fact that there has been little written on the enhancement or sequential validation findings from these expert panels.

The Delphi technique as an exploratory research method provides a platform for future research, but it is only one step in knowledge development, and its findings have certain limitations [11,26,27]. Delphi studies are actually research exercises held outside the context of real life. They have the potential to provide valuable information, yet the fact that few researchers have taken further steps to support or refine their findings is a threat to the applicability, or external validity, of the results. Strengths of the method may be viewed as limitations by some, depending on the person's overall view of credible knowledge. For example, anonymity and release from peer pressure could lead to lack of responsibility and accountability for responses [28]. Defining experts can also be problematic and arbitrary [27]. In Keeney's et al. (2001) critical review of the Delphi method, the definition of expert ranges from `informed individual' to `specialist in the field' and to `someone who has knowledge about a specific subject' [7].

Although increased bacterial resistance to antibiotics has several causes, two key factors are the overuse and misuse of antibiotics [6;29-32]. Antibiotics are frequently prescribed for indications in which their use is not warranted, or an incorrect or suboptimal dose of antibiotic is prescribed. Indeed, to transfer the responsibility to a panel of experts, or even better, to those in the hospital who are routinely dealing with antibiotic use is possible and can become a strong policing mechanism and recommendation when everyone participates in the process to construct the instrument. The criteria for usage of amikacin recommended by the ASHP were validated by the Delphi technique for utilization in Brazilian hospital settings. Changes in the duration of the treatment from 14 to 7 days, and the inclusion of a single daily dosing of amikacin were proposed by the eight specialists. The ASHP recommendation for the use of amikacin in nosocomial settings should be validated for each individual hospital and adapted to its own reality.

Acknowledgements

We thank to Dr. Edmundo Machado Ferray, Edwal Rodrigues, Flávia Rossi, Carlos Ernesto Satrling, Carlos Brites, Criatiana Nascimento, Marco Antonio Silveira, Júlio César Braga.

Received on 8 January 2006; revised 31 May 2006.

• 1. Cohen M.L. Epidemiology of drug resistance: implications for a post-antimicrobial era. Science 1992;257(5073):1050-5.
• 2. Cross A.S., Opal S., Kopecko D.J. Progressive increase in antibiotic resistance of Gram-negative bacterial isolates. Walter Reed Hospital, 1976 to 1980: specific analysis of gentamicin, tobramycin, and amikacin resistance. Arch Intern Med 1983;143(11):2075-80.
• 3. Hobson R.P., MacKenzie F.M., Gould I.M. An outbreak of multiply-resistant Klebsiella pneumoniae in the Grampian region of Scotland. J Hosp Infect 1996;33(4):249-62.
• 4. Holmberg S.D., Solomon S.L., Blake P.A. Health and economic impacts of antimicrobial resistance. Rev Infect Dis 1987;9(6):1065-78.
• 5. Leverstein-van Hall M.A., Fluit A.C., Blok H.E., et al. Control of nosocomial multiresistant Enterobacteriaceae using a temporary restrictive antibiotic agent policy. Eur J Clin Microbiol Infect Dis 2001;20(11):785-91.
• 6. Slama T.G., Amin A., Brunton S.A., et al. A clinician's guide to the appropriate and accurate use of antibiotics: the Council for Appropriate and Rational Antibiotic Therapy (CARAT) criteria. Am J Med 2005;118(Suppl 7A):1S-6S.
• 7. Keeney S., Hasson F., McKenna H.P. A critical review of the Delphi technique as a research methodology for nursing. Int J Nurs Stud 2001;38(2):195-200.
• 8. Kennedy H.P. Enhancing Delphi research: methods and results. J Adv Nurs 2004;45(5):504-11.
• 9. McKenna H.P. The Delphi technique: a worthwhile research approach for nursing? J Adv Nurs 1994;19(6):1221-5.
• 10. Greatorex J., Dexter T. An accessible analytical approach for investigating what happens between the rounds of a Delphi study. J Adv Nurs 2000;32(4):1016-24.
• 11. Hasson F., Keeney S., McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs 2000;32(4):1008-15.
• 12. Williams P.L., Webb C. The Delphi technique: a methodological discussion. J Adv Nurs 1994;19(1):180-6.
• 13. Palumbo D.J., Oliverio A. Implementation theory and the theory driven approach to validity. Evaluation and Program Planning 1989;12:337-44.
• 14. Champagne E., Contandriopoulos A.P., Pineault R. Un cadre conceptuel pour l'évalation des programmes de santé. Révue d'Épidemiologie et Santé Publique 1985;33:173-81.
• 15. Polit D.E., Hungler B.P. Fundamentos de Pesquisa em Enfermagem. 3ª Ed. Porto Alegre: Artes Médicas, 1995
• 16. Contandriopoulos A.P., Champagne E., Potvin L., et al. Saber Preparar uma Pesquisa: Definição, Estrutura e Financiamento. 2ª Ed. São Paulo: Editora Hucitec, 1997
• 17. ASHP American Society of Hospital Pharmacists. Criteria for Drug use Evaluations. Bethesda, , 1989. v. 1. 138p. In: ASHP, editor. Bethesda, 1989:1-138.
• 18. ASHP policy recommendations—invitation to comment. Am J Health Syst Pharm 2003;60(7):701-4.
• 19. Bowles N. The Delphi technique. Nurs Stand 1999;13(45):32-6.
• 20. Patton M.Q.. Utilization Focused Evaluation: The New Century Text. 3rd Ed. Thousand Oaks: Sage Publications, Inc., 1997
• 21. Holzheimer R.G., Quoika P., Patzmann D., Fussle R. Nosocomial infections in general surgery: surveillance report from a German university clinic. Infection 1990;18(4):219-25.
• 22. Singh A.K., Sen M.R., Anupurba S., Bhattacharya P. Antibiotic sensitivity pattern of the bacteria isolated from nosocomial infections in ICU. J Commun Dis 2002;34(4):257-63.
• 23. Hsueh P.R., Tseng S.P., Teng L.J., Ho S.W. Pan-drug-resistant Pseudomonas aeruginosa causing nosocomial infection at a university hospital in Taiwan. Clin Microbiol Infect 2005;11(8):670-3.
• 24. Kiffer C., Hsiung A., Oplustil C., et al. Antimicrobial susceptibility of Gram-negative bacteria in Brazilian hospitals: the MYSTIC Program Brazil 2003. Braz J Infect Dis 2005;9(3):216-24.
• 25. Todd M.W. Drug use evaluation. In: Brown T.R. [ed]. Handbook of Institutional Pharmacy Practice. Bethesda, ASHP, 1992:261-71.
• 26. Beech B. Go the extra mile—use the Delphi Technique. J Nurs Manag 1999;7(5):281-8.
• 27. Goodman C.M. The Delphi technique: a critique. J Adv Nurs 1987;12(6):729-34.
• 28. Sackman H. Delphi critique. Lexington Books, Lexington ,MA, 1975
• 29. Zarrilli R., Tripodi M.F., Di P.A., et al. Molecular epidemiology of high-level aminoglycoside-resistant enterococci isolated from patients in a university hospital in southern Italy. J Antimicrob Chemother 2005
• 30. Leverstein-van Hall M.A.,Paauw A., Fluit A.C., Verhoef J. Multidrug resistance among Enterobacteriaceae is strongly associated with the presence of integrons and is independent of species or isolate origin. J Infect Dis 2003;187(2):251-9.
• 31. Ramsay C., Brown E., Hartman G., Davey P. Room for improvement: a systematic review of the quality of evaluations of interventions to improve hospital antibiotic prescribing. J Antimicrob Chemother 2003;52(5):764-71.
• 32. Burnakis T.G. Facilitating drug-use evaluation with spreadsheet software. Am J Hosp Pharm 1989;46(1):84-8.

Publication Dates

• Publication in this collection
  18 Sept 2006
• Date of issue
  June 2006

History

• Reviewed
  31 May 2006
• Received
  08 Jan 2006