Using the Cusum curve to evaluate the training of orotracheal intubation with the Truview EVO2® laryngoscope

Abstracts

JUSTIFICATIVA E OBJETIVOS: As curvas de aprendizado têm se mostrado ferramentas úteis no monitoramento do desempenho de um trabalhador submetido a uma nova tarefa. Essas curvas vêm sendo utilizadas na avaliação de vários procedimentos na prática médica. O objetivo desta pesquisa foi avaliar o aprendizado da intubação orotraqueal (IOT) com o laringoscópio Truview EVO2® através da curva de aprendizado CUSUM. MÉTODO: Quatro aprendizes realizaram o treinamento da IOT com o laringoscópio Truview EVO2® em manequim. Eles foram orientados quanto aos critérios de sucesso e falha na IOT e alternaram-se nas tentativas, num total de 300 IOT para cada um deles. Quatro curvas de aprendizado foram construídas a partir do método da soma cumulativa CUSUM. RESULTADOS: O número calculado para adquirir proficiência na tarefa foi de 105 IOT. Os quatro aprendizes cruzaram a linha de taxa de falha aceitável de 5% antes de completar 105 IOT: o primeiro aprendiz alcançou a faixa de proficiência após 42 IOT, o segundo e o terceiro aprendizes, após 56 IOT, e o quarto aprendiz, após 97 IOT, mantendo-se constantes em seus desempenhos a partir de então. Não houve diferença na taxa de sucesso entre residentes e anestesiologistas experientes. CONCLUSÕES: A curva de aprendizado CUSUM é um instrumento útil para demonstração objetiva de habilidade na execução de uma nova tarefa. A laringoscopia com o Truview EVO2® em manequim demonstrou ser um procedimento fácil para médicos com experiência prévia em IOT, porém, a transposição dos resultados para a prática clínica deve ser cautelosa.

ANESTESIOLOGIA, Ensino; INTUBAÇÃO TRAQUEAL

SCIENTIFIC ARTICLE

Using the Cusum curve to evaluate the training of orotracheal intubation with the Truview EVO2^® laryngoscope^*

Aplicación de la curva CUSUM para evaluar el entrenamiento de la intubación orotraqueal con el laringoscopio Truview Evo2^®

Jaqueline Betina Broenstrup Correa, M.D.^I; José Ernani Flores Dellazzana, TSA, M.D.^II; Alexandre Sturm, M.D.^III; Dante Moore Almeida Leite, M.D.^III; Getúlio Rodrigues de Oliveira Filho, TSA, M.D.^IV; Rogério Gastal Xavier, M.D.^V

^IAnestesiologista

^IIAnestesiologista; Instrutor do CET/SBA do HCPA

^IIIME do CET/SBA do HCPA

^IVAnestesiologista; Responsável do CET/SBA Integrado de Anestesiologia da SES-SC

^VPneumologista; Doutor em Clínica Médica; Professor Associado do Departamento de Medicina Interna/UFRGS; Coordenador da Unidade de Broncologia e do Laboratório Experimental de Vias Aéreas e Pulmão do HCPA

Correspondence to

SUMMARY

BACKGROUND AND OBJECTIVES: Learning curves have proved to be useful tools to monitor the performance of a worker on a new assignment. Those curves have been used to evaluate several medical procedures. The objective of this study was to evaluate the learning of orotracheal intubation (OTI) with the Truview EVO2^® laryngoscope with the CUSUM learning curve.

METHODS: Four trainees underwent OTI training with the Truview EVO2^® laryngoscope in a mannequin. They received orientation on the successful and failure criteria of OTI and alternated during the attempts, for a total of 300 OTI for each one. Four learning curves were plotted using the CUSUM cumulative addition method.

RESULTS: It was calculated that the 105 OTIs were necessary to achieve proficiency. The four trainees crossed the line of acceptable failure rate of 5% before completing 105 OTIs; the first trainee reached proficiency after 42 OTIs, the second and third after 56 OTIs, and the fourth after 97 OTIs, and from then on their performance remained constant. Differences in the success rate between residents and experienced anesthesiologists were not observed.

CONCLUSIONS: The CUSUM learning curve is a useful instrument to demonstrate objectively the ability when performing a new task. Laryngoscopy with the Truview EVO2^® in a mannequin proved to be an easy procedure for physicians with prior experience in OTI; however, one should be cautious when transposing those results to clinical practice.

Keywords: ANESTHESIOLOGY, Teaching: Intubation; TRACHEAL INTUBATION.

RESUMEN

JUSTIFICATIVA Y OBJETIVOS: Las curvas de aprendizaje han sido herramientas útiles en el monitoreo del desempeño de un trabajador sometido a una nueva tarea. Esas curvas han venido siendo utilizadas en la evaluación de varios procedimientos en la práctica médica. El objetivo de esta investigación, fue evaluar el aprendizaje de la intubación orotraqueal (IOT) con el Laringoscopio Truview Evo2^® a través de la curva de aprendizaje CUSUM.

MÉTODO:Cuatro aprendices realizaron el entrenamiento de la IOT con el Laringoscopio Truview Evo2^® en un maniquí. Se les orientó en cuanto a los criterios de éxito y falla en la IOT y se intercambiaban los intentos, en un total de 300 IOT para cada uno de ellos. Cuatro curvas de aprendizaje fueron construidas a partir del método de la suma acumulativa CUSUM.

RESULTADOS: El número calculado para adquirir el desempeño en la tarea fue de 105 IOT. Los cuatro aprendices cruzaron la línea de rango de falla aceptable de un 5% antes de completar 105 IOT: el primer aprendiz alcanzó el rango de desempeño después del 42 IOT, el segundo y el tercer aprendiz después de 56 IOT, y el cuarto aprendiz, después de 97 IOT, manteniéndose constantes en sus desempeños a partir de ese momento. No se registró diferencias en la tasa de éxito entre residentes y anestesiólogos expertos.

CONCLUSIONES: La curva de aprendizaje CUSUM es un instrumento útil para la demostración objetiva de la habilidad en la ejecución de una nueva tarea. La laringoscopia con el TruviewEvo2^® en un maniquí, demostró ser un procedimiento fácil para médicos con experiencia previa en IOT, sin embargo, al llevar los resultados a la práctica clínica, eso deberá hacerse con cautela.

INTRODUCTION

There is a growing need to guarantee the quality of procedures in the medical field. Such abilities have impacts on patient care and results obtained by teams and institutions ¹.

In the first half of the XXth Century, mathematical models were applied to quality control of industrial production lines. The Cumulative Sum curve (CUSUM) is one of the statistical techniques known as Sequential Analysis. Sequential tests were developed as an instrument to evaluate whether the quality of a production process was "under control" (producing items within a predefined limit of quality) or "out of control" and based on a predetermined standard stop the process, correct it, and restart within an acceptable performance ^2,4.

The introduction of mathematical models for quality control in the biomedical field was based on the successful application of those models in industrial production lines. CUSUM analysis has been proved to be an objective and effective tool to help the learning process and monitor performance in procedures such as colonoscopies, ultrasound-guided biopsies, surgical and anesthetic procedures, and laboratorial tests, to mention only a few ^5-15.

The most difficult aspect on applying this technique to evaluate training is to establish an acceptable and unacceptable failure rate. This information may come from the guidelines of Expert Commissions in the intended area of study, studies on the same subject, or information collected at the study site from the performance of a more experienced team and using them as guidelines for the performance to be achieved by trainees. As long as success and failure can be strictly defined and the consistency of those definitions can be guaranteed, the procedure to be followed can be submitted to CUSUM analysis ³.

Orotracheal intubation (OTI) belongs to the different abilities developed in anesthesia. The number of attempts of OTIs necessary for residents to acquire proficiency varies ^5,11,13-15. Some factors can interfere with this process, such as prior experience with similar techniques. Studies on the care of trauma victims demonstrated that the failure rate of OTI can achieve up to 5% when performed by experienced professionals in a hospital setting ¹⁶. In general anesthesia, the frequency of difficult OTIs varies from 0.5% to 2%, but it might be greater when isolated surgical subspecialties are studied¹⁷. Failure of OTI is less frequent, ranging from 0.05% to 0.35%, depending on the study ^17-19.

The objective of this study was to evaluate, using the CUSUM curve, the learning of OTI in a mannequin with a new model of laryngoscope recently available in the national market, the Truview EVO2^® - Truphatek. Laryngoscopy and OTI with this model have differences when compared with the standard equipment, i.e., the Macintosh laryngoscope, which the authors are very familiar with, and this could influence the results of the study.

METHODS

During November and December 2007, seven training sessions of OTI with the Truview EVO2^® laryngoscope in the mannequin Laerdal Airway Mangement (Laerdal^®) were scheduled to plot the CUSUM learning curve.

Four of the investigators were the study subjects and they had to agree to participate in the project by signing an informed consent. The study was approved by the Ethics Committee of the Hospital das Clínicas de Porto Alegre.

Description of the Equipment:

The Truview EVO2^® (Truphatek - Israel) laryngoscope has of a standard handle which houses two alkaline C batteries that feed a 2.5 volts xenon bulb. The handle articulates with an angled blade (Figures 1 and 2). The blade has a 42-degree refraction optical lens: the 15-mm diameter optical visor, which allows the indirect view of the glottis and vocal cords is located in the proximal extremity. The visor can be connected to the handle of an endoscopic system to show the OTI on a video screen. The blade has an oxygen connecting port, which can be used to keep the lens mist-free. The use of medical anti-fog solutions can also be used for this end.

Description of the procedure:

The procedure began by reading the instruction in the User Manual that comes with the product.

Success and failure criteria of the OTI were explained to the trainees, who alternated on the attempts, totaling 300 attempts for each one distributed throughout the seven training sessions.

While one trainee performed the OTI, the second controlled the time, the third recorded it on a form, and the fourth ventilated the mannequin with a balloon-valve device (Figure 3).

Success and failure criteria for the OTI:

• Success: orotracheal intubation in 45 seconds or less, confirmed by visualizing the orotracheal tube in the trachea of the mannequin and lung expansion with manual ventilation with the balloon-valve device.

• Failure: orotracheal intubation in more than 45 seconds and/or failure to intubate.

The acceptable failure rate (p0) for OTI was 5% and the unacceptable failure rate (p1) was 10%. The acceptable and unacceptable failure rates established were stricter than those in the literature since residents and anesthesiologists experienced in OTI were being evaluated. The probability of a type I (α) and II (β) error was established at 0.1.

The calculated sample size (number of OTIs) for an acceptable failure rate of 5% was 105 attempts.

Using the data of the 300 OTIs, CUSUM curves were plotted for each trainee according to the formula in Chart I.

The success rate of experienced residents and anesthesiologists was compared by the Fisher Exact test, with a 5% level of significance.

On the graphic representation of the CUSUM curve, each trainee was represented by a geometric figure (▲, ■, ●, ♦) to guarantee anonymity.

RESULTS

Table I shows the calculated values to plot the CUSUM curve. Lines h0 and h1 (Figure 4) were derived from the determination of α and β of 0.1, 5% acceptable failure rate, and 10% unacceptable failure rate. Between those two horizontal lines shown in the chart is the undefined area of proficiency in the sequence of OTI attempts. The chart of the learning curve starts at zero. The calculated values for s (for each success) and 1-s (for each failure) determine the direction of the curve upwards and downwards, respectively. When the curve crosses the h1 line, it means the trainee has an unacceptable failure rate for the procedure in question, according to the consensus of the subspecialty, and the causes of those failures should be investigated if they persist. When the curve crosses the h0 line, it shows the trainee has acquired proficiency on the procedure, i.e., he presents an acceptable failure rate. Calculating the size of the sample is equivalent to the number of repetitions necessary to define the tendency of the trainee within those three possibilities: proficiency, undefined, or lack of proficiency. A very strict failure rate is reflected on a small calculated sample size, and may generate failure warnings too early in the training ⁷. To generate an intervention to determine and correct those failures, the choice of this value depends on the impact that failures will have on patients ¹².

The determination of equal alpha and beta values generates identical distances for h0 and h1 from zero. However, other values can be chosen for alpha and beta, according to the risk one is willing to take as to consider proficient someone who is not, or not proficient someone who is already trained enough within the criteria established ¹⁴.

The calculated number of repetitions necessary for our study was 105 OTI for an acceptable failure rate of 5% with the Truview EVO2^® laryngoscope. The four trainees, with different experience in Anesthesiology, completed the proposed training of 300 OTIs, divided in seven sessions, once a week.

All failures were seen on the first training day (Table II). All four trainees crossed the line of acceptable failure rate of 5% (h0) before completing 105 OTIs (Figure 4): trainee ▲ reached proficiency after 42 OTIs; trainees ■ and ♦ after 56 OTIs; and trainee ● after 97 OTIs, and their performance remained constant from there on.

The rate of success of residents and anesthesiologists showed no differences (Fisher Exact test, p = 0.2).

DISCUSSION

A mannequin was used to evaluate learning ability of OTI using a Truview EVO2^® laryngoscope. Transposition of the results to the daily practice has limitations that should be considered. Real life has more complex situations that might cause a reduction in efficiency when compared to simulations. The diversity of airways anatomy, the urgency of the procedure, and prior experience are examples of some of the factors that can influence the results of studies with patients. However, the repetitive execution of tasks can be initiated with simulations. The use of simulators and mannequins for training in the medical field is based on the idea of developing a minimal level of ability to enhance the performance when dealing with patients ^20,21. Classical examples of the use of mannequins and simulators include the regular training on cardiorespiratory arrest ²¹ and flight simulators used in the airline and aerospace industries ²². Applying the CUSUM curve to clinical situations that require risk stratification may need mathematical adaptations to more accurately reflect learning in those circumstances. Rogers et al. published an important review on the subject, discussing the different adaptations of the CUSUM curve according to what is being evaluated.

Learning curves have been the focus of attention of medical investigators who look for a systematic way of evaluating residents and to monitor the performance of experienced professionals ^1,2,515.

The CUSUM curve can determine when the trainee has achieved an acceptable performance on a specific task, similar to that of experienced professionals, which can influence the planning of training programs, both concerning the number of training sessions the trainee should be exposed to and establishing minimal requirements that allow safer progression to increasingly more complex tasks. Another positive aspect of learning curves is the possibility to evaluate performance within a specific period of time instead of on a single moment, which is the case of tests and exams that cannot detect the real performance of a trainee ^24,25.

Orotracheal intubation with the standard Macintosh laryngoscope is a well known technique in which all four subjects of this study had experience with at two different proficiency levels: two residents and two anesthesiologists. However, the Truview EVO2^® laryngoscope requires a different intubation routine. The main difference consists on the indirect view of the glottis and vocal cords through an optical lens, while traditional laryngoscopy allows direct view of the vocal cords. The tracheal tube has to be shaped with a metallic guide wire to be introduced with the Truview EVO2^®. The individual performing the procedure can only see the distal extremity of the tube when it reaches the entrance of the glottis, on a small field of vision projected by the lens. The trainees considered this the most difficult aspect of the procedure: the ability to place the distal end of the tube within the field of vision of the lens in order to insert it in the glottis through the vocal cords.

According to what was determined by the acceptable success and failure rates, trainees achieved the proficiency level between the 46^th and 97^th intubation in the mannequin. Those numbers can be compared with the performance of intubations in patients to determine whether prior training in mannequins could decrease the number of procedures necessary in daily practice. In a study ¹⁴ that evaluated the learning curve of anesthesiology residents on OTIs with the Macintosh laryngoscope, 57.14% crossed the h0 line (the line that limits the proficiency zone) after 43 ± 33.49 procedures. However, for one of the trainees the line remained in the undetermined zone after 144 procedures, demonstrating the variability that study subjects may present, as well as the need to identify trainees with greater difficulties in order to better manage training.

The consistency of the learning curve was high for the trainees, with few failures in the first ten intubations and no failures up to the 300^th intubation. This suggests that the Truview EVO2^® is easy to work with for those who already have prior laryngoscopy training. The easiness of learning was reflected on the similar performance of residents and anesthesiologists, without differences among their learning curves.

Training and acquisition of practical ability in the use of a new equipment are crucial before studies comparing it with a standard technique can be undertaken. It helps to avoid attributing the results to the lack of proficiency with the new equipment. Prior training with the Truview EVO2^® laryngoscope will allow investigators to compare it with the standard OTI technique in situations where a promise of its superiority over traditional laryngoscopy exists, such as in difficult intubations due to limitations of cervical extension and in patients with less visible vocal cords (Cormack 3 and 4). Being trained in alternative techniques of approaching difficult airways is critical, as recommended by the algorithms of several international Anesthesiology and Emergency Medicine societies ^17,27.

It was demonstrated that acquiring OTI ability with the Truview EVO2^® laryngoscope by physicians who already have some experience in this task is easy and stimulates the spread of the use of this equipment in the training of other professionals, such as Intensive Care and Emergency Care physicians, and as an integral part of the armamentarium for handling the airways.

The authors would like to acknowledge the contribution of the Cardiorespiratory Resuscitation Group of the Hospital das Clínicas de Porto Alegre for providing the Laerdal Airway Management (Laerdal^®) training mannequin.

REFERENCES

• 01. Steiner SH, Cook RJ, Farewell VT et al. - Monitoring surgical performance using risk-adjusted cumulative sum charts. Biostatistics 2000;1:441-452.
• 02. Bolsin S, Colson M - The use of the Cusum technique in the assessment of trainee competence in new procedures. Int J Qual Health Care 2000;12:433-438.
• 03. Anzanello MJ, Fogliatto FS - Curvas de aprendizado: estado da arte e perspectivas de pesquisa. Gest Prod 2007;14:109-123.
• 04. Williams SM, Parry BR, Schlup MMT - Quality control: an application of the cusum. Br Med J 1992;304:1359-1361.
• 05. Kestin IG - A statistical approach to measuring the competence of anaesthetic trainees at practical procedures. Br J Anaesth 1995;75:805-809.
• 06. Nan Rij AM, McDonald JR; Pettigrew RA et al. - Cusum as an aid to early assessment of surgical trainee. Br J Surg 1995;82:1500-1503.
• 07. Lim TO, Soraya A, Ding LM et al. - Assessing doctors' competence: application of CUSUM technique in monitoring doctors' performance. Int J Qual Health Care 2002;14:251-258.
• 08. Lim TO - Statistical process control tools for monitoring clinical performance. Int J Qual Health Care 2003;15:3-4.
• 09. Leandro G, Rolando N, Gallus G et al. - Monitoring surgical and medical outcomes: the Bernoulli cumulative SUM chart. A novel application to assess clinical interventions. Postgrad Med J 2005;81:647-652.
• 10. Chang WR, McLean IP - CUSUM: A tool for early feedback about performance? BMC Med Res Methodol 2006;6:8.
• 11. Konrad C, Schüpfer G, Wietlisbach M et al. - Learning manual skills in anesthesiology: is there a recommended number of cases for anesthetic procedures? Anesth Analg 1998;86:635-639.
• 12. Rogers CA, Reeves BC, Caputo M et al. - Control chart methods for monitoring cardiac surgical performance and their interpretation. J Thorac Cardiovasc Surg 2004;128:811-819.
• 13. Charuluxananan S, Kyokong O, Somboonviboon W et al. - Learning manual skills in spinal ansthesia and orotracheal intubation: is there any recommended number of cases for anesthesia residency training program? J Med Assoc Thai 2001;84(suppl 1):S251-5.
• 23. Cooper JB, Taqueti VR - A brief history of the development of mannequin simulators for clinical education and training. Qual Saf Health Care 2004;13(suppl 1):11-18.
• 24. Newble D - Techniques for measuring clinical competence: objective structured clinical examinations. Med Educ 2004;38: 199-203.
• 25. Sivarajan M, Miller E, Hardy C et al. - Objective evaluation of clinical performance and correlation with knowledge. Anesth Analg 1984;63:603-607.
• 26. Hagberg CA, Benumof JL - The American Society of Anesthesiologists' Management of the Difficult Airway Algorithm and Explanation: analysis of the algorithm, em: Hagberg CA - Benumof's Airway Management: Principles and Practice, 2 Ed, Philadelphia, Mosby-Elsevier 2007;236-251.

*

Recebido do Laboratório Experimental de Vias Aéreas e Pulmões do Centro de Pesquisas do Hospital de Clínicas de Porto Alegre (HCPA/UFRGS) - Serviço de Pneumologia, Porto Alegre, RS

Publication Dates

• Publication in this collection
  26 May 2009
• Date of issue
  June 2009

History

• Received
  06 Nov 2008
• Accepted
  28 Jan 2009