Print version ISSN 0034-7094
Rev. Bras. Anestesiol. vol.51 no.3 Campinas May/June 2001
Cumulative sum analysis applied to the teaching of spinal anesthesia*
Uso da soma cumulativa dos desvios para avaliação da proficiência no ensino do bloqueio subaracnóideo
Uso de la suma cumulativa de los desvíos para evaluación de la proficiencia en la enseñanza del bloqueo subaracnóideo
Getúlio Rodrigues de Oliveira Filho, TSA, M.D.I; Sérgio Galluf Pederneiras, TSA, M.D.II; Jorge Hamilton Soares Garcia, TSA, M.D.III
IIInstrutor Co-responsável pelo CET/SBA
IIIInstrutor do CET/SBA
BACKGROUND AND OBJECTIVES: Objective
performance measurements are necessary during the learning process of anesthetic
techniques. This study aimed at describing the cusum (cumulative sum) analysis
to build subarachnoid puncture learning curves.
METHODS: The successful subarachnoid space identification during the first attempt, in the first interspace approached, and the success of 275 spinal blocks performed by 5 residents during the initial six-month training period were used to build learning curves through the cusum analysis. Acceptable failure rates were taken from a sample of 264 spinal blocks performed by experienced anesthesiologists. The number of blocks needed for proficiency was calculated for each resident, for each attribute.
RESULTS: A wide variability in the number of blocks needed for attaining proficiency was observed, depending on the resident or on the attribute. However, most residents attained proficiency after 50 blocks. All residents achieved a 90% success rate after 30 blocks.
CONCLUSIONS: Cusum analysis may be used for objectively measuring performance during the learning phase of spinal anesthesia. A minimum of 50 blocks is required in order to attain proficiency in identifying subarachnoid space.
Key Words: ANESTHESIOLOGY: teaching; ANESTHETIC TECHNIQUES, Regional: spinal block
JUSTIFICATIVA E OBJETIVOS: Medidas
objetivas de desempenho são necessárias para o ensino de técnicas
anestésicas. Este estudo teve por objetivos aplicar e descrever o método
da soma cumulativa dos desvios para a construção de curvas de aprendizado
da punção subaracnóidea.
MÉTODO: O sucesso da punção subaracnóidea na primeira tentativa, no primeiro espaço abordado, e o sucesso da anestesia de 275 bloqueios subaracnóideos realizados por médicos em especialização (ME) durante os primeiros seis meses de treinamento foram utilizados para a construção de curvas de aprendizado, utilizando o método das somas cumulativas de desvios. As taxas aceitáveis de falha foram derivadas de uma amostra de 264 bloqueios subaracnóideos realizados por anestesiologistas. O número necessário de bloqueios para se obter proficiência foi calculado, para cada ME, para cada atributo.
RESULTADOS: Houve uma grande variabilidade em relação ao número de bloqueios necessários para se atingir a proficiência, dependendo do atributo e do ME. Contudo, a maioria dos ME atingiu proficiência para os atributos de sucesso durante a primeira tentativa ou no primeiro espaço inter-espinhoso abordado, após 50 bloqueios. Uma taxa de sucesso da anestesia de 90% foi obtida por todos os ME após 30 bloqueios.
CONCLUSÕES: O método da soma cumulativa de desvios pode ser utilizado para medir objetivamente o desempenho de médicos em especialização, durante a fase de aprendizado de anestesia subaracnóidea. Um número mínimo de 50 bloqueios é necessário para se obter proficiência em identificar o espaço subaracnóideo.
Unitermos: ANESTESIOLOGIA: ensino; TÉCNICAS ANESTÉSICAS, Regional: subaracnóidea
JUSTIFICATIVA Y OBJETIVOS: Medidas
objetivas de desempeño son necesarias para la enseñanza de técnicas
anestésicas. Este estudio tuvo por objetivo aplicar y describir el método
de la suma cumulativa de los desvíos para la construcción de curvas
de aprendizaje de la punción subaracnóidea.
MÉTODO: El suceso de la punción subaracnóidea en la primera tentativa, en el primer espacio abordado, y el suceso de la anestesia de 275 bloqueos subaracnóideos realizados por médicos en especialización (ME) durante los primeros seis meses de entrenamiento fueron utilizados para la construcción de curvas de aprendizaje, utilizando el método de las sumas cumulativas de desvíos. Las tasas aceptables de falla fueron derivadas de una muestra de 264 bloqueos subaracnóideos realizados por anestesistas. El número necesario de bloqueos para obtenerse proficiencia fue calculado, para cada ME, para cada atributo.
RESULTADOS: Hubo una grande variabilidad en relación al número de bloqueos necesarios para alcanzar la proficiencia, dependiendo del atributo y del ME. No obstante, la mayoría de los ME alcanzó proficiencia para los atributos de suceso durante la primera tentativa o en el primero espacio inter-espinoso abordado, después 50 bloqueos. Una tasa de suceso de la anestesia de 90% fue obtenida por todos los ME después de 30 bloqueos.
CONCLUSIONES: El método de la suma cumulativa de desvíos puede ser utilizado para medir objetivamente el desempeño de los médicos en especialización, durante la etapa de aprendizaje de anestesia subaracnóidea. Un número mínimo de 50 bloqueos es necesario para obtenerse proficiencia en identificar el espacio subaracnóideo.
Spinal block learning starts by mastering lumbar puncture techniques. Teaching starts with an intensive supervision which progressively decreases as the resident improves performance, until finally it is no longer needed. The transition point between intensive supervision and no supervision must be objectively defined through quantitative performance evaluation tools, that is, through the level of proficiency.
The ideal evaluation tool should be easily computed and provide data to follow the evolution of each resident, identifying performance levels apart of acceptable standards and measuring the results of corrective interventions.
The cusum (cumulative sum) analysis (CSA) is a statistical method used to measure the proficiency of residents during the learning process of procedures such as tracheal intubation, spinal and epidural anesthesia and central venous and arterial catheterization 1.
The method is easily computed and requires the calculation of three parameters: sigma deviation (s), upper control limit (UCL), calculated as from an unacceptable failure rate (p0) and lower control limit (LCL), calculated as from an acceptable failure rate (p1). To calculate CSA, one starts from a zero point, which represents the first procedure. At every success obtained in a subsequent procedure, the values is subtracted from the previous CSA and at every failure, the value of 1 - s is added. So a curve representing the learning process is obtained, which goes up when there are failures and goes down when successes are achieved. When the CSA line goes beyond UCL, performance is considered inadequate. While the CSA line is between LCL and UCL no statistical decision may be made and additional information is needed. When the CSA line goes beyond UCL coming from above, performance is considered acceptable for not being significantly different from p1 2.
So, this study aimed at building learning curves for spinal puncture techniques, using the cusum analysis to determine the minimum number of procedures performed by residents to attain proficiency.
The Medical Ethics Committee, Hospital Governador Celso Ramos, approved this study. Prospective data on the first 275 spinal blocks performed by 5 residents during the first six training months were collected (Group R). Residents were called A, B, C, D and E. During the same period, a control group was formed (Group A) with a sample of 264 spinal blocks performed by 22 experienced anesthesiologists.
In Group R, blockades were performed under the supervision of instructors duly qualified by the Brazilian Society of Anesthesiology. As to lumbar puncture, residents were oriented about patients positioning, antisepsis technique, placement of sterile drapes, interspace choice, local anesthesia, needle positioning and spinal space identification by CSF observation.
All blockades were performed with disposable Quincke needles. Age, gender, palpation quality of spinous processes (easy, difficult, impossible) spinal space access route (median or paramedian), position during lumbar puncture (lateral or sitting), positioning conditions (satisfactory, unsatisfactory) and lumbar puncture needle size were recorded for all patients.
The following parameters were recorded and coded as 1 if present and zero if absent:
1. Attribute 1: success on the first attempt - spinal space location with just one skin puncture.
2. Attribute 2: success on the first interspace - spinal space location in the first interspace approached although with more than one skin puncture.
3. Attribute 3 - success of anesthesia - induction of complete surgical anesthesia
To define control limits for attributes 1, 2 and 3, the following rates were derived from a sample of 264 spinal blocks performed by 22 anesthesiologists working in two hospitals during the same period (Group A).
1. Success rate for the first attempt - number of blockades where the spinal interspace was located with just one skin puncture divided by the total number of punctures multiplied by 100.
2. Success rate for the first space - number of blockades where spinal space was located by one or more skin punctures, but in the first interspace, divided by the total number of blockades and multiplied by 100.
3. Success rate for anesthesia - number of blockades where the surgical procedure was performed with no need for blockade supplementation with opioids or general anesthesia, divided by the total number of blockades and multiplied by 100.
For attribute 1, p1 was established as 0.35 because failure rate in Group A was 35%, and p0 was arbitrarily defined as 0.5. Similarly, p0 was established as 0.1 and p1 as 0.2 for attribute 2. For attribute 3, Group A's failure rate was 5%, resulting in p0 = 0.05 and p1 was arbitrarily defined as 0.2. For attribute 3, the same success rate of 90% was adopted and defined by a consensus of specialists1, resulting in p0 = 0.1 and p1 = 0.2. Values of a (type I error) and b (type II error) were established as 0.1.
Variables for CSA were computed using the following formulas:
a = ln[(1-b)/a]
b = ln[(1- a)/b]
P = ln (p1/p0)
Q = ln [(1 - p1)/(1- p0)]
LCL = -b/(P + Q)
UCL = a/(P + Q)
s = Q/(Q + P)
Learning curves for each attribute were calculated for each resident based on cusum analysis of deviations. Proficiency in a certain attribute (acceptance of null hypothesis) was identified by CSA values permanently below LCL. Number of blockades executed until proficiency was achieved, was compared among residents by the Chi-square and Fisher exact tests.
Patients demographic data and procedure technical characteristics were compared among groups by Student t (continuous variables) or chi-square (categorical variables) tests.
Groups A and R did not differ as to age. Group R had more male patients. In Group A, the paramedian approach and the sitting position were more frequently used than in Group R. 25G needles were more frequently used by Group R. There were no differences between groups as to 22 and 27G needles (Table I). Group R did not use 26 and 29G needles. Anatomic references quality (spinal processes) was equally distributed among groups (Table II).
Number of blockades performed by residents varied from 31 to 77.
Learning curves for attribute 1 - success in the first attempt - showed that A (21 blockades) and E (17 blockades) attained an early proficiency as compared to B (44 blockades) and C (57 blockades). D (31 blockades) did not attain proficiency at the end of six months (Table III and Figure 1).
Learning curves for attribute 2 - success in the first interspace - showed that A (25 blockades) and B (25 blockades) attained early proficiency as compared to C (61 blockades), while D (31 blockades) and E (50 blockades) did not attain proficiency (Table IV and Figure 2).
Learning curves for attribute 3 - surgical anesthesia success - showed that only C (40 blockades) and E (13 blockades) attained proficiency during the first six months, when LCL was calculated as from an acceptable failure rate of 5% (Table V and Figure 3). When the acceptable failure rate for such attribute was increased to 10% all residents attained proficiency although with significantly different numbers of blockades (Table VI and Figure 4).
There are major variations in the number of regional anesthesias performed by residents during training 3,4. This interferes with the level of self-confidence of residents with regards to blockade, which is higher when a higher number of procedures are performed 3.
In a previous study, learning curve analysis of spinal blocks has shown that performance is significantly increased after 20 blockades as compared to the first one although consistency - success rate maintenance - is only obtained after 45 attempts 5. In a different study, 90% success rate was only obtained after 71 attempts 6. The cusum method was proposed to evaluate proficiency of physicians specializing in anesthetic procedures. With this method, the number of spinal blocks necessary for proficiency varied from 39 to 67 attempts 1.
In those studies, success was defined as the level of orientation independence to perform spinal blocks1,5 or achieve a satisfactory anesthesia 6.
Considering that our teaching method anticipate that during the first six months of training all residents are under intensive supervision in all procedures and also that learning curves had never been studied, the level of independence of our residents was not used as a proficiency attribute. In its place, we used the success in identifying the subarachnoid space during the first puncture attempt for representing the ideal performance, the puncture success in the first interspace, regardless of the number of skin punctures, as proposed by other authors 7 and the success of the surgical anesthesia. Cusum analysis was used to identify the number of blockades needed to attain an acceptable level of proficiency for clinical practice.
Cusum analysis of an acceptable failure rate is a method similar to the sequential analysis used in lab experiments 1,2,8. Quality parameters are called attributes, which define the compliance requirements of a certain item, service or, in this case, with previously established specifications. The aim of such analysis is to graphically follow performance, identifying its trends and deviations as a function of time. The CSA analysis may be used to compare the performance of different people during a technique learning process, hence its potential application to the teaching of Anesthesiology procedures. This method can also be used by experienced people as a tool for individual performance control 2.
The cusum analysis is a simple procedure that starts with a definition of acceptable (p1) and unacceptable (p0) failure rates. These may be obtained by specialists consensus 1 or by calculating institutional failure rates for the procedure being studied 6. Then, acceptable magnitude of errors type I (a) and II (b) are determined. With such data, upper and lower control limits are calculated. Then, sigma (s) value, which represents the deviation value, is calculated. In a XY chart, the number of procedures is marked in the X axle and UCL and LCL are marked in the Y axle. The chart starts with 0, which represents the first procedure and the initial CSA value. With each success or failure a negative (s) or positive (1 - s) increment is added to obtain a cumulative score. So, CSA decreases with each success and increases with each failure. When CSA goes below LCL, the failure rate is not considered significantly different from p (null hypothesis), with a risk of error type II equals b. Conversely, when CSA crosses the line, which limits UCL coming from below, then failure rate is significantly higher than p0 (alternative hypothesis), with a chance of error type II equals a.
Graphically represented, CSA allows resident's performance follow up during the learning of a technique or procedure. If CSA goes beyond UCL, the resident's performance is significantly apart from p0 and corrective actions are indicated. If CSA remains between LCL and UCL, the performance cannot be considered either acceptable or unacceptable and more observations are needed. If CSA remains below LCL, it is assumed that the resident has attained proficiency.
This study aimed at building learning curves for lumbar puncture learning and at defining the minimum number of spinal blocks to be performed for the technique to be mastered by the resident during the first six months of training.
Similar to other studies 1,2,5,6, the minimum number of procedures for all residents to be proficient in locating the subarachnoid space was estimated in 50, using as attributes the puncture success in the first attempt or in the first interspace approached. When learning curves were built based on a 90% anesthesia success rate, it was seen that 30 blocks were needed for all residents to become proficient. This difference in the minimum number of blockades can be explained by the spinal anesthesia success rate. The performance variation among individuals in our sample reinforces the need for a customized performance follow up during the learning of subarachnoid puncture techniques 9.
In our study, only one teaching aspect of the spinal block was evaluated, that is, mastering subarachnoid puncture technique. So, it was not established the number of blockades needed to attain clinical proficiency in spinal anesthesia - an extremely relevant topic for which other follow up parameters have already been developed 10,11.
The conclusion is that the cusum analysis is a statistical method easily computed which allows an objective evaluation and the performance follow up of residents during the learning process of spinal anesthesia. Our results also led to the conclusion that at least 50 blockades are needed to attain spinal anesthesia proficiency.
01. Kestin IG - A statistical approach to measuring the competence of anaesthetic trainees at practical procedures. Br J Anaesth, 1995;75:805-809.
02. Williams SM, Parry BR, Schlup MMT - Quality control: an application of the cusum. Br Med J, 1992;304:1359-1361.
03. Smith MP, Sprung J, Zura A et al - A survey of exposure to regional anesthesia techniques in American anesthesia residency training programs. Reg Anesth Pain Med, 1999;24:11-16.
04. Bouaziz H, Mercier FJ, Narchi P et al - Survey of regional anesthetic practice among French residents at time of certification. Reg Anesth, 1997;22:218-222.
05. Kopacz DJ, Neal JM, Pollock JE - The regional anesthesia "learning curve". What is the minimum number of epidural and spinal blocks to reach consistency? Reg Anesth, 1996;21:182-190.
06. Konrad C, Schüpfer G, Wietlisbach M et al - Learning manual skills in anesthesiology: is the recommended number of cases for anesthetic procedures? Anesth Analg, 1998;86:635-639.
07. Sprung J, Bourke DL, Grass J et al - Predicting the difficult neuraxial block: a prospective study. Anesth Analg, 1999;89: 384-389.
08. Dixon WJ, Massey Jr FJ - Sequential Analysis, em: Dixon WJ, Massey Jr FJ - Introduction to Statistical Analysis. New York, McGraw-Hill, 1983;415-425.
09. Lawler PG, Patla VR, Garcia E et al - Assessment of training in anaesthesia and related skills. Anaesthesia, 1991;46:597.
10. Kopacz D - QA in regional anesthesia training. Quantity or quality? Reg Anesth, 1997;22:209-211.
11. Sivarajan M, Miller E, Hardy C et al - Objective evaluation of clinical performance and correlation with knowledge. Anesth Analg, 1984;63:603-607.
Dr. Getúlio Rodrigues de Oliveira Filho
Address: Rua José Cândido Silva 179/402
ZIP: 88075-250 City: Florianópolis, Brazil
Submitted for publicat September 11, 2000
Accepted for publication November 23, 2000
* Received from CET/SBA Integrado de Anestesiologia da SES-SC, Florianópolis, SC
01. Kestin IG - A statistical approach to measuring the competence of anaesthetic trainees at practical procedures. Br J Anaesth, 1995;75:805-809. [ Links ]
02. Williams SM, Parry BR, Schlup MMT - Quality control: an application of the cusum. Br Med J, 1992;304:1359-1361. [ Links ]
03. Smith MP, Sprung J, Zura A et al - A survey of exposure to regional anesthesia techniques in American anesthesia residency training programs. Reg Anesth Pain Med, 1999;24:11-16. [ Links ]
04. Bouaziz H, Mercier FJ, Narchi P et al - Survey of regional anesthetic practice among French residents at time of certification. Reg Anesth, 1997;22:218-222. [ Links ]
05. Kopacz DJ, Neal JM, Pollock JE - The regional anesthesia "learning curve". What is the minimum number of epidural and spinal blocks to reach consistency? Reg Anesth, 1996;21:182-190. [ Links ]
06. Konrad C, Schüpfer G, Wietlisbach M et al - Learning manual skills in anesthesiology: is the recommended number of cases for anesthetic procedures? Anesth Analg, 1998;86:635-639. [ Links ]
07. Sprung J, Bourke DL, Grass J et al - Predicting the difficult neuraxial block: a prospective study. Anesth Analg, 1999;89: 384-389. [ Links ]
08. Dixon WJ, Massey Jr FJ - Sequential Analysis, em: Dixon WJ, Massey Jr FJ - Introduction to Statistical Analysis. New York, McGraw-Hill, 1983;415-425. [ Links ]
09. Lawler PG, Patla VR, Garcia E et al - Assessment of training in anaesthesia and related skills. Anaesthesia, 1991;46:597. [ Links ]
10. Kopacz D - QA in regional anesthesia training. Quantity or quality? Reg Anesth, 1997;22:209-211. [ Links ]
11. Sivarajan M, Miller E, Hardy C et al - Objective evaluation of clinical performance and correlation with knowledge. Anesth Analg, 1984;63:603-607. [ Links ]