Assessing the quality of random clinical anesthesiology trials published on the Brazilian Journal of Anesthesiology from 2005 to 2008

Abstracts

JUSTIFICATIVA E OBJETIVOS: O ensaio clínico aleatório (ECA) é definido como um estudo que envolve grupos de intervenção e controle, com alocação aleatória dos participantes. O objetivo deste trabalho foi avaliar a qualidade dos artigos de ECA em anestesia publicados num determinado período. Desenho do estudo: descritivo. MÉTODO: Foi realizada busca manual dos artigos publicados na Revista Brasileira de Anestesiologia entre janeiro de 2005 e fevereiro de 2008, que tivessem características de ECA. A variável primária foi a qualidade dos ECA; as variáveis secundárias foram: encaminhamento da pesquisa para o Comitê de Ética em Pesquisa (CEP), utilização do termo de consentimento livre e esclarecido (TCLE), descrição da fonte de fomento, realização do cálculo do tamanho da amostra, número de autores, local de origem, teste estatístico utilizado, nível de significância adotado na pesquisa e classificação do tipo de estudo. Foi utilizada a escala de qualidade para avaliar a qualidade dos ECA, estatística descritiva e o cálculo do intervalo de 95% de confiança. RESULTADOS: Dos 114 artigos originais, 42 foram identificados como ECA. Destes, somente 3 (7,1%) foram classificados como de boa qualidade metodológica, considerando distribuição aleatória, encobrimento duplamente encoberto, perdas e exclusões. Dos 114 artigos, 107 foram encaminhados ao CEP, 67 utilizaram TCLE, em nenhum houve descrição do fomento, em 17 houve cálculo do tamanho da amostra, o número médio de autores por artigo foi 4,49; São Paulo contribuiu com 60 publicações, o teste t de Student foi o mais utilizado (47,4%), o nível de significância de 5% foi adotado em 97 e 42 foram ECA. CONCLUSÕES: Após a busca manual, 7,1% dos ensaios clínicos aleatórios foram considerados de boa qualidade metodológica.

ANESTESIOLOGIA; publicação; METODOLOGIA; ensaios clínicos controlados aleatórios

REVIEW ARTICLE

Assessing the quality of random clinical anesthesiology trials published on the Brazilian Journal of Anesthesiology from 2005 to 2008^*

Evaluación de la calidad de los ensayos clínicos aleatorios en anestesia publicados en la Revista Brasileña de Anestesiología en el período de 2005 a 2008

Fabiano Timbó Barbosa^I; Mário Jorge Jucá^II

^IMestrando em Ciências da Saúde pela Universidade Federal de Alagoas; Tutor da Liga de Anestesia, Dor e Terapia Intensiva do Estado de Alagoas

^IIPós-Doutorado em Coloproctologia pela Universidade do Texas, Dallas, EUA; Doutor em Gastroenterologia Cirúrgica na Área de Concentração em Coloproctologia; Membro Titular do conselho e do Colegiado do Curso de Pós-Graduação em Ciências da Saúde do Instituto de Ciências Biológicas e da Saúde da Universidade Federal de Alagoas

Correspondence to

SUMMARY

BACKGROUND AND OBJECTIVES: A random clinical trial (RCT) is defined as a study involving intervention and control groups with random distribution of the participants. The objective of the present study was to assess the quality of RCT in anesthesiology published during a specific time. Design of the study: descriptive.

METHODS: A manual search of the articles published by the Brazilian Journal of Anesthesiology between January 2005 and February 2008 was undertaken to identify studies with characteristics of RCTs. The quality of RCTs was the primary parameter; secondary parameters included: approval by the Ethics on Research Committee (ERC), use of the informed consent (IC), description of the source of the grant, the sample size was calculated, number of authors, place of origin, statistical tests used, level of significance adopted, and classification of the type of study. The quality scale, descriptive statistics, and calculation of the 95% confidence interval were used to evaluate the quality of the RCTs.

RESULTS: Out of 114 studies, 42 were identified as RCT. Only 3 (7.1%) of those were classified as having good methodological quality considering the random distribution, double blind, losses, and exclusions. One-hundred and seven out of 114 studies were submitted to the ERC, 67 used IC, none of them described the source of the grant, 17 calculated the size of the sample, the studies had a mean of 4.49 authors; 60 publications were from São Paulo; the Student t test was used more often (47.4%), a level of significance of 5% was adopted by 97 studies; and 42 were RCTs.

CONCLUSIONS: After the manual search, 7.1% of the random clinical assays were considered of good methodological quality.

Key Words: ANESTHESIOLOGY: publication; METHODOLOGY: randomized controlled clinical assays.

RESUMEN

JUSTIFICATIVA Y OBJETIVOS: El ensayo clínico aleatorio (ECA) se define como un estudio que involucra a grupos de intervención y control, con la ubicación aleatoria de los participantes. El objetivo de este trabajo fue evaluar la calidad de los artículos de ECA en anestesia publicados en un determinado período. Diseño del estudio: descriptivo.

MÉTODO: Se realizó una búsqueda manual de los artículos publicados en la Revista Brasileña de Anestesiología entre enero de 2005 y febrero de 2008, que tuviesen características de ECA. La variable primaria fue sobre la calidad de los ECA; las variables secundarias de los artículos fueron las siguientes: elevar la investigación al Comité de Ética en Investigación (CEP), utilización del término de consentimiento informado (TCI), descripción de la fuente de fomento, realización del cálculo del tamaño de la muestra, número de autores, local de origen, test estadístico utilizado, nivel de significancia adoptado en la investigación y clasificación del tipo de estudio. Se utilizó la escala de calidad para evaluar la calidad de los ECA, estadística descriptiva y el cálculo del intervalo de un 95% de confianza.

RESULTADOS: De los 114 artículos originales, 42 fueron identificados como ECA. De ellos, solamente 3 (7,1%) se clasificaron como siendo de buena calidad metodológica, considerando la distribución aleatoria, el encubrimiento doble ciego, las pérdidas y exclusiones. De los 114 artículos, 107 fueron elevados al CEP, 67 utilizaron TCLE, en ninguno de ellos hubo descripción del fomento, en 17 hubo cálculo del tamaño de la muestra, el número promedio de autores por artículo fue 4,49; São Paulo contribuyó con 60 publicaciones, el teste t de Student fue el más utilizado (47,4%), el nivel de significancia de 5% fue adoptado en 97 y 42 fueron ECA.

CONCLUSIONES: Después de la búsqueda manual, 7,1% de los ensayos clínicos aleatorios fueron considerados como siendo de buena calidad metodológica.

INTRODUCTION

The quality of random clinical trials can be defined as the probability of a study to generate unbiased results that are close to the therapeutic reality ¹. Three types of instruments can be used to evaluate the quality of randomized clinical trials: individual items, verification lists, and scales ². Scales transform information in numbers whose simplicity allows fast and easy understanding of the quality of clinical assays ¹.

Randomized clinical assays are defined as prospective studies that compare the effects and value of interventions in humans, in one or more groups compared with a control group ³. They should always test the effects of an intervention, may it be therapeutic, diagnostic, or prophylactic ³. Randomized clinical studies are the best type of primary study to clear doubts about specific interventions, related or not with the treatment or prevention of diseases ⁴. Advances in health care depend on the unbiased results of studies ⁵. Since randomized clinical trials represent the only hope to eliminate biases, they are considered the foundation of scientific development ⁵.

The hypothesis tested by our study was that 10% of original randomized scientific trials published in Brazil presented good methodological quality.

The objective of the present study was to evaluate the quality of original randomized clinical studies published in Brazil from January 2005 to February 2008.

METHODS

This study was submitted to the Ethics on Research Committee of the Universidade de Ciências de Alagoas. The informed consent does not apply to this type of study. Expenses inherent to the study were covered by the authors.

This is a descriptive study designed to evaluate the quality of randomized clinical trials. Controlled clinical anesthesiology study published in Brazil was the inclusion criterion. Exclusion criteria for the analysis of methodological quality included: original article on procedures that did not involve any type of anesthesia, original experimental study, and original anesthesiology studies classified as any type of study other than randomized clinical study.

The quality of randomized clinical trials, defined as the probability of a clinical study to generate unbiased results, was the primary parameter ¹. Secondary parameters included: approval by the Ethics on Research Committee, use of the informed consent, description of the source of the grant, calculation of the size of the study population, number of authors, place of origin, statistical test used, level of significance adopted, and classification of the type of study. Secondary parameters were collected from all original studies identified during the manual search.

The Brazilian Journal of Anesthesiology was chosen for the analysis of original studies because it is the official organ responsible for the dissemination of scientific knowledge among anesthesiologists in Brazil, being recommended and widely advertised by the Sociedade Brasileira de Anestesiologia as a source of updates on anesthesiology. Original articles published by the RBA from January 2005 to February 2008 were analyzed after a manual search focusing on the words random, randomized, randomly, random distribution, blind, double-blind, placebo, or any other word that suggested the study could be classified as a randomized clinical assay. Randomized clinical assays were separated and underwent quality evaluation. Box I shows the quality scale¹ used in this search.

Criteria described by the quality evaluation scale used in this study included ¹:

a) For random distribution: the method generating the random sequence was considered appropriate when it gave each study subject the same chance to receive each intervention and when the investigator could not predict what the next treatment would be.

b) For randomized double-blind studies: a study was considered double-blind when the expression double-blind was used. The method was considered appropriate when both patient and the investigator responsible for data collection could not identify the type of treatment received or, in the absence of this declaration, whether the use of identical or imitation placebo was mentioned.

c) Losses and exclusions: individuals enrolled in the study who did not finish the observation period or who were not included in the analysis and were described by the authors of the original study. The number and reasons for losses in each group have to be mentioned. The lack of losses should also be mentioned in the study. When losses were not mentioned, this item received a score of zero.

A maximum of five points could be obtained with this scale, in which: three points for each "yes" and an additional point for an adequate randomized distribution method. When the expression double-blind was not present, but the method was described, this item received a score in the quality scale. A study was considered of bad quality when it received two points or less in the quality scale ¹.

Original randomized clinical trials were also analyzed according to the random distribution and confidentiality of the allocation.

Secondary parameters are shown as percentages. The number of authors is expressed as mean ± SD. The CI 95% for each score was calculated. Calculation of the size of the study population considered the hypothesis that 10% of RCTs were of good quality, with an absolute precision of 9%, and a significance level of 5%, resulting in 42 original studies.

RESULTS

One hundred and fourteen studies were identified of which 36.8% (42/114, CI 95% 27.9% to 45.6%) were considered randomized clinical trials. Table I shows the results of the analysis using the scale.

In the quality assessment, 92.9% (39/42 95% CI 85.1% to 100%) received 2 points or less and were classified as having poor methodological quality, and 7.1% (3/42 CI 95% 0% to 14.1%) had scores greater than 2 and were classified as having good methodological quality ^6-8.

Random distribution was described in 35.7% (15/42 CI 95% 21.2 to 50.2%) studies and the confidentiality of the distribution was described in 21.4% (9/42 CI 95% 9% to 33.8%) of the studies.

As for analysis by the Ethics on Research Committee, 93.85% (107/114 CI 95% 89.4% to 98.2%) stated that the project was analyzed by this committee and in 6.15% (7/114 CI 95% 1.8% to 10.6%) this item was not described ^9-15.

Regarding the informed consent, 58.8% (67/114 CI 95% 49.8% to 67.8%) of the studies described its use and 25.4% (29/114 CI 95% 117.4% to 33.4%) did not; 14.9% (17/114 CI 95% 8.4% to 21.4%) were animal studies; and in 0.9% (1/114 CI 95% 0% to 2.63%) its use was not necessary.

The source of the grant was not mentioned by any of the studies.

The sample size was calculated in 14.9% (17/114 CI 95% 8.4% to 21.4%) of the studies, and it was not described in 85.1% (97/114 CI 95% 78.6% to 91.6%).

The mean number of authors per study was 4.49 (± 1.54).

Tables II and ^III show the place of origin of the studies.

Table IV shows the statistical tests used.

A level of significance of 5% was used in 85.1% (97/114 CI 95% 78.6% to 91.6%) of the studies; 0.9% used a level smaller than 1% (1/114 CI 95% 0% to 2.6%); 2.6% (3/114 CI 95% 0% to 5.5%) did not describe this parameter; and 11.4% (13/114 CI 95% 5.6% to 17.2%) did not use any statistical tests.

Classification of the original studies demonstrated that 36.9% (42/115 CI 95% 28.1% to 45.7%) were randomized clinical studies; 33;3% (38/114 VI 95% 24.7% to 41.9%) were prospective studies; 14.9% (17/114 CI 95% 8.4% to 2.4%) represented animal studies; 12.3% (14/114 CI 95% 6.3% to 18.3%) were transversal studies; and 2.6% (3/114 CI 95% 0% to 5.5%) represented other types of studies ^9,12,16.

DISCUSSION

The reader should aware of the design, conduction, analysis, and interpretation of the results of randomized clinical trials to understand why it was undertaken and how it was conducted and interpreted ¹⁷. This objective can only be achieved with transparency from researchers and those responsible for the publication of the journals ¹⁷. Inadequate description of a study can make its interpretation difficult or impossible ¹⁷, hindering the application of its results in daily clinical practice.

Evaluation of 114 original studies demonstrated that 36.8% were classified as clinical randomized studies. In the present study, randomized clinical trials could reach a score of five in the quality scale ¹ in which two points were attributed to the random distribution, to two points to double-blind, and one point for losses and exclusions. Only three original studies ^6-8 had a score above two and were classified as having good methodological quality. The lack of description of the randomized allocation method, as well as the description of losses and exclusions, were the greatest limitation of the clinical trials evaluated in the present study to score points in the quality scale ¹.

Fifteen of 42 randomized clinical studies (35.7%) described the method of random allocation. Lack of adequate description of the method of randomized allocation could be associated with biases in the assessment of the effectivity of the interventions ¹⁷. The choice of the expressions random distribution or randomized study was not enough to describe the sequence of distribution; for this reason, 27 out of 42 (64.3%) randomized clinical trials did not score points in this item of the quality scale ¹.

In the present study, 20 out of 42 (47.6%) studies were described as double-blind, and only 11 of them (26.2%) were considered adequate. A double-blind study protects the sequence of randomized distribution after allocation ¹⁸ and, consequently, the correct description of this item is extremely important. Open studies can have deleterious consequences, since the knowledge of the intervention received can affect the psychological and physical responses of the study subjects ¹⁸. Researchers' bias in favor or against the intervention can be transferred directly to the participants through their attitude, and this can even encourage or discourage participants to remain in the study ¹⁸. When the person responsible for data collection is not blinded to the procedures, his/her bias in favor or against the intervention can attribute a higher or lower value to the parameters analyzed ¹⁸. The expression double-blind was adopted in this study as an indication that the intervention was blinded in the study; however, in the absence of this expression, we looked in the description for an indication it was a double-blind study by looking for other terms that indicated the character of the study, to give it the proper score. But this did not increase the number of studies with positive scores in this item.

The concomitant description of losses and exclusions after the random distribution of patients was not found in any of the 42 original randomized clinical studies and, therefore, none of the studies had a positive score in this item. One study described the losses (2.4%), and seven described the exclusions (16.7%). Access of the reader to losses and exclusion rates and their causes is important for the proper evaluation of the viability of instituting the intervention in his/her daily practice.

The success of the random distribution depends on two processes: generation of the random sequence of distribution, and the confidentiality of the allocation ¹⁸. The confidentiality of the allocation was described in nine of 42 studies (21.4%) evaluated. An inadequate methodological design in controlled studies, especially in those with poor confidentiality of allocation, is associated with biases¹⁹. The confidentiality of the allocation should not be mistaken with the double-blind because the latter is intended at preventing selection bias and protecting of the allocation sequence before and until they are used ¹⁸.

In the present study, we observed that 7 out o 114 randomized clinical assays (6.15%) did not mention the approval by the Ethics on Research Committee; 29 (25.4%) of 114 studies did not mention the use of an informed consent; and 100% of the studies did not mention the source of the grant. Those items are also analyzed by other scales and it is listed as an integral part of the assessment of the methodological quality ²⁰. As for the Ethics Committee, among the studies with a negative classification, one referred to a literature review ⁹, two presented permits for clinical research ^10,11, one was approved by a laboratory ¹², and three did not mention this item ^13-15.

The level of significance of 5% was used in 97 of 114 studies (85.1%). This level indicates a lower than 5% probability that the result of the study can be attributable to chance ²¹. The choice of the level of significance in a study is arbitrary, and it is up to the investigator to choose the most adequate; however, 5% is the level used in most studies ²¹, including the present study.

Analysis of the statistical tests used showed that the Student t test was used more often (47.4%). Hypothesis-testing and the confidence interval can be used in statistical analysis ²¹. Hypothesis-testing is expressed as a p value, and means the probability that an event will occur in the sample, even if this event is null in the population that originated the sample ²¹. In the studies analyzed here, the p value was mentioned when a statistical test was used, even when the level of significance was not mentioned.

In 97 of 114 studies (85.1%), the calculation of the sample size was not described. The size of the study population influences inversely the value of p, and, for this reason, very large study populations tend to present low p values and induce errors in decision making regarding the differences found in the study ²¹. Absence of the calculation of the sample size raises doubts about the validity of the results, since the value of p can be over- or underestimated. The calculation of the sample size also has ethical implications because the use of the adequate number of participants prevents the exposure of a greater number of individuals to a specific intervention. The knowledge of the number of participants also allows prediction of expenses involved in a given study. Formulas to calculate the sample size usually give a numerical result that should be used in each group involved in the study; however, in one study it was observed that the result obtained was equally divided among the study groups and this might have affected the results.

The mean number of authors per article was 4.49. A specific rule defining the ideal number of authors per study does not exist. The habit of including the name of persons who contributed directly or indirectly with the study, and not just the name of the main investigator, could probably explain this number.

As for the place of origin, São Paulo contributed with the majority of the studies, 60 out of 114 (52.5%). The difference related to other places of origin does not seem to be qualitative but quantitative, since São Paulo has a greater number of anesthesiologists, medical schools, and post-graduate courses per region.

The evaluation of the classification of the studies showed that 42 out of 114 (36.9%) and 38 out of 114 (33.3%) original studies were classified as random clinical assays and prospective studies, respectively. This reflects the intention of the authors to undertake studies involving an intervention or to observe the intervention after it is done. Random distribution is used in clinical trials to eliminate the possibility of biases that could lead to the development of systematic differences among the study groups ²². In some circumstances, due to the critical state of the patient or for ethical reasons, randomization is not feasible and in those cases evidence from observational studies is admissible ²². Prospective studies can also be used to investigate the parameters of a population; however, a simple random distribution is recommendable ²².

The limitations of this study included: the use of only one scale and only one investigator to evaluate the quality of the studies. The quality scale used here ¹ only evaluates internal validity items, neglecting external validity items and the statistical method. Other means used to evaluate the methodological quality could have been used, such as the use of individual items and lists ²⁰. It is advisable that more than one investigator evaluate the studies and disagreements should be resolved in consensus meetings. Evaluation by two investigators would help decrease biases ¹.

The evaluation of the quality of randomized clinical trials is important to: observe the number of randomized clinical studies; avoid planning, execution, and dissemination flaws in future studies; and to determine the degree of reliability of the results of the studies. In the decade of 1990, an international group formed by statisticians, epidemiologists, and editors of biomedical journals developed the CONSORT (Consolidated Standards of Reporting Trials) statement to improve the quality of randomized clinical studies published ¹⁷. This instrument is composed of a list with 22 items to be verified and a flow chart with information on the four phases of a study (registration, random distribution, follow-up, and analysis). It should be consulted by every author of original studies who wants to publish randomized clinical trials.

After the manual search of original studies and evaluation of randomized clinical trials, 7.1% were considered as having good methodological quality.

REFERENCES

• 01. Jadad AR, Moore A, Carroll D et al. - Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials, 1996;17:1-12.
• 02. Moher D, Jadad AR, Nichol G et al. - Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials, 1995;16:62-73.
• 03. Castelo Filho A, Sesso RC, Atallah AN - Epidemiologia clínica: uma ciência básica para o clínico. J Pneumol, 1989;15:89-98.
• 04. Guyatt GH, Cook DJ, Sackett DL et al. - Grades of recommendation for antithrombotic agents. Chest 1998;114(suppl 5):441s-444s.
• 05. Schulz KF - Unbiased research and the human spirit: the challenges of randomized controlled trials. CMAJ 1995;153:783-786.
• 06. Helayel PE, Kroth L, Boos GL et al. - Efeitos da clonidina por via muscular e perineural no bloquio do nervo isquiático com ropivacaina a 0,5%. Rev Bras Anestesiol, 2005;55:483-490.
• 07. Helayel PE, Lobo G, Vergara R et al. - Volumes efetivos de anestésicos locais para bloqueio do compartimento da fáscia ilíaca: estudo comparativo duplamente encoberto entre ropivacaína a 0,5% e bupivacaina a 0,5%. Rev Bras Anestesiol, 2006; 56:454-460.
• 08. Contreras-Domingues V, Carbonell-Bellolio P - Profilaxia antiemética em cirurgia de abdome agudo. Estudo comparativo entre droperidol, metoclopramida, tropisetron, granisetron e dexametasona. Rev Bras Anestesiol, 2008;58:35-44.
• 09. Gonçalves LO, Cicarelli DD - Manobra de recrutamento alveolar em anestesia: como, quando e por que utilizá-la. Rev Bras Anestesiol, 2005;55:631-638.
• 10. Beato L, Camocardi G, Imbelloni LE. - Bloqueio de plexo braquial pela via posterior com uso de neuroestimulador e ropivacaína a 0,5%. Rev Bras Anestesiol, 2005;55:421-428.
• 11. Imbelloni LE, Beato L, Beato C et al. - Analgesia pós-operatória para procedimentos cirúrgicos ortopédicos de quadril e fêmur: comparação entre bloqueio do compartimento do psoas e bloqueio perivascular inguinal. Rev Bras Anestesiol, 2006;56:619-629.
• 12. Tanaka PP, Estèbe JP, Campos R et al. - Preparação, caracterização e avaliação in vitro de microesferas de bupivacaína em excesso enantiomérico de 50% (S75-R25). Rev Bras Anestesiol, 2008;58:15-22.
• 13. Carlos RV, Bittar CS, Lopes MR et al. - Variação da pressão sistólica como método diagnóstico da hipovolemia durante anestesia para cirurgia cardíaca. Rev Bras Anestesiol, 2005;55:3-18.
• 14. Bafatini A, Trindade RD, Gomes CR et al. - Anestesia para cirurgia bariátrica. Avaliação retrospectiva e revisão da literatura. Rev Bras Anestesiol, 2006;56:205-222.
• 15. Marcondes G, Soeiro FS, Ferreira EA et al. - Transporte de pacientes sem oxigenioterapia para a sala de recuperação pós-anestésica: repercussões na saturação de oxigênio e fatores de risco associados a hipoxemia. Rev Bras Anestesiol, 2006;56: 352-361.
• 16. Torres MLA, Yamaguchi ET, Fonseca US - Sistemas respiratórios com absorção de CO₂, circulares, valvulares: comparação do comportamento térmico entre sistema coaxial e convencional com diferentes fluxos de gás fresco. Rev Bras Anestesiol, 2005; 55:72-77.
• 17. Moher D, Schulz KF, Altman D - The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA, 2001;285:1987-1991.
• 18. Schulz KF, Chalmers I, Altman DG - The landscape and lexicon of blinding in randomized trials. Ann Intern Med, 2002;136:254-259.
• 19. Schulz KF, Chalmers I, Hayes RJ et al. - Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273:408-412.
• 20. Verhagen AP, De Vet HC, de Bie RA et al. - The Delphi list: a criteria list for quality assessment of randomized clinical trials conducting systematic reviews developed by Delphi consensus. J Clin Epidemiol 1998;51:1235-1241.
• 21. Cavalcanti AB, Akamine N, Sousa JMA - Avaliação Crítica da Literatura, em: Knobel E - Condutas no Paciente Grave. 3Ş ed. São Paulo, Atheneu, 2006;2635-2647.
• 22. Altman DG - Randomisation. BMJ, 1991;302:1481-1482.

*

Recebido do Hospital Geral do Estado Professor Osvaldo Brandão Vilela, Maceió, AL

Publication Dates

• Publication in this collection
  17 Apr 2009
• Date of issue
  Apr 2009

History

• Received
  04 Mar 2008
• Accepted
  25 Nov 2008