Open-access Evaluation of the initial response in clinical trial efforts for COVID-19 in Brazil

ABSTRACT:

Objective:   To describe the methodological characteristics and good research practices of COVID-19 interventional studies developed in Brazil in the first months of the pandemic.

Methods:  We reviewed the bulletin of the National Research Ethics Committee - Coronavirus Special Edition (Comissão Nacional de Ética em Pesquisa - CONEP-COVID) (May 28, 2020) and the databases of the International Clinical Trials Registry Platform (ICTRP), ClinicalTrials.gov, and Brazilian Clinical Trials Registry (Registro Brasileiro de Ensaios Clínicos - ReBEC) to identify interventional studies registered in Brazil that assessed drug type, biological therapy, or vaccines. We described their methodological characteristics and calculated their power for different effect magnitudes.

Results:   A total of 62 studies were included, 55 retrieved from the CONEP website, and 7 from registry databases. The most tested pharmacological interventions in these studies were: chloroquine/hydroxychloroquine, azithromycin, convalescent plasma, tocilizumab, sarilumab, eculizumab, vaccine, corticosteroids, anticoagulants, n-acetylcysteine, nitazoxanide, ivermectin, and lopinavir/ritonavir. Out of 22 protocols published on registry databases until May 2020, 18 (82%) were randomized clinical trials, and 13 (59%) had an appropriate control group. However, 9 (41%) of them were masked, and only 5 (24%) included patients diagnosed with a specific laboratory test (for example, reverse transcription polymerase chain reaction - RT-PCR). Most of these studies had power > 80% only to identify large effect sizes. In the prospective follow-up, 60% of the studies available at CONEP until May 2020 had not been published on any registry platform (ICTRP/ReBEC/ClinicalTrials) by July 21, 2020.

Conclusion:   The interventions evaluated during the Brazilian research response reflect those of international initiatives, but with a different distribution and a large number of studies assessing hydroxychloroquine/chloroquine. Limitations in methodological design and sample planning represent challenges that could affect the research outreach.

Keywords:  Coronavirus infections; Clinical trials as topic; Efficacy; Resource allocation Brazil; Clinical protocols

RESUMO:

Objetivo:   Descrever as características metodológicas e de boas práticas em pesquisa dos estudos de intervenção para COVID-19 desenvolvidos no Brasil nos primeiros meses da pandemia.

Métodos:   Revisamos o boletim da Comissão Nacional de Ética em Pesquisa - edição especial Coronavírus (CONEP-COVID) (28 de maio de 2020) e as bases International Clinical Trials Registry Platform (ICTRP), ClinicalTrials.gov e Registro Brasileiro de Ensaios Clínicos (ReBEC) para identificar estudos registrados no Brasil que avaliassem intervenções de tipo de medicamento, terapia biológica ou vacinas. Descrevemos as características metodológicas e calculamos o poder para diferentes magnitudes de efeito.

Resultados:   Foram incluídos 62 estudos, 55 identificados no site da CONEP e mais sete nas bases de registro. As intervenções medicamentosas mais frequentemente testadas nesses estudos foram: cloroquina/hidroxicloroquina, azitromicina, plasma convalescente, tocilizumabe, sarilumabe, eculizumabe, vacina, corticoides, anticoagulantes, n-acetilcisteína, nitazoxanida, ivermectina e lopinavir/ritonavir. De 22 protocolos publicados até maio de 2020 nas bases de registro, 18 (82%) eram ensaios clínicos randomizados e 13 (59%) tinham grupo controle adequado. Entretanto, nove (41%) eram mascarados e somente cinco (24%) incluíam pacientes diagnosticados com teste de laboratório específico (por exemplo, transcrição reversa seguida de reação em cadeia da polimerase - RT-PCR). A maioria desses trabalhos teria poder > 80% apenas para identificar grandes tamanhos de efeito. Em seguimento prospectivo, observamos que 60% dos estudos disponíveis na CONEP até maio de 2020 não estavam em nenhuma das plataformas de registro (ICTRP/ReBEC/ClinicalTrials) até o dia 21 de julho de 2020.

Conclusão:   As intervenções avaliadas durante a resposta brasileira em pesquisa refletem iniciativas internacionais, porém com distribuição diferente, tendo número elevado de estudos que avaliam hidroxicloroquina/cloroquina. Limitações no delineamento metodológico e planejamento amostral representam desafios que podem afetar o alcance dos trabalhos.

Palavras-chave:  Infecções por coronavírus; Ensaios clínicos como assunto; Eficácia; Alocação de recursos; Brasil; Protocolos clínicos

INTRODUCTION

The COVID-19 pandemic has led researchers from different areas to produce relevant scientific results in record time. One example is the vaccine from the University of Oxford, which began to be tested in humans a few months after the start of its development1. The demand for prompt answers popularized the open science movement and preprint platforms2. However, this "acceleration of science" can lead to the hasty implementation of works without peer review, the publication of studies with biases and low statistical power, and the concealment of conflicts of interest, among other potential issues3.

Brazil is a regional leader in research, with the largest number of citations and original publications in Latin America4. In response to the pandemic, Brazilian research support agencies published notices specific for COVID-195. This investment meets an urgent demand since the country has reached the top of global estimates of new cases of SARS-CoV-2 infection a few months after the first confirmed case in March 20206.

Brazilian research is backed by the action of agencies that ensure ethics in research. The National Research Ethics Committee (Comisssão Nacional de Ética em Pesquisa - CONEP) is responsible for elaborating regulatory norms and guidelines for research involving human beings, as well as evaluating whether these regulations are respected. Research protocols should be submitted to a Research Ethics Committee (REC) for consideration through the Brazil Platform, a national digital registry database7.

In addition, according to the Good Clinical Practices (GCP) and following the Declaration of Helsinki, all research involving humans should provide its protocol before recruiting the first participant8, ensuring data transparency and favoring the reduction in duplicated efforts. The International Clinical Trials Registry Platform (ICTRP), a World Health Organization (WHO) database, compiles different public domain platforms, including the Brazilian Clinical Trials Registry (Registro Brasileiro de Ensaios Clínicos - ReBEC)9 and the ClinicalTrials.gov, a database widely used in Brazil whose headquarters is located in the United States of America10. Prior protocol registration allows a later comparison with the scientific article to assess the methodological differences and their possible impacts on the results obtained11.

Amid the COVID-19 pandemic, all countries were challenged to elaborate health policies promptly, based on evidence, and effective in mitigating damages. However, the processes of knowledge production should be continually evaluated to identify challenges and enhance research. This investigation aimed to describe the characteristics of COVID-19 interventional studies started in Brazil during the first months of the pandemic, including the assessment of compliance with good practices in clinical research, and measure the statistical power to identify effects of different magnitudes.

METHODS

We conducted a systematic review to identify COVID-19 clinical trial protocols whose development started at the beginning of the pandemic (up to May 2020). Studies were found based on the CONEP-COVID Bulletin and a search in registry databases for clinical trial protocols. The eligibility criteria included the evaluation of drugs, biological therapies, or vaccines, and the recruitment of patients in Brazilian territory. Consequently, studies that assessed invasive procedures (for example, dialysis and surgery) and non-pharmacological therapies (e.g., psychological support and physical activity) were excluded.

NATIONAL RESEARCH ETHICS COMMITTEE BULLETIN

The 20th CONEP-COVID bulletin, issued on May 28, 2020, was accessed to identify studies that evaluated therapeutic interventions for COVID-19 through a title analysis conducted independently by two researchers (TBR and DOM or NAOS). Disagreements were resolved by consensus. The data extracted on May 30, 2020, included title, date of approval, CONEP identification number (Certificate of Presentation for Ethical Consideration/Certificado de Apresentação para Apreciação Ética - CAAE), and identification of the intervention.

REGISTRY DATABASES FOR CLINICAL TRIAL PROTOCOLS

An independent and triplicate search (TBR, TAM, and NAOS) was performed in the ICTRP/WHO database until May 30, 2020, with no language restrictions, to collect registered clinical protocols that met the eligibility criteria. Although the ICTRP/WHO includes ReBEC and ClinicalTrials, an additional search was carried out directly on the websites of these databases to retrieve any records that might not have been included in the ICTRP indexing.

Characterization of studies with published protocols

Two independent researchers (TBR and NAOS) extracted data from all clinical trials with protocols published on any of the registry databases investigated (ClinicalTrials.gov, ReBEC, and ICTRP/WHO). Only studies with their full protocol available until May 30 were included in the methodological characterization. The extracted data were: protocol identification number, title of the work, date of registration, institution responsible for the registration, federative unit of the recruitment institutions, study status, study design, total sample (n), clinical trial phase (that is, according to categories I to IV of intervention assessment), masking, characteristics of the population included, drugs allocated to the intervention and control arms, and data from primary and secondary outcomes.

Regarding secondary outcomes, this review presents only those that correspond to core outcomes for patients hospitalized due to COVID-19, which are the ones suggested by the initiative Core Outcome Measures in Effectiveness Trials (COMET) as relevant to these patients: all-cause mortality at hospital discharge and respiratory support (for instance, oxygen by mask or nasal cannula, oxygen with noninvasive ventilation or high flow, intubation and mechanical ventilation, or extracorporeal membrane oxygenation)12.

Works were classified as multicenter if they involved more than one recruitment center13. Interventions were defined as "preventive" or "treatment". In studies of this last category, patients were characterized according to the disease inclusion criteria (that is, only clinical suspicion or confirmed diagnosis by a specific laboratory test), the severity of symptoms, and hospitalization conditions.

As no SARS-CoV-2 treatments had been scientifically validated during the observation period, studies that compared the intervention with placebo or standard treatment were considered "relevant comparators".

STATISTICAL ANALYSIS AND POWER CALCULATION

Measures of central tendency for the main characteristics of the studies, as well as graphical representations, were obtained using Microsoft Excel®.

Given the sample sizes, we evaluated the statistical power to identify the difference of proportions using Yates's correction (with the software EPIDAT 3.1). We established the parameters for this calculation by considering two hypothetical scenarios: scenario 1, with moderate effect size (relative risk - RR of 0.6, incidence of 7.2 vs. 12%), inspired by a feasible magnitude reported in a study on remdesivir14; scenario 2, with large effect size (RR of 0.4, incidence of 10 vs. 25%)15. In these calculations, parallel randomized clinical trials were evaluated according to the comparison of two arms for independent samples; in non-randomized before-and-after clinical trials, we calculated the power for the comparison of paired proportions. Considering that a value above 80% is often regarded as sufficient16, we interpreted and classified the power according to the following categories: high ≥ 90%; good: between 80 and < 90%; low: between 50 and < 80%; and very low < 50%.

Analysis of the adherence to registration

The studies were included until May 30, 2020, but we followed these works until July 21 to verify if they were registered at both CONEP and protocol registry platforms (ClinicalTrials/ReBEC and ICTRP). Investigations were considered duplicates when the intervention, proposing institution, and number of participants were the same. Two independent researchers (TBR and NAOS) conducted the evaluation.

We also calculated the number of clinical trials that might not have been included in both CONEP and registry platforms, using the capture-recapture methodology17. According to this method, the number of studies not included in the two systems would be directly proportional to the product of the number of works registered at only one system and inversely proportional to the number of studies included in both. To that end, we used the following formula (with correction for small samples): studies not identified = (a*b)/(c + 1), in which "a" corresponds to the number of studies found only in CONEP, "b" to those retrieved only from registry platforms, and "c" to those available at both sources (CONEP and any registry platform).

Ethical aspects

This research does not require assessment by the Ethics Committee because it conforms to the first article, "research that uses public domain information," of the National Health Council (NHC) Resolution No. 510 from April 7, 2016.

RESULTS

Among the 370 studies approved in Brazil by CONEP until May 28, 2020, 55 were eligible. Fifteen of these studies were also available at registry platforms. Seven other works not found in the CONEP Bulletin were retrieved from registry platforms. Therefore, 22 full protocols were included from March 23 to May 30, 2020 (Supplementary Material, Figura S1). Among them, only one, by the WHO with an arm in Brazil18, does not have a Brazilian as its main researcher. Different interventions were evaluated in Brazilian studies, but, until mid-April, most of them involved hydroxychloroquine/chloroquine and azithromycin (Figure 1).

Figure 1.
Cumulative number of COVID-19 clinical trials in Brazil according to the main ongoing interventions, with CONEP approval or protocol published on RCT registry databases.

Only one of the registered protocols assessed a preventive intervention (hydroxychloroquine + zinc) for COVID-19; all others evaluated treatments. Seven (32%) studies were multicenter, including 4 with more than 20 research centers. Most works (82% or 18 of 22) were randomized clinical trials (RCT), 9 (41%) were masked at any level, and a little over half (13 or 59%) presented a placebo/standard treatment comparator (Table 1). Figure 2 demonstrates the comparison network of COVID-19 interventional studies in development in Brazil.

Table 1.
Characteristics of COVID-19 research protocols in development in Brazil available at clinical trials registry databases until May 2020.

Figure 2.
Comparison network of COVID-19 clinical trial protocols in development in Brazil until May 2020*.

Out of the 21 protocols that assessed treatment options for COVID-19, 16 (76%) proposed including patients with clinical suspicion - 5 included only patients with COVID-19 confirmed by a specific diagnostic test (for example, reverse transcription polymerase chain reaction - RT-PCR). Although only 38% of studies considered important outcomes for COVID-19 as primary ones, 86% regarded them as secondary. Individual study details are described in the complementary material.

Power calculation was performed for 21 studies because the SOLIDARITY protocol18 did not mention the number of Brazilian patients recruited. Seventeen (81%) of these works presented very low power (< 50%) in a conservative scenario (moderate effect). In scenario 2, 7 studies (33%) had very low power, 2 had low power, 2 had good power, and 10 (48%) had high power (≥ 90%) for associations of large magnitude (Table 1).

Considering the prospective follow-up, out of the 62 studies identified until May 2020, only 29 (47%) had their protocol registered at any of the platforms (ClinicalTrials.gov/ReBEC/ICTRP) by July 21, 2020. They included 22 of the 55 (40%) studies previously identified in CONEP. Using the capture-recapture method, we estimated that there could be another four works not registered at both CONEP and the registry platforms at the time of data collection. The details of the individual studies and power calculation can be accessed in the Supplementary Material.

DISCUSSION

Our study evaluated essential elements of the scientific community's initial reaction to the pandemic in Brazil: volume of studies, compliance with good research practices, and the power to identify associations that may contribute to the decision-making process.

Research in developing countries faces major challenges, such as financing difficulties and scarcity of capacity building19,20; nonetheless, even amid the crisis experienced by support agencies, Brazil stands out in Latin America3. As a response to the pandemic, we identified studies about drugs, vaccines, or biological therapies, many of which are also being studied around the world21.

We emphasize that the hospitals associated with major multicenter research initiatives are linked to the Support Program for the Institutional Development of the Public Health System (Programa de Apoio ao Desenvolvimento Institucional do Sistema Único de Saúde - PROADI-SUS). They receive federal incentives through tax exemptions, which they must return as actions for SUS22. Additionally, many patient recruitment sites are public hospitals and/or connected to educational institutions, such as universities or federal and state institutes, so these facilities stand out in their support for Brazilian research and its maintenance.

Some clinical trials with published protocols were not listed in the CONEP Bulletin, which might mean they were still under analysis. We underline that, by law, these protocols must be approved by an ethics committee prior to patient recruitment. Recently, we had a case of great repercussion in which a study was sanctioned due to ethical issues, which resulted in its suspension by CONEP23.

Among the studies available at the CONEP-COVID Bulletin, 60% were not found in protocol registry platforms (ICTRP/ReBEC/ClinicalTrials) during our follow-up. The explanation may be related to the duration of the process, but this finding is worrying, as this procedure is necessary for the transparency of research processes. Studies that do not have their protocols publicly available before starting have more difficulty in being accepted by journals with peer review.

The requirement to register the protocol helps prevent publication bias, among other benefits24. A survey revealed that, from 2010 to 2015, 10% of the clinical trial protocols published had not been previously registered, and that these studies were 38% more likely to present favorable results (95% confidence interval - 95%CI 20 - 58)25. The AllTrials initiative was launched in 2013 and, ever since, has become a global movement advocating the registration of study protocols and publication of their data26.

Despite the diversity of the interventions described in the protocols identified, the drugs assessed more often were chloroquine/hydroxychloroquine. Conversely, in a review of 155 ongoing studies on COVID-19 conducted worldwide, and that started until March 27, 2020, only 1.2% of the works evaluated this intervention21. Except for this antimalarial drug, the frequency of other interventions in Brazilian studies is similar to that observed in other parts of the world.

Some of the interest in chloroquine/hydroxychloroquine may have been motivated by their availability and price. However, their use is controversial since we currently have no evidence of its efficacy for treatment or prophylaxis. Also, some studies have suggested an increased incidence of severe adverse events, such as cardiovascular ones27. Examples such as these justify the reflection on the relevance of a high number of studies on the same intervention. In 2014, a series published by The Lancet ("Research: increasing value, reducing waste") highlighted some factors related to the waste in research, including inadequate methods, insufficient power, and analysis with risk of bias, among others11. Moreover, it emphasizes the importance of a comprehensive review of the evidence available to properly guide research efforts, focusing on interventions with higher benefit potential.

According to our sample size analysis, most of the evaluated studies had low or very low power to identify moderate effect sizes. This effect is feasible in the context of COVID-19 interventions since the estimate originates from the approximate difference observed for the mortality outcome in a clinical trial15. This finding reveals the need for planning and sample selection, as well as for the strengthening of cooperation, exchange of information, and the integration of research centers, which will allow a joint effort for developing investigations with greater power, increasing research value, and using resources efficiently.

Another important challenge is the definition of eligibility aspects. The inclusion of patients based solely on COVID-19 suspicion may induce a bias of access to diagnosis28, as many suspected cases are not confirmed29,30. This scenario might reflect the variation in the availability of COVID-19 diagnostic tests in Brazilian health services. Nevertheless, the inclusion of a very heterogeneous population may lead to validity issues due to the dilution of effects.

Another key planning aspect is the election of research outcomes. Those that are clinically relevant directly affect the data applicability to clinical practice. On the other hand, moderate outcomes, such as imaging or clinical tests, may not have enough clinical relevance and reduce the significance of the evidence produced, given that they do not always correlate with the target result or the result more important for the patient.

Studies should have a comparator group, randomization, allocation concealment, and masking to reduce the risk of bias. Most studies identified in our review used an appropriate control arm. However, the lack of masking was common (59%) and may introduce biases related to detection, performance, and outcome reporting, as well as represent greater chances of deviation by randomization violation and unplanned interim analysis31.

When extrapolating the study distribution from CONEP systems and registry platforms, we estimated that there could be around four works not reported in either of them. This estimate warns us about the necessity of monitoring good research practices to avoid problems such as publication bias. In this regard, we stress that scientific journals and manuscript reviewers must pay attention to the requirements for protocol registration and timely approval requests from research ethics committees before including the first patient.

The knowledge produced by research should contribute to society, and, accordingly, studies should be irreproachable and reliable. To this end, the recently published "Hong Kong Principles" elucidates the importance of responsible research practices, transparent reports, and the general need for an open science32. Thus, the publication of robust protocols is the first step toward scientific dissemination, and their wide circulation eliminates duplicates and promotes transparency.

As limitations of this study, we underline that the analysis of the report issued by CONEP was based on title review. This decision was made because the Brazil Platform provides a limited amount of free data, and additional information on ethical aspects was not evaluated. In addition, the scope of the present study was restricted to drug and biological therapies and vaccines. Therefore, other COVID-19 interventions were not considered, and the data were limited to the material available until the end of May 2020, which we regarded as the initial research response.

In conclusion, the allocation of research resources in a pandemic like the COVID-19 one should take into account ethical and logistic aspects, as well as the potential contribution to knowledge about effective health interventions. In this sense, we believe that the studies found reflect the response capacity of Brazilian research for tackling a health emergency. The selection of candidate interventions seems to mirror that of international initiatives, but with a different distribution, characterized by a higher proportion of studies evaluating hydroxychloroquine and chloroquine in Brazil than in other countries.

With respect to methodological aspects, most studies were randomized clinical trials and had an adequate control group. Nonetheless, we detected significant challenges, such as the lack of (or delay in) protocol registration in a relevant number of studies. Also, designs without masking and the inclusion of patients without laboratory-confirmed diagnosis posed a risk of biases related to detection/performance and selection, respectively. Furthermore, the power of most studies was appropriate only for effects of great magnitude, which indicates that documenting moderate effects would require meta-analyses or increasing collaborations for larger studies.

We believe that the continuous research evaluation will improve the quality of evidence, as well as strengthen the scientific response capacity in the face of major public health problems.

ACKNOWLEDGMENTS

We would like to thank Professor Dr. Moacyr R. Cuce Nobre for his contributions to the first draft of the manuscript, as well as Amanda Luzia Dias dos Santos Augusto and Stephanie Mary Zoltay for the notes on ethics in clinical research.

References

  • Financial support: No specific funding. FADQ is the recipient of a research productivity grant from the National Council for Scientific and Technological Development (Conselho Nacional de Desenvolvimento Científico e Tecnológico - CNPq), under No. 312656/2019-0.
  • ERRATUM
    https://doi.org/10.1590/1980-549720200104erratum In the manuscript “Evaluation of the initial response in clinical trial efforts for COVID-19 in Brazil”, DOI: 10.1590/1980-549720200104, published in the Rev bras epidemiol. 2020; 23: e200104. On page 7, Table 1.
    Where it reads:
    Table 1.
    Characteristics of COVID-19 research protocols in development in Brazil available at clinical trials registry databases until May 2020.
    It should read:
    Table 1.
    Characteristics of COVID-19 research protocols in development in Brazil available at clinical trials registry databases until May 2020.

Publication Dates

  • Publication in this collection
    06 Jan 2021
  • Date of issue
    2020

History

  • Received
    28 July 2020
  • Reviewed
    08 Sept 2020
  • Accepted
    21 Sept 2020
  • Preprint posted on
    25 Sept 2020
location_on
Associação Brasileira de Saúde Coletiva Av. Dr. Arnaldo, 715 - 2º andar - sl. 3 - Cerqueira César, 01246-904 São Paulo SP Brasil , Tel./FAX: +55 11 3085-5411 - São Paulo - SP - Brazil
E-mail: revbrepi@usp.br
rss_feed Stay informed of issues for this journal through your RSS reader
Acessibilidade / Reportar erro