Reflections upon the article “Evaluation of the impact that the changes in tuberculosis treatment implemented in Brazil in 2009 have had on disease control in the country”

TO THE EDITOR:

In the article by Rabahi et al.¹ published in volume 43, issue 6, of the JBP in 2017, the authors concluded that “The changes in tuberculosis treatment [fixed-dose combination implemented by the Brazilian Ministry of Health] were unable to contain the decrease in cure rates, the increase in treatment abandonment rates, and the increase in [multidrug-resistant tuberculosis] MDR-TB rates, being associated with increased mortality from pulmonary tuberculosis during the study period.” Considering that this statement published in the major means of scientific dissemination of the Brazilian Thoracic Association may have a major impact not only on the Brazilian medical community but also on the health professionals engaged in the fight against tuberculosis, we would like this letter to be likewise published in the JBP. Some comments must be made about methodological issues that certainly influenced the conclusions of the aforementioned study.¹

It is known that evaluation of the level of scientific evidence should be a routine activity of health professionals, but various barriers prevent this from happening. Studies on the impact of public health program interventions require the application of specific methods that consider both the use of an appropriate study design and well-constructed theoretical causal models. Since the discovery of Mycobacterium tuberculosis as the causal agent of tuberculosis, various models of disease determination have been proposed.² Initially, these models were uni-causal, based only on this etiologic agent. However, successive failures to control tuberculosis have led to the recognition of a broad range of potential disease determinants, and the uni-causal models have been replaced by complex models, which, in addition to the aspects related to the agent, include determinants ranging from those related to the person with tuberculosis to those related to the social and programmatic context that surrounds him or her.² Complex causal models have also been proposed to study interventions. Therefore, attributing solely to a new treatment the outcomes of an intervention, that is, stating a single cause relationship, is an important conceptual limitation since it disregards the multi-causal complexity at play, especially if observational studies are proposed instead of studies with experimental designs, such as randomized clinical trials or even cluster randomized trials.

Conversely, the use of interrupted time series analysis techniques requires meeting some conditions, the most important of which being that the only change affecting the outcome measure in the period is the intervention of interest.^3,4 An article by Linden,⁵ which was used by Rabahi et al.¹ as a reference for performing interrupted time series analysis, also reinforces that caution is needed in drawing inferences when potential confounding factors, such as concomitant policies and programs, vary during the study period. It is known that, during the period studied by Rabahi et al.,¹ other important changes occurred that could affect treatment outcomes, such as the lack of nationwide use of tuberculin testing within the health care system; the improvement in diagnosis, with the implementation of the Xpert MTB/RIF assay⁶; and the economic crisis that unequally affected the population at highest risk for unfavorable treatment outcomes because of their social vulnerability.

In addition to the limitation that potential confounding factors were disregarded, there is the fact that the treatment was not implemented uniformly in Brazil, with implementation occurring early in some states and later in others. In the study by Rabahi et al.,¹ the intervention time frame chosen does not seem appropriate, given that the study that validated the implementation of the supervised treatment in the health care system was completed only in September 2010, in five cities surveyed.⁷ Therefore, during data analysis, line fitting should consider heterogeneity in the treatment’s adoption and use (whether treatment was supervised or not) and Family Health Program coverage by city, as well as socioeconomic variables.⁸

The inferences drawn by the authors must also be considered, since not detecting a relationship between an exposure and an outcome should not be interpreted as “there is no relationship between them.” The study by Rabahi et al.¹ could not detect the impact of the new treatment on cure and treatment abandonment rates, and it is not correct to state that “the changes in tuberculosis treatment were unable to contain the decrease in cure rates, the increase in treatment abandonment rates, (. . .)” because the inability to verify this relationship may be due to the low statistical power of the study. Additionally, some results were presented in a format that is difficult to interpret, such as in Figure 2,¹ given that there are confidence intervals that include null values but show p values less than 0.05 (Figures 2C and 2G). Furthermore, Figure 2G presents a line with a positive slope and a negative estimate for the parameter.

Therefore, we consider that important methodological limitations and misinterpretation of results have led to conclusions with a low level of scientific evidence, and disseminating this knowledge without criticism is inconsistent with good practices of collective health and health research.

REFERENCES

• ¹
  Rabahi MF, da Silva Júnior JLRD, Conde MB. Evaluation of the impact that the changes in tuberculosis treatment implemented in Brazil in 2009 have had on disease control in the country. J Bras Pneumol. 2017;43(6):437-444. https://doi.org/10.1590/s1806-37562017000000004
  » https://doi.org/10.1590/s1806-37562017000000004
• ²
  Maciel EL, Reis-Santos B. Determinants of tuberculosis in Brazil: from conceptual framework to practical application. Rev Panam Salud Publica. 2015;38(1):28-34.
• ³
  Penfold RB, Zhang F. Use of interrupted time series analysis in evaluating health care quality improvements. Acad Pediatr. 2013;13(6 Suppl):S38-44. https://doi.org/10.1016/j.acap.2013.08.002
  » https://doi.org/10.1016/j.acap.2013.08.002
• ⁴
  Bernal JL, Cummins S, Gasparrini A. Interrupted time series regression for the evaluation of public health interventions: a tutorial. Int J Epidemiol. 2017;46(1):348-355. https://doi.org/10.1093/ije/dyw098
  » https://doi.org/10.1093/ije/dyw098
• ⁵
  Linden A. Conducting interrupted time-series analysis for single-and multiple-group comparisons. Stata J. 2015;15(2):480-500.
• ⁶
  Durovni B, Saraceni V, van den Hof S, Trajman A, Cordeiro-Santos M, Cavalcante S, et al. Impact of replacing smear microscopy with Xpert MTB/RIF for diagnosing tuberculosis in Brazil: A stepped-wedge cluster-randomized trial. PLoS Med. 2014;11(12):e1001766. https://doi.org/10.1371/journal.pmed.1001766
  » https://doi.org/10.1371/journal.pmed.1001766
• ⁷
  Braga JU, Trajman A. Effectiveness of RHZE-FDC (fixed-dose combination) compared to RH-FDC + Z for tuberculosis treatment in Brazil: a cohort study. BMC Infect Dis. 2015;15:81. https://doi.org/10.1186/s12879-015-0820-4
  » https://doi.org/10.1186/s12879-015-0820-4
• ⁸
  Maciel EL. Sales CM, Bertoldi A, Reis-Santos B. O Brasil pode alcançar os novos objetivos globais da OMS para o controle da TB? Rev Epidemiol Servico Saude. [Epub ahead of print]

Authors’ reply

Authorship SCIMAGO INSTITUTIONS RANKINGS

In the first part of our article, we presented the annual data for the period from January 2003 to December 2014 regarding all 861,901 reported cases of pulmonary tuberculosis (PTB) in individuals aged 10 years or older, as recorded in the Brazilian Ministry of Health National Case Registry Database on October 18, 2016.¹ These data showed a decrease in the total number of cases of PTB, a decrease in the number of new cases of PTB, and a continuous decrease in cure rates, as well as a steady increase in the rates of recurrence, multidrug-resistant TB, and mortality (death from tuberculosis among patients diagnosed with PTB), together with high rates of treatment abandonment.¹ There was no sample selection, and, therefore, selection bias is not possible. We included all available official data, that is, the entire population treated in the period.

Subsequently, we used interrupted time series analysis (ITSA) to determine whether or not there was an association between the irrefutable worsening in the figures regarding PTB and the changes in tuberculosis treatment implemented in 2009.^2,3 At no time did we use the word “causality” mentioned in the letter to the editor. The distinction between “association” and “causality” has been well described in the literature.⁴

The most important variable in assessing infectious disease treatment outcomes is the treatment itself. Therefore, this condition was met in our study. According to Linden,⁵ when multiple observations of an outcome variable are available in the pre- and post-intervention periods, ITSA offers a quasi-experimental research design with a high degree of internal validity. According to the literature, one of the strengths of ITSA is the low interference from typical confounding variables that remain reasonably constant (e.g., socioeconomic variables) or change slowly (e.g., Family Health Program coverage or supervised treatment), because these variables are taken into consideration in the long-term trend model.⁶ Naturally, the use of the entire population (rather than a sample) strengthened the validity of our study by allowing the control of confounding variables omitted from the statistical analysis, which rejects the low statistical power hypothesis. In fact, because of its robustness, ITSA is used to assess the effects of community interventions, public policies, and regulatory actions; in addition, systematic reviews of the literature have increasingly been including studies that used ITSA as a data analysis tool.⁷

The lack of tuberculin testing (which is used to diagnose latent tuberculosis) and the implementation of the Xpert MTB/RIF assay (which is used to diagnose tuberculosis in patients with negative sputum smear microscopy results) do not affect treatment outcomes in patients with active PTB.

In the “Methods” section of our study,¹ it can be seen that the time frame considered for the implementation of the changes in tuberculosis treatment was from December 2009 to December 2010 (three months after the implementation validation study’s date of completion mentioned by the authors of the letter to the editor).

The suggestion that the inferences drawn in our article could not be drawn is in contrast with the literature.⁸ In the interpretation of a statistical test, the rejection of the null hypothesis (i.e., when the p value is significant) means that the variables are not independent (i.e., there is a relationship among them), and therefore the opposite is true.⁸ Thus, the terms used regarding the inference used in our text are entirely appropriate and correct.

The observation that there are confidence intervals that include null values but show p values less than 0.05 (Figures 2C and 2G) is valid.¹ However, the conclusion from this observation is wrong. It can be easily perceived that there was a misprint (a minus sign is missing before “4.76”). This can be proved by calculating the p value from the confidence interval.⁹ When placing a minus sign (β = −8.20; 95% CI: −11.58 to −4.76), we find a p value of 0.000003300, that is, p < 0.0001, as described in our study; this shows that the reported interval does not include null values. The same is true for Figure 2G and for the slope of the line where the parameters are positive (p = 0.00001356, i.e., p < 0.0001). Therefore, there were no misinterpretations or methodological limitations in our study, and the data analyzed allow all of the inferences and conclusions drawn in our article.

We thank the authors of the letter to the editor for the critical review of our article. The review allowed us to dispel doubts, clarify concepts, address aspects that we had not addressed, and contribute to the better understanding of ITSA, thereby significantly increasing the strength of the evidence that we presented.

REFERENCES

• ¹
  Rabahi MF, da Silva Júnior JLRD, Conde MB. Evaluation of the impact that the changes in tuberculosis treatment implemented in Brazil in 2009 have had on disease control in the country. J Bras Pneumol. 2017;43(6):437-444. https://doi.org/10.1590/s1806-37562017000000004
  » https://doi.org/10.1590/s1806-37562017000000004
• ²
  Soumerai SB, Starr D, Majumdar SR. How Do You Know Which Health Care Effectiveness Research You Can Trust? A Guide to Study Design for the Perplexed. Prev Chronic Dis. 2015;12:E101. https://doi.org/10.5888/pcd12.150187
  » https://doi.org/10.5888/pcd12.150187
• ³
  Wagner AK, Soumerai SB, Zhang F, Ross-Degnan D. Segmented regression analysis of interrupted time series studies in medication use research. J Clin Pharm Ther. 2002;27(4):299-309. https://doi.org/10.1046/j.1365-2710.2002.00430.x
  » https://doi.org/10.1046/j.1365-2710.2002.00430.x
• ⁴
  Gordis L. Epidemiology. 2nd ed. Philadelphia: Elsevier Saunders; 2000.
• ⁵
  Linden A. Conducting interrupted time series analysis for single-and multiple-group comparisons. Stata J. 2015;15(2):480-500.
• ⁶
  Bernal JL, Cummins S, Gasparrini A. Interrupted time series regression for the evaluation of public health interventions: a tutorial. Int J Epidemiol. 2017;46(1):348-355. https://doi.org/10.1093/ije/dyw098
  » https://doi.org/10.1093/ije/dyw098
• ⁷
  Cochrane Effective Practice and Organisation of Care (EPOC). London: EPOC; c2018 [cited 2018 May 20]. EPOC Resources for review authors, Interrupted time series (ITS) analyses. Available from: http://epoc.cochrane.org/epoc-specific-resources-review-authors
  » http://epoc.cochrane.org/epoc-specific-resources-review-authors
• ⁸
  Smith LF, Gratz ZS, Bousquet SG. The Chi-Square Test: Hypothesis Tests for Frequencies. In: Smith LF, Gratz ZS, Bousquet SG, editors. The Art and Practice of Statistics. 1st ed. Belmont: Wadsworth Cengage Learning; 2009. p. 407-20.
• ⁹
  Altman DG, Bland JM. How to obtain the confidence interval from a P value. BMJ. 2011;343:d2090. https://doi.org/10.1136/bmj.d2090
  » https://doi.org/10.1136/bmj.d2090

Publication Dates

• Publication in this collection
  May-Jun 2018