Does my patient have chronic Chagas disease ? Development and temporal validation of a diagnostic risk score

Introduction: With the globalization of Chagas disease, unexperienced health care providers may have difficulties in identifying which patients should be examined for this condition. This study aimed to develop and validate a diagnostic clinical prediction model for chronic Chagas disease. Methods: This diagnostic cohort study included consecutive volunteers suspected to have chronic Chagas disease. The clinical information was blindly compared to serological tests results, and a logistic regression model was fit and validated. Results: The development cohort included 602 patients, and the validation cohort included 138 patients. The Chagas disease prevalence was 19.9%. Sex, age, referral from blood bank, history of living in a rural area, recognizing the kissing bug, systemic hypertension, number of siblings with Chagas disease, number of relatives with a history of stroke, ECG with low voltage, anterosuperior divisional block, pathologic Q wave, right bundle branch block, and any kind of extrasystole were included in the final model. Calibration and discrimination in the development and validation cohorts (ROC AUC 0.904 and 0.912, respectively) were good. Sensitivity and specificity analyses showed that specificity reaches at least 95% above the predicted 43% risk, while sensitivity is at least 95% below the predicted 7% risk. Net benefit decision curves favor the model across all thresholds. Conclusions: A nomogram and an online calculator (available at http://shiny.ipec.fiocruz.br:3838/ pedrobrasil/chronic_chagas_disease_prediction/) were developed to aid in individual risk estimation.


INTRODUCTION
Chagas disease is increasingly under control in Latin America (1) , but it is spreading with the migration of Latin Americans (2) (3) (4) .The burden created by Chagas disease is currently similar to or exceeding those of other prominent diseases globally (5) .Several countries in which Chagas disease was not considered to be endemic until the 1990's are identifying cases of Chagas disease among immigrants, and occasional transmission through blood transfusion or organ transplantation has also been observed (4) .Therefore, unexperienced health care providers may eventually need to decide whether or not to screen or conduct diagnostic investigations for chronic Chagas disease.
Current guidelines for diagnosing chronic Chagas disease recommend mainly serological tests and occasional molecular tests (2) (6) (7) (8) (9) (10) (11) (12) (13) .However, explicit guidelines for which patients should undergo diagnostic investigation are scarce.This lack of formal recommendations makes screening, diagnostic investigations, and decision-making less rigorous and leads to more individual choices by physicians.
The problem with diagnostic investigation of Chagas disease is that up to half of patients with chronic Chagas disease have the indeterminate form (14) (15) , and many of those with cardiac or gastrointestinal involvement are asymptomatic (14) .Therefore, it is challenging, even for health care providers with many years of experience in the field.This study aimed to develop and validate a diagnostic decision support tool for chronic Chagas disease.

Participants
This is a phase 3 diagnostic research project (16) , which was conducted between April 2008 to May 2012 (development phase) and from June 2012 to July 2014 (validation phase) at Evandro Chagas National Institute of Infectious Diseases [Instituto Nacional de Infectologia Evandro Chagas (INI)] -Oswaldo Cruz Foundation [Fundação Oswaldo Cruz (FIOCRUZ)] in Rio de Janeiro, Brazil.Volunteers were selected sequentially from all medical appointments in the study period.These patients sought Chagas disease diagnosis and follow-up after referral or of their own volition.The inclusion criteria were (a) suspicion of chronic Chagas disease and (b) written consent to serve as a volunteer.The exclusion criteria were (a) previously diagnosed Chagas disease with supporting test results, (b) inability to comply with procedures of the research protocol, (c) suspected acute Chagas disease, (d) and pregnancy.

Procedures
Potential volunteers were interviewed for research screening by a physician.Once written consent was obtained, the volunteer was evaluated using a structured interview with a template questionnaire, which was recorded in the medical chart.Serological tests, electrocardiogram (ECG), and chest radiographs were ordered at this first evaluation.Within 20 to 40 days after the screening visit, the volunteers returned to the outpatient clinic.At the end of the medical evaluation, the ECG, radiographs, and serological tests were assessed and discussed with the patient.During the medical interviews, pictures of kissing bugs, mud houses, leishmaniasis-mediated ulcers, and Romaña's signs were shown to the patients.Professionals conducting clinical interviews, ECG results, and radiographs, as well as the serological test results were blinded to serological tests results and vice-versa.Therapy and follow-up were offered as judged necessary, according to current guidelines (6) .

Predictors
Information about potential Trypanosoma cruzi infection, symptoms, and findings on complementary tests were investigated as potential predictors of Chagas disease, along with history of living in a rural area, history of living in mud houses, recognizing the kissing bug in pictures, history of blood transfusions, history of siblings with Chagas disease, dysphagia or persistent constipation, ECG findings, and enlarged heart on chest radiographs with oral barium contrast.

Reference standard
The following commercial serological tests were used as reference tests for Chagas disease classification in different research periods according to test availability at INI: Wiener lab's ELISA (Wiener Lab, Rosario, Argentina), Pathozyme Chagas (Omegam Diagnostics, Scotland, UK), ELISA Biozima Chagas (Lemos Lab, Argentina), WAMA's Immuno-con Chagas (WAMA, São Paulo, Brazil), Biocientifica Immunofluor Chagas (Biocientifica, Buenos Aires, Argentina), Chagas-ELISA (Ebram Produtos Laboratoriais Ltda, São Paulo, Brazil), and Anti-Chagas Symbiosys (Symbiosys Ltda, São Paulo, Brazil).These tests were performed according to the respective manufacturer's instructions in the immunodiagnosis laboratory at INI. Diagnostic investigations were conducted and interpreted as recommended by the Brazilian consensus on Chagas disease (6) .Briefly, patient samples were submitted for two serological tests conducted in parallel, an enzyme linked immune-sorbent assay (ELISA) and an indirect immunofluorescence (IIF) test.Patients were classified as with Chagas disease if both serological tests were positive, and they were classified as without Chagas disease if both serological the tests were negative.If there was disagreement among serological tests results leading to an inconclusive diagnosis, additional blood samples were collected to perform the serological tests until a definitive diagnosis was reached.

Ethical considerations
The project was evaluated and approved by the institutional review board/ethic committee for research with human subjects, registered at SISNEP with the number 0045.0.009.000-07.Procedures followed were in accordance with the ethical standards and all volunteers signed a written consent.

Data analysis plan
Multiple imputations with chain equations were conducted to fill missing data (17) .After comparison of several models, a logistic regression was chosen.Potential predictors matching the following conditions were not explored: a) unacceptable reliability (data not shown); b) less than 10 events; or c) colinearity in the full model with a variance inflation factor higher than 10.Continuous predictors were tested for functional form with restricted cubic splines and were truncated if a range without relationship to the outcome was detected.Backwards removal of predictors from a full model was applied, and only predictors identified as significant at 5% using the Akaike information criteria (AIC) were retained in the final model, even if after penalizing the model the p values were higher than 5%.The full model was initially composed of signs, symptoms, and history of exposure available at the moment of diagnostic investigation.Additionally, other information (e.g., referral and comorbidities) that we thought could be relevant in clinical decision-making was also evaluated.This was initially based on chronic Chagas disease guidelines and our own practice.
The full model initially included sex, age, referral from blood bank, history of living in a rural area, history of living in mud houses, recognizing the kissing bug from pictures, history of Romaña's sign, history of blood transfusion, systemic hypertension, history of coronary disease, history of stroke, number of siblings, number of siblings with Chagas disease, mother with heart disease, number of relatives (brothers, sisters, mother, or father) with history of stroke, use of medicines for congestive heart failure, any evidence of dysphagia, any evidence of constipation, heart rate, ECG with low voltage, 1 st degree atrioventricular block, anterosuperior divisional block, 3 rd degree left bundle branch block (LBBB), pathologic Q wave, altered repolarization, right bundle branch block, atrial fibrillation, any kind of extrasystole, and any evidence of heart failure on radiographs.Internal validation was conducted using a bootstrap procedure.This procedure estimates the model optimism for later penalization, and it provides bias-corrected indices.Non-parametric area under the receiver operating characteristic (ROC) curve, Brier score, and Nagelkerke-Cox-Snell-Maddala-Magee R-squared were estimated as internal validity performance measures.Two-graphic receiver operating characteristic (TGROC) analysis (18) was conducted to analyze the trade-offs between sensitivity and specificity across the range of estimated risks.An inconclusive range of predicted risks was defined as the range for which both the sensitivity and specificity were below 0.95.Net benefit decision curves were also plotted (data not shown) to estimate the number of true positives gained from using the model, compared to results without using a model, in the range of risk thresholds (19) .This analysis allows the identification of a range of thresholds, which when used result in a model that is superior in correctly classifying patients compared to treat-all and treat-none strategies.Calibration was tested using several statistics, including the calibration belt (20) .A nomogram and an online calculator were constructed to estimate the probability of having chronic Chagas disease.R-project software (R Foundation for Statistical Computing, Vienna, Austria) (21) (with packages epicalc, rms, givitiR and shiny) was used.

RESULTS
Most of the patients who were not included after screening did not consent or were aware of a previous Chagas disease diagnosis (Figure 1).In the end, 740 patients (602 in the development cohort and 138 in the validation cohort) were included in the analysis.The prevalences of patients with initial inconclusive results were 4.5% in the development cohort and 0.7% in the validation cohort.All initially inconclusive diagnoses were either classified as with Chagas or without Chagas after conducting serological tests in a third blood sample.
Chagas disease prevalences were 19.9% in the development cohort and 17.4% in the validation cohort.Most patients were from several different Brazilian states, but there was also one patient from Peru, two from Bolivia, one from Portugal, and two from the United States of America.Most patients had sought health care because of physician referrals, while smaller numbers had relatives diagnosed with Chagas disease or were referred from blood banks.When patients were referred from other physicians, the most common reason for referral was heart disease, followed by esophageal disease (Table 1).
There were slightly more women than men, (Table 1) with a mean age of 47.69 years (standard deviation = 15.72 years).All patients were currently living in urban areas, but the majority reported lived in rural areas and/or in mud houses at least once in their lifetime, and they recognized the kissing bug from pictures.Few reported previous blood transfusions, but a relevant number of volunteers reported prior blood donations.Almost one-third of the volunteers reported having a mother with Chagas disease.
More than half of the ECGs were considered abnormal (Table 2).The most frequent findings were sinus bradycardia, anterosuperior divisional block, right bundle branch block, altered repolarization, extrasystole, and sinus dysrhythmia.Although most ECGs were considered abnormal, the prevalence of individual ECG diagnosis (e.g., pathologic Q wave) was low.Signs of heart disease on radiographs were noted in nearly 20% of volunteers (Table 2).
Sex, age, referral from blood bank, history of living in a rural area, recognizing the kissing bug from pictures, systemic hypertension, number of siblings with Chagas disease, number of relatives (brothers, sisters, mother, or father) with history of stroke, ECG with low voltage, anterosuperior divisional block, pathologic Q wave, right bundle branch block, and any kind of extrasystole remained as predictors after applying the predictor selection strategy in the full model (Table 3).The bootstrap procedure estimated an optimism of 0.0542 for R 2 , 0.1013 for the intercept, and 0.1189 for the slope.After penalization, the model resulted areas under the ROC curve (c statistic) of 0.904 and 0.912, R 2 values of 0.537 and 0.477, and Brier scores of 0.087 and 0.095 (from a maximum of 0.159), for the development and validation cohorts, respectively.(Figure 2) The calibration plots and their statistics show an excellent relationship between the actual and predicted values in the development and validation cohorts.
TGROC (data not shown) shows that when the predicted risk was between 43.5% and 100%, the model had at least a 95% probability of correctly identifying those without Chagas disease (specificity).When the predicted risk was between the 0 and 7.7%, it has at least 95% probability of correctly classify those with Chagas disease (sensitivity).Although there is uncertainty in a considerable range of predicted probabilities, the decision curves (data not shown) show higher net benefit with use of the model compared to use of individual variables alone, and the test-all (or treat-all) strategy, even with a decision threshold as low as 2%.This finding indicates clinical utility at any decision threshold.
Scores were assigned for each of the predictors (Table 3).To help the reader to determine the risk of a patient having chronic Chagas disease using this score, a nomogram was provided (Figure 3).The patient's individual clinical characteristic scores must be identified by drawing a vertical line from the characteristic axis toward the Points upper axis.These individual scores, which are found for each clinical characteristic, must be manually summed, and a vertical line should be drawn from the Total Score axis toward the Chronic Chagas probability axis.Alternatively, one may access the online calculator at http://shiny.ipec.fiocruz.br:3838/pedrobrasil/chronic_chagas_disease_prediction/.According to the preset minimum required 95% sensitivity and specificity of the model, a range of chronic Chagas disease risks between 7% and 43% results in an inconclusive characterization, and further testing would be recommended.Below or above this range, further testing would not be recommended, as it would not substantially change the predicted risk.

DISCUSSION
We initially wondered if (a) it is possible to estimate individual risk of Trypanosoma cruzi infection with reasonable accuracy before serological tests and (b) if there is evidence that the model is clinically useful when compared to the treat-all (or test-all) strategy.
When conducting a clinical evaluation of patients suspected of having chronic Chagas disease, physicians may realize that signs and symptoms may be misleading, as more than half of the patients have no symptoms or signs (14) .In the natural history of Chagas disease progression, ECG signs of Chagas disease (e.g., complete right branch block) are detectable before symptoms, such as palpitations and syncope, for the   cardiac form of the disease.On the other hand, for the digestive form, symptoms, such as dysphagia and chest pain, are detectable before signs on complementary tests (e.g.achalasia in images with barium contrasts) (14) .Thus, the history of exposure to infection is one of the main factors in the diagnostic investigation of chronic Chagas disease.Exposures to T. cruzi may occur through contact with the kissing bug, blood transfusion, mother-to-child or oral transmission, or by other less-frequent events, including laboratory accidents and organ transplants.Intuitively, physicians will try to characterize to the method of infection.Most patients currently live in urban areas and lived in rural areas only during childhood and adolescence.However, the mean age at diagnosis is very advanced, and because there may be a period as long as sixty years between potential exposure and diagnostic investigation, patient recall concerning events of interest may be limited.For the same reason, it is challenging to patients to recall signs and symptoms potentially related to the acute phase of Chagas disease, such as Romaña's signs or persistent febrile illness with liver enlargement.Determination of T. cruzi infection among relatives of patients with Chagas disease may suggest that the disease is commonly spread among households.Variables, such as number of siblings and having a relative with heart disease, indirectly predict exposure.This is also challenging because many of the relatives were never investigated, because of either personal desires or excessive distance from a health facility.Even when patients definitively state that their siblings do not have Chagas disease, it is likely that they have never been tested.
There are also some difficulties involving use of ECG abnormalities to diagnose disease.There is no general consensus on which abnormalities could be attributable to Chagas disease.This is particularly true for those abnormalities that are loosely correlated to poor prognosis of chronic Chagas heart disease, such as sinus bradycardia and sinus dysrhythmia, for which guidelines are not explicit and compatible (2) (6) .
Serological testing for Chagas disease is relatively simple and inexpensive.However, the prediction tool is intended for non-specialists and health care providers who are less experienced with Chagas disease, for settings where this condition is not frequent, or for settings where laboratory testing is not easily accessible and screening may be advisable.External validation and its impact on patient care will provide further support of its clinical utility.Such evaluations are desirable as a part of decision-making in clinical practice and increase the strength of the evidence provided here.
Many will agree intuitively that as more information is required, there is more difficulty in using a model in clinical practice.However, the reduction of the number of predictors also progressively reduced the overall accuracy, the calibration quality, and the net benefit across a range of decision thresholds.In addition, although some patients sought diagnostic investigation due to gastrointestinal signs or symptoms, it was not possible as part of this research study to conduct imaging tests to explore esophageal or intestinal   involvement.Therefore, this model may be less accurate for patients with gastrointestinal involvement exclusively.Despite its limitations, the evidence supports using this tool in decision-making.It is intended to be used to screen patients suspected of chronic Chagas disease with an evidence-based rationale by healthcare providers with less experience with this condition or in settings where further laboratory tests are not easily accessible.Further external validation studies with a fully independent sample and data impact studies on the improvement of patient care will improve this model and/or to support its clinical utility and widespread use.Such evaluations are recommended in order for the decision-making tools to become widely adopted in clinical practice (35) .
In conclusions, the results presented here show that a combination of variables within a clinical evaluation, including ECG findings, allow clinicians to accurately estimate chronic Chagas disease risk, either indirectly through summing scores or directly through the online calculator, before serological testing.

TABLE 3 -
Scores, regression coefficients, standard errors, Wald's test values, and odds ratios for chronic Chagas disease predictors in the final model.odds ratio; CI LL: confidence interval lower limit; CI UL: confidence interval upper limit; BB: blood bank; RBBB: right bundle branch block.

FIGURE 2 -
FIGURE 2 -Calibration plot and validation statistics for the development and validation cohorts.Dxy: Somers's D{xy} rank correlation between predicted and observed outcomes; C (ROC): area under the ROC curve; R 2 : Nagelkerke-Cox-Snell-Maddala-Magee R-squared index; D: discrimination index D; U: unreliability index U; Q: quality index Q; Brier: Brier score (average squared difference in predicted and observed outcomes); Intercept: calibration curve intercept; Slope: calibration curve slope; Emax: maximum absolute difference in predicted and calibrated probabilities; S: the Spiegelhalter Z-test for calibration accuracy and its two-tailed p-value; p-value: value for the GiViTI calibration test related to the calibration belt.Triangles in the development cohort are groups of 50 ordered predicted values, and in the validation cohort, triangles are groups of 10 ordered predicted values.

FIGURE 3 -
FIGURE 3 -Nomogram derived from a logistic model of estimation of chronic Chagas disease risk.Note:To use the nomogram one must find the corresponding points for each patient's individual characteristics by drawing a vertical line from the characteristic to the upper points axis.After doing so for every characteristic, one must sum the points and draw a vertical from the total score axis toward the chronic Chagas probability axis to find the patient's risk of having chronic Chagas disease.BB: blood bank; RBBB: right bundle branch block.