TG-ROC analysis of immunofluorescence assays in canine visceral leishmaniasis diagnosis

OBJECTIVE: To analyze the accuracy of the diagnosis of two protocols of indirect immunofluorescence assays for canine visceral leishmaniasis. METHODS: Dogs from the seroepidemiological survey conducted in an endemic area of the cities of Araçatuba and Andradina, in Northwestern São Paulo state, in 2003, and in a non-endemic area of the metropolitan region of São Paulo, were used to assess two protocols of indirect immunofluorescence assay (IFA) for leishmaniasis: one using a Leishmania major heterologous antigen (IFA-BM) and another using a Leishmania chagasi homologous antigen (IFA-CH). Two-graph receiver operating characteristic (TG-ROC) analysis was used to estimate accuracy. TG-ROC analysis compared 1:20 dilution readings of the homologous antigen (IFA-CH), considered as reference test, with IFA-BM dilutions (heterologous antigen). RESULTS: The 1:20 dilution used in the IFA-CH test showed the best contingency coefficient (0.755) and the highest strength of association between the two variables studied (chi-square=124.3). Thus, it was considered the test reference dilution in comparisons with different IFA-BM test dilutions. The best IFA-BM results were obtained from 1:40 dilutions with the best contingency coefficient (0.680) and highest strength of association (chi-square=80.8). With the change in the cut-off point, recommended for the IFA-BM 1:40 dilution in this analysis, the specificity parameter value rose from 57.5% to 97.7%, even though the 1:80 dilution showed the best sensitivity estimate (80.2%), with the new cut-off point. CONCLUSIONS: TG-ROC analysis can provide important information about diagnostic tests, in addition to offering suggestions on cut-off points that can improve test sensitivity and specificity estimates and assessing these tests in terms of the best cost-benefit ratio. DESCRIpTORS: Leishmaniasis, Visceral, diagnosis. Dogs. Fluorescent Antibody Technique. Diagnostic Techniques and Procedures. Sensitivity and Specificity. Seroepidemiologic Studies. 2 TG-ROC analysis in CVL diagnosis Silva RM et al American visceral leishmaniasis (AVL) affects 88 countries, with 90% of cases occurring in India, Bangladesh, Nepal, Sudan and Brazil.a In Brazil, AVL is considered endemic in 19 states, especially in the Northeast region, where a higher number of cases are reported and transmission patterns have changed. At first, this disease was considered predominantly rural and peri-urban. Nowadays, it has been recorded in major urban centers such as Rio de Janeiro, Belo Horizonte, Araçatuba, Southeastern Brazil, and Corumbá, Central-West Brazil.10,b,c In the state of São Paulo, the first report of the human disease’s autochthony was made in the metropolitan area of São Paulo, in 1978.10 However, at that time, it was not possible to identify the reservoir and vector in the links of transmission chain. In 1998, in the city of Araçatuba, the presence of Leishmania sp. was detected in direct parasitological examination of dogs with clinical suspicion of canine visceral leishmaniasis (CVL), subsequently identified as Leishmania chagasi. These facts, in addition to the presence of the vector insect, Lutzomyia longipalpis, notified in 1997, confirmed the autochthony of CVL in dogs living in the urban area of Araçatuba. In 1999, in this same city, the first autochthonous human case was reported.b In general, official CVL surveillance and control programs recommend canine serological surveys should be made, aiming to know the disease’s epidemiological situation in areas with active transmission or with potential transmission, with the concomitant identification of serologically positive dogs for subsequent destruction.b,c Studies emphasize the importance of test sensitivity and specificity parameters to assess the impact of destroying dogs, when compared to the serological methodology employed.3 Studies assessing the results obtained from epidemiological surveys performed in Belo Horizonte,1,2 using a kit obtained from the Leishmania major antigen, suggest that the use of indirect immunofluorescence assay (IFA) could compromise the effectiveness of the CVL Control Program. This is because IFA does not detect infected animals, due to false negative results, while it identifies non-infected dogs (false positive) with the resulting recommendation for destruction, according to the disease’s Control Program. INTRODUCTION a World Health Organization. Tropical disease research: progress 2003-2004. Seventeenth Programme Report of the UNICEF/UNDP/World Bank/WHO Special Programme for Research & Training in Tropical Diseases. TDR 2005. Geneva; 2005. (Programme Report, 17). Disponível em: http://www.who.int/tdrold/publications/publications/pr17.htm b Camargo-Neves VLF, Glasser CM, Cruz LL, Almeida RG. Manual de Vigilância e Controle de Leishmaniose Visceral Americana do Estado de São Paulo. São Paulo: Ministério da Saúde. Secretaria de Estado da Saúde; 2006. c Ministério da Saúde. Departamento de Vigilância Epidemiológica. Secretaria de Vigilância em Saúde. Manual de Vigilância e Controle de Leishmaniose Visceral; 2006. The present study aimed to analyze the accuracy of two protocols of indirect immunofluorescence assays for canine visceral leishmaniasis.

American visceral leishmaniasis (AVL) affects 88 countries, with 90% of cases occurring in India, Bangladesh, Nepal, Sudan and Brazil.a In Brazil, AVL is considered endemic in 19 states, especially in the Northeast region, where a higher number of cases are reported and transmission patterns have changed.
At first, this disease was considered predominantly rural and peri-urban.b,c In the state of São Paulo, the first report of the human disease's autochthony was made in the metropolitan area of São Paulo, in 1978. 10However, at that time, it was not possible to identify the reservoir and vector in the links of transmission chain.In 1998, in the city of Araçatuba, the presence of Leishmania sp. was detected in direct parasitological examination of dogs with clinical suspicion of canine visceral leishmaniasis (CVL), subsequently identified as Leishmania chagasi.These facts, in addition to the presence of the vector insect, Lutzomyia longipalpis, notified in 1997, confirmed the autochthony of CVL in dogs living in the urban area of Araçatuba.In 1999, in this same city, the first autochthonous human case was reported.b In general, official CVL surveillance and control programs recommend canine serological surveys should be made, aiming to know the disease's epidemiological situation in areas with active transmission or with potential transmission, with the concomitant identification of serologically positive dogs for subsequent destruction.b,c Studies emphasize the importance of test sensitivity and specificity parameters to assess the impact of destroying dogs, when compared to the serological methodology employed. 3Studies assessing the results obtained from epidemiological surveys performed in Belo Horizonte, 1,2 using a kit obtained from the Leishmania major antigen, suggest that the use of indirect immunofluorescence assay (IFA) could compromise the effectiveness of the CVL Control Program.This is because IFA does not detect infected animals, due to false negative results, while it identifies non-infected dogs (false positive) with the resulting recommendation for destruction, according to the disease's Control Program.The present study aimed to analyze the accuracy of two protocols of indirect immunofluorescence assays for canine visceral leishmaniasis.

METHODS
Tests were performed in two sample groups.
Group A was comprised of 94 serum samples of male and female dogs of different ages and breeds, of which 74 came from a seroepidemiological survey conducted in the cities of Araçatuba and Andradina, an endemic area in Northwestern São Paulo state.
A total of 20 seronegative dog samples from the city of São Paulo were added, an area considered to be without transmission at the time of collection (2003).
All dogs were assessed in terms of their clinical status, in addition to their having been serologically assessed for CVL diagnosis using IFA and direct parasitological examination.In this way, group A dogs were considered the reference population, of which 50 showed positive diagnosis, clinically classified as: symptomatic (30 dogs), oligosymptomatic (18 dogs) and asymptomatic (two dogs).There were 24 dogs with negative diagnosis that belonged to the endemic area and 20 that belonged to the non-endemic area.
Group B was comprised of 160 paired samples of serum and blood, collected on filter paper, from dogs belonging to the same cities and participating in the AVL serological surveys.A total of two IFAs were compared: one using promastigote Leishmania major forms (heterologous antigen), (IFA-BM, Biomanguinhos, Fiocruz -Oswaldo Cruz Foundation Immunobiological Technology Institute), and the other using promastigote Leishmania chagasi forms (homologous antigen) (IFA-CH, Laboratório de Patologia de Doenças Infecciosas, Faculdade de Medicina da USP -São Paulo University School of Medicine Laboratory of Infectious Disease Pathology).In both assays, samples were assessed semi-quantitatively, in 1:20, 1:40, 1:80 and 1:160 dilutions.Assays were performed in duplicate and samples were re-numbered, so that readings could be made as a blind test.
IFA readings were made in a fluorescence microscope with a magnification of x400, with the reading criterion adopted according to the density of fluorescent parasites and defined by a number of plus signs: 1+, a The readings of the immunoflorescence reaction on the microscope (x 400) can be seen in the online version of Revista de Saúde Pública, available from www.scielo.br/rspb Schoojans F. MedCalc statistic for biomedical research: software manual.Mariakerke: Medcalc Statistical Software; 1998.
2+, 3+ and 4+, resulting in a score.This score was converted into percentages, so that the quantitative variable became continuous, thus enabling the use of the Two-Graph Receiver Operating Characteristics (TG-ROC) technique. 7The reading a made with IFA-CH, which served as parameter for comparison: as a general rule, the microscope visual field was divided into four quadrants, each filled with fluorescent parasites equivalent to 25% fluorescence or 1+.Frequency distribution analyses for each serum dilution versus clinical forms, calculating chi-square and contingency coefficients extracted from contingency tables, were made for each set of results obtained from IFA-BM and IFA-CH readings.MedCalc statistical package b was used for this analysis.
Cut-off point estimates, in addition to sensitivity and specificity parameters, positive and negative predictive values and effectiveness were obtained using graphic analysis, created by the CMDT statistical package 8 's TG-ROC technique. 7

RESULTS
The readings of the IFA-CH and IFA-BM a indicate the immunofluorescence reaction with the IFA-CH homologous antigen is more specific than that with the IFA-BM heterologous antigen, which shows a nonspecific background reaction that usually prevents a correct reading.
Results found on Tables 1 and 2, referring to group A samples, show the distributions of readings of the clinical diagnosis variable versus the serum dilution variable in the different categories, in both assays (IFA-CH and IFA-BM).
The IFA-CH 1:20 dilution (Table 1) showed the best contingency coefficient (0.755) and the highest strength of association between the two variables studied (chi-square = 124.3).In addition, it was con-sidered the assay's reference dilution and used for the comparative study with different IFA-BM dilutions.For this assay, the best results were obtained with the 1:40 dilution, which showed the best contingency coefficient (0.680) and the highest strength of association (chi-square=80.8)(Table 2).
TG-ROC graphic analysis on Figure 1 shows the sensitivity and specificity parameter values, represented on the y axis, whereas the x axis shows the values of readings in plus signs, which were converted into percentages (values of cut-off points).TG-ROC graphic analysis compares the homologous antigen (IFA-CH) 1:20 dilution readings, considered as reference assay, with the readings of each heterologous antigen (IFA-BM) dilution (1:20, 1:40, 1:80 and 1:160) (Figure 1).1B), established by the analysis of group A TG-ROC, to group B's canine population.All the 160 paired samples of serum and blood, collected on filter paper, had already been assessed in 1:40 dilution, once this dilution is recommended on the IFA-BM kit label.However, at this dilution, fluorescence-reactive results that showed 1+ or 25% fluorescence were not seen as positive, once the cut-off point considered was equal to 26.70%, thus >25%.Figure 2A shows that the estimates of sensitivity and specificity parameters were 98% and the new cut-off point was 39.5%.The same cut-off point procedure was performed with the same dogs, whose blood was collected on filter paper, with sensitivity and specificity values of 73% and new cut-off point value of 37.2% (Figure 2B).
Table 3 shows the new estimates of sensitivity and specificity values, positive and negative predictive values, effectiveness and IFA-BM incorrect classification, based on the cut-off point provided by the TG-ROC analysis and using the IFA-CH 1:20 dilution as reference.The new cut-off points shown on Figure 1 resulted from new values for sensitivity and specificity parameters.

DISCUSSION
The TG-ROC graphic analysis 7 enables the cut-off point value, associated with the combination of sensitivity and specificity parameters, to be found.In this way, the diagnostic assay accuracy allows the distinction between ill individuals and non-ill (healthy) individuals.
In the present study, the TG-ROC graphic analysis was applied to group A population, comparing two diagnostic assays: one specific, using homologous antigens (IFA-CH), and the other using a heterologous antigen (IFA-BM).The best distinction between seronegative dogs (healthy) and seropositive dogs (probably ill) occurred with the IFA-CH 1:20 dilution, which showed the best contingency coefficient and chi-square (Table 1), whereas the 1:40 dilution was the one that best separated these two sub-populations for IFA-BM (Table 2).
In this way, the new cut-off point suggested at IFA-BM 1:20 and 1:40 dilutions (Figure 1), above 25% (1+), disregards 1+ results as positive, resulting in a negative character.In this case, with group A population, there is a significant difference between assays when their results are analyzed comparatively, without the cut-off point adjustment.This can be explained by the characteristics of cases near the cut-off point, which usually have low antibody titers and represents the range where cross-reactivity is usually observed.Studies that assessed the immunoenzymatic assay (ELISA), using different antigen extracts and including dogs from the CVL-endemic area, found the existence of cross-reactions with other parasitic diseases (Chagas' disease, dirofilariasis and babesiosis), with both L. chagasi homologous antigen and L. amazonensis heterologous antigen.However, when recombinant rk-39 and rk-26 antigens were used, no crossreactivity was observed. 15Another study 5 showed the occurrence of cross-reactions when three serological methods for CVL were compared: IFA, ELISA, and direct agglutination test (DAT).After analyzing 234 dog samples in the endemic area of Minas Gerais, IFA and ELISA specificity indices were low, 52% and 64% respectively, showing cross-reactions with dog serums infected with T. cruzi, L. braziliensis and E. canis, whereas DAT specificity was high, 95%, showing only one animal with E. canis, seropositive for CVL.Likewise, in other studies performed in Corsega, in the Mediterranean Sea, authors found 100% specificity for DAT. 5,13In contrast, there were yet other studies, using serum samples from dogs infected with Babesia sp. and from a dog with ehrlichiosis, that did now show cross-reactivity when samples were submitted to IFA for CVL. 16e best estimate for the specificity parameter value was found in the 1:40 dilution (Sp=97.7%),even though the 1:80 dilution shows the best sensitivity parameter estimate (Se=80.2%)(Table 3).These results show the possibility of choosing dilution cut-offs and making more adequate decisions for the cut-off point.As an example, the 1:40 dilution with a cut-off point higher than 26.7% shows the best specificity estimates and positive predictive value, preventing truly negative dogs from being destroyed.On the other hand, when the 1:80 dilution is selected, with better sensitivity index (80.2%)and lower specificity (79.5%),   the positive predictive value also decreases (53.2%), increasing the possibility of healthy dogs being destroyed.Regardless, the decision to be made implies knowledge about the costs and benefits the selected cut-off point may offer.
Table 3 shows values for incorrect classification of about 20%, 18% and 17%, in 1:20, 1:40 and 1:80 dilu-tions, respectively, indicating that this parameter must also be assessed in combination with sensitivity and specificity.Comparison of results obtained in each of the IFA-BM dilutions with the IFA-CH (reference assay) 1:20 dilution shows that many serums did not show agreement, probably due to antigenic differences in the two antigens and the decrease in antibody avidity, or yet due to a question of equivalence zone of the antigen-antibody reaction between serums analyzed.Thus, a decrease in sensitivity in the 1:20 and 1:40 dilutions was observed, compared to 1:80 and 1:160 dilutions, which show the incorrect classification parameter of 11%.The choice of the 1:40 dilution with a cut-off point above 25% would be better, because, although the estimate of sensitivity is 68.3%, the confidence interval is within the 1:80 dilution confidence interval, which shows a specificity of 80.2%.In addition, in the 1:40 dilution, the specificity parameter estimate is the one that shows the highest value, 98.0%, and the incorrect classification varies very little between the two dilutions (0.18 and 0.17).
In the literature, some authors consider that the best IFA dilution cut-off is 1:80. 12,14,15In the case of the IFA-BM, used in visceral leishmaniasis surveillance and control programs, the 1:40 dilution is usually recommended as dilution cut-off, although considering the 1+ reading as positive.Thus, the estimated sensitivity and specificity value of 57.50% results in a difference in specificity of almost 40% and, after reclassification, specificity was 97.50%.
Studies have observed high levels of sensitivity and specificity in serological tests used in canine surveys.
Review studies 1    Another relevant question concerning the use of IFA-BM in serological surveys is their use in samples collected on filter paper.This assessment was made with group B serums (Figure 2).This group was comprised of paired samples of serum and filter paper, although the majority of cases showed strongly positive results, that is, IFA readings above 2+ (50%).In this way, TG-ROC analysis showed very high sensitivity and specificity estimate values (98%) for serum samples, whereas the referred values were 73% for blood samples on filter paper, indicating a 25% difference between serum use and filter paper.There are several studies comparing results of IFA performed in paired samples of serum and eluates.Some studies 2,4 report low values for the sensitivity parameter of IFA with eluate, when compared to the ELISA method with serum.On the other hand, different authors found high agreement of results of this assay in serum and eluate samples. 15ese considerations are important when assessing diagnostic tests, especially as regards the sample used in the comparison.To avoid such bias, studies on validation of diagnostic tests 9 suggest guidelines on all variables that must be controlled to assess these tests.
In conclusion, the present study shows how TG-ROC analysis can provide important information about diagnostic tests, in addition to its offering suggestions on cut-off points that can improve the estimates of test specificity and sensitivity parameters and assess them in terms of the best cost and benefit.Finally, it provides more consistency when making decisions in the analyses of seroepidemiological survey results.

Figure 2
Figure 2 was constructed by applying the new cut-off point value of the 1:40 dilution (Figure1B), established by the analysis of group A TG-ROC, to group B's canine population.All the 160 paired samples of serum and blood, collected on filter paper, had already been assessed in 1:40 dilution, once this dilution is recommended on the IFA-BM kit label.However, at this dilution, fluorescence-reactive results that showed 1+ or 25% fluorescence were not seen as positive, once the cut-off point considered was equal to 26.70%, thus >25%.Figure2Ashows that the estimates of sensitivity and specificity parameters were 98% and the new cut-off point was 39.5%.The same cut-off point procedure was performed with the same dogs, whose blood was collected on filter paper, with sensitivity and specificity values of 73% and new cut-off point value of 37.2% (Figure2B).

Table 1 .
Distribution of frequencies observed in the semi-quantitative IFA reading