Accuracy of visual and image-based ICDAS criteria compared with a micro-CT gold standard for caries detection on occlusal surfaces

: The aim of this study was to check the in vitro accuracy of ICDAS criteria on digital images compared to visual examination for the diagnosis of occlusal caries against a micro-CT gold standard. ICDAS was scored in 40 extracted permanent molars by means of visual inspection and stereomicroscopic images. Visual examinations were performed in duplicate and at a one-week interval by three different calibrated examiners. The analysis of digital images by ICDAS criteria was also performed in duplicate, 1 month after visual examinations. The detection methods were compared by means of sensitivity, specificity, area under the curve, predictive positive and negative values, and accuracy for two different thresholds (1- sound vs. carious teeth; 2- tooth requiring operative vs. non-operative treatment). Sensitivity and accuracy values for threshold 1 in the visual ICDAS and image-based ICDAS methods were high for sensitivity (0.93 and 0.97) and for accuracy (0.83 and 0.85), but low for specificity (0.55 for both methods). Specificity values for threshold 2 were 0.77 and 0.82, while sensitivity was 0.33 and 0.28 for each method. Spearman’s rank correlation coefficient was 0.53 and 0.43 (p<0.05) for visual and image-based ICDAS compared to the gold standard scores. Both visual and image-based ICDAS scores were similar to each other in terms of diagnostic accuracy when compared to the micro-CT gold standard. Low specificity for the presence of caries and sensitivity for the detection of caries requiring operative treatment were found.


Introduction
The ICDAS (International Caries Detection and Assessment System) was developed to measure changes at the tooth surface level as related to potential histological depth of carious lesions. 1 This new scoring system was aimed at aiding in the initial detection of carious lesions during clinical examination, increasing the chance of arresting their progression. 2 Compared to histological gold standards, ICDAS has shown good accuracy in occlusal caries identification, Declaration of Interests: The authors certify that they have no commercial or associative interest that represents a conflict of interest in connection with the manuscript. especially in lesions located in the external half of the enamel. 3 Moreover, compared to different contemporary diagnostic methods, including fiber-optic transillumination (FOTI), quantitative light-induced fluorescence (QLF), autofluorescence, and optical coherence tomography (OCT), visual inspection and scoring with ICDAS resulted in similar performance compared to histological gold standard scores. 4 The application of digital photography in dentistry has improved treatment outcomes, especially when aesthetics is involved. Moreover, digital photographs improve perception and simplify interpretation of tooth-related characteristics by providing optimal conditions for examination, such as illumination and magnification. Finally, absence of patientrelated confounding factors, such as behavior, also contributes to a more objective evaluation. 5 When applied to the diagnosis of carious lesions, digital photography has increased caries detection sensitivity compared to visual examination. 6 In fact, DMFT scores obtained from the examination of digital photographs have been used to identify treatment needs, especially in remote areas or where dentists are not available. 7 Although most studies on sensitivity, specificity, reproducibility, and accuracy of dental caries diagnostic tools are performed against a histological gold standard (stereomicroscopy), 8 more recently, laboratory non-destructive methods have been proposed as possible surrogate gold standards for detection of carious lesions. 9 Histological detection of caries has some limitations, including the destructive and two-dimensional approach, difficulties in discriminating between histological dentin reactions and true demineralization, 10 and the applied qualitative-based scoring system, which may vary among studies. 11,12 Micro-CT is a laboratory non-destructive technique which allows reconstruction of linear attenuation coefficients measured from a series of X-ray projections into a pattern of gray scale values throughout the object's volume. If the gray scale values are calibrated against mineral density values of the material of interest, a map of mineral density profiles can be obtained. 13 Considering that the micro-CT technique has been successfully used to evaluate levels of mineral loss in dentin or enamel samples 14,15,16,17 and that digital images may help to provide dental care in remote or underprivileged areas, the aim of this study was to compare the in vitro accuracy of ICDAS in occlusal carious lesions scored by visual examination to ICDAS scored on digital images against a micro-CT gold standard.

Methodology Tooth samples
The present study was approved by the local Research Ethics Committee (Plataforma Brazil CAAE 54270016.3.0000.5257). Teeth with suspected carious lesions on their occlusal surfaces were selected from a pool of extracted erupted third molars collected during a three-month period after patients' written consent. Sound or restored teeth, those with extensive cavities with visibly exposed carious dentin, or teeth with color changes not related to carious involvement were excluded from the study.
Sample size was calculated using a formula established for diagnostic tests. 18 Mean sensitivity (0.73) and specificity (0.82) values for both diagnostic methods (visual and image-based ICDAS) were obtained from a previously published study, 6 assuming a 95% confidence interval and precision of 0.25. A total of 37 samples were then required, and 40 teeth were then selected. The specimens were cleaned with ultrasonic tips to remove calculus and debris and embedded by the roots in gypsum blocks to facilitate manipulation.

Visual and image-based examination
The occlusal surface was visually examined by three previously calibrated examiners. All examiners were pediatric dentists with a maximum of 5 years of clinical practice. Calibration included following the ICDAS e-learning program (90-minute course on ICDAS examination protocol and coding system application) 1 week before the start of the study (Table 1). During this training, a senior experienced researcher was available to establish consensus in case of disagreement between the examiners. For each diagnostic method (visual or image-based), ICDAS scores were obtained in duplicate, at a one-week interval, but the order in which the teeth were presented for being scored was modified by an independent researcher between the examinations to reduce possible bias. The specimens were examined after drying, with illumination from a dental unit and without using a probe. After visual examination and scoring, the occlusal surface of each tooth was imaged under 25X magnification, artificially illuminated with a white LED using a 10-megapixel digital camera coupled to a stereomicroscope (Opticam, São Paulo, SP), and stored in tiff format with a 300-dpi resolution and RGB color format.
One month after the visual examination, the images were sent by e-mail to each examiner, when new sets of ICDAS scores were to be attributed. For the image-based method, the images were also scored by each examiner twice at a one-week interval, and before the second examination, the examiners received new sets of images where the order of teeth to be scored was also randomly modified in relation to the first examination.
The final (agreement) score for each diagnostic method (visual or image-based) was obtained by selecting the mode value as given by all the examiners and examination period. If the mode resulted in two score values, the most aggressive one was used as the final score.

Micro-CT gold standard for carious lesions
After visual examination and color digital image acquisition, the specimens were scanned using high-energy micro-CT (1173, Bruker, Kontich, Belgium) with the following acquisition parameters: 70kV, 114µA, 7.12µm pixel size, 1-mm thick Al filter, 1s exposure, 0.5° rotation step at 360°, and 20-line random movements. Reconstruction of the projections into cross-sectional slices was performed using a proprietary software program (Nrecon, v.1.6.9.4 Bruker micro-CT) and the following parameters: ring artifact correction (5), beam hardening correction (75%), and standardized contrast limits between 0 and 0.1.
S even dipot assiu m hyd rogen phosphate (K 2 HPO 4 ) phantoms were prepared at the following concentrations: 0.3, 0.6, 0.9, 1.2, 1.5, 1.8, and 2.1 g/cm 3 . They were scanned and reconstructed using the same acquisition and reconstruction parameters as described above. A calibration curve was built to convert micro-CT attenuation (gray scale values) to hydroxyapatite mineral density, as described previously. 19 The reconstructed 3D images were visualized using an open-source software product (FIJI/ImageJ). 20 Crosssectional slices were resampled into buccolingual slices to enable visualization of caries penetration into the occlusal fissure. Two thresholds were established for caries detection: a. Threshold 1: presence vs. absence of caries (including enamel or dentin lesions); b. Threshold 2: presence vs. absence of a dentin lesion.
To detect any sign of caries (threshold 1), positive cases were identified by setting a cutoff point of 20% deviation from the mean gray scale level of sound enamel based on the histogram distribution of each specimen. 21 For dentin caries (threshold 2), the cutoff point was set as the demineralized dentin tissue that should be removed from the cavity (biomechanically failed dentin), based on previously published micro-CT data 17,22 and on biomechanical evaluations of demineralized dentin tissue. 23 In this case, values above 1.2 g/cm 3 of hydroxyapatite were considered as sound dentin. The whole stack of images for each tooth was analyzed.
After applying the cutoff points to the whole volume of each specimen, the following classification was used: E0 -no enamel lesion; E1 -enamel lesion without cavitation; E2 -cavitated enamel; D1 -demineralization in the outer half of dentin; D2 -demineralization in the inner half of dentin; and D3 -cavitated dentin lesion. Figure illustrates the representative samples of each ICDAS and micro-CT scores. Table 2 shows the micro-CT threshold definitions used in the present study to calculate specificity, sensitivity, and accuracy between the diagnostic methods.

Statistical analysis
Statistical analysis was performed using SAS ® statistical software package (SAS Institute Inc., Cary, NC, USA). Intra-examiner reproducibility of visual and image-based ICDAS was assessed using linearly weighted kappa coefficients. Kappa coefficients between 0.80 and 1.00 were considered excellent; those between 0.60 and 0.79 were considered substantial, and those between 0.40 and 0.59 indicated moderate agreement. 24 Inter-examiner linearly weighted kappa coefficients were calculated and compared against the calculated maximum kappa value. Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), accuracy, area under the curve (AUC), and Spearman's rank correlation coefficient were calculated as thresholds for both diagnostic methods.

Results
Mean kappa coefficient values for intra-examiner agreement are shown in Table 3 for the visual and image-based methods. Substantial or excellent agreement was found for both methods and examiners, except for examiner 2, who presented moderate agreement for the visual examination method. Mean kappa coefficient values were slightly increased for the image-based method compared to the visual method. Inter-examiner kappa calculations are shown in Table 4.
Cross tabulations of visual and image-based ICDAS mode values with micro-CT scores are presented in Table 5. Table 6 shows specificity, sensitivity, PPV, NPV, AUC, and Spearman's rank correlation coefficient for visual and image-based methods and the micro-CT gold standard scores.

Discussion
The current study compared ICDAS scores given to occlusal surfaces of extracted carious molars after visual inspection and image-based examination in order to investigate whether the use of digital images for caries diagnosis can be reliable compared to clinical examination.   It was possible to observe that both visual and image-based inspection methods showed excellent sensitivity (0.93 and 0.97, respectively) in distinguishing sound from carious elements (threshold 1: caries presence), thus corroborating the findings of previous studies in which a histological gold standard was used. 4,25 For detection of carious lesions requiring operative treatment, both methods showed poor sensitivity values (0.33 and 0.28, respectively). This may be partly explained by the fact that most accuracy studies testing ICDAS have been designed using stereomicroscopy as a gold standard for caries presence. It has been recently shown that stereomicroscopy has low accuracy in detecting dentin demineralization, underestimating the real depth of dentin involvement. 26 In fact, in the present study, when ICDAS 1-3 scores (enamel caries) were viewed according to the micro-CT gold standard, specimens were mostly already showing clear signs of dentin demineralization (Table 5), being therefore included in threshold 2.
Regarding specificity, both methods were similarly low for threshold 1 (caries presence), indicating that ICDAS might not be so effective in distinguishing truly sound teeth (0.55 for both methods). This has been also found in other studies, as visual examination presents some difficulties in distinguishing between active caries and non-carious hard tissue defects or inactive lesions. 25,27 Moreover, we also attribute this result to the type of gold standard used in the present study, which evaluated the lesions on a three-dimensional basis and relied on quantitative cutoff mineral density values. Collectively, the low sensitivity values for threshold 2 and low specificity values for threshold 1 found in the present study may emphasize the effect of an alternative micro-CT gold standard. This does not downplay the importance of ICDAS as an attempt to categorize dental carious lesions in the clinical setting, but draws attention to the need of comparing different gold standards for caries detection in terms of their ability to truly represent the diseased tissue.
For teeth included in threshold 2 (caries requiring operative treatment), both visual and photographic methods showed better specificity, demonstrating that both were effective in detecting teeth that do not need operative intervention. This is a very significant finding, since modern approaches for the control of carious lesions rely on minimally invasive interventions, thereby preserving as much sound tissue as possible. 28 The application of digital photography in dentistry has improved treatment outcomes, especially when aesthetics is involved. 5 When applied to the diagnosis of carious lesions, this method increases caries detection sensitivity compared to visual examination. 6 In fact, DMFT scores obtained from the examination of digital photographs have been used to identify treatment needs, especially in remote areas or where dental professionals are not available. 7 Thus, digital photography can be certainly employed to improve care for populations living in areas with poor access to health services, in addition to serving as a diagnostic tool for public health strategies.
Substantial or excellent intra-examiner agreement was observed in both methods and among almost all examiners in the present study. However, systematically lower values were found for inter-examiner agreement (Table 4). This may be attributed to the calibration method employed in the present study, which only required e-learning training in ICDAS. One of the main limitations on the more widespread use of ICDAS as a caries scoring system is the difficulty in achieving inter-examiner agreement, since each examiner is expected to have a different interpretation of visual signs based on personal visual acuity, neural signal processing, experience, and education. 25 A previous study found that e-learning sessions on ICDAS along with training sessions performed with instructors on extracted teeth improved the ability of dental students to apply the ICDAS criteria for diagnostic purposes. 29 In this study, calibration with e-learning sessions on ICDAS was performed to check whether a less demanding learning curve could be expected from this scoring system involving relatively experienced dental professionals. However, a low level of interexaminer agreement was found in the present study, corroborating the need of joint in vitro and/or in vivo sessions for an in-depth discussion about ICDAS scores before applying them to the general population.
In the present study, a micro-CT method was used as the gold standard. In fact, a recent study has shown that this method correlates well with a histological gold standard (0.917 for accuracy and 0.949 for AUC) when visual scores for the presence of caries are compared. 9 Some advantages of the micro-CT method is that it is non-destructive and provides mineral density values as outcomes of caries presence instead of possibly relying on dentin reactions to caries rather than on the effect of tissue demineralization. 10 Moreover, micro-CT allows threedimensional visualization and quantification of sound and diseased tissues. 14 In the reconstruction phase of the present study, tissue demineralization was precisely obtained with the use of dipotassium hydrogen phosphate (K 2 PO 4 ) phantoms to calibrate micro-CT values against hydroxyapatite mineral density values. We emphasize, here, the need of setting objective cutoff points for dentin demineralization on micro-CT images of carious teeth since visual detection of the intensity of gray scale values also implies subjectivity and ambiguity. 30 The present study demonstrated the feasibility of using a quantitative micro-CT approach to set gold standards for the presence of carious lesions.

Conclusion
Visual and image-based ICDAS criteria were similar to each other in terms of diagnostic accuracy when compared to the micro-CT gold standard. However, low specificity values for caries detection and low sensitivity values for detection of caries requiring operative treatment were found, corroborating the need of in-depth ICDAS training before clinical application of these criteria and also justifying more appropriate definitions of gold-standards for caries detection.