The use of e-ASPECTS in acute stroke care: validation of method performance compared to the performance of specialists

Background: The rapid evaluation of non-contrast-enhanced computed tomography (NCCT) brain scans in patients with anterior stroke symptoms saves time and favors optimal and prompt treatment. e-ASPECTS is a tool that automatically calculates the Alberta Stroke Program Early CT Score (ASPECTS) values, leading to a more accurate and timely image evaluation. Objective: To determine the ability of e-ASPECTS in differentiating images with and without injury. Methods: One-hundred sixteen patients admitted to a stroke unit in a Brazilian tertiary hospital underwent a CT scan at admission and at least one control brain imaging (NCCT or magnetic resonance imaging – MRI) 24 hours after admission. ASPECTS evaluation was performed by three neuroradiologists, three neurologists, and three neurology residents, all blinded to the symptoms and the injury side. The scores were compared to the ground truth, and an ASPECTS score was provided by two independent non blinded evaluators. Sensitivity and specificity were analyzed, and receiver operating characteristic curves, Bland-Altman plots with mean error score, and Matthews correlation coefficients (MCCs) were obtained for ASPECTS scores, assuming values equal to 10 for images without injury and values other than 10 for images with ischemic injury. Results: e-ASPECTS demonstrated similar performance to that of neuroradiologists and neurologists, with an area under the curve of 0.78 and an MCC value of 0.48 in the dichotomous analysis. The sensitivity and specificity of e-ASPECTS were 75% and 73%, respectively. Conclusion: e-ASPECTS is a validated and reliable tool for determining early signs of ischemia in NCCT.


INTRODUCTION
Acute ischemic stroke management has advanced in the last three decades, from the introduction of intravenous thrombolysis to the validation of mechanical thrombectomy 1 . In all studies, brain image analysis has become a significant tool for verifying the presence of early ischemic changes 2,3,4,5 in order to exclude the presence of intracranial hemorrhage 2 and indicate the correct reperfusion therapy management 6,7 .
In recent years, a software-based machine learning algorithm (e-ASPECTS, Brainomix ® , Oxford, UK; www. brainomix.com) has been developed based on the Alberta Stroke Program Early Computed Tomography Score (ASPECTS), a simple and reliable method for assessing early ischemic stroke signs in non-contrast-enhanced computed tomography (NCCT) brain scan. e-ASPECTS is a 10-point quantitative score used in patients with symptoms suggestive of middle cerebral artery (MCA) ischemic stroke. The scores range from 10 (normal image) to 0 (all regions present early signs) 8 . e-ASPECTS has demonstrated a good correlation compared to assessments by neuroradiologists, stroke neurologists, and residents in neurology, but current analyses have been performed only in high-income countries. In low-and middle-income countries, the use of this software may improve time-totreatment, which is still far less than ideal 6,9 . The current study aimed to evaluate the ability of e-ASPECTS in identifying injuries in NCCT in a tertiary care hospital and compare the performance of e-ASPECTS with that of three groups of trained professionals.

METHODS
All patients with suspected acute ischemic stroke admitted to the Hospital de Clínicas of the Universidade Federal do Paraná between March 2017 and February 2018 were evaluated and considered for the current study. The inclusion criteria were as follows: 1) hospital admission within the first 24 hours of symptom onset; 2) NCCT performed upon admission; 3) anterior circulation stroke symptoms; 4) NCCT or magnetic resonance imaging (MRI) performed to confirm anterior circulation ischemic stroke 24 hours after admission.
The exclusion criteria were a diagnosis other than ischemic stroke, such as hemorrhagic stroke, the presence of transient ischemic attack (TIA), and the absence of stroke (stroke mimics).
All patients underwent brain NCCT upon admission and NCCT or MRI to confirm the ischemic stroke 24 hours after admission. All patients were treated according to the current guidelines.
Patients were examined on a multislice CT system using a 0.5-mm thick slice with 64 simultaneous detector rows (Aquilion 64, Toshiba Medical Systems, Japan). Axial brain NCCT images were reconstructed in 2-mm and 5-mm thick slices using FC64 kernels.
NCCT images were retrospectively analyzed by three neuroradiologists (NR1, NR2, NR3) with at least 5 years of experience in stroke imaging analysis, three neurologists (N1, N2, N3) with at least 5 years of experience in acute stroke care, and three neurology residents (R1, R2, R3) with one year of experience in stroke medicine. All groups were trained on the ASPECTS method prior to the analysis of the images, and they were able to discern acute and chronic signs of ischemic stroke. All evaluators were blinded to the symptoms and the injury side.
e-ASPECTS is a fully automated software that quickly calculates the ASPECTS value based on machine learning algorithms. This software can detect acute and non-acute hypodensity in NCCT, but only early signs of acute ischemic stroke are scored in the ASPECTS final value 10 .
The ground truth corresponded to the e-ASPECTS values provided by an independent stroke neurologist and a neuroradiologist who were not blinded to any clinical information. The values were based on CT or MRI images after 24 hours, as well as all other imaging or clinical information in the patient medical records.
A receiver operating characteristic (ROC) curve was calculated for each evaluator to obtain a dichotomized ASPECTS score based on normal (ASPECTS=10) or abnormal (ASPECTS<10) findings. The objective was to evaluate whether the observer had identified anormalCTwhen the CT was in factnormal, and, similarly, whether they had identified an injury where there was an injury.
Sensitivity, specificity, positive predictive value, negative predictive value, and accuracy were calculated for e-ASPECTS and all three groups, using the standard method and the sandwich estimator, the generalized estimating equation (GEE). While the standard method presents the results obtained by the simple proportion of individuals without considering other effects of the sample, GEE reveals the average sensitivity and specificity across all patient-specific estimates of sensitivity. GEE, therefore, is interpreted as the weighted average across the study population 11 and is more robust than the traditional model 12 . The Matthews correlation coefficient (MCC), which ranges between -1 and +1, was calculated for each data point. For an optimal demonstration of error distribution, Bland-Altman plots and histograms were constructed for the three neuroradiologists and e-ASPECTS. Data

RESULTS
Among the 445 initially selected patients, 116 fulfilled the inclusion criteria of the study. We excluded 279 patients due to hemorrhagic stroke, non-anterior stroke, symptom improvement, or lack of NCCT or MRI data after 24 hours. Another 100 patients had different final diagnoses (TIA and stroke mimics). Table 1 presents the baseline characteristics of the sample. e-ASPECTS showed an accuracy of 74.14%, which was only lower than that of one neuroradiologist (83.62%) and higher than the accuracies of all other experts and neurology residents ( Table 2).
When analyzing the ROC curve, e-ASPECTS had area under the curve values similar to those of all experts and higher than those of two neurology residents (Figure 1). MCC demonstrated a positive correlation (0.48) for e-ASPECTS, which was only lower than that for one neuroradiologist (0.68). One neurology resident presented a negative MCC (-0.04).

DISCUSSION
The present study showed that, considering all diagnostic criteria, e-ASPECTS has similar performance to that of neuroradiologists and neurologists in evaluating early signs of ischemia from NCCT; thus, e-ASPECTS is a safe method that can be used in stroke care. e-ASPECTS is a fast and reliable tool for the initial evaluation of stroke images. Furthermore, recent investigations have shown a strong relationship between e-ASPECTS and clinical outcomes 13 .
The current study confirms previous noninferior results of e-ASPECTS compared to neuroradiologists in the analysis of early signs of acute ischemic stroke from NCCT 14 . In addition, this study corroborates the good performance of e-ASPECTS that had been previously reported by Herweh et al. 15 .
e-ASPECTS does not seem to have the same performance as neuroradiologists when the images demonstrate other lesions besides the early signs of ischemia, including leukoaraiosis, old infarcts, and atypical parenchymal defects such as calcification. Although e-ASPECTS is able to differentiate between old and recentinfarcts, previous studies have not found a significant correlation for patients with pre-existing changes 10 . Our research did not exclude patients with old infarcts or any other lesion. Nevertheless, these confounders did not affect the software performance 10 .

A B
As a reliable tool comparable to neuroradiologist assessments, e-ASPECTS can assist neurology residents in their acute stroke image analysis training. The lower performance observed among neurology residents may be improved with the use of e-ASPECTS, which can check their mistakes in real time.
To the best of our knowledge, this study was the first to validate e-ASPECTS in a large developing country. Nonetheless, this study has limitations. This was a single-center investigation with a small sample size. The independent evaluators used different machines and screening tools to assess the images. Due to operational and availability issues, diffusionweighted MRI (DW-MRI) was not the only method used for ground truth determination, as was the case in other studies 15 . Some follow-up images were obtained after thrombolysis, which may have altered the final area of infarction.
In conclusion, the current study found that e-ASPECTS demonstrated similar performance to that of neuroradiologists and neurologists, but it performed better than neurology residents in differentiating NCCT images with or without early signs of ischemia, thus making it a reliable method in Brazil.