Validity and reproducibility of retinal arteriole and venule diameter measurements: ELSA-Brasil study. A cross-sectional study

ABSTRACT BACKGROUND: Investigation of alterations to retinal microvasculature may contribute towards understanding the role of such changes in the pathophysiology of several chronic non-communicable diseases. The objective here was to evaluate the validity and reproducibility of retinal arteriole and venule diameter measurements made by Brazilian Longitudinal Study of Adult Health (ELSA-Brasil) graders. DESIGN AND SETTING: Cross-sectional study at six teaching and research institutions. METHODS: To evaluate validity, each of 25 retinal images from the University of Wisconsin (gold standard) was measured by five ELSA-Brasil graders. To evaluate reproducibility, 105 images across the spectrum of vessel diameters were selected from 12,257 retinal images that had been obtained between 2010 and 2012, and each image was reexamined by the same grader and by an independent grader. All measurements were made using the Interactive Vessel Analysis (IVAN) software. Bland-Altman plots, paired t tests and intraclass correlation coefficients (ICCs) were analyzed. RESULTS: Mean differences between ELSA-Brasil and gold-standard readings were 0.16 µm (95% CI -0.17-0.50; P = 0.31) for central retinal artery equivalent (CRAE), -0.21 µm (95% CI -0.56-0.14; P = 0.22) for central retinal vein equivalent (CRVE) and 0.0005 (95% CI -0.008-0.009; P = 0.55) for arteriole/venule ratio (AVR). Intragrader ICCs were 0.77 (95% CI 0.67-0.86) for CRAE, 0.90 (95% CI 0.780.96) for CRVE and 0.70 (0.55-0.83) for AVR. Intergrader ICCs were 0.75 (95% CI 0.64-0.85) for CRAE, 0.90 (95% CI 0.79-0.96) for CRVE and 0.68 (95% CI 0.55-0.82) for AVR. CONCLUSIONS: Retinal microvascular diameter measurements are valid and present moderate to high intra and intergrader reproducibility in ELSA-Brasil.


METHODS
The Brazilian Longitudinal Study of Adult Health (ELSA-Brasil) is a multicenter study on 15,105 volunteer staff members (aged 35-74 years) at public institutions of higher education in six Brazilian state capital cities who were enrolled between 2008 and 2012.The overall objective of ELSA-Brasil was to investigate the epidemiological, clinical and molecular aspects of non-communicable chronic diseases, especially cardiovascular diseases and diabetes.ELSA Brasil was approved by the Ethics Committees of Hospital de Clínicas de Porto Alegre (under the registration number 06-194), Hospital Universitário da Universidade de São Paulo (669/06), Fundação Oswaldo Cruz (343/06), Universidade Federal de Minas Gerais (186/06), Universidade Federal da Bahia (027-06) and Universidade Federal do Espírito Santo (041/06), and all participants consented to their participation in the research. 15,16ciodemographic and health data from the ELSA participants were collected through interviews and examinations, as previously described in greater detail. 17Retinal images were obtained using a Canon CR-1 digital non-mydriatic retinal camera, coupled to a Canon EOS 40D digital camera (10 MP), and the images were shot at a 45º angle.Photographs of the macula and optic disc were acquired from both eyes of each participant.The retinal images were obtained in accordance with pre-established protocols, 18 without compression or zooming of images, so as not to lose quality. 1,19,20The images were then transferred in Digital Imaging and Communications in Medicine (DICOM) format to the study's central server, from which they were retrieved by the reading center.
A small fraction of the images were stored in .jpegformat, without compression, for transfer.
The supervisory ophthalmologist of the ELSA-Brasil retinal reading center was trained and certified by professionals in the Ocular Epidemiology Research Group of the Department of Ophthalmology and Visual Sciences, University of Wisconsin-Madison (UW), to make microvascular retinal measurements using the Interactive Vessel Analysis (IVAN) software. 20He then trained and supervised the graders who performed the analyses in ELSA-Brasil, who were healthcare professionals without prior knowledge of ocular fundus images.The five readers who performed the greatest numbers of readings (28% of the total) were chosen to participate in the evaluations of validity and reproducibility.
For the validity evaluation, 25 retinal images received from the Ocular Epidemiology Research Group (UW) that had previously been read were considered to be the gold standard for ELSA-Brasil readings.Five evaluators from the ELSA-Brasil study read each of the 25 images received, totaling 125 readings.
For the reproducibility evaluation, images from ELSA-Brasil participants that had previously been read by these five graders were selected for new readings.In order to select images representing the range of values obtained, the images were classified into quintiles of their values, separately for central retinal artery equivalent (CRAE), central retinal vein equivalent (CRVE) and arterio-venous ratio (AVR, i.e.CRAE/CRVE).Seven images were selected from each quintile of each measurement for the study, thus totaling 105 images, of which 21 had been originally measured by each of the five graders.
Since a high correlation of retinal vessel caliber between the right and left eyes (0.78 to 0.99 intra and intergrader) had previously been shown, 21

Statistical analysis
Shapiro-Wilk and Kolmogorov-Smirnov tests were used to evaluate the normality of the data.Differences between the readings were evaluated through Bland-Altman plots and were tested using the t test for paired samples.
The intraclass correlation coefficient (ICC) was used to evaluate reproducibility.Since each image was read twice by one grader and once by another grader, we used a two-factor random effects model (not nested), with estimation of variance components by means of the method of moments, 22

Validity
The retinal microvascular measurements of the sample of The Bland-Altman graph (Figure 1) showed that the differences obtained between the means of the microvascular readings performed by the five ELSA-Brasil graders, in comparison with the Wisconsin readers, were distributed in a similar way over the spectrum of values of these measurements.The maximum value for the difference was 1.78 μm for the arteriole diameter (CRAE) and, in 84% of the images, the differences did not reach 1.0 μm.Venule measurements (CRVE) showed a maximum difference of 1.87 μm and 80% of the images had differences smaller than 1.0 μm.For AVRs, the maximum difference was 0.009 and, for 76% of the images, the differences were smaller than 0.005.Bland-Altman plots (Figure 2) showed that the differences obtained between the repeated readings in ELSA-Brasil were distributed in a similar way over the spectrum of values.In the intragrader measurements, the differences were < 10 μm in 75.2% of the arteriole measurements (CRAE) and in 86.7% of the venous measurements (CRVE).For the variable arteriole/venule ratio (AVR), 90.5% of the sample showed a difference < 0.1.In the intergrader measurements, the differences did not reach 10 μm in 71.4% of the arteriole measurements (CRAE) and 74.3% of the venule measurements (CRVE).For the variable arteriole/venule ratio (AVR), 89.5% of the sample showed a difference < 0.1.Regarding validity, the differences in comparison with the gold standard were on average less than 1% of the absolute values of the diameters, as can be seen by comparing mean differences (Y axis) with mean values (X axis) in the Bland-Altman graphs.
These differences, besides being epidemiologically irrelevant, were not statistically significant.
In general, the reproducibility of measurements in epi-  stroke, obesity and dyslipidemia) 9,21,25,26 that seem to involve microvascular alterations in association with their development and/or progression.Aside from these better studied associations, retinal microvascular measurements have more recently also been shown to be associated with cognitive dysfunction, [31][32] the prevalence of complications in type 1 diabetes, 33 the internal carotid artery pulsation index 34 and changes to adiponectin levels. 35

CONCLUSION
In summary, our study suggests that the retinal microvascular measurements in the ELSA-Brasil have strong validity and moderate to high reproducibility within and across graders.The capacity for generalization of these findings to other studies is restricted to the use of the same techniques, software and training procedures.Future studies analyzing associations of these microvascular retinal measurements with clinical and subclinical measurements of disease in ELSA-Brasil may contribute towards better understanding of changes within the disease-health process and towards prediction of the onset of these diseases.

SUPPLEMENTARY MATERIAL
The statistical basis for measurement of the intraclass correlation coefficient when different numbers of measurements are read by each grader, but there is a constant number of readings per person, as developed by Dr. Lloyd E. Chambless, was used in this study.We applied the method of moments, as follows.
Y is the variable whose repeatability we study.
For any ID =i, m i = j=1 p Σ n ij for all i, so m i = obs per i.
Assume m i =m for all ID.(Exclude observations for which this is false or keep only m if there are more).We will work with the mean sums of squares: where ) is 1 when n ij ≥ 2 and 0 otherwise, and Note that + cross-product terms with zero expectation. Where + cross-product terms with zero expected value.The coefficient of σ 2 β in the E (MSS β ) formula simplifies as follows: What is the coefficient of σ 2 β in the E (MSS α ) formula if n ij = w for all i, j? u = OBS per reader = nw, w = OBS per reader x ID, m = OBS per person = pw What is the coefficient of σ 2 α in the E (MSS β ) formula n ij = w for all i, j?
Thus, if n ij = w for all i, j then the formula for E (MSS β ) and E (MSS α ) simplifies to what SAS procglm would get.
We return to the other expected squares.
optical disc images of a single eye were used.The right eye was given preference for this.In this selection, images that did not open in the software, or which did not meet the minimum criteria for grading (low light, poor visual quality or less than four venules and/or arterioles present) were excluded.As mentioned earlier, to assess reproducibility, each of the five graders re-read each of their 21 selected images, thus totaling 105 repeat readings.To evaluate intergrader reproducibility, each image was re-read by another grader, again totaling 105 repeated readings.Thus, considering both the original readings and the intra and intergrader repeated readings, the reproducibility evaluation was based on 315 readings (105 original readings + 105 readings repeated by the same reader + 105 readings repeated by another reader).The third readings were performed on average two years after the first readings (on the same images).The readers on this occasion were blinded with regard to the identity of the previous reader and to the values previously obtained.

Figure 2 .DISCUSSION
Figure 2. Reproducibility evaluation on Bland-Altman plots of the intra and intergrader differences in retinal microvascular measurements.Dashed lines indicate the 95% confidence interval (CI) of the mean difference, which is indicated by the black line.
demiological studies involves both biological variability and the variability of the entire data collection method.This can include the biological variability of the retina between repeated measurements, variability in capturing the images through photography and variability in the process of grading those images.Thus, a hypothetical overall ICC would represent the correlation between repeated retinal evaluations on an individual involving photography at different times, use of different cameras, assessment of the various images by different graders and repeated gradings by each grader.In this study, we only considered the reproducibility of grading.Regarding this reproducibility, the agreement between measurements performed repeatedly by the same evaluator was high for both arteriolar and venular diameter, with ICC > 0.77 and ICC > 0.90 respectively, and the concordance between observers was high, with ICC > 0.70 for both measurements.The intra and intergrader agreements for the AVR (ICCs of 0.70 and 0.68, respectively) were at the top of the "moderate" range of agreement.These results are compatible with those of other studies described in the literature.In the Atherosclerosis Risk Communities Study (ARIC), the reproducibility of the readings was estimated as 140 retinal images for intragrader agreement and 151 images for intergrader agreement.The concordance was high for the venous diameter, with ICCs of 0.89 and 0.77 for intra and intergrader agreement, respectively.
n ij = w for all i, j.Then m = j=1 p Σ n ij = pw = OBS per person, u = nj = i=1 n Σ nij = nw = OBS per reader, and mn = pwn = pu.

From©
2018 by Associação Paulista de Medicina This is an open access article distributed under the terms of the Creative Commons license.

Table 1 .
Comparison of microvascular measurements of the retina in the Brazilian Longitudinal Study of Adult Health (ELSA-Brasil) and at the University of Wisconsin Department of Ophthalmology (gold standard), n = 25 P-value of the t test for paired data.SE = standard error; CI = confidence interval.Figure 1.Validity evaluation on Bland-Altman plots of retinal microvascular measurements, comparing Brazilian Longitudinal Study of Adult Health (ELSA-Brasil) graders with those of the University of Wisconsin.Dashed lines indicate the 95% confidence interval (CI) of the mean difference, which is indicated by the black line.

Table 3 .
Additionally, although the average differences between the original and repeat readings were at times statistically significant, they were always less than CReproducibilityAmong the 105 images selected, 63.8% were from women, 44.8% from whites, 48.6% from participants aged 45-54 years and 61.0% from those who had completed a university degree; 25.7% were from individuals with hypertension and 20.0% from individuals with diabetes (Table2).The distribution of retinal microvascular values in the reproducibility sample (n = 105) approximated the values found overall in ELSA-Brasil (n = 12,257), as shown in

Table 2 .
Sociodemographic, anthropometric and disease characteristics of the Brazilian Longitudinal Study of Adult Health (ELSA-Brasil) sample that was used to evaluate microvascular measurement reproducibility, n = 105

Table 3 .
Distribution of retinal microvascular measurements in the sample that was used to calculate reproducibility (n = 105) and in the whole cohort of the Brazilian Longitudinal Study of Adult Health (ELSA-Brasil) (n = 12,257)
26 71 and 69 subjects, respectively, with ICCs of 0.67 and 0.91, but also without indicating the microvascular measurement of the retina to which these related.26In the Singapore Cohort Study of Risk Factors for Myopia (SCORM), 25n the Beaver Dam Eye Study (BDES), reproducibility was evaluated in relation to 40 retinal images, with ICCs between 0.78 and 0.99 for inter and intragrader agreement,25presumably considering all three microvascular measurements. Te Cardiovascular Health Study (CHS) described intra and intergrader reproducibility in relation