Reliability of a digital system for models measurements in BBO grading: A pilot study

ABSTRACT Introduction: Currently, no method is considered effective for the evaluation of digital models in the Certification Examination of the Brazilian Board of Orthodontics (BBO), considering the parameters of the currently used manual method. Objective: Thus, the aim of this study is to verify the reliability of an evaluation method for digital models that could be used in the BBO exam, compared to the gold standard. Methods: Measurements were performed by five previously calibrated examiners. Samples of ten sets of plaster models of the final phase of orthodontic treatment were measured using a manual method (Objective Grading System, OGS). These models were digitized using a 3D scanner and exported to Geomagic Qualify software, in which the measurements were made with the proposed digital method. These measurements were repeated using five models, after fifteen days. The intra-examiner performance with this method was analyzed with a paired t-test, whereas the inter-examiner analysis was carried out with analysis of variance and Tukey’s test. To compare the manual and digital methods, a paired t-test and Pearson’s correlation analysis were performed. Results: A statistically significant difference was found. The results showed that, when compared to the manual method, the digital method was effective in measuring the OGS in four of the seven variables studied: Marginal Ridge, Overjet, Occlusal Contact, and Interproximal Contact. The variables Alignment, BL inclination, and Occlusal Relationship showed a great amount of dispersion in the findings. Conclusion: Further studies are needed to develop an adequate digital methodology that can be used for all OGS variables.


INTRODUCTION
The Brazilian Board of Orthodontics and Facial Orthopedics (BBO) is an entity that promotes clinical excellence within the specialty of orthodontics. During the certification process, a candidate presents the results from six clinical cases after orthodontic treatment is completed. The plaster models are evaluated under the Objective Grading System (OGS), which is used to judge the orthodontic case using a metric system developed by the American Board of Orthodontics (ABO). 1,2,3 Also, the ABO system is considered an excellent way to self-assess cases treated in private orthodontists' offices. 4 Due to the increasing demand for digital models, OrthoCAD has developed a tool 7 to enable the inclusion of digital models in the final clinical case assessment performed in the ABO candidate examination. 8 However, some studies evaluating this feature have determined that the tool cannot replace manual measurements in plaster models. 5,6,9 With the emergence of scanning and digital modeling, there is a growing interest regarding the accuracy of the measures obtained from digital models, as compared to the conventionally used plaster models. 5 No digital method was considered efficient after evaluating every item from the OGS. 6 Miranda PMB, Fernandes LQP, Sevillano MGC, Carvalho FAR, Capelli Junior J -Reliability of a digital system for models measurements in BBO grading: A pilot study 5 Considering the above-mentioned concerns, this study aimed to determine the accuracy of a digital measurement system under the OGS, compared to the conventional manual approach.

MATERIAL AND METHODS
Five examiners were designated, who were all specialists in orthodontics. Before starting this study, the examiners were calibrated by a BBO board member, and five different sets of study models were evaluated by them. Subsequently, ten sets of plaster models from patients who attended the Orthodontics Specialization Clinic of the State University of Rio de Janeiro (Brazil) for orthodontic treatment were selected to this study sample. Inclusion criteria were that the plaster models needed to be in good condition and the models needed to be from patients with finished orthodontic treatment that presented complete permanent dentition, except for the third molars.
At first, the sample was measured with the manual method using the ABO measuring gauge (OGS). Afterwards, for the digital evaluation method, the models were digitized using a 3D scanner (Maestro 3D Dental Scanner -AGE Solutions, Potedera, Italy). After the images were captured, they were stored in STL open formats and exported to Geomagic Qualify 2013 software (Raindrop Geomagic, Inc., Cary, NC, USA), where the proposed digital measurement methodology was employed. An interval of 15 days was considered suitable to repeat the method in 5 Miranda PMB, Fernandes LQP, Sevillano MGC, Carvalho FAR, Capelli Junior J -Reliability of a digital system for models measurements in BBO grading: A pilot study 6 sets of models of the sample, to obtain inter-examiners and intra-examiner comparisons. Lastly, the measurements of the manual and digital methods were compared to evaluate the reliability of the proposed digital method.

PROPOSED DIGITAL METHOD
The proposed digital methodology used the same reference points that are applied to the manual evaluation system. Instead of using the ABO measuring gauge that is applied on plaster models, the Geomagic Qualify software was employed to take the measurements on the digital models. Therefore, when an alteration was detected on the ideal finalization pattern, points were placed on the sites where the ABO measuring gauge would be, and the software showed the distance between those points in the three axes (vertical, transversal, and anterior-posterior), as well as it showed the total distance. The axis of interest depended on the OGS evaluated item, which was determined by this method.
Initially, for each model, three axes were created: the Y (anterior-posterior) axis, X (transverse) axis, and Z (vertical) axis. Two reference planes were created in the lower part of the model,  For the Alignment variable, points were placed on the teeth that were considered to be misaligned. For the posterior teeth, the transverse X-axis was observed, and for the anterior teeth, the total distance was generated (Fig 2A). For the Marginal Ridge variable, points were placed on the ridges with a difference in height, and the distance generated on the vertical Z-axis was observed ( Fig 2B). In the Buccolingual Inclination (BL inclination), as seen through the height difference of the buccal and palatal cusps, the vertical Z-axis was observed ( Fig   3). For the Overjet, when there was a lack of contact in the anterior teeth, points were placed in the incisal edge of the upper teeth and in the buccal face of the lower teeth, and the total distance generated was considered ( Fig 4A). For the posterior teeth, the distance where the teeth should be occluding, if not in the ideal position, was observed, and the transverse X-axis was the focus of interest ( Fig 4B). The Occlusal Contact  Finally, in the Interproximal Contact, if there was no contact between any teeth, the points were placed on the distal and mesial area of the teeth in question, and the distance between them was measured. For the anterior teeth, the total distance was considered, and for the posterior teeth, the anteroposterior Y-axis was considered ( Fig 6B). In the digital method, a paired t-test was used to check the reproducibility of the scores, and the null hypothesis was that there was no correlation between the measurements made the first time (T 1 ) and the second time (T 2 ). A strong correlation was r > 0.7, a moderate correlation was r between 0.7 and 0.3, and a weak correlation was r < 0.3. All three conditions were considered.
To analyze inter-examiners reproducibility, an analysis of variance (ANOVA) was used, and the null hypothesis was that there was no difference between the groups. In case of a significant difference, Tukey's test was applied to the variable in question. In the analysis that compared the manual and digital methods, two tests were applied for each evaluated variable: a paired t-test to evaluate the scores between the methods and Pearson's correlation test to verify if there was a correlation between the measurements.
Correlations were classified according to the value of r, with r > 0.7 indicating a strong correlation, r between 0.7 and 0.3indicating a moderate correlation, and r < 0.3 indicating a weak correlation.

RESULTS
The Shapiro-Wilk test showed that the data formed a normal distribution.
In the intra-examiner analysis (

DISCUSSION
A study was conducted with the Geomagic Qualify software, 10 which allows the creation of a coordinate system through three axes in space and the projection of the distance between two points, which is a feature that is not available in OrthoCAD.
However, some disadvantages with the methodology were described by the authors, such as the time-consuming execution, since points were recorded for every tooth, regardless of its position. In the present study, although digital method took longer, it was not that expressive. We observed approximately 17 minutes for executing the manual method and 21 minutes for the digital (2 minutes for the creation of the coordinate system and 19 minutes for the measurements of the variables).
However, the addition of five minutes to each model can make a difference to someone who is evaluating a lot of models in the certification process of the BBO.
In the present study, in intra-examiner analysis, despite the low correlations found (Table 1), the measures of the variables were similar, indicating that a larger sample could be more efficient in identifying significant differences. In addition, the digital method may need to be further calibrated, as orthodontists have less experience with this technology than with the manual approach. In the inter-examiner agreement, there was no reproducibility of the method for the variables Alignment, Occlusal Relationship, and Interproximal Contact ( Table 2).
For Alignment, the correlation in Pearson's correlation analysis could not be confirmed based on the p-value (Table 4), and a statistically significant difference was found only with the paired t-test (Table 3). Thus, the digital method was not compatible with the manual method in this domain. This difference may have occurred due to the angle of this variable in relation to the coordinate axis, since it does not follow the shape of the dental arches. Since the front teeth are in front of that shape, it became a challenge to measure their misalignment on one of the horizontal axes (X or Y), so the total distance was considered. However, this did not eliminate the vertical variation of the points, and if there is an angle between them, it can generate an increase in the true distance, thus causing a possible higher score. This also occurred in the posterior teeth, which continue shaping the sides of the arch, and although less curved, they were not parallel to the anteroposterior X-axis, which was used to measure the misalignments. This may have been one of the reasons for the large difference in this variable. Of three studies evaluating the OrthoCAD program, two found statistically significant differences for this variable, 5,9 and one study proposed a method using Geomagic Qualify software. 10 Another study, however, found consistency between the manual and non-manual digital measurements. 6 For the Marginal Ridge, no statistically significant difference was found with the paired t-test (Table 3), and there was a moderate correlation in Pearson's correlation analysis (Table 4).
Therefore, it can be considered that there was a consistency in its measurement, which is a result compatible with studies evaluating OrthoCAD 5,6,9 and a previous study that employed Geomagic Qualify. 10 For the BL Inclination, the paired t-test showed a statistically significant difference (Table 3), although Pearson's correlation analysis showed a moderate correlation (Table 4). This indicated that although the measurements differed regarding the methods, as one increased, the other also increased. Thus, for the application of this variable, a new scoring table is suggested, since the digital method scored higher than the manual method, but with a similar proportion. This variable showed a statistically significant difference in a study evaluating OGS in digital models by OrthoCAD. 9 In the study with Geomagic Qualify, there was consistency between the manual and digital methods, 10 as demonstrated in another study. 5 For the Overjet, no statistically significant difference was found with the t-test (  5 while the other studies found consistency between the manual and digital measurements. 6,9 For Occlusal Contact, the methodologies were considered to be comparable, as no statistically significant difference was found with the paired t-test (Table 3), in addition to obtaining a strong correlation with Pearson's correlation analysis (Table 4).
Of the other OrthoCAD studies, only one did not find a statistically significant difference, 9 and the study that used Geomagic Qualify did not find a difference as well. 10 The others found a statistically significant difference. 5,6 For the Occlusal Relationship variable, a statistically significant difference was found with the paired t-test (Table 3) and a moderate correlation was found with Pearson's correlation analysis (Table 4). Thus, as in the BL Inclination, for its application, a new scoring table is suggested, since the digital method scored higher than the manual method, but with a similar proportion. This result was similar to that found in the study using Geomagic Qualify 10 and in a study evaluating OrthoCAD, 6 whereas in the other two studies, no statistically significant differences were found. 5,9 For Interproximal Contact, no statistically significant difference was found with the paired t-test (Table 3), and the correlation was moderate by Pearson's correlation analysis (Table 4), indicating that it is an alternative to the digital methodology. These results concurred with the studies already cited. 5,6,9,10 The differences found between the manual and digital methodologies should be analyzed with some considerations. Although evaluation using plaster models is considered the gold standard, this method was created for clinical purposes and has some methodological limitations. Among them is the parallax effect, which is a different assessment depending on the angle at which the observer looks at the model. This effect would be negated with digital models using the method suggested in a study where points were placed on all teeth. 10