Comparison of assessment methods in dental education

Comparação de métodos de avaliação no ensino odontológico

Abstract

Introduction  Assessment in dental education, especially in competency-based curricula, should reflect the different learning domains by combining methodologies for a more complete assessment of students.

Objective  To compare the outcomes of three assessment methods (written, clinical and Objective Structured Clinical Exam - OSCE) used in Dental Clinics disciplines.

Material and method  Two samples were used for this performance analysis: 179 students attending four different dental clinics disciplines in the same semester; and 33 students from the same class during their 5th, 6th, 7th and 8th periods, taking the four disciplines along four semesters sequentially. From the mean grades obtained by the students in each examination, the comparative analyses were carried out using repeated measures analysis of variance.

Result  The results showed similarity between the theoretical and OSCE evaluations, which differ significantly from the results obtained in the clinical evaluation. Comparing the performance of the same students during sequential periods, the results showed a statistically significant difference in all evaluations, with periods of positive progression and decrease in student performance along the semesters.

Conclusion  Written and OSCE assessments were similar while clinical evaluations presented significantly higher grades. Therefore, teachers should be aware of the particularities of the assessment methods and apply them, solely or combined, in order to obtain the best outcomes from the students. Evaluation methods should value teacher orientation and consider the validity of subjective evaluation, paying attention to the educational value of the assessments.

Descriptors:
Educational measurement; teaching; academic performance

Resumo

Introdução  A avaliação no ensino odontológico, especialmente em currículos baseados em competências, deve refletir os diferentes domínios de aprendizagem combinando metodologias para uma avaliação mais completa dos estudantes.

Objetivo  Comparar os resultados de três métodos de avaliação (escrito, clínico e Exame Clínico Objetivo Estruturado - OSCE) utilizados em disciplinas de Clínica Odontológica.

Material e método  Duas amostras foram utilizadas para esta análise de desempenho: 179 alunos que cursavam quatro disciplinas diferentes de clínica odontológica no mesmo semestre; e 33 alunos da mesma turma durante seus 5º, 6º, 7º e 8º períodos, cursando as quatro disciplinas ao longo de quatro semestres sequencialmente. A partir das notas médias obtidas pelos alunos em cada exame, as análises comparativas foram realizadas utilizando análise de variância de medidas repetidas.

Resultado  Os resultados mostraram similaridade entre as avaliações teórica e OSCE, que diferem significativamente dos resultados obtidos na avaliação clínica. Comparando o desempenho dos mesmos alunos durante períodos sequenciais, os resultados mostraram diferença estatisticamente significativa em todas as avaliações, com períodos de progressão positiva e queda no desempenho dos alunos ao longo dos semestres.

Conclusão  As avaliações escritas e do OSCE foram semelhantes, enquanto as avaliações clínicas apresentaram notas significativamente mais altas. Portanto, os professores devem estar cientes das particularidades dos métodos de avaliação e aplicá-los, isoladamente ou combinados, para obter os melhores resultados dos alunos. Os métodos de avaliação devem valorizar a orientação do professor e considerar a validade da avaliação subjetiva, atentando para o valor educacional das avaliações.

Descritores:
Avaliação educacional; ensino; desempenho acadêmico

INTRODUCTION

The objective of higher education should be to stimulate the development of learning features such as critical thinking, autonomy and problem solving ability¹. This is also valid for dental education². The learning process is recorded throughout the course by assessments that determine whether students can begin dental practice independently³. Evaluation is a programmatic task that needs to become a living part of any educational program. For competence-based educational programs, the design of the evaluation system must reflect the programmatic philosophy and its context⁴.

Competence is the ability to handle a complex professional task integrating cognitive, psychomotor and affective skills. In educational practice, curricular programs have been built around these competencies, idealized by modern educational theory, which postulates that learning is facilitated when tasks are integrated⁵,⁶.

It is recognized that hardly all dimensions and elements of clinical learning can be, in an adequate and holistic way, evaluated using traditional forms of oral and written assessments⁷, and cannot be determined by a single method³. Undoubtedly, these forms of assessments are valid for testing clinical knowledge and thinking, but they are insufficient when assessing clinical skills and abilities⁷. Thus, the evaluation practices in this educational model should be designed to reflect the specific characteristics of competence-based education⁴,⁸. Therefore, the selection of the evaluation method should be aligned with the specific learning outcome expected³.

To evaluate the different learning outcomes in a valid and rigorous way, different assessment methods should be used. For example, performance skills cannot be evaluated through reports or multiple-choice tests. For this purpose, the use of simulations with laboratory exercises and Objective Structured Clinical Exam (OSCEs) are more appropriate⁹. OSCE is an evaluation method that assesses higher levels of cognition compared to basic memorization. Regarding the Bloom taxonomy of educational goals in cognitive domain¹⁰, OSCE is designed to evaluate the levels of application, analysis, synthesis and evaluation¹¹.

Another recommended method is the portfolio, which enables multiple and ongoing evaluations with multiple evaluators providing the best strategy for overall assessment of the student's skills in a valid and reliable way¹².

An ideal evaluation would require the application of a multitude of evaluative methods and the combination of information obtained would ensure their validity and reliability. Therefore, each evaluation method must present specific learning outcomes³ and there is no method better than other. In this sense, the overall assessment of the required skills in the training of a dental professional occurs with the combination of the outcomes obtained by different methods³. Thus, the research objective was to evaluate the dental student performance in three assessment methods (theoretical, clinical practice and Structured Objective Clinical Exam – OSCE) in multidisciplinary dental clinic disciplines, as well as to analyze their progress over a two-year period.

MATERIAL AND METHOD

This cross-sectional observational study was approved by the Research Ethics Committee and an initial sample of grade reports of multidisciplinary dental clinic disciplines was selected. This comprised assessments of 183 students from the 5th to the 8th period of the School of Dentistry of a private University.

Grading reports of the students who attended Dental Clinic disciplines (Dental Clinic II, III, IV and V) regularly in the 2019.2 academic semester were included. Also, evaluation reports of students who attended Dental Clinic II in 2018.1, Dental Clinic III in 2018.2, Dental Clinic IV in 2019.1 and Dental Clinic V in 2019.2 were included to analyze the student performance progression.

Exclusion criteria were students who did not complete the disciplines in the second semester of 2019 and students who did not complete the sequence of Integrated Clinics II to V disciplines regularly in the period 2018.1 to 2019.2.

At the beginning of the academic semester, the semester plans were presented to the students to inform them about the evaluation process, which consisted of three examinations. Each examination included a written test (0 to 100 points) and a clinical exam (0 to 100 points). The third examination also included the OSCE (0 to 100 points).

Grades obtained in all examinations were collected, the mean of the assessment methods were calculated for each student and the values obtained for the different evaluations (written, clinical and OSCE) were compared using repeated measures analysis of variance (ANOVA) and Tukey-Kramer post-hoc test. Statistical analyses were performed using Jamovi software (The Jamovi Project, Version 1.2, 2020) with a significance level of α = 0.05 for all tests performed.

RESULT

The final sample of the study was composed by the grading reports of 179 students from the 5th to the 8th period in the subjects of Dental Clinics II (n=60), III (n=35), IV (n=45) and V (n=39) since four students were excluded from the study. The age ranged from 19 to 35 years, with an average age of 22 years. Sex ratio was predominantly female, with 133 women and 46 men. For the analysis of students’ performance progress, a group of 33 students (20 female and 13 male subjects were evaluated.

In the present study two analyses were performed in different samples. The first one compared grades in Dental Clinics subjects in three evaluation methods (n= 179); and the second analysis compared grades from the same group of students during four consecutive semesters of Dental Clinics disciplines (n= 33).

Regarding the first analysis, the results presented a mean value of 62.5 points in the written evaluation, 82.2 points in the clinical and 59.2 points in the OSCE evaluation (Table 1), with statistically significant difference between the clinical evaluation and the other two methods (p<0.05). For each assessment method, the results presented variations, and OSCE was the one that presented the greatest variation, followed by written and clinical evaluation (Figure 1).

Table 1
Comparison of means of student performance results in each evaluation method

Method of Assessment	Results (grades)
	Mean	Minimum	Maximum
Written (Multiple choices)	62.5 A	47.3	81.3
Clinical	82.2 B	74.2	91.4
OSCE	59.2 A	15.0	93.8

• Different letters in superscript following values indicate statistical significance (level of significance set at p < 0.05).

Figure 1
Box plot showing the distribution of written, clinical and OSCE grades.

In the second analysis, evaluating the progress of the students in different semesters (2018.1, 2018.2, 2019.1 and 2019.2), statistically significant differences were observed in the written evaluation (p ≤ 0.001), with grade improvement between 2019.1 and 2019.2 (Table 2). Between 2018.1 and 2019.1, a decrease in the grades was observed in the written methodology.

Table 2
Mean grades showing student´s performance evolution throughout clinical disciplines

	Written	Clinical	OSCE
2018.1	64.0 AC	83.7 A	70.8 A
2018.2	61.7 AB	81.1 B	51.3 B
2019.1	59.6 B	83.9 A	64.0 A
2019.2	65.9 C	83.8 A	70.8 A

• Different letters in superscript following values indicate statistical significance in the column (level of significance set at p < 0.05).

Students performance in clinical evaluation presented significant increase between 2018.2 and 2019.1/2019.2 semesters. Between 2018.1 and 2018.2, it could be observed a decrease in the grades in the practical assessment (Table 2). The same analysis in the OSCE evaluation was positive between 2018.2 and 2019.1/ 2019.2, showing a drop in performance between 2018.1 and 2018.2 (Table 2). The oscillations observed in the three evaluation methodologies are presented in Figure 2.

Figure 2
Distribution of the averages obtained in written, clinical and OSCE evaluations over four periods with clinical disciplines.

DISCUSSION

The results obtained showed that clinical evaluations presented better results when compared to the written and OSCE evaluations. Clinical evaluation was performed using a portfolio, where clinical procedures, reports and attitudinal factors were considered. According to Patel et al.³, there has been a search for reliability and objectivity in the evaluations, OSCE being an example of this. However, the evaluation of complex skills such as professionalism, management and leadership requires subjectivity. The assessments’ reliability and the objective assessment limitations should be considered, so that it does not interfere in the validity of practical assessments even considering the use of portfolios. Reliability and validity are inversely related and a balance between these elements can be achieved by combining different assessment methods³.

Despite all efforts, teachers in clinical disciplines often find it difficult to assess complex skills, and “competence-based education” has received criticism in the literature¹³-¹⁵. For example, the use of portfolios to assess attitudinal factors with pre-defined grades interferes the assessment validity, standardizing this assessment method. The evaluation of complex skills should be performed in a non-standardized way, being the teacher a vital component in this process³. However, subjective evaluations are considered as low reliability, since in many situations they are performed by a single evaluator. For this, it is suggested that subjective evaluations are carried out by multiple teachers, with different subjective judgments, which allows validating the evaluation of complex skills³,¹⁶.

Considering this study, although the clinical evaluation is performed by several teachers from different areas of dentistry, the questioning of the validity and reliability of the portfolio by the student intimidates the teacher in the subjective evaluation of complex skills. This consideration pointed out by the literature, supports the results presented in this study, where clinical methods present better results than written and OSCE's evaluations³.

Although practical evaluations presented positive results, showing satisfactory performance of the students (mean = 82.2 points), in the written evaluations this result was significantly lower (mean = 62.5 points; p≤0.05). Written assessment based on traditional multiple-choice exams are valuable in determining a student's ability to remember basic principles or to recognize and make fundamental associations but are not ideal for assessing levels of order of thought¹⁷. Traditional assessment methods in dental education often focus on the student's knowledge and memorization skills rather than the cognitive skills needed for clinical practice¹¹. The difference in performance results between clinical and written evaluations presented in this study justifies the use of written evaluations aiming the student’s effort to learn the basic concepts, such as precepts for clinical practice.

Considering this, the results showed a correlation between students’ performance in written and OSCE's evaluations, showing a statistically significant difference when compared to clinical evaluations. The study results, presenting the performance in clinical assessments with higher averages than written and OSCE assessments can be questioned, based on the objective way in which the complex competencies are being assessed. The literature points out that OSCE's and written evaluations are significantly related to the clinical performance of students. OSCE, as an evaluation method, is considered useful to identify students who may be underperforming in a clinical environment¹⁸, which may be being masked by the way the portfolio has been used in the context of this study.

The students’ performance throughout the course of Dentistry showed similar results to those pointed out in the comparison of the three evaluation methods. The average analysis of each student in the methods of the study did not present significant difference during their progress in clinical disciplines. According to Tonni et al.¹³, evaluations focusing on quantifiable evaluation data (e.g., grades and performance ratings) may have a detrimental effect on learning and decrease students' intrinsic motivation.

Therefore, graduate professionals committed to excellence in health care and continuing education may require a change in evaluation systems. Evaluation methods should value teacher orientation and consider the validity of subjective evaluation of the faculty, paying attention to the educational value of evaluations.

CONCLUSION

The performances results obtained by students in written and OSCE evaluation methodologies showed similarity, differing from the results observed in clinical evaluations, which showed higher grades when compared to the others. Therefore, teachers committed to excellence in health care and continuing education should be aware of the particularities of the assessment methods and apply them, solely or combined, in order to obtain the best outcomes from the students. Evaluation methods should value teacher orientation and consider the validity of subjective evaluation, paying attention to the educational value of the assessments.

ACKNOWLEDGEMENTS

The authors deny any conflicts of interest related to this study. All authors disclose any financial, economic or professional interests that may have influenced the design, execution or presentation of the scholarly work.

• How to cite:
  Amaral DC, Cruvinel DR, de Rezende Barbosa GL, Gomes CC. Comparison of assessment methods in dental education. Rev Odontol UNESP. 2025;54:e20250005. https://doi.org/10.1590/1807-2577.00525

REFERENCES

• ¹ Savin-Baden M. Problem-based learning in higher education: untold stories. Buckingham: The Society for Research into Higher Education & Open University Press; 2000.
• ² Plasschaert AJ, Holbrook WP, Delap E, Martinez C, Walmsley AD. Profile and competences for the european dentist. Eur J Dent Educ. 2005 Aug;9(3):98-107. http://doi.org/10.1111/j.1600-0579.2005.00369.x PMid:15982379.
  » http://doi.org/10.1111/j.1600-0579.2005.00369.x
• ³ Patel US, Tonni I, Gadbury-Amyot C, Van der Vleuten CPM, Escudier M. Assessment in a global context: An international perspective on dental education. Eur J Dent Educ. 2018 Mar;22(S1 Suppl 1):21-7. http://doi.org/10.1111/eje.12343 PMid:29601682.
  » http://doi.org/10.1111/eje.12343
• ⁴ van der Vleuten CP, Schuwirth LW. Assessing professional competence: from methods to programmes. Med Educ. 2005 Mar;39(3):309-17. http://doi.org/10.1111/j.1365-2929.2005.02094.x PMid:15733167.
  » http://doi.org/10.1111/j.1365-2929.2005.02094.x
• ⁵ van Merrienboer JJG. Training complex cognitive skills: a four-component instructional design model for technical training. Englewood Cliffs, NJ: Educational Technology Publications; 1997.
• ⁶ Cowpe J, Plasschaert A, Harzer W, Vinkka-Puhakka H, Walmsley AD. Profile and competences for the graduating European dentist - update 2009. Eur J Dent Educ. 2010 Nov;14(4):193-202. http://doi.org/10.1111/j.1600-0579.2009.00609.x PMid:20946246.
  » http://doi.org/10.1111/j.1600-0579.2009.00609.x
• ⁷ Conceição HCF. Avaliação de competências através de OSCE. [Internet]. 2009 [cited 2024 Dec 1]. Available from: https://www.uc.pt/site/assets/files/1469069/essencias13.pdf
  » https://www.uc.pt/site/assets/files/1469069/essencias13.pdf
• ⁸ Bok HG, Teunissen PW, Favier RP, Rietbroek NJ, Theyse LF, Brommer H, et al. Programmatic assessment of competency-based workplace learning: when theory meets practice. BMC Med Educ. 2013 Sep;13(1):123. http://doi.org/10.1186/1472-6920-13-123 PMid:24020944.
  » http://doi.org/10.1186/1472-6920-13-123
• ⁹ Nulty DD, Short LM, Johnson NW. Improving assessment in dental education through a paradigm of comprehensive care: a case report. J Dent Educ. 2010 Dec;74(12):1367-79. http://doi.org/10.1002/j.0022-0337.2010.74.12.tb05012.x PMid:21123504.
  » http://doi.org/10.1002/j.0022-0337.2010.74.12.tb05012.x
• ¹⁰ Bloom BS, Englehart MD, Furst EJ, Hill WH, Krathwohl DR. Taxonomy of educational objectives: the classiication of educational goals. Handbook 1: cognitive domain. New York: Longman; 1956.
• ¹¹ Graham R, Zubiaurre Bitzer LA, Mensah FM, Anderson OR. Dental student perceptions of the educational value of a comprehensive, multidisciplinary OSCE. J Dent Educ. 2014 May;78(5):694-702. http://doi.org/10.1002/j.0022-0337.2014.78.5.tb05721.x PMid:24789829.
  » http://doi.org/10.1002/j.0022-0337.2014.78.5.tb05721.x
• ¹² Gadbury-Amyot CC, Overman PR. Implementation of portfolios as a programmatic global assessment measure in dental education. J Dent Educ. 2018 Jun;82(6):557-64. http://doi.org/10.21815/JDE.018.062 PMid:29858251.
  » http://doi.org/10.21815/JDE.018.062
• ¹³ Tonni I, Gadbury-Amyot CC, Govaerts M, Ten Cate O, Davis J, Garcia LT, et al. ADEA-ADEE shaping the future of dental education iii: assessment in competency-based dental education: Ways forward. J Dent Educ. 2020 Jan;84(1):97-104. http://doi.org/10.1002/jdd.12024 PMid:31977092.
  » http://doi.org/10.1002/jdd.12024
• ¹⁴ Lurie SJ. History and practice of competency-based assessment. Med Educ. 2012 Jan;46(1):49-57. http://doi.org/10.1111/j.1365-2923.2011.04142.x PMid:22150196.
  » http://doi.org/10.1111/j.1365-2923.2011.04142.x
• ¹⁵ Touchie C, Ten Cate O. The promise, perils, problems and progress of competency-based medical education. Med Educ. 2016 Jan;50(1):93-100. http://doi.org/10.1111/medu.12839 PMid:26695469.
  » http://doi.org/10.1111/medu.12839
• ¹⁶ Burrows RS. Understanding self-assessment in undergraduate dental education. Br Dent J. 2018 Jun;224(11):897-900. http://doi.org/10.1038/sj.bdj.2018.437 PMid:29855594.
  » http://doi.org/10.1038/sj.bdj.2018.437
• ¹⁷ Albino JE, Young SK, Neumann LM, Kramer GA, Andrieu SC, Henson L, et al. Assessing dental students’ competence: best practice recommendations in the performance assessment literature and investigation of current practices in predoctoral dental education. J Dent Educ. 2008 Dec;72(12):1405-35. http://doi.org/10.1002/j.0022-0337.2008.72.12.tb04620.x PMid:19056620.
  » http://doi.org/10.1002/j.0022-0337.2008.72.12.tb04620.x
• ¹⁸ Terry R, Hing W, Orr R, Milne N. Do coursework summative assessments predict clinical performance? A systematic review. BMC Med Educ. 2017 Feb;17(1):40. http://doi.org/10.1186/s12909-017-0878-3 PMid:28209159.
  » http://doi.org/10.1186/s12909-017-0878-3

Publication Dates

• Publication in this collection
  19 May 2025
• Date of issue
  2025

History

• Received
  20 Feb 2025
• Accepted
  07 Apr 2025