Objective structured clinical evaluation as an assessment method for undergraduate chest physical therapy students: a cross-sectional study

Avaliação clínica objetiva e estruturada como um método para avaliar estudantes de graduação em fisioterapia respiratória: um estudo transversal

Objective structured clinical evaluation as an assessment method for undergraduate chest physical therapy students: a cross-sectional study

Avaliação clínica objetiva e estruturada como um método para avaliar estudantes de graduação em fisioterapia respiratória: um estudo transversal

Cibele C. B. M. Silva; Adriana C. Lunardi; Felipe A. R. Mendes; Flavia F. P. Souza; Celso R. F. Carvalho

Physical Therapy Department, School of Medicine, Universidade de São Paulo (USP), São Paulo, SP, Brazil

^{Correspondence}

ABSTRACT

BACKGROUND: The Objective Structured Clinical Evaluation (OSCE) has been considered a reliable method for the evaluation of students' clinical skills in health sciences, but it has been rarely applied in the teaching of physical therapy.

OBJECTIVE: To assess the use of the OSCE as a tool to evaluate the abilities of undergraduate chest physical therapy students and to verify the internal consistency of the OSCE exam.

METHODS: Forty-seven students were evaluated using two types of exams: the traditional exam and the OSCE. Independent educators elaborated the exams. Each question (traditional) or station (OSCE) was given a score ranging from 0.0 to 2.0; being 10.0 the highest possible score of both exams. The relationship between the total score that were obtained from both exams was analyzed using Bland-Altman analysis and Pearson's correlation coefficient. The internal consistency of the OSCE stations was evaluated by four experienced chest physical therapists and it was tested using Cronbach's alpha.

RESULTS: The students' average score on the OSCE ranged from 4.4 to 9.6. The internal consistency of the OSCE stations was considered good (0.7). The agreement between exams was analyzed, and it was determined that the exams are not comparable. Examiners also observed a low agreement between the two exams (r=-0.1; p=0.9).

CONCLUSION: Our results showed that OSCE and traditional exams are not interchangeable. The OSCE exam had good internal consistency and is able to evaluate aspects that the traditional exam fails to evaluate.

Keywords: physical therapy; clinical competence; educational measurement; professional competence; program evaluation.

RESUMO

CONTEXTUALIZAÇÃO: A avaliação clínica objetiva e estruturada (OSCE) é considerada um método confiável para avaliar as competências clínicas de estudantes na área de saúde, entretanto tem sido pouco aplicada no ensino de fisioterapia.

OBJETIVO: Analisar o uso da OSCE como uma ferramenta para avaliar as habilidades dos alunos de graduação em fisioterapia respiratória e verificar sua consistência interna.

MÉTODOS: Quarenta e sete alunos foram avaliados por meio de dois exames: tradicional e OSCE. As provas foram elaboradas por educadores independentes. Para cada questão (tradicional) ou estação (OSCE), atribuiu-se uma pontuação de 0 a 2. A pontuação máxima possível em ambos os exames foi de 10. As relações entre as pontuações obtidas nos dois exames foram analisadas por meio de Bland-Altman e Correlação de Pearson. A consistência interna das estações da OSCE foi avaliada por quatro fisioterapeutas experientes e foi testada pelo Alfa de Cronbach.

RESULTADOS: A pontuação média dos estudantes na OSCE oscilou entre 4,4 e 9,6. A consistência interna das estações da OSCE foi considerada boa (0,7). A concordância entre os exames foi analisada, e determinou-se que os exames não são comparáveis. Os examinadores também observaram uma baixa concordância entre os dois exames (r=-0,1, p=0,9).

CONCLUSÃO: Nossos resultados mostraram que a OSCE e o exame tradicional não são intercambiáveis. O exame da OSCE teve uma boa consistência interna, sendo capaz de avaliar aspectos que o exame tradicional não avalia.

Palavras-chave: fisioterapia; competência clínica; avaliação educacional; competência profissional; avaliação de programas; projetos de saúde.

Introduction

Traditional methods that are used to assess clinical skills in health science fields have been deemed insufficient¹, and the paucities noted in this type of evaluation include the lack of objectivity and standardization by the examiners; the inability to test communication skills; the possible practice of favoritism and the failure to predict the future performance of the students². Therefore, the use of more sophisticated examination methods to assess competencies and clinical skills has been suggested¹.

The Objective Structured Clinical Evaluation (OSCE) has been used to evaluate clinical skills³ and has been considered a reliable method to assess the clinical competence of health science students⁴ because it objectively measures technical skills, attitudes and decision-making strategies⁵. In this type of exam, students are assessed in diverse clinical situations. This seems to address the challenge of assessing clinical performance in real-life situations², although these situations are bounded, controlled and pre-defined^5-8. The OSCE is considered to be a more effective way to assess problem-solving abilities, critical thinking and communication skills³. It is also considered to be more objective⁷ due to a reduced risk of examiner bias⁹ and the provision of parity among all students¹⁰.

Although the OSCE has been used in Canada to evaluate the clinical knowledge of physical therapists seeking a professional license⁵, its use to evaluate undergraduate chest physical therapy students is still limited, and studies evaluating its relevance are scarce. Presently, we are aware of only a single study that explores the use of the OSCE with physical therapy students, and it shows a poor internal consistency of the stations and an inability to predict a student's clinical performance¹¹. Therefore, the aims of this study were to assess the use of the OSCE as a tool to evaluate the abilities of undergraduate chest physical therapy students and to verify its internal consistency.

Methods

Participants

Forty-seven undergraduate physical therapy students in the 3^rd year (6^th semester) from two consecutive classes from a public Universidade de São Paulo (USP), São Paulo, SP, Brazil were assessed using traditional and OSCE exams. All students consented to participate in this study.

Study design

This cross-sectional study evaluated students in 2 different days at the end of the chest physical therapy course. In the first day, they performed theoretical and practical exams (traditional exam) prepared by an educator. In the second day, they performed OSCE exam prepared by other educators from the USP. The students and the educators who prepared the OSCE exams were blinded to the grades obtained on the traditional exam to avoid interference in the exam performance. The contents of the exams included the following: clinical evaluation of the respiratory system, bronchial hygiene and lung expansion techniques, pulmonary rehabilitation, oxygen therapy and discussion of clinical cases based on hospital visits. The study design was approved by the ethics committee of the School of Medicine of the USP (protocol number 7362).

Traditional exam

The traditional exam included four theoretical questions and one practical question. Two points were given for each correct answer, with a maximum score of 10.0 points. The practical question was randomly chosen by the student (by drawing), and the student was required to perform, for example, a technique that was specific to bronchial hygiene or lung expansion on another student while the examiner observed. The questions consisted of clinical assessment of the respiratory system, techniques of bronchial hygiene and lung expansion, pulmonary rehabilitation and a case report, which assessed cognitive and psychomotor skills. By definition, in medicine, a case report is a detailed report of the symptoms, signs, diagnosis, treatment, and follow-up of an individual patient.

Objective structured clinical evaluation

The OSCE was composed of five stations. Each station was arranged in one of five classrooms that were all located in the same corridor, in a circuit arrangement, and were identified by a letter placed on the door (from A to E). Before entering the station, students had one minute to read a description of the task to be performed. After entering the station, they had five minutes to complete the requested task. Four minutes after they had entered the station, the coordinator gently knocked on the door to inform the students that they had one minute left to complete the task. After five minutes, the coordinator opened the door, and students moved on to the next station. Inside each station there was an examiner who was responsible for completing the checklist of activities that the student should have performed; however, this was done only after the students left the station. Throughout the circuit, no comments were allowed between the students, the coordinator and the examiners.

At each station, there was an actor who simulated a real clinical practice situation. The simulated activity included a situation involving a multi-professional interaction (which is needed for decision-making), an intervention that is performed on a patient or instructions to the caregiver. All examiners and actors were professionals who had at least five years of experience. The distribution of skills that were assessed in each station is described in Table 1. The skill classification that was used in the present study (cognitive, psychomotor and attitudinal) was adapted from the classification that is used for medical students¹¹. Briefly, attitudinal skill was evaluated by the student´s reaction and speech as well as verbal command and intonation with the volunteer-actor and their initiative to solve the task in each station. Psychomotor skill was evaluated as the student´s ability to perform a respiratory maneuver and/or a clinical examination. Cognitive skill was evaluated as the student´s capacity to integrate intellectual knowledge to perform a task or to elaborate an adequate clinical conduct to solve a clinical case. To ensure the validity of this classification for the OSCE that is used in chest physical therapy, four experienced chest physiotherapists reviewed each item on the checklist independently and rated the skills that were evaluated in each station. There was a meeting to evaluate and discuss specific questions if there was no consensus about the validity.

Statistical analysis

The data were tested for normality using the Kolmogorov-Smirnov test and were presented as means and standard deviations. The internal consistency of the OSCE stations was tested using Cronbach's alpha. The agreement between the total score obtained in both tests was analyzed using

Bland-Altman analysis, and the correlation was assessed using Pearson's correlation coefficient. The level of significance for all analyses was set at 5% (p<0.05).

Results

Our results show that the OSCE stations evaluated psychomotor, cognitive and attitudinal skills (53.6, 14 and 32.4%, respectively; Figure 1), and the internal consistency of the OSCE stations was considered good (Cronbach's alpha=0.7). Students' performance was considered similar for attitudinal and cognitive skills at all stations (on average 76 and 84%, respectively). Students' performance of psychomotor skills showed greater variation between stations (between 36 and 66% on average) (Figure 2).

The average score on the stations (OSCE) was 66.6% ranging from 50 to 73%. The students had poorer performances on station C (Figure 3A).

The traditional exam evaluated only cognitive and psychomotor skills, and the students' grades ranged from 2.0 to 9.5 points (mean score=7.3) (Figure 3B). Only 8.6% of the students scored below 5.0 points. For the OSCE exam, students' grades ranged from 4.4 to 9.6 points (mean score=8.0), and only one student's scored below 5 points. It was observed that there was an agreement between the two exams grades for 93% of the students (Figure 4), and the variation between the grades that were obtained for the traditional and the OSCE exams occurred independently of the student's mean final score. The agreement between both exams was estimated using Bland-Altman analysis; the grades were not comparable in 5 out of 47 students (10.6%) and most students (93.7%) were within 2 standard deviations of the mean (Figure 4). There was no correlation between the two exams (r=-0.1; p=0.9).

Discussion

Our results show that students presented distinctly different performances on traditional and OSCE exams. In addition, the OSCE exam showed good internal consistency and seemed to assess all of the skills and the competencies that are expected for a chest physical therapy student. These results suggest that the OSCE could be a good complement to the traditional exam.

The traditional exam focuses on cognitive and psychomotor skills, whereas the OSCE exam was developed to address the attitudinal and psychomotor skills that are required to perform basic procedures that are used in clinical practice. The evaluation of students based on cognitive skill tends to promote a learning strategy of memorize information, whereas evaluations based on the ability to integrate their knowledge tend to develop a more efficient learning strategy¹². In the present study, the lack of association between the students' performances on a traditional exam and on an OSCE can be explained by the specificity of each exam, and this may suggest that both exams are complementary and not mutually exclusive, in agreement with other studies^13,14.

The OSCE exam was initially proposed by Harden and Gleeson⁴ in 1975, and since then it has been considered to be an efficient way to evaluate the clinical competence of various healthcare professionals, including dentists¹⁵, physicians¹⁶ and nurses¹⁷. Clinical competence includes the ability to solve problems, to think critically in order to apply clinical reasoning, to work as a team and to communicate effectively in both verbal and written forms¹⁸. All of these skills are also necessary for a physical therapist; however, to our knowledge, the OSCE exam was described in research only once as an evaluation of undergraduate learning for these students¹¹. In contrast, the OSCE has been used as a proficiency exam for physical therapists in Canada¹⁹. The OSCE has also been used to evaluate the skills of other healthcare professionals, such as dietitians²⁰ and dentists²¹, because it reproduces the actual tasks that a professional must perform in a clinical situation²².

Despite the increasing use of the OSCE, it is important that educators pay close attention to the stations and the checklist preparation to ensure the reliability and validity of this exam^16,23. The internal consistency of the OSCE exam in our study was considered good, as it was higher than the cut off value (0.4) that was suggested by Barman¹⁶ and the minimal limit (0.6) that was proposed by Linn and Gronlund²⁴. The consistency of the OSCE exam in our study was even higher than the consistency that was described in other studies^10,11,23. In our opinion, this is due to the wide range of skills that were assessed in the OSCE stations in the other studies. Another possible explanation for the higher internal consistency in our OSCE exam is the agreement in the skill for each station (cognitive, psychomotor and attitudinal). Skill analysis was performed prior to the study in an independent manner by four experienced chest physical therapists; after independent analysis, the therapists discussed possible differences in their opinions and reached a consensus for all stations²⁴.

The main benefit of using the OSCE is the possibility of linking a set of core competencies with the setup of simulated clinical situations that allows the measurement of determined skills. This implies that the OSCE can be considered as a "gold standard" for evaluating healthcare professionals²⁵; however, some disadvantages of the OSCE have been indentified. This is the most stressful type of exam for the students, and stress could adversely affect their performance⁹. This stress did not seem to occur in our study; students obtained higher scores and demonstrated less variation when evaluated with the OSCE than when evaluated with the traditional exam. Another drawback of the OSCE is the need for a complex organization process including time, cost and assurance of confidentiality. In this study, the OSCE exam required at least 7 chest physical therapists, 5 classrooms and 4 hours to evaluate 12 students.

One limitation of this study is the relatively small sample size; however, the number of students evaluated is similar to previously described¹¹. In addition, we did not assess whether the OSCE exam was able to predict the students' performance in practical training or their successful entry into the labor market. Lastly, whether administering the traditional exam along with the OSCE can improve the evaluation of chest physical therapy students was not directly assessed.

Our results show that the OSCE exam evaluated distinctly different skills than the traditional exam for undergraduate chest physical therapy students. The study also showed that our OSCE exam had good internal consistency. These results suggest that the use of the OSCE to evaluate skills that traditional exam fails and that the OSCE could be used as a complement to a traditional exam.

Received: 06/09/2011

Revised: 08/22/2011

Accepted: 08/26/2011

• 1. Wass V, Van der Vleuten C, Shatzer J, Jones R. Assessment of clinical competence. Lancet. 2001;357(9260):945-9.
• 2. Agarwal A, Batra B, Sood AK, Ramakantan R, Bhargava SK, Chidambaranathan N, et al. Objective structured clinical examination in radiology. Indian J Radiol Imaging. 2010;20(2):83-8.
• 3. Newble D. Techniques for measuring clinical competence: objective structured clinical examinations. Med Educ. 2004;38(2):199-203.
• 4. Harden RM, Gleeson FA. Assessment of clinical competence using an objective structured clinical examination (OSCE). Med Educ. 1979;13(1):41-54.
• 5. Nayer M. An overview of the objective structured clinical examination. Physiother Can. 1993;45(3):171-8.
• 6. Mavis BE, Henry RC. Between a rock and a hard place: finding a place for the OSCE in medical education. Med Educ. 2002;36(5):408-9.
• 7. Schuwirth LW, van der Vleuten CP. ABC of learning and teaching in medicine: Written assessment. BMJ. 2003;326(7390):643-5.
• 8. Troncon LEA. Clinical skills assessment: limitations to the introduction of an "OSCE" (Objective Structured Clinical Examination) in a traditional Brazilian medical school. Sao Paulo Med J. 2004;122(1):12-7.
• 9. Bartfay WJ, Rombough R, Howse E, Leblanc R. Evaluation. The OSCE approach in nursing education. Can Nurse 2004;100(3):18-23.
• 10. Roberts J, Brown B. Testing the OSCE: a reliable measurement of clinical nursing skills. Can J Nurs Res. 1990;22(1):51-9.
• 11. Wessel J, Williams R, Finch E, Gémus M. Reliability and validity of an objective structured clinical examination for physical therapy students. J Allied Health. 2003;32(4):266-9.
• 12. Martenson D, Hakan A, Kerstin G. An integrated final examination in preclinical subjects for medical students: 10 years of experience. Teaching and Learning in Medicine. 1999;11(1):26-33.
• 13. Ilic D. Assessing competency in Evidence Based Practice: strengths and limitations of current tools in practice. BMC Med Educ. 2009;9:53-8.
• 14. Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9 Suppl):S63-7.
• 15. Larsen T, Jeppe-Jensen D. The introduction and perception of an OSCE with an element of self- and peer-assessment. Eur J Dent Educ. 2008;12(1):2-7.
• 16. Barman A. Critiques on the Objective Structured Clinical Examination. Ann Acad Med Singapore. 2005;34(8):478-82.
• 17. Aggarwal R, Mytton OT, Derbrew M, Hananel D, Heydenburg M, Issenberg B, et al. Training and simulation for patient safety. Qual Saf Health Care. 2010;19 (Suppl) 2:i34-43.
• 18. Simpson EJ. The classification of educational objectives in the psychomotor domain. Washington, DC: Gryphon House; 1972.
• 19. Miller PA, Cooper MA, Eva KW. Factors predicting competence as assessed with the written component of the Canadian Physiotherapy Competency Examination. Physiother Theory Pract. 2010;26(1):12-21.
• 20. Pender FT, de Looy AE. The testing of clinical skills in dietetic students prior to entering clinical placement. J Hum Nutr Diet. 2004;17(1):17-24.
• 21. Gerrow JD, Murphy HJ, Boyd MA, Scott DA. Concurrent validity of written and OSCE components of the Canadian dental certification examinations. J Dent Educ. 2003;67(8):896-901.
• 22. Reteguiz J, Cornel-Avendaño B. Mastering the OSCE/CSA. New York: McGraw-Hill; 1999.
• 23. Turner JL, Dankoski ME. Objective structured clinical exams: a critical review. Fam Med. 2008;40(8):574-8.
• 24. Linn RL, Gronlund NE. Measurement an Assessment in Teaching. 8^th ed. New York: Prentice-Hall; 2000.
• 25. Miller JK. Competency-based training: objective structured clinical exercises (OSCE) in marriage and family therapy. J Marital Fam Ther. 2010;36(3):320-32.

Publication Dates

• Publication in this collection
  21 Nov 2011
• Date of issue
  Dec 2011

History

• Accepted
  2011
• Reviewed
  2011
• Received
  06 Sept 2011