Length of internship influences performance on medical residency exam

Duração do internato influencia o desempenho no exame de residência médica

ARTIGO ORIGINAL

Length of internship influences performance on medical residency exam

Duração do internato influencia o desempenho no exame de residência médica

Itamar de Souza Santos^I; Joaquim Edson VieiraII, ^*; Maria do Patrocínio Tenório Nunes^III

^IMédico pela Faculdade de Medicina da Universidade de São Paulo, São Paulo,SP

^IIMédico pela Faculdade de Medicina da Universidade de São Paulo, São Paulo,SP

^IIILivre-docente - Professora universitária pela Faculdade de Medicina da Universidade de São Paulo, SP

SUMMARY

OBJECTIVES: Medical education encompasses globally diverse context and conditions. The Brazilian scenario seemed a natural environment to study the influence of medical education programs and internship duration on the entrance exam for medical residency. This investigation evaluates some methods used during the entrance exam for medical residency as a means to make a distinction between candidates with longer clerkships.

METHODS: Candidates selected for a residency program performed a multiple-choice (MC), an open question (OQ) and OSCE-like tests, an interview and a curriculum analysis for participation in scientific meetings, papers published and voluntary activities. Groups were compared for gender, year of graduation, tests and OSCE scores.

RESULTS: Participants were distributed into two groups based on clerkship duration: 2 years or less than 2 years. There was no difference for the MCT score among groups or any of the activities from interview and curriculum analysis. The 2 years clerkship group showed significantly higher OQ (p=0.009) and OSCE-like affective (p=0.025) and knowledge (p=0.002) scores.

CONCLUSION: The OSCE test identified some aspects related to competence acquisition and assessed basic skills and attitudes essential to the supervised practice of medicine during residency. OSCE discriminated aspects not perceived by the sole use of knowledge tests

Key words: Internship and residency. Education medical. Curriculum. Questionnaires. Education measurement.

RESUMO

OBJETIVOS: A educação médica mostra contextos e condições globalmente diversas. O cenário no Brasil pode ser considerado um ambiente natural para se estudar a influência da diversidade dos programas de educação médica bem como a duração do internato no exame de entrada para residência médica. Esta investigação avalia alguns métodos usados no exame de entrada para residência médica como métodos para diferenciar os candidatos com internatos mais longos.

MÉTODOS: Candidatos selecionados para um programa de residência executaram um teste múltiplas-escolhas (MCT), teste com perguntas abertas (OQ) e o OSCE, além de uma entrevista e uma análise de currículo para avaliar participações em reuniões científicas, artigos publicados e atividades voluntárias. Foram comparados grupos pelo gênero, ano de graduação, resultados dos testes e do OSCE.

RESULTADOS: Os participantes foram distribuídos em dois grupos baseados na duração do internato: 2-anos ou menos de 2 anos. Não houve nenhuma diferença para a pontuação no MCT entre os grupos ou por quaisquer das atividades de entrevista e análise de currículo. O grupo de internato de 2-anos mostrou OQ mais alto (p =0.009) bem como os resultados do OSCE afetivo (p =0.025) e de conhecimento (p =0.002).

CONCLUSÃO: O exame OSCE diferenciou alguns aspectos relacionados a aquisição de competências e pode avaliar habilidades básicas e atitudes que seriam essenciais à prática supervisionada de medicina durante residência médica. OSCE separou aspectos não notados pelo uso de testes de conhecimento ou múltiplas escolhas.

Unitermos: Internato e residência. Educação médica.Questionários.Currículo. Avaliação educacional.

INTRODUCTION

To a large extent, medical education is often perceived as globally comparable.. However, context and conditions are usually diverse¹. Programs have included from one to two or even three years of internship.

During undergraduate years, students undergo assessment processes by a wide variety of tests, from multiple-choice (MCT) to short answers, essay questions (EQ), and oral examinations. On the other hand, assessment of clinical competence is becoming increasingly complex as well as demanding since it is patient centered and student driven². Non-cognitive goals of medical education that include values, attitudes and skills continue do be difficult to define and even more complicated to measure.

Teamwork and interprofessional skills; duty and responsibility; communication and interpersonal skills; professionalism and values; and trustworthiness and ethical behavior were all considered important non-cognitive goals in a recent survey and could be designated as Professionalism³. The OSCE (Objective Structured Clinical Examination) has been proposed as a better way to evaluate competences and perhaps even one of the best⁴.

The current Brazilian scenario for medical education includes a large number of new medical schools. length of internship among traditional and new institutions may vary. There is evidence in literature as well as some opinion from educators recommending longer clerkships as a method to improve skills for the practice of medicine^5,6. The background of medical education in Brazil seems a natural environment to study the influence of a range of programs and the internship duration on performance of the entrance exam for medical residency at the University of São Paulo Medical School.

The purpose of this observational study was to evaluate the capacity of some tests used in the residency entrance exam to identify candidates with longer clerkships during graduation.

METHODS

A multiple-choice test (MCT) and short-answer open questions (OQ) exam selected candidates for an Internal Medicine residency program at University of Sao Paulo Medical School. Performance of these candidates is described in this investigation. All candidates received written information for the exam. Their anonymity was ensured by a serial number on the application form.

Candidates selected were submitted to an OSCE-like test, an interview and curriculum analysis. The OSCE-like test consisted of five station sections contemplating the core areas (Internal Medicine, Obstetrics&Gynecology, Pediatrics, Public Health, Surgery): 1 - an upper digestive bleeding presentation, 2 - a supraventricular tachycardia, 3 - a community-acquired pneumonia, 4 - a N. meningitidis meningitis clinical case documented in a school and 5 - a pregnancy follow-up. In sections 2 to 5, candidates interacted with a professional actor/actress. Evaluation of the candidates encompassed two competences: cognitive associated with psychomotor abilities and affective abilities desirable to approach clinical situations (eye-to-eye contact, calming an anxious mother, explaining an electrical cardioversion procedure).

The interview with candidates sought evidences of participation in activities such as scientific meetings and paper/abstract presentations and published papers, as well as voluntary activities in patient care.

All test results were transformed into a score of 0 to 1000 points. The OSCE-like test score was subdivided. Those based on cognitive associated with psychomotor abilities were labeled OSCE knowledge and those based on affective abilities, OSCE affective scores.

Statistical analysis

Groups were compared for gender, year of graduation, MCT, OQ, OSCE knowledge and OSCE affective scores. Student t test, Chi-square and Mann-Whitney U tests were used accordingly from a statistical package (SPSS 10.0 for Windows). Candidates were arranged into quartiles for a post-hoc statistical analysis.

RESULTS

Five hundred and seventy eight candidates applied to the residency entrance exam that selected 111 candidates (19.2%) by means of results from MCT and OQ. These candidates (45% male) had an undergraduate education with clerkships of 1 year (8.1%), 1.5 years (29.7%) or 2 years (62.2%). For statistical purposes, they were distributed into two groups: with clerkship undergraduate programs of 2 years or less than 2 years. Most applicants finished their medical education just before taking the exam (76.6%) (Table 1).

There was no difference for the MCT score among groups. The 2 years clerkship group showed a significantly higher OQ score (740 vs. 720 points, p=0.009, Mann-Whitney U test). The OSCE-like results revealed a higher score for the 2 year clerkship group, for the affective (850 vs. 793 points, p=0.025) and the knowledge scores (808 vs. 736 points, p=0.002) (Table 2).

Performance during the OSCE-like exam could not be predicted from results in the MCT or OQ, nor from the previous clerkship duration. The OSCE scores were higher when comparing the two clerkships groups (Table 3). These groups did not differ for any of the activities assessed during the interview and curriculum analysis (Table 4).

DISCUSSION

This investigation suggests that OSCE is reliable to differentiate aspects difficult to detect in multiple choice tests. It is an exam technique capable of assessing basic skills and attitudes essential to the practice of supervised medicine that characterizes medical residency.

Literature on residency selection criteria appears to be limited, and perhaps very specific according to national educational systems. General tests, USMLE or related tests' scores, ranking in medical school and interview performance may be reported. Notwithstanding, there are different perspectives between applicants and program directors about what might be more important or valued for residency admission⁷. Also, assessment exercises and interview performance could be related to selecting doctors for postgraduate training in pediatric medicine, but erformance provided a greater breadth and depth of information about candidates than the structured interview⁸. Results obtained in this sample showed no difference in the knowledge exam with MCT, but there was a better performance with open questions and the OSCE, among applicants who had had a longer internship.

The non-cognitive attributes, although considered equally important for admission to residency programs, are also the subject of debate on recruitment and selection. There is an ongoing debate regarding ways to promote student recruitment and selection, aiming for a better match between the medical student population and the healthcare requirements of the population.This includes greater access for applicants from underprivileged areas and lower socio-economic groups⁹.

A review of pertinent literature pointed to the low discrimination capacity of grade point average and Medical College Admission Test scores because these offer no measurement of important non-cognitive attributes¹⁰. The OSCE provided a method of examining the skill acquisition of medical students¹¹. However, effects of such assessment on performance of in-training physicians could still be under evaluated¹². Brazilian medical education has had different conditions for offering internship among medical schools. Nowadays, this education system faces a national turmoil about implementation of new medical schools without appropriate evaluation of their need and even their quality, according to some professional associations.¹³ Some schools are offering internships that vary from 1, to 1.5 and 2 years. This scenario, as previously stated, seemed to be a natural environment for study of the influence of such a variety of internship durations upon the performance in the entrance exam for medical residency.

Development of practical methods for assessment of selection for residency programs could identify candidates with extensive skills and better attitudes, The OSCE has been related to these possibilities, although it is not yet a complete test¹⁴. Results presented here point to a reasonable understanding that OSCE-based admission exams, despite being time-consuming and more expensive, could focus on the objective of encouraging students to participate more intensely in daily practical activities during internship or even before¹⁵.

In addition, this model of exam may also influence medical teachers to review teaching plans, directing more energy to practical activities and attitude counseling. Contrary to such efforts, a previous report stressed that cultural aspects and lack of a favorable educational environment could delay expected improvements in traditional medical schools that adopt new assessment plans¹⁶. On the other hand, those teaching efforts may be easily recognized by resident physicians who pointed to quality of teaching as the best indicator when evaluating a learning environment¹⁷.

This investigation advocates that exams for residency selection need to emphasize and enhance importance of competence acquisition, besides knowledge content but distinct from scientific production or received honors. In addition, some authors have found data suggesting that performance during residency training was not predicted by scientific production. There are evidences revealing that personal qualities, performance during medical studies and clinical skills are better for predicting higher achievement during residency, in addition to a mentoring/coaching program as a powerful factor for career success, supporting the quality of teaching ^{4, 17, 18}.

The present study has some limitations. Sample size was not adequate to perform the proposed post-hoc analysis, resulting in non-significant tendencies. Also, it only focused on candidates for the internal medicine residency program, from whom full data could be obtained.

CONCLUSION

In conclusion, this investigation suggests that OSCE is a reliable instrument to distinguish some aspects difficult to perceive when using tests that measure knowledge or even from open questions. The OSCE can be an exam technique to assess basic skills and attitudes that are essential for the practice of medicine under supervision that characterizes medical residency.

Artigo recebido: 18/06/09

Aceito para publicação: 18/07/09

Conflict of interest: none

Trabalho realizado na Faculdade de Medicina da Universidade de São Paulo, Centro de Desenvolvimento de Educação Médica "Prof. Eduardo Marcondes" - CEDEM, São Paulo, SP

• ¹
  World Federation for Medical Education. Statement on the Bologna process and medical education. Association for Medical Education in Europe. [cited 2009 jan 9]. Available from: http://www.aic.lv/ace/ace_disk/Bologna/contrib/Statem_oth/WFME-AMEE.pdf 2005.
• 2. Howley LD. Performance assessment in medical education: where weve been and where were going. Eval Health Prof. 2004;27(3):285-303.
• 3. Mann KV, Ruedy J, Millar N, Andreou P. Achievement of non-cognitive goals of undergraduate medical education: perceptions of medical students, residents, faculty and other health professionals. Med Educ. 2005;39(1):40-8.
• 4. Epstein RM. Assessment in medical education. N Engl J Med. 2007;356(4):387-96.
• 5. Klamen DL, Willians RG The effect of medical education on students' patient-satisfaction ratings. Acad Med. 1997;72(1): 57-61.
• 6. Lind DS, Marum T, Ledbetter D, Flynn TC, Romrell LJ, Copeland EM. The effects of the duration and structure of a surgery clerkship on student performance. J Surg Res. 2005;84(1):106-11.
• 7. Bernstein AD, Jazrawi LM, Elbeshbeshy B, Della Valle CJ, Zuckerman JD. An analysis of orthopaedic residency selection criteria. Bull Hosp Jt Dis. 2002-2003;61(1-2):49-57.
• 8. Randall R, Davies H, Patterson F, Farrell K. Selecting doctors for postgraduate training in paediatrics using a competency based assessment centre. Arch Dis Child. 2006;91(5):444-8.
• 9. Powis D, Hamilton J, Gordon J. Are graduate entry programmes the answer to recruiting and selecting tomorrows doctors? Med Educ. 2004;38(11):1147-53.
• 10. Reede JY. Predictors of success in medicine. Clin Orthop Relat Res. 1999;362(1):72-7.
• 11. Harden RM, Stevenson M, Downie WW, Wilson GM. Assessment of clinical competence using objective structured examination. Brit Med J. 1975;1(5955):447-51.
• 12. Hodges B. OSCE! Variations on a theme by Harden. Med Educ. 2003;37(12):1134-40.
• ¹³
  Proteja-se: lute pela proibição da abertura de novos cursos de medicina. Carta aberta da entidades médicas (2004). [citado 11 mar 2009]. Disponível em: http://www.proteja-se.org.br/?siteAcao=Cartaaberta&id=4
• 14. Barman A. Critiques on the objective structured clinical examination. Ann Acad Med Singapore. 2005;34(8):478-82.
• 15. Vieira JE, Bellodi PL, Marcondes E, Martins MA. Practical skills are the most popular elective choice. Med Educ. 2004;38(9):1015-6.
• 16. Troncon LE. Clinical skills assessment: limitations to the introduction of an "OSCE" (Objective Structured Clinical Examination) in a traditional Brazilian medical school. São Paulo Med J. 2004;122(1):12-7.
• 17. Vieira JE. The postgraduate hospital educational environment measure (PHEEM) questionnaire identifies quality of instruction as a key factor predicting academic achievement. Clinics. 2008;63(6):741-6.
• 18. Buddeberg-Fischer B, Stamm M, Buddeberg C, Klaghofer R. Career-success scale: a new instrument to assess young physicians academic career steps. BMC Health Serv Res. 2008;8:120.

Publication Dates

• Publication in this collection
  25 Feb 2010
• Date of issue
  2009

History

• Accepted
  18 July 2009
• Received
  18 June 2009