National examination of Brazilian residents and specialization trainees in radiology and diagnostic imaging: a tool for evaluating the qualifications of future radiologists

Prova Nacional dos Residentes e Especializandos em Radiologia e Diagnóstico por Imagem no Brasil: instrumento de avaliação da qualificação do futuro radiologista

CLINICAL SCIENCE

National examination of Brazilian residents and specialization trainees in radiology and diagnostic imaging: a tool for evaluating the qualifications of future radiologists

Prova Nacional dos Residentes e Especializandos em Radiologia e Diagnóstico por Imagem no Brasil: instrumento de avaliação da qualificação do futuro radiologista

Fernando Alves Moreira^I; Luciana de Pádua Silva Baptista^II; Aldemir Humberto Soares^III; Henrique Manuel Lederman^IV; Sergio Aron Ajzen^IV; Jacob Szejnfeld^IV

^IColégio Brasileiro de Radiologia: Av. Paulista 491, Paraiso, 01311909. Email: [e-mail]

^IIDepartment of Radiology, Hospital Nove de Julho: Rua Peixoto Gomide n° 569, Cerqueira Cesar, 01409-001

^IIIDepartment of Radiology, Hospital do Servidor Publico Estadual, São Paulo Brazil

^IVDepartment of Image Diagnosis, Universidade Federal de São Paulo, São Paulo Brazil

ABSTRACT

PURPOSE: This is a study of performance based on an In-training Examination for Radiology and Diagnostic Imaging targeting residents (R) and specialization trainees (ST) in Radiology. The radiological training may differ between R and ST in some centers. The authors present their experience and thoughts regarding the first three years of application of the In-training Examination administered by The Brazilian College of Radiology.

METHODS: Three hundred and eight-six tests were analyzed in 1999, 715 in 2000, and 731 in 2001. The yearly tests consisted of multiple-choice answers, some with interpretation of digital images, and were divided into 9 specialties: neurology, thorax, physics, pediatrics, digestive system, urinary system, musculoskeletal system, mammography, and gynecology-obstetrics. Each specialty was analyzed separately. The tests were given simultaneously in 12 Brazilian cities. The subspecialty scores of examinees at different stages of training were compared (1^st, 2^nd, and 3^rd year residents and specialization trainees), by the Kruskal-Wallis test (P<0.001).

RESULTS: The 1999 examination did not reveal any significant difference between the R and ST groups (P>0.05). Generally, in 2000 and 2001, R achieved higher scores than ST (P<0.001). The performance in physics was poor for both groups for the 3 years covered by the study.

CONCLUSION: The performance of residents was better than that of the specialization trainees in the majority of the subspecialties, mainly in the last two years. The In-training Examination provides a system for evaluating future specialists and identifying the centers that need to revise their teaching methods and the regional differences in radiological training.

Keywords: Medical Education. Radiology and Diagnostic Imaging

RESUMO

OBJETIVO: Estudo comparativo entre o desempenho dos residentes e especializandos em radiologia por meio da Prova Nacional dos Residentes e Especializandos em Radiologia e Diagnóstico por Imagem (PNRERADI), durante os três primeiros anos de sua aplicação. O ensino nos centros de formação em radiodiagnóstico pode diferir entre residentes e especializandos.

MÉTODOS: Foram analisadas 386 provas em 1999, 715 em 2000 e 731 em 2001. As provas foram divididas em nove subespecialidades: neurologia, tórax, digestivo, física, pediatria, urinário, músculo-esquelético, mamografia e ginecologia-obstetrícia, cada uma delas avaliada separadamente, constando de testes de múltipla escolha, algumas com interpretação de imagens digitalizadas. As provas foram aplicadas simultaneamente em 12 cidades distribuídas no território nacional. As subespecialidades foram comparadas nos diversos níveis (residentes e especializandos de 1º, 2º e 3º anos) através do teste não-paramétrico de Kruskal-Wallis (p<0,001).

RESULTADOS: Na prova de 1999 não foi observada diferença significativa entre residentes e especializandos na maioria das áreas. Em 2000 e 2001, no geral, os residentes apresentaram aproveitamento superior aos especializandos. O aproveitamento na área de física foi ruim para residentes e especializandos, durante esses três anos de aplicação da prova.

CONCLUSÕES: O desempenho dos residentes em relação aos especializandos foi melhor na maioria das subespecialidades, principalmente nos dois últimos anos avaliados. A PNRERADI representa um meio de avaliação dos futuros especialistas, identificando os centros que necessitem de revisão do modelo de ensino, além de prover informação da diferença regional no treinamento da radiologia.

Unitermos: Educação Médica. Radiologia e Diagnóstico por Imagem.

INTRODUCTION

Until the early 1970s, radiology progressed relatively slowly compared to some other medical specialties. The advent of ultra-sonography (US), computerized tomography (CT), and magnetic resonance imaging (MRI) provided a marked stimulus for the specialty, broadening the diagnostic tools and the inclusion of radiological procedures in medicine. These new methods renewed, revolutionized, and modified the diagnostic protocols for a significant number of diseases.

Besides the technological progress, medical schools have made an effort to investment in training for these new tools. Otherwise, it would be impossible to provide the technical-scientific information required for the students who would choose radiology as a medical specialty.¹

Medical demand for radiology has been increasing, as well as for the subspecialty of diagnostic imaging. With the increasing expansion of medical knowledge, sub-specialization has become a natural course in medicine based on the available equipment or areas of the human body². The resulting growth in radiological technology led to the establishment of sub-specialties.³

However, the growth of sub-specialization has resulted in a complex structural organization of departments with frequent under-specialization or over-specialization in certain organs and systems or diagnostic techniques.⁴ The expansion of radiological knowledge has been followed by greater reliance on the newer diagnostic techniques and their clinical applications.

Residency training provides adequate assistance for radiology specialty; however, the assessment of this training is not always duly analyzed, either due to theoretical-practical or interpretative and personal parameters. Although the educational training for residents (R) and specialization trainees (ST) should be the same and based in a program accredit by the National Commission of Medical Residence (Portuguese, "Comissão Nacional da Residência Médica – CNRM"), in some centers there are no official teaching programs for ST group.

In 1999, the National Commission of Teaching and Improvement of the Medical Residency of the Brazilian College of Radiology (Colégio Brasileiro de Radiologia - CBR) proposed that the proper way to evaluate R, ST, and radiology centers would be by testing them. This is how the National in-Training Examination for Radiology and Diagnostic Imaging (Portuguese, PNRERADI: "Prova Nacional dos Residentes e Especializandos em Radiologia e Diagnóstico por Imagem") came into being. The exam is administered yearly during the medical radiological residency, permitting follow-up of the learning progress, and it has since been applied to all Brazilian R and ST groups.

Purpose

A comparative study of the performance of the residents and specialization trainees in the various sub-specialties of radiology was undertaken by the PNRERADI during the first 3 years of its application.

METHODS

Brazilian R and ST were able to register for the annual tests administered in 1999, 2000, and 2001, at 12 national cities.

The multiple choices tests included 9 sub-specialties: neurology, thorax, physics, pediatrics, urinary system, musculoskeletal system, digestive system, mammography, and gynecology-obstetrics. Each exam had 180 questions per year, 20 per sub-specialty, with separate analysis of image interpretation questions. The image interpretation questions were digitized, with most of the questions on the second and third examinations.

Head-neck, US, CT and MRI sub-specialties were not included separately in the study, although questions referring to these techniques and area were included in all tests.

To ensure secrecy, the test seals were broken at the time and place of the examinations, in the presence of the R and ST, by the presidents of the regional CBR societies, who were responsible for execution and delivery of the written tests to the CBR.

All the tests were scored electronically. The 1999 and 2000 grades were computed via a table, in which each group's highest grade was considered equivalent to 100% and the remainders were proportional to the highest grade. In 2001, each test was analyzed based on absolute values and percentages relative to grades at each level.

Every R and ST received his absolute grade and was made aware of his level of training in each sub-specialty and his rank compared to the others at his level. The grades were mailed to each candidate to ensure confidentiality.

The various areas of the sub-specialties (n=9) were compared applying the Kruskal-Wallis non-parametric pos test. The level of rejection of the null hypothesis for all tests was set at 0.05 or 5% and the significant values were marked with an asterisk. The SPSS software (9.0 version) was used for application of the statistical tests.

RESULTS

The results displayed for 1999, 2000, and 2001 tests are in Tables 1 through 9, taking into account the comparative analysis of the examinations among the R and ST at each level and in each area according to the average absolute value of points (Portuguese, "VAMDP"-valor absoluto da média de pontos) and the minimal significant difference (Portuguese, "DMS"-diferença mínima significante).

No significant difference in the majority of areas was detected between R and ST in the 1999 PNRERADI (P>0.05). Nevertheless, it should be noted that the number of ST registered was one-third the number of residents. This was the first test applied and few trainees took it. It is not known whether these trainees came from programs that were accredited by the National Commission of Medical Residence (Portuguese, "Comissão Nacional da Residência Médica – CNRM"). This topic is not discussed in this report.

A significant increase in performance by the R was evident in 2000 (P<0.001), particularly in the areas of pediatrics, urology, musculoskeletal, and digestive systems compared to ST of the same level for all three years. In mammography and thorax, the R performed better than the 1^st and 2^nd year ST (P<0.001).

In 2001, a significant increase in performance by R compared to ST of the same level occurred over the 3 years in pediatrics, thorax, neurological, urological, musculoskeletal, and digestive systems (P<0.001). The 1^st and 2^nd years R performed better than ST in gynecology-obstetrics (P<0.001). In mammography, R exhibited a better performance than 2^nd and 3^rd year ST (P<0.001). The superior performance by R, especially in their 2^nd and 3^rd year, compared to ST was confirmed in almost all areas.

No significant difference was observed in physics on the tests grades between the three levels in 1999 (P=0.171) and 2001 (P=0.034). It is important to point out that these results were similar in these years, although the tests were equal for all levels with various degrees of difficulties. Only the ST for the 1^st year did a better performance then 1^st R in 2000 test. The ST group did not achieve a better performance in no other area during the 3 years of testing.

DISCUSSION

The education in some radiological centers could be influenced not just by the length of the medical residency for R and ST, but also by the quality of the teaching. For example, the performance of both R and ST in physics was poor, even though the candidates of the 3^rd year. The low scores in physics could be due to a lack of interest of the radiologists, inadequate preparation, or a combination of both, since the grades in this area remained consistently low at all levels over the 3 years. Lately, the CBR has been promoting physics courses and virtual physics classes for interested radiology centers in an attempt to improve radiological physics for future specialists.

The quality of radiology teaching should be considered. The intention to evaluate doctors trained in radiology residency centers began with the examination taken to obtain the Certification of Specialist in Radiology and Diagnostic Imaging (Portuguese, "Título de Especialista em Radiologia e Diagnóstico por Imagem"–TERDI). In briefing, this title could indicate the quality of the candidate's residency center. The skills which radiology students should acquire during their training were the subject of Koch's thesis in 1987.⁵

In Souza's thesis on the evaluation of radiology residency programs the following mandatory areas were suggested: mammography, conventional radiology, US, CT, MRI, and interventional radiology. Optional areas were also suggested including: densitometry, nuclear medicine, and radiotherapy. Other issues discussed by Souza were: examinations under supervision, manuals of routine protocols, exchanges with residents at other hospitals, priority for scientific events, medical ethics/professional responsibility, and participation in meetings, classes, and case discussions.⁶

Although there is a standard medical residency program in radiology, the experience gained by both R and ST varies among centers and often among students from the same center. The ST groups represent a heterogeneous team. Some radiology residencies accredited by the CNRM offer specialization in Radiology and Diagnostic Imaging (RDI). In most of these centers, the ST activities are similar to those of the R group. However, in some unaccredited centers, radiology training is offered with undetermined teaching methods, sometimes resulting in inadequate specialty training.

Even though radiology residencies were instituted at the end of the 1960´s, structured information about medical residencies filed with the CNRM only became available in 1985. FUNDAP (Portuguese, "Fundação Nacional de Apoio e Pesquisa") data show that from 1985 to 1998, the number of accredited institutions offering medical residency programs grew at a rate of about 51%, principally at the expense of private entities (90%). During this time, medical residency programs in RDI grew by 37%.⁷

Many teaching techniques are employed for radiological residency education. A commonly utilized system is review by assistants of radiological reports issued by residents. A study by Seltzer et al.⁸ found that residents under this study system exhibited improved performance as early as the first 6 months of practice. In addition to improving the residents' education, it also ensures greater accuracy in the assistants' final reports. The reports by the second and third year residents were more accurate than those issued by first year residents.

Night shifts are also necessary and are considered an essential educational experience in radiological residencies and are used in the majority of the 166 programs analyzed in the USA.⁹ In the majority of cases, 1^st year residents together with 2^nd year residents are on call during the first 6 months and only after this period are allowed to take over some responsibilities on their own.^10,11

A study correlating the level of discrepancies in interpretation of emergency body CT scans among residents and assistants reported that this rate ranged from 1.2% to 6.5%, for major and minor discrepancies, respectively. Major differences were considered as those likely to affect the patient's therapy and minor ones were those that did not meet this criterion. The residents were more inclined to overlook some details of the image such as incidental findings rather than interpreting normal images as pathological ones. This data suggests good interpretation of emergency images by residents.¹²

The PACS (Picture Assistance Computer System) is an opportunity for radiologists to increase their interaction with other specialists, in addition to providing better diagnostic service.¹³

Many studies evaluated the effect of the introduction of PACS on diagnostic productivity and its effect on teaching during medical residency. Studies revealed an improved in academic activities productivity by teaching/research. Besides a change in the profile of residents who modify the position from active medical participation in reports to observer assistance. In the study by Redfern et al.,¹⁴ the proportion of images interpreted by residents dropped from 38% with film images to 17% when using digital images. Additionally, 45% of the proportion of images interpreted by residents fell to 4% with image digitalization. The effects of this change on teaching are still unknown. Nevertheless, PACS has the advantage of storing a greater number of images for teaching purposes.¹⁴

A study of 92 radiology residency programs approved by the Accreditation Council for Graduate Medical Education has shown that development of research should be an important priority to meet the Council's requirements. About 53% of residency programs require research by residents; however, only 23% of them dedicate specific time for it.¹⁵ It is up to the resident to take advantage of the residency for doing research, giving lectures, honing skills, and finding additional educational opportunities.¹⁶

Interestingly, a study examining why junior academic radiologists leave academia for private practice showed that significant changes should be made to motivate junior radiologists to stay in academia. Academic radiologists should not be expected to read private practice volumes and still have time to produce quality research and teach. In this situation, at least 1 day a week should be protected academic time.¹⁷

The impact of the course by the Armed Forces Institute of Pathology (AFIP/Washington DC, USA) regarding residents' scores on written tests of the American College of Radiology (ACR) has demonstrated better results by residents who took the ACR test during the AFIP course compared to those who had not yet taken the course. This may affect the residencies that supervise the accomplishment of residents for the American Board of Radiology examination.¹⁸

An educational activity based on case studies was well received by students and met its educational objectives. Activities of this type have potential for widespread application in radiology education. Application of this method in radiology education centers reminded the importance of providing clinical histories when ordering radiological examinations.^19,20

Radiology education is heavily dependent on visual images. Currently, hardcopy film is being replaced by digital radiological images in teaching hospitals and an electronic teaching file library would be desirable.^21,22 The ACR "Index for Radiological Diagnoses" is hierarchically organized and thus an ideal classification system for digital teaching files.²³

The digital revolution in radiology continues to advance rapidly. Whereas the implementation of an institutional radiological digital image database can require significant developmental effort and programming expertise, there are a number of web-based solutions which are freely available and can be relatively easily employed to establish a contemporary electronic image library.^24,25

Radiologists are responsible for providing patients with proper assistance in spite of the demands of their work schedule; for example, imaging specialists should be able to view general radiology studies. There are differences between radiologists working in urban and rural environments, between minor and major services, and between academic and private practice.²⁶

The PNRERADI is not only an annual method for evaluating future radiologist, but is included in a pedagogical training process of continuing education for R, ST, medical residencies, and teaching committees.

The proposal of this evaluation is to guide R and ST through their most difficulties in RDI. Additionally, it should stimulate interest and provide motivation for them to obtain the Certification of Specialist in Radiology and Diagnostic Imaging (TERDI) in Brazil.

This test has been applied since 1999 and the number of R and ST has been increasing over the years. The training of good professionals and their continuous evaluation are fundamental for the best practice of radiology.

CONCLUSION

The performance of the R compared to that of the ST was better for the majority of the radiology specialties, demonstrating that R may receive better overall training, enabling them to obtain better average scores on tests. The PNRERADI provides a system for evaluating future specialists and for identifying the centers that need to revise their teaching methods. It could be used to inform accreditation decisions and to identify regional differences in radiological training, in addition to providing support for improvement of the imaging specialist.

Received for puiblication on May 16, 2007

Accepted for publication on August 12, 2007

• 1. Karpovas L. O "Status Atual da Radiologia em Nosso Meio. Radiol Bras. 1998;31:V.
• 2. Covey AM, Sunshine J, Forman HP. The Job Market in Diagnostic Radiology 1999: Updated Findings from a Help Wanted Index of Job Advertisements. AJR. 2000;175:957-961.
• 3. Secaf F. Caminhos da Radiologia. Radiol Brás. 1988;21:XI-XIV.
• 4. Henschke CI, Steiner RM, McLoud T, Westcott JL. The Practice of Chest Radiology in Departments with Residency Programs. Invest Radiol. 1994;29:48-53.
• 5. Koch HA, Pereira AA (orientador). A radiologia nos cursos de graduação em medicina: uma contribuição no processo ensino-aprendizagem. [Tese de doutorado]. Rio de Janeiro: UFRJ, 1987.
• 6. Sousa EG, Koch HA (orientador). Avaliação dos programas de residência médica em radiologia. [Tese de doutorado]. Rio de Janeiro: UFRJ, 2000.
• ⁷
  Fundação do Desenvolvimento Administrativo. Oportunidade de Ingresso na Residência Médica. São Paulo: FUNDAP, 1998 (Documentos de Trabalho, 77).
• 8. Seltzer SE, Doubilet PM, Sheriff CR, Katz JM. Educational Impact of Faculty Review on Radiology Residents´ Radiographic Interpretations. Invest Radiol. 1984;19:61-64.
• 9. Townsend RR, Manco-Johnson ML. Night call in US Radiology Residency programs. Acad Radiol. 1995;2:810-815.
• 10. Lederman HM, Clemente Filho AS. A Importância do Aprendizado das Técnicas Radiológicas (de Imagem) pelo Residente de Radiologia. Radiol Brás. 1989;22:IX-X.
• 11. Stubbs DM. Emergency Radiology Services at Medical Schools in the United States. Invest Radiol. 1992;27:648-652.
• 12. Wechsler RJ, Spettell CM, Kurtz AB, Lev-Toaff AS, Halpern EJ, Nazarian LN, et. al. Effects of Training and Experience in Interpretation of Emergency Body CT Scans. Radiology. 1996;199:717-720.
• 13. Thrall JH. The Radiologist in the 1990s: new practice expectations and management responsibilities. AJR. 1994;163:11-15.
• 14. Redfern RO'C, Lowe RA, Kundel HL. From being a radiologist to watching a radiologist: impact of filmless operation on the training of radiology residents. Acad Radiol. 2001;8:1154-1158.
• 15. Mcguire CW, Heberman HB. Research in radiology programs. Acad Radiol. 1998;5:698-700.
• 16. Harolds JA. Resident's be aggressive in your own education. Acad Radiol. 1998;5:655-657.
• 17. Taljanovic MS, Hunter TB, Krupinski EA, Alcala JN, Fitzpatrick KA, Ovitt TW. Academic radiology: the reasons to stay or leave. Acad Radiol. 2003 Dec;10:1461-8.
• 18. Wise SW, Mauger DT, Mathews AE, Hartman DS. Impact of the armed forces institute of pathology radiologic pathology course on radiology resident performance on the acr in-training and abr examinations. Acad Radiol. 2000;7:693-699.
• 19. Chew FS, Ochoa ER, Relyea-Chew A.Application of the case method in medical student radiology education. Acad Radiol. 2005;12:746-51.
• 20. Dawes TJ, Vowler SL, Allen CM, Dixon AK.Training improves medical student performance in image interpretation. Br J Radiol. 2004 ;77:775-6.
• 21. Yang GL, Lim CC. Singapore National Medical Image Resource Centre (SN.MIRC): a world wide web resource for radiology education. Ann Acad Med Singapore. 2006;35:558-63.
• 22. Silverstein JC, Walsh C, Dech F, Olson E, Papka ME, Parsad N, Stevens R. Immersive virtual anatomy course using a cluster of volume visualization machines and passive stereo. Stud Health Technol Inform. 2007;125:439-44.
• 23. Trumm C, Dugas M, Wirth S, Treitl M, Lucke A, Küttner B, Pander E, Clevert DA, Glaser C, Reiser M. Digital teaching archive. Concept, implementation, and experiences in a university setting. Radiologe. 2005;45:724-34.
• 24. Scarsbrook AF, Foley PT, Perriss RW, Graham RN. Radiological digital teaching file development: an overview. Clin Radiol. 2005;60:831-7.
• 25. Scarsbrook AF, Graham RN, Perriss RW. Radiology education: a glimpse into the future. Clin Radiol. 2006;61:640-8.
• 26. Vydareny KH. Radiology 1997: The Ethics of Change. AJR. 1998;171:311-312.

Publication Dates

• Publication in this collection
  25 Nov 2008
• Date of issue
  2007

History

• Accepted
  12 Aug 2007
• Received
  16 May 2007