Research methodologies to assess teaching in psychiatric residency: a literature review

Metodologias de pesquisa para avaliação do ensino na residência de psiquiatria: uma revisão da literatura

COMUNICAÇÃO BREVE

Research methodologies to assess teaching in psychiatric residency: a literature review

Metodologias de pesquisa para avaliação do ensino na residência de psiquiatria: uma revisão da literatura

Ibiracy de Barros Camargo; José Onildo B Contel

Mental Health Doctorate Program of the Ribeirão Preto Medical School of the University of São Paulo. Ribeirão Preto, SP, Brazil

^{Correspondence}

ABSTRACT

OBJECTIVES: Research methodologies in psychiatry have followed both the changes in mental health care and the need of updating programs of medical residency. To identify empirical articles in the indexed literature from 1997 to 2002, with the aim of analyzing and discussing methodological aspects of research dealing with the description and assessment of residency programs in psychiatry.

METHOD: The bibliographic survey was performed using MedLine, PsycLit, Web of Science, and Lilacs. Twenty-one articles were identified.

RESULTS: Nineteen studies were characterized as exploratory-descriptive and two as experimental. Data collection used questionnaires in 12 of them, and combined techniques in the other seven and the two experimental studies had data collected by tests applied before and after the teaching intervention. Most of the subjects were residents and program directors. Fifteen studies used statistical analysis.

CONCLUSIONS: All the articles outlined the problems based on literature reviews. Most of the studies made use of standard techniques of social research and only two used experimental procedures. Only three studies employed external measures in order to establish correlations with the collected data. Procedures to validate and assess the reliability of the instrument by means of pilot-studies were absent in 11 studies, what may indicate methodological biases.

Keywords: Training. Academic psychiatry. Skills. Knowledge. Medical education. Medical training. Residency. Psychiatry.

RESUMO

OBJETIVOS: As metodologias de pesquisa na psiquiatria têm acompanhado tanto as transformações nos sistemas de atendimento à saúde mental como a necessidade de atualização dos programas de residência médica. Identificar, analisar e avaliar na literatura indexada entre 1997 e 2002 artigos empíricos relativos à avaliação de programas de residência em psiquiatria.

MÉTODO: O levantamento bibliográfico foi realizado através dos indexadores MedLine, PsycLit, Web of Science e Lilacs. Foram identificados 21 artigos.

RESULTADOS: 19 estudos caracterizam-se como descritivos exploratórios e dois, experimentais. Em 12, a coleta de dados utilizou questionários, em sete, técnicas combinadas e, nos dois estudos experimentais, os dados foram coletados por meio de testes aplicados antes e depois da intervenção. Residentes e diretores de programas constituíram a maioria dos sujeitos. Em 15 trabalhos, a análise de variáveis utilizou procedimentos estatísticos.

CONCLUSÕES: Todos os artigos delinearam os problemas com base em revisão da literatura. A maioria dos trabalhos utilizou técnicas comuns na pesquisa social e apenas dois empregaram recursos experimentais. Somente três estudos utilizaram medidas externas para estabelecer correlações com os dados coletados. Procedimentos de validação e aferição de confiabilidade do instrumento, por meio de estudos-piloto, não foram realizados em 11 estudos, o que pode indicar vieses metodológicos.

Descritores: Treinamento. Psiquiatria Acadêmica. Habilidades. Conhecimento. Educação Médica. Treinamento Médico. Residência. Psiquiatria.

Introduction

The role of the research methodology in psychiatry has undergone recent modifications¹ and knowledge originated from research had become increasingly crucial for psychiatrists. The reviews that have teaching in psychiatry as their object of study have become relevant to assess its current state.²

The concern with research has been constant in the field of Psychiatric Residency teaching, in the several rotations of the Services of the Program of Psychiatric Residency (MRP) of the Clinical Hospital of Ribeirão Preto.^3,4

The authors suppose that a similar concern may be present in many of the 45 MRP, accredited by the National Commission of Medical Residency.⁵ Researches about quality, program content and teaching methodology in MRP will be essential for the development of Brazilian psychiatry.

The complexity of a research project recommends that its development be preceded by certain precautions, such as a review of the literature which sums up the scientific breakthroughs, generates knowledge and creates hypotheses.^6-8 Its advantage is to apprehend what is the most current and valuable in the scientific research, immediately divulged for professionals all over the world.

Objectives

To identify empirical articles in the indexed literature between 1997 and 2002 aiming at analyzing and discussing methodological aspects of researches focused on the description and assessment of residency programs in psychiatry.

Method

The bibliographical research was performed with the indexes MedLine, PsycLit, Web of Science and Lilacs with four sets of keywords: 1) training, academic psychiatry, skills; 2) training, medical education, skills; 3) medical training, residency, psychiatry; 4) residency, psychiatry, knowledge. Review articles and those about teaching on non-psychiatric residencies and on medical graduation were excluded.

Results

We found 21 articles about the chosen subject, 19 of which described researches about existent programs and two described experimental methodologies.

Exploratory descriptive studies

Out of 19 articles, 15 obtained data regarding the quality and features of functioning of MRP^{9-17,20,22,24,25,28,29} and four developed or tested instruments to assess the quality of the programs.^18,19,23,28

Experimental studies

One of the articles tested the efficacy of the Problem-Based Learning,²⁶ and other verified the efficiency of a training about cultural sensitivity.²¹

Research outlining and formulation of hypotheses

Eleven studies had not outlined their researches based on pilot studies,^{10-15,17,20,24,25,29} whereas nine had done it^{9,16,18,19,21,23,26-28} and one was presented as a pilot.²²

All articles outlined the research issues based on a review of the literature, eight of which had explicitly divulged their hypotheses^{10,15,21,23-26,28} and 13 had not.^{9,11-15,17-20,22,27,29}

Data collection

Among the 19 exploratory descriptive studies, 12^{9-14,16,17,20,22,28,29} used the questionnaire as a sole instrument, with a predominance of shipment, whereas seven^{15,18,21,23-25,27} used combined techniques, such as questionnaires, phone or face-to-face interviews, PRITE examination and evaluation card of the resident by professors.

The two experimental studies collected data by means of pre-tests or post-tests using questionnaires²² and Likert attitudes scale.²³

Subjects

Subjects used in the researches reported in the articles were: residents in 10 studies;^{9,11,17-19,21,23,26,27,29} program directors, in 7;^{10,12,13,16,20,22,28} preceptors and residents, in two;^24,25 directors and residents, in one¹⁴ and patients professors and residents, in another one.¹⁵

Data analyses

In 15 studies, the analysis of inter and intra group variables used statistical procedures (ANOVA, Chi-square, Kappa, Mann-Whitney U Test, among others),^{10,12-14,17-27} while six presented variables using only percentage distribution.^{9,11,15,16,28,29}

Inter group, intra group and external comparisons. Eighteen studies performed the analysis of variables by means of intra group and inter group comparisons.^{9-17,19-22,24-26,28,29} In three studies^18,23,27 PRITE exams were used as external comparisons.

Discussion

Psychiatric residency occurs in a sequential number of years, being four in the US and Europe and two, exceptionally three, in Brazil.

All articles assessed teaching results or characteristics of MRP profiting from the design of the expected sequential progress. This characteristic favor transversal, and longitudinal studies and intra and inter group comparisons. In the US external comparison using the PRITE Exam,³⁰ noted in three articles,^18,23,27 provided an objective external measurement of the resident's performance.

Methodologically, the use of external comparisons is an advance as it utilizes more than one measure instrument, besides those developed within the local investigation of MRP, to check the focused variables.

The analyzed articles depict researches conducted by means of common techniques in the area of social research. The fact that we found only two experimental studies^21,26 might be an indication that current researches do not dare to present and test the validity of new modes of teaching psychiatry, focusing mainly in the description of what is already existent.

In some studies it was possible to detect traces of methodological flaws. In one study, the title of the article¹⁷ was too restrictive regarding the objectives of the research, which were much wider than analyzing the attitude of residents regarding psychotherapy. What we observe is the much wider opinion of subjects regarding the biological, psychological and social areas which compose the process of formation of psychiatrists.

One of the analyzed studies¹² does not explicit the way in which subjects answered the questionnaire, what prevents conclusions regarding the homogeneity of the application procedures of the instrument and may have influenced the answers.

The most common methodological flaw was the absence of pilot studies to validate their research instruments in 11 articles, as they did not provide precision assurances of their instruments of data collection.

One article was developed with methodological strictness.²² Its outlying and the investigated theme were based on previous researches, using the same instrument and a significant number of respondents, with a wide support in theoretical and methodological references.

Conclusions

This review encompasses a wide gamut of aspects related to MRP in several countries. The wide range of themes suggests the lack of a common direction in the researches and a lack of focus and a dispersion of objectives, what suggests the inexistence of well-defined research lines. This varied description of MRP ends hampering the generalization of obtained results and the adoption of measures aimed at changing the psychiatric teaching.

The two studies of Brazilian authors may indicate that in Brazil there is still no consistent development of research lines in this field.

Most of the studies uses common techniques to social research and only two used experimental resources. The fact that only three studies used external measures may indicate that this procedure, although desirable from the perspective of its validity as an assessment instrument, be underestimated by the researchers in the field.

None financial support and conflict of interests

Received in 17/1/2003

Approved in 31/3/2003

• 1. Weiss DS. Research, methodology and statistics. In: Tasman A, Key J, Liebirman JA, eds. Psychiatry. 1^st ed. Philadelphia: WD Saunders; 1997. p. 1876-91.
• 2. Camargo IB. Habilidades e conhecimentos sobre a prática profissional em psiquiatria: tradução, adaptação e aplicação de questionários em psiquiatras brasileiros [dissertação de mestrado]. Ribeirão Preto: Faculdade de Medicina de Ribeirão Preto da USP; 2001.
• 3. Camargo IB, Vansan GA, Contel JOB. História do início do ensino de psiquiatria na residência médica do Hospital das Clínicas da FMRP-USP. J Bras História Med 2002;5 Supl 1:38.
• 4. Camargo IB, Contel JOB. Influências relevantes na procura por formação em psiquiatria no Hospital das Clínicas da FMRP-USP. J Bras História Méd 2002;5 Supl 1:38.
• ⁵
  Brasil. MEC/CNRM/SESU. 5 de julho de 2000. Relação de Instituições e Programas de Residência Médica por Região. Brasília (DF): CNRM; 2000. p. 119.
• 6. Berrios GE. The review paper in psychiatry. In: Iremon C, ed. Research methods in psychiatry: a beginner's guide. Washington (DC): American Psychiatric Press; 1992. p. 12-23.
• 7. Cobos RPG, Hernandes JR, Morales CL. Aptitud para la lectura crítica de artículos de investigación clínica en residentes de psiquiatria. Rev Investigación Clínica 2001;53:28-34.
• 8. Fu CHY, Hodges B, Reghr G et al. Is a journal club effective for teaching critical appraisal skills: a controlled trial with residents in psychiatry. Acad Psychiatry 1999;23:205-9.
• 9. Carney CP, Pitkin AK, Malis R, Yates WR. Combined internal medicine/psychiatry and medicine practice/psychiatry training programs 1999-2000. Acad Psychiatry 2002;26:110-6.
• 10. Greenberg WM, Ritvo JI, Fazzio L, Bridgeford D, Fong T. A survey of addiction training programming in psychiatry residencies. Acad Psychiatry 2002;26:105-9.
• 11. Houghtalen R, Olivares T, Greene Y, Booth H, Conwell Y. Residents' morning report in psychiatry training description of a model and survey of resident attitudes. Acad Psychiatry 2002;26(1):9-16.
• 12. Margariti MM, Kontaxakis VP, Christodoulou GN. Toward a European harmonization of psychiatric training in Greece: the prospects of residency training in Greece. Acad Psychiatry 2002;26:117-24.
• 13. Sudak DM, Beck JS, Gracely EJ. Readiness of psychiatry residency training programs to meet the ACGME requirements in cognitive-behavioral therapy. Acad Psychiatry 2002;26:96-101.
• 14. Yudkowsky R, Elliott R, Schwartz, A. Two perspectives on the indicators of quality in psychiatric residencies: program directors' and residents'. Acad Med 2002;77:57.
• 15. Dobscha SK, Ganzini L. A program for teaching psychiatric residents to provide integrated psychiatric and primary medical care. Psychiatric Services 2001;52:1651.
• 16. Doebbeling CC, Pitkin AK, Malis R, Yates WR. Combined internal medicine psychiatry and family medicine psychiatry training programs, 1999-2000: program directors' perspectives. Acad Med 2001;76(12):1247-52.
• 17. Margariti MM, Kontaxakis VP, Christodoulou GN. Psychotherapy in postgraduate psychiatric training: attitudes of residents. Psychother Psychosom 2001;70:112-4.
• 18. Sierles FS, Daghestani A, Weiner CL, Vito R, Fichtner CG, Garfield DAS. Psychometric properties of ABPN-Style oral examinations administered jointly by two psychiatry residency programs. Acad Psychiatry 2001;25:214-22.
• 19. Elliott RL, Yudkowsky R, Vogel RL. Quality in psychiatric training: development of a resident satisfaction questionnaire. Acad Psychiatric 2000;24:41-6.
• 20. Isaacson JH, Fleming M, Kraus M, Kahn R. A national survey of training in substance use disorders in residency programs. J Stud Alcohol 2000;61(6):912-5.
• 21. Lobo-Prabhu S, King C, Albucher R, Liberzon I. A cultural sensitivity training workshop for psychiatry residents. Acad Psychiatry 2000;24:77-84.
• 22. Yudkowsky R, Schumartz A. Content, culture and contexts determinants of quality in psychiatry residency programs. Acad Med 2000;75(10):99-101.
• 23. Albucher RC, Maixner SM, Riba MB, Liberzon I. Neurology training in psychiatry residency. Acad Psychiatry 1999;23:77-81.
• 24. Calil LC. Verificação do cumprimento das exigências feitas pela comissão nacional de residência médica nos programas de residência médica em psiquiatria do Estado de São Paulo no ano de 1993. J Bras Psiq 1999;48(8):367-74.
• 25. Calil LC, Contel JOB. Estudo dos programas de residência médica em psiquiatria no Estado de São Paulo. Rev Bras Psiquiatr 1999;21(3):139-44.
• 26. Schultz-Ross RA, Kline AE. Using problem-based learning to teach forensic psychiatry. Acad Psychiatry 1999;23:37-41.
• 27. Woodman C, Schultz SK. Faculty assessment of residents and the psychiatry resident in-training examination. Acad Psychiatry 1999;23:137-41.
• 28. Ellis TE, Dickey TO, Jones EC. Patient suicide in psychiatry residency programs. Acad Psychiatry 1998;22:181-9.
• 29. Guillem-Mirales JL, Cuquerella-Benavent MA, Hernández-Viadel M, Pérez-Prieto, JF, Raya-Chamorro N, Conesa-Burguet M de los LI, Vila-Gómes ML. Formación em adicciones em el programa mir (Médico interno residente) de psiquiatria em España. Resultados de uma encuesta. Adicciones 1997;9(4):571-8.
• ³⁰
  The American College of Psychiatrists. PRITE Coordinator (online) 2002. Available from www.acpsych.org

Publication Dates

• Publication in this collection
  01 Oct 2003
• Date of issue
  Sept 2003

History

• Accepted
  31 Mar 2003
• Received
  17 Jan 2003