Documenting Outcomes of a Faculty Training Workshop on Problem-Based Learning

Documentando os Resultados de um Seminário de Treinamento de Docentes em Aprendizagem Baseada em Problemas

Abstract:

A faculty training workshop on problem-based learning (PBL) methods was provided to educators at a public medical school in Brazil. Wprkshop activities were designed to address specific knowledge, skill and attitudinal objectives that had been defined beforehand. Participants completed a structured questionnaire before and after the workshop to document attainment of the objectives. Twelve participants completed the 20-hour workshop, and questionnaire results. Though there were certain limitations on interpretation of the pre/post questionnaire, it does provide some assurance to program organizers and sponsors that objectives are attained. Participants' responses also help organizers improve their delivery techniques.

Keywords:
Problem-based learning; Medical education

Resumo:

Uma oficina de trabalho para treinamento de docentes sobre métodos de aprendizado baseado em problemas (PBL) foi realizada com educadores em uma escola pública de ensino médico no Brasil. As atividades da oficina de trabalho foram planejadas para atingir objetivos especificas de conhecimentos, de habilidades e de atitudes previamente estabelecidos. Os participantes preencheram um questionário estruturado antes e após a oficina de trabalho para documentar o alcance dos objetivos. Doze participantes completaram a oficina de trabalho de 20 horas e os resultados do questionário demonstraram que os objetivos foram alcançados. O conhecimento dos participantes sobre a metodologia PBL aumentou significativamente, quando avaliado pela recordação de fatos e pela própria percepção desta compreenção. Os participantes adquiriram habilidades em concluir módulos de ensino e tornaram-se significavamente mais confiantes em sua capacidade de construir módulos. Embora a interpretação do pré e do pós-questionário tenha certas limitações, ela proporciona alguma garantia aos organizadores e patrocinadores de tais programas de treinamento de que os objetivos são alcançados. As respostas dos participantes também ajudam os organizadores a melhorar suas técnicas de apresentação.

Descritores:
Educação médica; Aprendizado baseado em problema

Brazilian medical educators, like those in many parts of the world, are currently seeking ways to improve the delivery and outcomes of medical education. Innovative educational methods such as problem based learning (PBL) have been introduced in some Brazilian medical schools¹ and are being considered by others.^2),(3

On-site educational workshops are a recognized way to increase faculty understanding of new methods such as PBL and to develop the faculty skills necessary to implement them.^4),(5 However, carrying out workshops is a costly undertaking. The fees and transportation costs of guest trainers, facility costs, educational materials and time investment by local organizers are substantial. In addition, the time investment by participants is significant. Brazilian medical educators usually have more than one source of employment⁶ and are under considerable professional stress.⁷ Taking time off for educational training activities usually involves making complex arrangements and accepting increased time pressures. Thus educators willing to volunteer for these training experiments represent a dedicated and potentially influential resource for the institution. Considering the investments involved in providing a training workshop, planners have the responsibility to assess and meet the needs of the institution through specific objectives, to use appropriate methods for delivering and evaluating the experience and to provide evidence to the host institution that objectives have been met.⁸

Training goals can be identified in cognitive, behavioral and affective domains, and workshops which address a combination of objectives, i.e. in knowledge, skill and attitudes, are more likely to result in lasting change.⁹ Furthermore, training modalities and evaluation methods differ for each of these types of objective.¹⁰ Explicit evaluation of training outcomes is an essential component of workshop design, both to provide a deliverable for the sponsoring institution and to assist workshop presenters in improving their training efforts.⁴ We present here the design and outcomes of a brief PBL training workshop at a Brazilian public medical school (Table 1). A pre-test/post-test evaluation questionnaire was used to demonstrate that workshop objectives were met.

Table 1
Objectives, Training Modalities and Evaluation Methods used in a Problem-Based Learning Faculty Development Workshop

Area of learning	Workshop objective: At the end of the workshop, participantes will	Training Modalities	Evaluation Methods
Knowledge	Correctly recall basic characteristics of PBL methodology and differences between PBL and traditional medical education methods	• Lectures	• Accuracy of factual recall
		• Open discussion sessions	• assessment of self efficacy16
		• Demonstration of tutorial group
Skills	Work effectively with colleagues in the development of PBL case modules	8 supervised practice in case	• Behavioral observation of skill acquisition
		9 development formative feedback	•assessment of self-efficacy
Attitudes	Display an open but appropriately criticai attitude regarding advantages, disadvantages and potential modes of implementation of PBL in their own educational milieu.	• modeling	• Affirmation of acceptance
		•critical reflection17	• goal-setting

METHODS

Participants: The workshop was open to faculty, and students of the Master's Program in Hematology, at the Pernambuco State Medical School (UPE). The workshop was announced through the Faculty Post-Graduate Coordination of the institution. Participants had to register in advance, but there was no charge for participation. Workshop faculty included one guest coordinator / trainer (PLM) and two coordinator /trainers from UPE (FA, RM). Medical students from UPE volunteered to participate in a demonstration tutor group. Teaching Support Program personnel from the Pernambuco State Hematology Foundation (HEMOPE) provided organizational assistance.

Program: A total of 20 hours of scheduled activities was carried out in two and one half consecutive days. During these three days seven brief formal presentations (20 - 40 min) by trainers were interspersed with open discussion sessions of the same length. Day One activities also included a demonstration of a tutor group studying a patient problem. On Day Two, participants worked in small groups to develop and present patient problem modules, using actual case records as resources. On Day Three one session focused on ways in which participants could continue to act as change agents¹¹ to promote further development of PBL at UPE. Copies of a recent medical education text describing PBL methods¹² were distributed to key educators and were donated to the UPE and HEMOPE libraries. Day Three activities also included a final workshop evaluation session. All activities and evaluations were carried out in Portuguese. Those attending all sessions of the workshop received certificates of completion issued by the Dean of the Medical School.

Sponsorship: The workshop was jointly sponsored by the Hematology Discipline, UPE; the Didactic-Clinical Training Disci­pline of the Inter-institutional Master's Program of UPE/University of Sao Paulo; and HEMOPE. The educational facilities of the HEMOPE Foundation, Recife, were used for all training activities.

Workshop Evaluation: A structured questionnaire was administered before and after the workshop to evaluate participants' knowledge, attitudes and perceptions of their own skills regarding PBL. Content of individual questionnaire items is described in Results. Since questionnaires were anonymous, pre- and post­tests were matched by a number code system. To encourage participation and to reinforce remembering the number code, the number was used in a drawing at the end of the workshop, with a copy of a recent medical education text¹² awarded to the winner. Paired Student's t-tests (two-tailed) were used to evaluate differences between pre-test and post-test responses on individual items, with p < 0.05 considered significant.

RESULTS

Ten faculty members from Hematology (4) and other Disciplines (6), plus two post-graduate students, completed all workshop activities. An additional eight faculty members and one post-graduate student participated in some segments. Ali questionnaire analyses are based on results from the twelve participants who completed the workshop.

Of those who did not complete the course, two were faculty from institutions other than the UPE, and six were faculty from UPE. One post-graduate student could not complete the course due to conflicting schedules.

Three of the nine did not complete the pre-test. Of those who did, none felt that students were resistant to PBL, while two of six felt that faculty were somewhat resistant and one of six felt that administration was somewhat resistant to PBL. One of five who answered the question felt that PBL methods were somewhat inconsistent with the objectives of UPE, the rest indicating that methods were consistent with the objectives of the institution.

Knowledge Objective: Attainment of this objective was evaluated from participants' answers to eight multiple choice questions regarding PBL curricula and activities. Total correct, total incorrect and total 1ight-mi.nus-wrong scores were compared (Table 2). At the beginning of the workshop, participants had considerable misinformation about PBL as well as limited correct information. Both total correct answer scores (mean correct answers = 82 ± 18%) as well as total right-minus-wrong scores were significantly improved at the end of the workshop. However, participants still retained some misinformation about PBL (Table 2).

Table 2
Effects of a training workshop m participants' responses on a multiple choice quiz assessing knowledge of problem-based learning.*

Score	Pre-test (mean ± sd)**	Post-test (mean ± sd)
Total correct responses (maximum =12)	6.0 ± 4.0	9.8 ±2.11
Total incorrect responses (maximum = 20)	± 5.1	7.8 ± 3.2
Total right minus wrong (maximum = 12)	-4.2 ± 6.3	1.8 ± 4.11

• *responses to 8 questions, 4 options per question, with instructions to choose all options wich applied;**mean ± standard deviation. n = 12; ¹slatistically significant difference from pre-test, paired t-test p < 0.02; ¹statistically significant difference from pre-test, paired t-test p < 0.01.

In addition to multiple choice scores, attainment of the knowledge objective was judged by changes in participants' own perceptions of their understanding of various aspects of PBL, which they rated on a 1- 5 scale (from no understanding to complete understanding). As shown in Figure 1, those attending the workshop felt that their understanding had increased significantly (p < 0.01) by the workshop's end for all aspects assessed.

Skills Objective: Attainment of the skills objective was assessed by the production of teaching modules and by participants' increased confidence in preparing such modules. A total of five draft modules were produced by the groups, complete with history, physical examination and laboratory test findings. Participants agreed that with a similar amount of time spent on editing, the modules could be used by medical student PBL groups in hematology or pediatrics. Furthermore, when asked to rate their confidence (on a scale of 1 = noconfidence to 5 = complete confidence) in their ability to develop such modules, participants' ratings increased significantly (p < 0.01) from the beginning (mean rating 1.7 ± 1.0) to the end (mean rating 3.3 ± 0.9) of the workshop.

Attitudes Objective: As a result of workshop activities, participants increased their estimation of PBL as an appropriate format for several didactic activities at UPE (Fig. 2). At the end of the workshop over70% of participants considered PBL as an acceptable teaching method for individual courses and in the postgraduate curriculum. By contrast, after the workshop more participants thought the methodology unsuitable as a teaching method for the entire preclinical curriculum, indicating that altered views did not merely involve uncritical acceptance of PBL. In one session, participants were asked to define specific activities or situations in which they could act as change agents to encourage use of PBL methods at UPE. Both faculty members and postgraduate students were able to identify specific ways in which they were willing and intended to promote the use of PBL.

Workshop activities did not change attitudes regarding the general acceptability of PBL at UPE. Participants were asked at the beginning and end of the workshop to indicate whether they considered PBL methods to be consistent with the goals and objectives of the institution. On a scale of 1 = completely inconsistent to 5 = completely consistent, they considered the methodology to be consistent with UPE goals (mean before and after ratings, 3.8/3.8). Similarly, participants ranked resistance to adoption of PBL methods among faculty, students and administration of UPE (from 1 = totally resistant to 5 = completely in favor). Participants felt that students (mean before and after ratings, 3.6/3.7) would be more in favor of PBL than faculty (2.7 /2.7) or administrators (2.8/2.7).

On the post-test, participants were asked to rate (from l= no change to 5 = completely changed) how much they thought their questionnaire responses had changed since the pre-test. They were also asked to what extent they felt any changes were due to the workshop and to what extent they were due to other events that had occurred during the period. Participants felt their answers had changed substantially (mean rating 3.8 ± 0.8) and that the changes were almost exclusively due to the workshop (mean rating 4.7 ± 0.5).

DISCUSSION

Though use of an evaluation instrument was helpful to document the results of the workshop, interpretation of these outcomes must take into account the limitations of both the study design and the instrument itself. The simple pre/post, one-group design is a relatively weak format for assigning causality,¹³ and thus some caution is warranted in attributing differences in pre- and post-tests to the workshop intervention. Nevertheless, confidence in the role of the workshop activities in producing d1ange is increased by two factors. First, the compact time frame of the intervention left little time for confounding occurrences to contribute to post-test differences, and second, the participants themselves attributed changes in their responses almost exclusively to the workshop.

The instrument itself was designed specifically for this workshop and thus has undergone no formal validity and reliability analysis. Workshop developers must face this dilemma in tailoring workshops to local needs, since the instrument can only be developed after the objectives of the particular workshop are defined. Thus, formal analysis, e.g. of international consistency of questions in the knowledge section of the questionnaire, becomes difficult. One way to strengthen the probability that the assessment method detects attainment of objectives is to include at least two (preferably three) separate types of evaluation for each objective. This triangulation, along with face validity of the instrument, may be the only assurance that workshop developers and clients have that the instrument adequately indicates if goals have been met.

The fact that nine of those who attended the first day of the experience did not complete all activities may also introduce bias into the results obtained. Pretest results suggest that at least two of these may have had negative preconceptions regarding the acceptability or relevance of PBL for the institution. Considering U1e ti.me constraints on medical school faculty, it is not surprising that some were unable to attend on the third day when post-tests were administered. Workshop organizers should calculate such attrition into their estimates of attendance for intensive or lengthy training and should consider drop-outs to include both those who were not convinced of the value of the endeavor and those whose other commitments took priority. The phenomenon is not unique to UPE. Pretest questions can help distinguish the two motives for dropping out.

With these caveats in mind, the instrument used in this workshop indicated that all three objectives were attained. Knowledge of various aspects of PBL was increased as indicated by the multiple choice portion of the test, and participants also indicated they felt their knowledge had increased. Despite these changes, those attending the workshop still retained numerous misconceptions regarding PBL whi.ch need to be addressed in further activities. The skill objective was also attained, as judged by the production of teaching modules and by an increase in participants' ratings of self-efficacy in carrying out this activity. The moderate level of confidence which faculty expressed after training is not surprising, since module production is a complex activity. Since participants actually prepared drafts of all sections of a PBL module, and the local library contains examples of completed modules, further formal training should not be necessary to consolidate these skills.

Attitudinal objectives are probably the most difficult to attain and to assess.⁹ After this workshop participants apparently increased their positive attitudes regarding PBL and were able to discriminate certain teaching situations in which the method might be appropriate. Nevertheless, participants continued to feel that general attitudes toward PBL at UPE were moderately positive to slightly resistant. Teachers who are willing to dedicate their time to volunta1y training activities probably represent the most amenable sector of the faculty. A "politeness bias" may also influence responses to attitudinal questions, especially in small groups. Thus, the actual impact of the workshop on use of PBL may require a more formal follow-up procedure to assess implementation of PBL methods in didactic activities. In active PBL programs, follow-up of tutor training has shown that behavior is best sustained with yearly reinforcement of trainning.^14),(15 As in the present case, workshop outcomes can be used, during the initial development stages of programs, to indicate to funding agencies the level of commitment from the institution involved, availability of suitable resources (e.g. modules) and receptivity of administration and faculty to innovative methods.

Our own use of the assessment instrument has provided lessens that will guide future workshops. First, though attention was given to misconceptions which appeared on the knowledge portion of the pre-test, these misconceptions could have been more methodically addressed in training activities. Thorough analysis of pervasively incorrect answers on the pre-test can highlight specific points to be stressed in workshop presentations. Similarly, goal-setting activities were carried out but not formally documented in the workshop, e.g. through development of action plans.⁹ However, written action plans prepared by participants can reinforce the long-term impact of the workshop and can provide the operational criteria for assessing this impact in the future.

Figure 1. Participants' ratings of their understanding of different aspects of PBL methodology before and after attending a training workshop. Participants (n = 12) felt their understanding had increased significantly (p < 0.01) after training.

Figure 2. Percent of participants in a training workshop who considered PBL methodology to be appropriate for different curricular segments at their institution. After training, a larger proportion considered PBL an appropriate educational method for some, but not all, segments of the curriculum.

REFERENCES

• ¹
  Komatsu RS, Zanolli MB, Lima VV. Aprendizagem baseada em problemas. In: Marcondes E, Gonçalves EL , editors. Educação médica. São Paulo: Sarvier, 1998: 223-37.
• ²
  Rodrigues MLV, Figueiredo JFC. Aprendizado centrado em problemas. Medicina, Ribeirão Preto 1996; 29:396-402.
• ³
  Moriearty PL. Problem-based learning at the Ribeirão Preto Faculty of Medicine? Weighingthe pros and cons. Medicina, Ribeirão Preto 1996;29: 403-6.
• ⁴
  Bland CJ. Faculty development through workshops. Springfield, IL: CC Thomas, 1980.
• ⁵
  Steinert Y. Twelve tips for conducting effective workshops. Med Teacher 1992; 14:127-31.
• ⁶
  Batista NA. Formação do professor de medicina: desafios e perspectivas. In: Marcondes E, Gonçalves EL, editors. Educação médica. São Paulo: Sarvier , 1998: 223-37.
• ⁷
  Machado MH. Os médicos no Brasil. Um retrato da realidade. Rio de Janeiro: Editora Fiocruz, 1997.
• ⁸
  Hitchcock MA, Stritter Ff, Bland CJ. Faculty development in the health professions: conclusions and recommendations. Med Teacher 1993; 14:295-309.
• ⁹
  Silberman M, Auerbach C. Active training. A handbook of techniques, designs, case examples, and tips. San Francisco: Jossey Bass, lnc., 1990
• ¹⁰
  Kraiger K, Ford JK, Salas E. Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. J Appl Psychol 1993;78:311-28.
• ¹¹
  Rogers EM. Diffusion of innovations. 4th Ed. New York: Free Press, 1995.
• ¹²
  Venturelli J. Educación médica: nuevos enfoques, metas y métodos. Washington DC: Organización Panamericana de la Salud, 1997.
• ¹³
  Campbell DT, Stanley JC. Experimental and quasi-experimental designs for research. Chicago: Rand McNalley College PublishingCompany, 1963.
• ¹⁴
  Benor DE, Mahler S. Training medical teachers: rationale and outcomes. ln: Schmidt HG, Lipkin M Jr, de Vries MW, Greep JM, Eds. New directions for medical education. Problem-based learning and community-oriented medical education. New York: Springer Verlag, 1989, 248-59.
• ¹⁵
  Wilkerson LA, Irby D. Strategies for effecting change in teaching practices: a review of current models. In: Scherpbier AJJA, van deer Vleuten CPM, Rethans JJ, van der Steeg AFW, Eds. Advances in medical education. Boston: Kluwer Academic Publ, 1997, 23-9.
• ¹⁶
  Gist ME, Mitchell TR. Self-efficacy: a theoretical analysis of its determinants and malleability. Academy Manag Rev 1992;17:183-211.
• ¹⁷
  Vella J. Learning to listen, learning to teach. The power of dialogue in educating adults. San Francisco: Jossey-Bass Publishers, 1994.

Publication Dates

• Publication in this collection
  23 Sept 2020
• Date of issue
  May-Dec 1999