Empowerment evaluation at the Stanford University School of Medicine: using a critical friend to improve the clerkship experience

Evaluación de Concesión Legal de Poder de la Escuela de medicina de la Universidad de Stanford: usando un amigo crítico para mejorar la experiencia en la residencia

Avaliação da concessão legal de poder da escola de medicina da Universidade de Stanford: usando um amigo crítico para melhorar a experiência na residência

Empowerment evaluation at the Stanford University School of Medicine: using a critical friend to improve the clerkship experience

Avaliação da concessão legal de poder da escola de medicina da Universidade de Stanford: usando um amigo crítico para melhorar a experiência na residência

Evaluación de Concesión Legal de Poder de la Escuela de medicina de la Universidad de Stanford: usando un amigo crítico para mejorar la experiencia en la residencia

David Fetterman

Formerly, Stanford University; Professor, University of Arkansas at Pine Bluff and Director, Arkansas Evaluation Center. E-mail: [e-mail]

ABSTRACT

Empowerment evaluation was adopted by Stanford University's School of Medicine to engage in curricular reform. It was also used to prepare for an accreditation site visit. Empowerment evaluation is a guided form of self-evaluation. It was selected because the principles and practices of empowerment evaluation resonated with the collaborative and participatory nature of the curricular reform in the School. This article highlights one of the most important features of an empowerment evaluation: a critical friend. This individual has evaluation expertise but serves as a coach, advisor, or guide, rather than "the expert". The evaluation is in the hands of the people in the program, but a critical friend helps to keep it on track and rigorous. As one indication of the importance of this role, student ratings on selected clerkship rotations, where a critical friend was requested to respond to student concerns, resulted in significant increases in student ratings.

Keywords: Empowerment. Evaluation. Empowerment evaluation. "Critical friend". Student ratings.

RESUMO

A avaliação de concessão legal de poder foi adotada pela Escola de Medicina da Universidade de Stanford para participar da reforma curricular. Também foi usada para preparar um credenciamento em um site de visita. A avaliação de concessão legal de poder é uma forma guiada de autoavaliação. Ela foi selecionada, pois os princípios e práticas de avaliação legal de poder correspondem harmoniosamente à natureza colaborativa e participativa da reforma curricular na escola. Este artigo destaca uma das mais importantes características da avaliação de concessão legal de poder: o amigo crítico. Este indivíduo possui experiência em avaliação, porém serve como instrutor, conselheiro e guia em vez de "o perito". A avaliação está nas mãos das pessoas do programa, porém o amigo crítico ajuda a manter o bom caminho e a rigorosidade. Como um indicador da importância desse papel, as avaliações dos alunos nas escalas de residência selecionadas, onde um amigo crítico foi solicitado para responder às preocupações dos alunos, resultou em um aumento significante na classificação dos alunos.

Palavras-chave: Concessão legal de poder. Avaliação. Avaliação de concessão legal de poder. "Amigo crítico." Classificação dos alunos.

RESUMEN

La evaluación de concesión legal de poder fue adoptada por la Escuela de Medicina de la Universidad de Stanford para participar de la reforma curricular. También fue usada para preparar un credenciamento en un site de visita. La evaluación de concesión legal de poder es una forma guiada de autoevaluación. Los principios y prácticas de evaluación legal de poder corresponden harmoniosamente a la naturaleza colaboradora y participativa de la reforma curricular en la escuela. Este artigo destaca una de las más importantes características de la evaluación de concesión legal de poder: el amigo crítico. Este individuo tiene experiencia en evaluación, pero sirve como instructor, consejero y guía en lugar del "perito". La evaluación esta en las manos de las personas del programa, pero el amigo crítico ayuda a mantener el bueno camino y la precisión. Como un indicador de la importancia de ese papel, las evaluaciones de los alumnos en las escalas de residencia seleccionadas, donde un amigo crítico fue solicitado a responder a las preocupaciones de los alumnos, resultó en aumento significante en la clasificación de los alumnos.

Palabras clave: Concesión legal de poder. Evaluación. Evaluación de concesión legal de poder. Amigo crítico. Clasificación de los alumnos.

Purpose

Empowerment evaluation is used throughout the world, including Australia, Brazil, Canada, Finland, Israel, Japan, Mexico, Nepal, New Zealand, South Africa, Spain, United Kingdom, and the United States. It is being applied in townships, public schools, foundations, Native American reservations, government settings, and private industry. Empowerment evaluations are being conducted with HIV prevention programs, teenage pregnancy prevention programs, battered woman's shelters, digital villages, and even the Mars Rover (FETTERMAN; BOWMAN, 2002). The principles have been elaborated in Empowerment Evaluation Principles in Practice (FETTERMAN; WANDERSMAN, 2005). The steps have been detailed in the Foundations of Empowerment Evaluation (FETTERMAN, 2001). However, insufficient attention has been paid to the role of the critical friend. This discussion highlights this role within the context of accountability or outcomes. This discussion specifically focuses on the power of empowerment evaluation and the critical friend role to help transform the quality of medical education.

Background

Medical education throughout the world is changing. Most schools are based on a Flexnerian model (COOKE et al., 2006) in which the first two years are focused on basic sciences and the second involve intensive clinical experience. Students have grown inpatient with this approach (divorcing knowledge from practice) and many faculty agree that more years of clinical experience are necessary to improve the quality of medical education.

Stanford University's School of Medicine was part of this wave of change. They implemented a new curriculum that introduced more clinical activity into the curriculum much earlier. The School also developed a more integrated curricular experience throughout medical school. The impetus for this sea change was in part a result of student and faculty dissatisfaction. It was also in response to accrediting agency concerns about how well the School was preparing future physicians. There was a consensus that something substantive had to be done to revamp the curriculum.

One of the most important tools in any curricular redesign is feedback, specifically evaluation feedback. The only way to determine if changes are working as desired is to collect, analyze, report, and act on data. In this case, student performance and satisfaction data was needed, (as well as faculty views and assessments of the curricular innovations). One approach that is consonant with a more participatory and integrated curricular approach is empowerment evaluation.

Empowerment evaluation is a form of guided self-evaluation. It requires student, faculty, and administrator participation in curricular development and refinement, facilitated by an empowerment evaluator.

The School recruited the founder of the approach, Dr. David Fetterman, to create a Division of Evaluation and serve as the founding director of the division. The Division played a leadership role (as a facilitation agent) in the School's accreditation self-study. The accreditation study was one of the driving forces in the School's curricular redesign and continual refinement or improvement.

Empowerment evaluation

"Empowerment evaluation is the use of evaluation concepts, techniques, and findings to foster improvement and self-determination" (FETTERMAN; KAFTARIAN; WANDERSMAN, 1996). It is an approach that "aims to increase the likelihood that programs will achieve results by increasing the capacity of program stakeholders to plan, implement, and evaluate their own programs" (FETTERMAN; WANDERSMAN, 2005, p. 27). There are many tools to implement an empowerment evaluation¹.

For example, Fetterman (2001) typically uses a 3-step model: 1) mission; 2) taking stock; and 3) planning for the future. Faculty, students, and administrators collectively help develop a mission (or sub-mission) associated with the medical school curriculum. Then they are asked to prioritize (using dots) to vote for the most important things to evaluate as a group. Once the abbreviate list is established (typically by selecting the activities which received the highest number of dots or votes), the group rates how well the school is doing in each area using a 1 (low) to 10 (high) rating scale. This helps the group launch into a discussion or dialogue about the ratings. Bohm (1996) refers to this as dialogue. It often unearths basic concerns and insights into the school that otherwise remain buried and unexamined, facilitating fundamental change.

The students, faculty, and administrators build on the "taking stock" exercise to create a plan for the future or intervention. School of Medicine students, faculty, and administrators created new goals, related to the activities they just rated and discussed, and new strategies to accomplish those goals. Collectively, as guided by the empowerment evaluator, the group comes up with credible evidence to determine if the new strategies are working.

Conventional (and novel) evaluation tools are used to determine if the new strategies are working and that information is fed directly back to the group to help them make mid-course corrections as needed. Periodically the group takes stock of where they are again and compares the second data point with their initial baseline ratings. This provides a vivid illustration (often using a bar graph) of any curricular change over time.

In addition to these steps, there is a close, nurturing relationship that is established between the group and their empowerment evaluation. That is the focus of this discussion.

Coaching: the critical friend

Steps are useful and necessary. However, empowerment evaluation is about relationships. The role of the critical friend merits attention because it is a like a fulcrum in terms of fundamental relationships. Applied improperly it can be like a wedge inhibiting movement and change, applied correctly, this role can be used to leverage and maximize the potential of a group. The empowerment evaluator differs from many traditional evaluators. Instead of being the "expert" and completely independent, separate, and detached from the people they work with, so as not to get "contaminated" or "biased", the empowerment evaluator works closely with and along side program staff members and participants. Empowerment evaluators are not in charge. The people they work with are in charge of the direction and execution of the evaluation. Empowerment evaluators are critical friends or coaches. They believe in the merits of a particular type of program but they pose the difficult questions (in a diplomatic fashion). Some people ask how can an empowerment evaluator be objective and critical if they are friends and in favor of a type of program? The answer is simple: an empowerment evaluator is critical and objective because they want the program to work (or work better).

Empowerment evaluators are trained evaluators with considerable expertise. They provide it as needed to keep the evaluation systematic, rigorous, and on track. They are able to function in this capacity by advising, rather than directing or controlling an evaluation. They provide a structure or set of steps to conduct an evaluation. They recommend, rather than require, specific activities and tools. They listen and rely on the group's knowledge and understanding of their local situation. The critical friend is much like a financial advisor or health trainer. Instead of judging and making pronouncements about success or failure, compliance or non-compliance, the empowerment evaluator serves the group or community in an attempt to help them maximize their potential and exponentially unleash their creative and productive energy for a common good. Important attributes of a critical friend include: creating an environment conducive to dialogue and discussion; providing or requesting data to inform decision making; facilitating rather than leading; and being open to ideas, inclusive, and willing to learn.

A case example of how a critical friend is used in empowerment evaluation is presented in the next section of this article. It highlights an important principle of empowerment evaluation in the process: accountability and outcomes.

Stanford University School of Medicine Clerkships

The first part of medical education at Stanford focuses on basic sciences. A clinical dimension has been woven into it since the curricular reform was implemented. However, the second half of the curriculum remains similar to its original configuration. It consists largely of medical students rotating from one specialty to the next through the hospital. For example, a student might spend one month in pediatrics and another in neurology. Faculty and residents evaluate the students during each rotation. However, the students also evaluate each clerkship as they complete it. Two clerkships in particular received very low marks by students. The director of each of these clerkships requested special assistance from the Division of Evaluation to respond to these critiques and low ratings. A critical friend approach was adopted since it is an instrumental part of the overall empowerment evaluation approach in the School.

Critical friend activities

The Director of Evaluation, served as the critical friend. He convened a meeting of the faculty representing each clerkship. He presented the student ratings and comments. In addition, he presented faculty views (and self-assessments). He facilitated a discussion about the evaluation data and the groups identified many of the reasons for the low ratings. By the end of the discussion the groups were able to highlight themes and common denominators between them. Both clerkships identified three areas that merited immediate attention: 1) orientation; 2) mid-clerkship feedback; and 3) timely feedback at the end of the clerkship rotation.

Each group generated ideas or strategies to improve their performance. For example, one group decided to put all the orientation information online. They also assembled practical information, such as a detailed schedule and contact information, that could all be downloaded into a medical student's personal digital assistant (PDA). Another activity involved working collaboratively to evaluate each student. One group met as a faculty, sitting in a semi-circle, solicited comments from each faculty or resident, about each medical student. The reviewed each student on the rotation as a group - one at a time. They discussed the cumulative wisdom about the students and then provided students with mid-clerkship feedback on their performance, engaging the student in a dialogue about their work.

Some of the activities or strategies worked and some did not work as well. More important than any individual strategy was the process. The critical friend helped the groups monitor and assess the newly implemented strategy or intervention. They helped the groups conduct online surveys, focus groups, interviews, and observations to inform clerkship director and faculty decision making. If the new strategy was not viewed as timely or effective, another idea was generated and implemented. This process allowed for mid-course corrections, often before the end of the rotation.

This process proved to be both popular and successful. Students had data to document that the faculty were listening to them. Faculty had the data they needed to pinpoint students concerns and make immediate changes, including unintended redundancies in the curriculum.

The two clerkships that requested a critical friend made the most significant gains in student ratings. In one case, ratings increased by 26% and the second by 16% in less than one year (see Clerkship A and B in Figure 1). These clerkships had previously been decreasing each year.

Conclusion

Empowerment evaluation is considered a part of the intellectual landscape of evaluation (FETTERMAN, 2001; PATTON, 2005). It has matured and developed over the last decade. Contributors to the approach have enhanced conceptual clarity, provided greater methodological specificity, and highlighted empowerment evaluation's commitment to accountability and producing outcomes (FETTERMAN; WANDERSMAN, 2007). This brief discussion highlights the role of the critical friend in facilitating an empowerment evaluation. The role is not as an expert or director. Instead, a critical friend is a coach and an advisor, enhancing the probability of success and enabling people to accomplish their goals and objectives. The critical friend helps people make more informed decisions data driven decisions with meaningful outcomes. They help cultivate or foster an environment conducive to people empowering themselves.

Recebido em: 03/06/2009

Aceito para publicação em: 29/06/2009

• BOHM, D. On dialogue. London: Routledge, 1996.
• COOKE, M. et al. American medical education 100 years after the Flexner report. The New England Journal of Medicine, Boston, MA, v. 355, n. 13, p. 1339-1344, 2006.
• FETTERMAN, D. M. Foundations of empowerment evaluation. Thousand Oaks, CA: Sage, 2001.
• FETTERMAN, D. M.; BOWMAN, C. Experiential education and empowerment evaluation: Mars Rover educational program case example. Journal of Experiential Education, St. Catherines, Ontario, v. 25, n. 2, p. 286-295, 2002.
• FETTERMAN, D. M.; KAFTARIAN, S.; WANDERSMAN, A. Empowerment evaluation: knowledge and tools for self-assessment and accountability. Thousand Oaks, CA: Sage, 1996.
• FETTERMAN, D. M.; WANDERSMAN, A. Empowerment evaluation: yesterday, today, and tomorrow. American Journal of Evaluation, Thousand Oaks, CA, v. 28, n. 2, p. 179-198, 2007.
• FETTERMAN, D. M.; WANDERSMAN, A. Empowerment evaluation principles in practice. Thousand Oaks, CA: Sage, 2005.
• PATTON, M. Toward distinguishing empowerment evaluation and placing it in a larger context: take two. American Journal of Evaluation, Thousand Oaks, CA, n. 26, p. 408-414, 2005.

1

There are other equally effective tools that can be used in empowerment evaluation, including Wandersman's 10 step Getting to Outcomes model.

Publication Dates

• Publication in this collection
  16 Nov 2009
• Date of issue
  June 2009

History

• Received
  03 June 2009
• Accepted
  29 June 2009