Applicability of open book examination as a teaching strategy in medical schools

Abstract:

Introduction:

In the last decades, pedagogical practices and teaching methodologies have experienced radical changes, culminating today with the exponential growth of active teaching methodologies. Despite the growing use of these active methodologies in medical courses, the theoretical evaluation models changed little, if at all, in the last decades, which should not occur, as the evaluation systems need to evolve concomitantly to ensure a cohesive, quality teaching-learning process.

Objective:

To demonstrate how open-book examination is inserted in medical courses.

Methods:

An integrative review was developed by searching for publications indexed in the last 10 years in the databases of the U.S. National Library of Medicine (PubMed), “Scientific Electronic Library Online” (SciELO), Science Direct, and the “Education Resources Information Center” (ERIC), as full articles written in Portuguese or English. The following descriptors were used in combination, respectively in English and Portuguese: “medicine”, “assessment”, “open book examination”, “open book exam”, “open book test” and “open book assessment”; “medicina”, “avaliação”, “prova com consulta”, “avaliação com consulta” and “teste com consulta”.

Results:

For this review, a total of eleven publications that met the eligibility criteria were selected, and after their theoretical immersion and analysis, two main categories emerged: “Open-book examination as an educational resource for medical education” and “Open-book examination as a pedagogical alternative in the pandemic scenario”, which were analyzed in the light of an integrative review.

Conclusion:

Although there are still doubts and further studies are required, it is evident that Open-Book examination is very pertinent to medical education, as it contributes to the training of professionals with high capacity for questioning and debating, who are prepared for permanent study and education routines, and who understand that there is constant evolution of knowledge in the medical field.

Keywords:
Undergraduate Medical Education; Medical Education; Higher Education; Teaching; Educational Measurement

Resumo:

Introdução:

Nas últimas décadas, as práticas pedagógicas e as metodologias de ensino têm experimentado mudanças radicais, culminando hoje com o crescimento exponencial de metodologias de ensino ativas. Apesar da crescente utilização dessas metodologias ativas nos cursos de Medicina, as formas de avaliação teórica foram pouco, ou nada, alteradas nas últimas décadas, o que não deveria ocorrer, pois os sistemas de avaliação precisam evoluir concomitantemente para garantir um processo de ensino-aprendizagem coeso e de qualidade.

Objetivo:

Este estudo evidencia como está inserida a avaliação com consulta em cursos de Medicina.

Método:

Foi desenvolvida uma revisão integrativa por meio de consulta de publicações indexadas no período dos últimos dez anos nas bases de dados da Biblioteca Nacional de Medicina dos Estados Unidos (PubMed), Scientific Electronic Library Online (SciELO), ScienceDirect e Education Resources Information Center (ERIC), em formato de artigos completos nas línguas portuguesa ou inglesa. Utilizaram-se os descritores a seguir de forma combinada, respectivamente em inglês e português: “medicine”, “assessment”, “open book examination”, “open book exam”, “open book test” e “open book assessment”; “medicina”, “avaliação”, “prova com consulta”, “avaliação com consulta” e “teste com consulta”.

Resultado:

Para a composição deste estudo, selecionaram-se 11 publicações que atenderam aos critérios de elegibilidade traçados, e, após a imersão teórica e análise delas, emergiram duas categorias principais: “A avaliação com consulta como recurso educacional para o ensino médico” e “A avaliação com consulta como alternativa pedagógica no cenário pandêmico”, as quais foram analisados à luz de uma revisão integrativa.

Conclusão:

Ainda que haja dúvidas e sejam necessários novos estudos, fica evidente que a avaliação com consulta é muito pertinente ao ensino médico, uma vez que contribui para a formação de profissionais com alta capacidade de questionamento e argumentação, preparados para rotinas de estudo e educação permanentes e que entendam que há constante evolução do conhecimento na área médica.

Palavras-chave:
Educação de graduação em Medicina; Educação médica; Ensino superior; Ensino; Avaliação educacional

INTRODUCTION

The teaching-learning process is based on the acquisition of pre-established skills by students. Education and practice in health professions require multiple cognitive, psychomotor, attitudinal and relational skills¹, which must be developed throughout the course. The assessment of learning plays a crucial role in this process, as it allows assessing the students’ progress throughout the educational efforts, aiming at understanding and outlining the necessary actions to obtain better results. For this reason, it is a frequent topic of discussion and questioning, since the evaluation is an ethical responsibility of the training institution, by guaranteeing to society that each new graduate has been certified in the essential competences required for a quality professional performance².

Specifically in the assessment of medical students, one should not only be concerned with assessing the retention of factual knowledge, but also bear in mind the importance of assessing more complex skills, related to clinical reasoning, the application of acquired knowledge in the solution of concrete problems³, to the development of communication skills, leadership and teamwork. As a result, teaching and assessment in medical courses require greater complexity and have started to demand diversified and innovative strategies². Scholars have demonstrated that by using several assessment methods, we can provide greater coverage of learning outcomes, as it is possible to measure different components of the “universe” of characteristics and competences to be acquired¹. These needs are in line with the National Curriculum Guidelines (DCN, Diretrizes Curriculares Nacionais) for undergraduate medical courses of 2014⁴, which mention in Art. 31 that “student assessments will be based on knowledge, skills, attitudes and the developed curricular contents”. Nevertheless, teacher training deals little with assessment⁵ and even less with complex and quality evaluations, capable of in-depth analysis of the acquired knowledge. Although advances in teaching-learning methodologies are evident in all medical schools in recent decades, the evaluation systems have not kept up with this evolution, which should not be perpetuated, since no pedagogical innovation can ignore the evaluation system⁶.

Evaluations in medical courses tend to be momentary, little comprehensive and predominantly centered on memorizing factual knowledge, disregarding the incorporation of skills and other more complex competences. It is known that pass-and-fail decisions should not be made based on isolated evaluations, single methodologies and individual data points; however, this is often seen in traditional regulations⁷.

Moreover, there is rarely any concern that the conditions of using different evaluation methods can guarantee validity and reliability⁸. The use of evaluations in which the use of external sources (such as notebooks, articles, notes, books and even the internet) is allowed seems to be more authentic and closer to professional practice, with the evaluation success not linked solely to memorization. Students in settings that allow open-book exams⁹ apply critical thinking, prepare detailed responses, work creatively, produce their own thoughts, and feel more confident about the exam results than when taking a closed-book exam.

In recent years, we have been experiencing an atypical situation caused by the Covid-19 pandemic, with severe impacts on different levels of education. Specifically, the pandemic created unprecedented challenges for evaluation in medical courses¹⁰, showing that it is necessary to know how to evaluate medical students with quality and reliability. The experience arising from this period must persist and be scientifically based to result in an increasingly better use, with more safety and knowledge.

Therefore, the present study aimed to show how open-book evaluations are included in medical courses.

METHOD

An integrative review was developed, which aimed at answering the guiding question:

• How is open-book evaluation included in medical courses?

The choice for this methodological design was due to the fact that the integrative review has tools that value its applicability in the health area, since it synthesizes studies in their specificities and denotes the scientific nature of professional practice¹¹.

Searches in the databases took place in May 2021 and indexed publications were searched in the databases of the US National Library of Medicine (PubMed), Scientific Electronic Library Online (SciELO), Science Direct and Education Resources Information Center (ERIC). For searches carried out in English, the following descriptors were used in combination: “medicine”, “assessment”, “open book examination”, “open book exam”, “open book test” and “open book assessment” . For the searches carried out in Portuguese, the following descriptors were used, also in combination: “medicina”, “avaliação”, “prova com consulta”, “avaliação com consulta” e “teste com consulta”. Searches in Portuguese used the Boolean operators (medicina) AND (avaliação) AND (“prova com consulta” OR “avaliação com consulta” OR “teste com consulta”). Searches in English used the Boolean operators (medicine) AND (assessment) AND (“open book examination” OR “open book exam” OR “open book test” OR “open book assessment”).

The adopted criteria included studies published in the last ten years, in Portuguese or English, written as full articles, related to the topic and non-redundant. The articles that did not meet the criteria established above were excluded. The choice of publications in the aforementioned period was intended to gather a number of relevant publications, enough to generate knowledge and discussion on the subject and capable of answering the research question.

To meet the outlined criteria, after completing the searches, a careful analysis of the full articles ensued. Initially, we had a total of 6286 results, of which 3,712 were eliminated by the criterion of year of publication. After this phase, another 360 articles that did not fit the publication type criterion were excluded. In addition to these, 2,194 publications that were not related to the proposed topic and two others that were repeated in the searched databases were also excluded. After this phase, 18 articles were selected, which were read in full to confirm the eligibility. At that time, seven studies were excluded due to incompatibility with the topic. Finally, 11 publications remained and were included in this review. The processes of analysis and selection of the publications are shown in Table 1.

Table 1
Analysis of publications retrieved for review.

Results of searches in databases and evolution of the analysis process
	PUBMED	SCIELO	SCIENCEDIRECT	ERIC
Search Results	213	2.602	2.243	1.228
Number of articles excluded by exclusion factor:
Publication period (last ten years):	28	1.232	1.538	914
Publication type (full articles)	0	0	360	0
Relationship with the topic	173	1.366	342	313
Redundancy	0	0	2	0
Total publications excluded	201	2.598	2.242	1.227
Total publications included	12	4	1	1
Number of articles excluded after read in full
Exclusion due to lack of relationship with the topic	3	4	0	0
Total of analyzed publications	9	0	1	1

• Source: Elaborated by the authors.

RESULTS

This review found 11 publications that met the proposed eligibility criteria. The methodological path of the search and analysis process is shown in Table 2.

Table 2
Methodological path for article selection

Total publications retrieved: 6286	Total publications excluded: 6275	Total publications included: 11
PubMed: 213	PubMed: 204	PubMed: 9
SciELO: 2.602	SciELO: 2.602	SciELO: 0
ScienceDirect: 2.243	ScienceDirect: 2.242	ScienceDirect: 1
ERIC: 1.228	ERIC: 1.227	ERIC: 1

• Source: Elaborated by the authors.

Frame 1 lists all the publications included in this review. When evaluating it, it can be observed that 45.4% of the publications (n=5) address the topic of open-book exams in the context of the Covid-19 pandemic.

Frame 1
Descriptive list of the publications included in the review.

Nº	Título	Periódico	Autores e ano	País
1	“Because life is open book: an open internet family medicine clerkship exam”	PRiMER	Erlich (2017)	USA
2	“Assessing open-book examination in medical education: the time is now”	Medical Teacher	Zagury-Orly et al. (2021)	USA
3	“Adaptation to open-book online examination during the Covid-19 pandemic”	Journal of Surgical Education	Eurboonyanun et al. (2021)	Thailand
4	“Comparing open-book and closed-book examinations: a systematic review”	Academic Medicine	Durning et al. (2016)	USA
5	“Online open-book examination of undergraduate medical students: a pilot study of a novel assessment method used during the coronavirus disease 2019 pandemic”	The Journal of Laryngology and Otology	Sarkar et al. (2021)	India
6	“Medical teaching and assessment in the era of Covid-19”	Journal of Medical Education and Curricular Development	Monaghan (2020)	UK
7	“Open book exams: a potential solution to the ‘full curriculum’?”	Medical Teacher	Teodorczuk et al. (2018)	Australia
8	“Perceptions of clinical years’ medical students and interns towards assessment methods used in King Abdulaziz University, Jeddah”	Pakistan Journal of Medical Sciences	Ibrahim et al. (2015)	Pakistan
9	“Could the future of medical school examinations be open-book: a medical student’s perspective?”	Medical Education Online	Mathieso et al. (2020)	UK
10	“Remote e-exams during Covid-19 pandemic: a cross-sectional study of students’ preferences and academic dishonesty in faculties of medical sciences”	Annals of Medicine and Surgery	Elsalem et al. (2021)	Jordan
11	“A comparison of open-book and closed-book formats for medical certification exams: a controlled study”	American Educational Research Association	Brossman et al. (2017)	USA

• Source: Elaborated by the authors.

Regarding the country of origin, it is clear that there are no Brazilian publications. The distribution of countries of origin is detailed in Table 3 and shows a predominance of publications from countries in the northern hemisphere, with no representative from Latin America. As for the year of publication, Table 3 shows that all publications are recent, demonstrating the growing interest in the topic in recent years, with 54.5% of the studies published between 2020 and 2021.

Tabela 3
Análise quantitativa por país de origem, ano e periódico de publicação

Country of Origin	N	%
USA	4	36,4
UK	2	18,2
Australia	1	9,1
India	1	9,1
Jordan	1	9,1
Pakistan	1	9,1
Thailand	1	9,1
Year of publication	N	%
2015	1	9,1
2016	1	9,1
2017	2	18,18
2018	1	9,1
2020	3	27,27
2021	3	27,27
Journal	N	%
Medical Teacher	2	18,18
Academic Medicine	1	9,09
American Educational Research Association	1	9,09
Annals of Medicine and Surgery	1	9,09
Journal of Medical Education and Curricular Development	1	9,09
Journal of Surgical Education	1	9,09
Medical Education Online	1	9,09
Pakistan Journal of Medical Sciences	1	9,09
PRiMER	1	9,09
The Journal of Laryngology and Otology	1	9,09

• Fonte: Elaborada pelos autores.

Finally, we performed an analysis of each publication per journal, with the Medical Teacher journal showing the highest number of publications, as shown in Table 3.

DISCUSSION

When analyzing the publications in this review, predominant characteristics arose, and the perception of two categories emerged, which didactically standardized the central evidence identified in the studies: “Open-book examination as an educational resource for medical education” and “Open-book examination as a pedagogical alternative in the pandemic scenario”. To facilitate the analysis of the studies, the discussion will be divided into the two aforementioned categories.

Open-book examination as an educational resource for medical education

The evaluation, as previously demonstrated, is a crucial step in any educational effort, which cannot be disregarded by any of its actors. Thus, in this topic we will discuss the studies that analyzed the inclusion of open-book exams as an educational resource for medical education. A first important point is to note that Erlich¹² and Durning et al.¹³ carried out comparisons between open- and closed-book evaluations, aiming to find out if there is any difference in student performance between these modalities, but they found similar results in both. Additionally, Durning et al.¹³ also demonstrated that the students’ performance in open-book evaluations could be improved after carrying out practical preparatory tests and instructions on this evaluation modality, since there is less experience with this type of evaluation. Equally important, Erlich¹² defends that most students who perform below average in open-book exams also have low scores in clinical evaluations by preceptors, specifically in the information domain area. This demonstrates that there is no superiority of one type of evaluation over the other, and both can be used, depending on the objective of the evaluation. Therefore, both evaluation tools are able to identify students with low performance.

Regarding the influence of the evaluation on how students prepare and study, there is a disagreement among scholars, since Durning et al. ⁽¹³ showed that students do not change their study tactics when open-book exams are allowed. On the other hand, Sarkar et al.¹⁴ evaluated student feedback after carrying out an evaluation that allowed online consulting and reported that students spent more time understanding the topic instead of just memorizing it, in addition to realizing that they would not be able to write answers to the evaluation questions if the topics had not been previously read and studied. This may demonstrate that open-book exams do not affect the way students study and prepare or, even better, it may show that students carry out a more in-depth study, worrying less about memorizing concepts and more about higher functions, such as correlation, debate and synthesis. This viewpoint is shared by Erlich¹², who emphasizes that, in an era of evolution of internet-based knowledge, physicians and medical students must be competent to quickly access, synthesize and apply information that is always updated for decision-making.

The studies that evaluated the time taken to undertake the evaluations and solve the questions are consistent in showing an increase in the time required for open-book exams. In a study carried out by Durning et al. ⁽¹³, students took 10% to 60% longer to complete open-book exams, when compared to similar closed-book evaluations. In agreement with that, Brossman et al.¹⁵ reported the need for 40% longer to resolve open-book exams. This finding has direct implications for the operationalization of open-book exams, since the additional resolution time must be taken into account, so it does not become a factor that negatively influences the evaluation result¹⁵.

Other scholars assessed the degree of anxiety related to the evaluation. Sarkar et al.¹⁴ analyzed the students’ feedback and demonstrated a lower level of stress during the open-book exam, in agreement with Mathieso et al.¹⁶. As for Durning et al.¹³, they observed that the students associated open-book exams with a lower degree of anxiety, but only a minority of them actually reported less anxiety when they actually carried out this evaluation modality. Thus, there does not seem to be a consensus on the fact that students actually show a reduction in anxiety with open-book exams. However, no study has shown an increase in the level of stress or anxiety related to open-book exams.

Regarding the depth of the topics addressed in the open-book and closed-book exams, we observed that written clinical examinations are adequate for open-book exams, since the questions require a distinct synthesis of a great dela of information from the clinical scenarios provided to students and, therefore, the answer cannot be simply searched on the internet¹³.

The discriminating power of a test or evaluation is the capacity to differentiate students who have the required proficiency from those who do not. To perform the analysis of the discrimination power, Brossman et al.¹⁵ used the Item Response Theory (IRT) methodology, which considers three characteristics of the questions (or items): the capacity to assess whether students have the necessary knowledge to answer them, the degree of difficulty and the possibility of the correct answer happening at random. Using this methodology, they demonstrated that the open-book exam has greater discrimination capacity than similar evaluations using the closed-book method. This means that the questions in an open-book exam are better able to differentiate the best performing candidates from the worst ones, apparently related to the depth and complexity of the questions in an open-book exam, which tend to require superior mental functions. At this point, it is worth questioning whether the better discriminating results are due to consulting external sources of information in itself or whether this is caused by the creation of questions that demand greater clinical reasoning.

It can also observed that one of the advantages of open-book exams is that they not only prevent students from temporarily memorizing superfluous information for repetition during evaluations, but, crucially, they closely mirror the actual clinical practice, where such information is easily acquired from the available resources, such as digital libraries of evidence-based medicine^13),(17. It is evident that using this type of evaluation also has the advantage of putting students in contact with situations closer to the professional practice they will have to face in the future. As a gap, Eurboonyanun et al.¹⁸ point out the need for further studies to assess the effects of open-book exams taken online on knowledge retention and application in the long term.

Open-book examination as a pedagogical alternative in the pandemic scenario

Since the beginning of 2020, major challenges have occurred due to the Covid-19 pandemic. There have been global impacts in all sectors and activities, including education, since social distancing was necessary to contain viral circulation, resulting in the sudden interruption of in-person teaching activities. To avoid major losses, adaptations were made to teaching methodologies and evaluation formats¹⁹, since this period of remote learning was longer than what was initially expected. Next, data from the discussion of the studies that addressed student evaluation in this specific scenario will be presented.

In a study carried out by Elsalem et al.²⁰, the results showed that only one-third of the students preferred open-book exams taken online as the evaluation method during the Covid-19 pandemic. The authors associate the students’ low acceptance in relation to the open-book exam to the need for more time to prepare for it, difficulties in the previous preparation and inadequacy of the questions in relation to the study material provided to them. These findings are valuable to allow the design of academic strategies that can help overcome the difficulties with remote open-book exams. This may include improving distance learning methodologies, reorganizing evaluation strategies and revising academic curricula to suit the actual situation.

This period, despite the difficulties, constituted an excellent opportunity for medical educators to carefully explore the use of open-book exams in the online environment¹⁹, since this modality has the capacity to assess the students’ ability to efficiently search and translate information, which is a necessary skill for future clinical practice. The same authors also suggest mixed evaluation strategies, including a first part using the closed-book approach, evaluating the learning of concepts they should know without consulting external sources, followed by a second, open-book exam, about topics the students are expected to research and demonstrate the capacity for synthesis, correlation, debate and clinical reasoning.

In this sense, Eurboonyanun et al.¹⁸ carried out the evaluation at the end of surgical training in an online environment with open-book examinations and correlated these data with the results of the predecessor groups, in the same in-person closed-book exam, using the same question database. Medical students who took the online open-book exam had a higher average score on the multiple-choice and dissertation questions, but a lower average score on the short-answer questions. This result is important, as it shows the need to define cut-off points and comparable minimum passing scores for open- and closed-book exams, procedures that should be repeated in other institutions that wish to change their evaluation methodologies. Similarly, Sarkar et al.¹⁴ carried out a study evaluating an online open-book exam in an Otorhinolaryngology discipline, also due to adaptations to remote teaching during the pandemic. The authors compared these results with those of previous traditional in-person closed-book exams and demonstrated similar passing rates in both methodologies.

Additionally, scholars questioned the occurrence of academic dishonesty or “cheating” in remote open-book exams. Elsalem et al.²⁰ investigated the occurrence of misconduct and dishonesty during these exams and demonstrated that 55.07% of the students reported no dishonesty or misconduct, while 20.41% reported seeking help from friends and 24.52% used other unauthorized sources of information. Furthermore, we can point out that Sarkar et al.¹⁴ found similar results, with 72.2% of students stating they had not consulted classmates, answering the questions independently. Monaghan¹⁷ defends that, to inhibit “cheating” or complicity, randomization strategies related to the order of questions should be used for each student, making communication between them ineffective. This demonstrates that the occurrence of dishonesty and “cheating” during remote open-book exams does not seem to be so frequent and there are ways to inhibit its occurrence. However, the current literature lacks studies that compare the frequency with which these cheating actions occur between in-person or remote evaluation strategies, as well as comparisons between open- and closed-book exams, aiming to know whether the permission to have open-book examinations inhibit or reduce the occurrence of “cheating”.

Changes in the teaching and assessment methods during the Covid-19 pandemic provided the opportunity to implement, test and better understand open-book exams, either remotely or not, in medical courses^14),(17),(19. Most of these educational changes have occurred on an urgent basis, but many are likely to remain, in a more refined form, as preferred methods of teaching and evaluation in the future¹⁷. Also, Zagury-Orly and Durning¹⁹ point out that the opportunity that arose in this period should be seized to advance our understanding of the student’s holistic assessment.

It can also be noted that, prior to the scenario of the Covid-19 pandemic, Teodorczuk et al.²¹ had already pointed out the need for such changes, when they stated that the time had come to open the books, demonstrating the need to include open-book exams. In doing so, the authors say that this change in the evaluation philosophy could benefit students by putting them in touch with deeper and more fun ways of learning. This fact is also pointed out by Ibrahim et al.²² when they mentioned that it is necessary to add more innovative evaluation methods such as open-book exams, self-evaluation and evaluation by peers.

As a challenge to the adoption of open-book exams, scholars agree that it is necessary to convince medical educators, who may not like the change, and prefer to maintain the existing and already tested evaluation system¹⁴. Likewise, other authors agree that the challenge is that educators might not like the changes and prefer to continue using the same evaluation methods¹⁷. To overcome these challenges, a change in assessment philosophy is necessary, which can result in students engaged with deeper and more detailed knowledge and innovative ways of teaching and evaluation.

FINAL CONSIDERATIONS

This review showed that open-book exams constitute an efficient, reliable and consistent method with active teaching methodologies, focusing on the central figure of the student, since it allows an in-depth assessment, with quality, without losing in discrimination and reliability capacity. Thus, it seems to be quite useful for the evaluation of medical students. Nevertheless, their inclusion in medical courses is still discreet, having increased during the Covid-19 pandemic, mainly through remote tests (online platforms). Several potentialities of this type of evaluation have been demonstrated, since the studies show that it contributes to the training of professionals with a high capacity for questioning and debate, prepared for routines of permanent study and education and who understand that there is a constant evolution of knowledge in the medical area. Likewise, there are several challenges, such as the need to redefine cutoff points and grades, the operationalization for using these exams and training teachers and students to use the evaluation modality, among others. As limitations, the present research points out that the number of studies on the subject of open-book exams in medical courses is still very low, which culminated in a small group of articles that were part of this review. Nevertheless, there are robust studies with good technical quality, which allow us to draw important conclusions on the subject and encourage further research in the future. Therefore, it is expected that the knowledge generated herein will encourage further discussions on the subject, as well as support future studies.

REFERÊNCIAS

• ¹
  Norcini J, Anderson MB, Bollela V, Burch V, Costa MJ, Duvivier R, et al. 2018 Consensus framework for good assessment. Med Teach. 2018;40(11):1102-9.
• ²
  Gontijo ED, Alvim CG, Lima MECC. Manual de Avaliação da Aprendizagem no curso de graduação em Medicina. Revista Docência do Ensino Superior. 2015;5(1):205-326.
• ³
  Troncon LEA. Avaliação do estudante de medicina. Medicina (Ribeirão Preto Online). 1996;29(4):429-39.
• ⁴
  Brasil. Resolução CNE/CES nº 3, de 20 de junho de 2014. Institui Diretrizes Curriculares Nacionais do Curso de Graduação em Medicina e dá outras providências. Diário Oficial da União. 23 jun. 2014. Seção 1, p. 8-11.
• ⁵
  Perrenoud P. A avaliação entre duas lógicas. In: Perrenoud P. Avaliação: da excelência à regulação das aprendizagens: entre duas lógicas. Porto Alegre: Artes Médicas Sul; 1999. p. 9-24.
• ⁶
  Perrenoud P. Os procedimentos habituais de avaliação, obstáculos à mudança das práticas pedagógicas. In: Perrenoud P. Avaliação: da excelência à regulação das aprendizagens: entre duas lógicas . Porto Alegre: Artes Médicas Sul ; 1999. p. 65-76.
• ⁷
  Van Der Vleuten CPM, Schuwirth LWT, Driessen EW, Govaerts MJB, Heeneman S. Twelve tips for programmatic assessment. Med Teach . 2015;37(7):641-6.
• ⁸
  Troncon LEA. Avaliação programática do estudante: estratégia institucional para melhor cumprir as funções da avaliação educacional. Revista de Graduação USP. 2016;1(1):53-8.
• ⁹
  Theophilides C, Koutselini M. Study behavior in the closed-book and the open-book examination: a comparative analysis. Educ Res Eval. 2000;6(4):379-93.
• ¹⁰
  Sam AH, Reid MD, Amin A. High-stakes, remote-access, open-book examinations. Med Educ. 2020;54(8):767-8.
• ¹¹
  de Souza MT, da Silva MD, de Carvalho R. Revisão integrativa: o que é e como fazer. Einstein (Sao Paulo). 2010;8(1):102-6.
• ¹²
  Erlich D. Because life is open book: an open internet family medicine clerkship exam. PRiMER. 2017;1:1-6.
• ¹³
  Durning SJ, Dong T, Ratcliffe T, Schuwirth L, Artino AR, Boulet JR, et al. Comparing open-book and closed-book examinations: a systematic review. Acad Med. 2016;91(4):583-99.
• ¹⁴
  Sarkar S, Mishra P, Nayak A. Online open-book examination of undergraduate medical students: a pilot study of a novel assessment method used during the coronavirus disease 2019 pandemic. J Laryngol Otol. 2021;135(4):288-92.
• ¹⁵
  Brossman BG, Samonte K, Herrschaft B, Lipner RS. A comparison of open-book and closed-book formats for medical certification exams: a controlled study. American Educational Research Association. 2017;1:1-15.
• ¹⁶
  Mathieson G, Sutthakorn R, Thomas O. Could the future of medical school examinations be open-book: a medical student’s perspective? Med Educ Online. 2020;25(1).
• ¹⁷
  Monaghan AM. Medical teaching and assessment in the era of Covid-19. J Med Educ Curric Dev. 2020;7:1-3.
• ¹⁸
  Eurboonyanun C, Wittayapairoch J, Aphinives P, Petrusa E, Gee DW, Phitayakorn R. Adaptation to open-book online examination during the Covid-19 pandemic. J Surg Educ. 2021;78(3):737-9.
• ¹⁹
  Zagury-Orly I, Durning SJ. Assessing open-book examination in medical education: the time is now. Med Teach . 2021;43(8):972-3.
• ²⁰
  Elsalem L, Al-Azzam N, Jum’ah AA, Obeidat N. Remote e-exams during Covid-19 pandemic: a cross-sectional study of students’ preferences and academic dishonesty in faculties of medical sciences. Ann Med Surg (Lond). 2021;62:326-33.
• ²¹
  Teodorczuk A, Fraser J, Rogers GD. Open book exams: a potential solution to the “full curriculum”? Med Teach . 2018;40(5):529-30.
• ²²
  Ibrahim NK, Al-Sharabi BM, Al-Asiri RA, Alotaibi NA, Al-Husaini WI, Al-Khajah HA, et al. Perceptions of clinical years’ medical students and interns towards assessment methods used in King Abdulaziz University, Jeddah. Pak J Med Sci. 2015;31(4):757-62.

• 6
  Evaluated by double blind review process.

• SOURCES OF FUNDING

  The authors declare no sources of funding.

Chief Editor: Rosiane Viana Zuza Diniz. Associate editor: Kristopherson Lustosa Augusto.

Publication Dates

• Publication in this collection
  10 Mar 2023
• Date of issue
  2023

History

• Received
  14 Mar 2022
• Accepted
  03 Nov 2022