Flipped classroom for learning clinical examination

Abstract:

Introduction:

Flipped Classroom (FC) is an Active Learning Methodology characterized by the sending of teaching materials to students in advance, so that the classroom moment is entirely dedicated to non-expository activities. The FC was implemented in 2019 for the teaching of Clinical Examination (CE) aimed at undergraduate medical students at a Higher Education Institution in the Northeast region of Brazil.

Objective:

To analyze the implementation of FC for CE learning compared to the mini-exposure methodology followed by practice.

Method:

Analysis of educational intervention with historical control over the implementation of the FC methodology carried out in three phases. In the first, the two tutors involved in the implementation or who acted as teachers were evaluated through a semi-structured interview about the FC implementation process and its initial operation. The second consisted in the assessment of 44 medical students, through a Likert questionnaire, on learning with the new methodology. The third consisted in evaluating the grades obtained by 66 students who experienced learning with FC in relation to 142 students who experienced the previous methodology.

Result:

The evaluated tutors knew little about the methodology before it was implemented and believe that its implementation promoted gains, such as students’ greater dedication to individual study. The initial difficulty arose from the creation of an extensive database of questions for the pre-test, which was carried out at the beginning of the in-person moments. The evaluated students reported being well adapted and agree with the benefits of FC, including: feeling stimulated to study, developing the practice more easily and the presence of pre-tests that help to improve individual study. The results of the comparison of grades showed a significant increase in performance when comparing students who experienced the FC (9,11 - SD 0,45) with those who experienced the previous methodology (8,49 - SD 0,91).

Conclusion:

The methodology was satisfactorily implemented, promoted gains in learning and optimized the in-person time to be entirely dedicated to practical learning.

Keywords:
Physical Examination; Medical Education; Learning; Anamnesis; Methodology

Resumo:

Introdução:

A sala de aula invertida (SAI) é uma metodologia ativa de aprendizagem caracterizada pelo envio antecipado de materiais didáticos aos estudantes, de modo que o momento sala de aula seja inteiramente dedicado a atividades não expositivas. A SAI foi implementada em 2019 para aprendizagem do exame clínico (EC) de alunos da graduação de Medicina em uma instituição de ensino superior da Região Nordeste do Brasil.

Objetivo:

Este estudo teve como objetivo analisar a implantação da SAI na aprendizagem do EC comparando com a metodologia de miniexposição seguida de prática.

Método:

Trata-se de análise de intervenção educacional com controle histórico sobre a implantação da metodologia da SAI realizada em três fases. Na primeira, os dois tutores envolvidos na implantação ou que atuaram como docentes foram avaliados por meio de entrevista semiestruturada sobre o processo de implantação da SAI e seu funcionamento inicial. A segunda fase foi a avaliação de 44 estudantes de Medicina, por meio de questionário Likert, sobre a aprendizagem com a nova metodologia. A terceira consistiu na avaliação das notas obtidas por 66 estudantes que vivenciaram o aprendizado com SAI em relação aos 142 discentes que vivenciaram a metodologia anterior.

Resultado:

Os tutores avaliados conheciam pouco sobre a metodologia antes da implantação e acreditam que sua implantação promoveu ganhos, como uma maior dedicação dos estudantes ao estudo individual. A dificuldade inicial decorreu da elaboração de um extenso banco de questões para o pré-teste que era realizado no início dos momentos presenciais. Os estudantes avaliados relataram que estavam bem adaptados e que concordavam com os benefícios da SAI, como: sentir-se estimulado a estudar, desenvolver a prática com mais facilidade e a presença dos pré-testes que ajudam a aprimorar o estudo individual. Os resultados da comparação das notas mostraram um aumento significativo no desempenho ao compararem os estudantes que vivenciaram a SAI (9,11 - DP 0,45) com os que vivenciaram a metodologia anterior (8,49 - DP 0,91).

Conclusão:

A metodologia foi implantada de maneira satisfatória, promoveu ganhos na aprendizagem e otimizou o momento presencial para ser totalmente dedicado à aprendizagem prática.

Palavras-chave:
Exame Físico; Educação Médica; Aprendizagem; Anamnese; Metodologia

INTRODUCTION

The traditional learning method, in which the teacher and the content that is taught are the protagonists of the teaching-learning process has been challenged by new teaching methodologies, whose center of the process is now the student. For centuries, students have remained silent, in rows of chairs, attentively listening to the contents taught by the teachers, the owners of the knowledge that the students did not yet possess¹.

However, traditional methodologies have proved to be less effective in the learning of attitudes and skills for the new generations, whose access to information is a large-scale one and who, in the face of all the distractions of the contemporary world, find it difficult to maintain the necessary concentration to understand the expository classes. Moreover, the leading role in learning should be given to the most interested party, which is the student².

In this context, some more innovative teaching techniques have emerged, more broadly called active methodologies, in which the student becomes more active in the teaching-learning process, with demonstrated effectiveness for the teaching of subjects in high school and higher education environments^3)-(5.

The Flipped Classroom (FC) is one of these methodologies, which promotes a modification in the sequence of the learning process, since the content is learned by the student in advance instead of being taught as an expository lecture in the classroom, through audiovisual and written material. At the in-person moment, the previously studied content can be shared and evaluated through different ways, such as debates, quizzes, experiments and oral presentations^6)-(8.

The FC has been applied in the teaching of Medicine with good results, both at undergraduate and graduate levels, demonstrating gains in the learning of theoretical and practical contents in procedural skills teaching laboratories^9)-(13.

Learning the skills, competences and attitudes involved in the Clinical Examination, for instance, requires training that is usually carried out in simulated practice laboratories. These procedures go beyond the technical training and are characterized by being the first contact with the doctor-patient relationship^{14), (15}.

In this context, the FC was implemented for learning the Clinical Examination in a medical school in Northeastern Brazil. Its implementation process and its impact are analyzed in this study.

METHOD

An educational intervention analysis was carried out with historical control over the implementation of the flipped classroom methodology. The study was carried out in 2020 at Faculdade Pernambucana de Saúde (FPS), in 3 stages.

The first stage consisted of an individual semi-structured interview, with previously selected questions, with the two tutors who participated in the FC implementation process in the teaching of Clinical Examination, with one of them being the teacher currently responsible for the area of knowledge at the institution. The selection of tutors was carried out by convenience. The method was chosen because it promotes the collection of more details of the process, in a more flexible and comprehensive way. The interviews were conducted with broad questions aiming to understand how the implementation of the FC was carried out for learning Clinical Examination in the FPS medical course and its current impact on learning. The interviews were audio-recorded, with the participants’ permission¹⁶.

The second stage was carried out with the application of a survey on the use of FC in the teaching of Clinical Examination, for students who experienced learning in the area of knowledge through this methodology. The research was presented, in the classroom, to 66 students and an invitation to participate in the study was made. The sample was obtained by convenience. Forty-four students answered a digital form divided into two blocks. The first one was used to collect sociodemographic characteristics, knowledge and experience with active learning methodologies and the second block was structured in a five-point Likert scale containing statements based on the experience with the FC.

In the third stage, information was collected on the performance of 208 students who developed the Clinical Examination learning process using the FC methodology (66) in 2019, and using the previous methodology (142), in 2018, through the results of the assessments. Information was obtained from the medical school’s academic records on the participants’ grades in the formal assessments of the academic semester.

In the first stage, the interviews were transcribed and analyzed according to the answers obtained for each asked question. The answers were grouped into direct citations and summarized in tables. Subsequently, a narrative was created based on these data.

For the evaluation of the survey carried out with the students, the collected data were stored and organized in an Excel^® spreadsheet, version 2013, with double data entry, checking for typing errors. The program EpiInfo^® version 7.1 was used for the statistical analysis. For the data analysis, measures of central tendency and dispersion were initially obtained for continuous variables and measures of frequency distribution for categorical variables. To analyze the opinion of students in the Likert scale, the Average Ranking (AR) value above 3.0 was adopted as a consensus criterion for agreement. Cronbach’s alpha was used to analyze the form consistency.

Grades obtained by students who experienced FC were compared with grades obtained by students who did not experience it; these were collected and organized in an Excel® spreadsheet, version 2013, and arranged in means and standard deviations. The statistical test used was Kruskal-Wallis test, considering a significance level of p <0.05 and a confidence interval (CI) of 95%, using the statistical program EpiInfo® version 7.1.

The project was approved by the Ethics Committee of Faculdade Pernambucana de Saúde (FPS), through CAAE number 26405319.0.0000.5569.

RESULTS

Semi-structured interview with the tutors

Interviews were carried out with 2 teachers from Faculdade Pernambucana de Saúde. The first interviewee (A) actively participated in the process of implementing the FC and was teaching using the other methodology prior to its implementation; the second interviewee (B) actively participated in the implementation and is the current teacher in the field of knowledge. Neither had practical contact with the methodology prior to its implementation but knew it in theory and had experience with other active learning methodologies.

The respondents reported that the implementation arose from difficulties encountered while learning the clinical examination, because as it was previously based on mini-exposures followed by practice, the time of exposures greatly reduced the time of practice. With the change in the teaching methodology, the learning objectives were modified, and the module started to integrate the theoretical and practical contents. The implemented teaching model was based on the previous submission of materials (videos, texts and slides) and an in-person moment, which begins with a pre-test followed by a learning activity in a practical simulation laboratory.

The need for a pre-test at each meeting was one of the difficulties encountered at the start of the process, as it was necessary to prepare an extensive database of questions for its application, which needed to stimulate the student’s clinical reasoning and not just their memory capacity. The digital tool http://www.quizes.com was used to store the constructed questions and facilitate the correction process.

Regarding the participation of students in learning activities, the respondents report that students are more participative and more dedicated to individual studies than before. The presence of mini-tests made students more anxious during the initial period of adaptation to the new method, which resulted in lower grades in the pre-tests and inhibited student participation in practical activities, but these difficulties were soon overcome.

One difficulty pointed out by the assessed tutors was related to student monitors, who carried out their activities through expositions on the study subject. With the new methodology, in which expositions were no longer held, this monitoring style became inconsistent with the model. Despite the initial guidelines, the monitors still persisted carrying out their activities using the previous model, a difficulty that was only resolved after the teachers provided comprehensive directions on the method and, later, with the selection of new monitors who had already been admitted within a new perspective.

For Interviewee A: “in essence, the methodology is very interesting, because if the student who is interested in the resource, who studies at home, actively participates in monitoring, they come to the meeting and take the mini-test, which assesses their knowledge and will practice it; for this student, the basis is very interesting.” Interviewee B reports: “it is an active teaching methodology, very good. Today I feel a great satisfaction in working with this method”.

Student survey

The total number of study participants comprised 44 first-year medical students at FPS, corresponding to a response rate of approximately 66.6%. The age ranged from 17 to 46 years, with a predominance of the age group under 25 years. The majority (75%) were females, about 20% studied at another college and of these, 4 (9%) had finished another degree. Regarding the use of the active methodology before joining FPS, about 90% had never used it and although 28 students (60%) reported having knowledge of the FC methodology, only 3 students reported having used it before (Table 1).

1
Sociodemographic characteristics of medical students who evaluated the Flipped Classroom method in the study of the Clinical Examination. FPS, 2020.

Sociodemographic Characteristics	N (44)	%
Age
< 25 years	39	88.63
25-30 years	3	6.81
30-40 years	2	4.56
Gender
Female	31	70.45
Male	13	29.55
Have you studied at another college?
Yes	9	20.45
No	35	79.55
Do you have another degree?
Yes	4	9.09
No	40	90.91
Have you previously used any active methodology before joining Faculdade Pernambucana de Saúde?
Yes	2	4.55
No	42	95.45
Have you previously used the Flipped Classroom as a learning method before joining Faculdade Pernambucana de Saúde?
Yes	3	6.82
No	41	93.18
Did you have any knowledge of the Flipped Classroom method before joining Faculdade Pernambucana de Saúde?
Yes	28	63.64
No	16	36.36

As for the students’ opinion regarding the use of the FC methodology, the results were grouped into three blocks: statements about preparation, guidance and adaptation, statements about the attitude towards the study and learning process, and statements about the use of pre-test (Table 2). Overall, the students have a positive view of the implementation of the FC for learning the Clinical Examination, with a Cronbach’s alpha of 0.87.

Table 2
Opinion of 1^st-year medical students on the use of the Flipped Classroom methodology in the study of Clinical Examination in simulated environments. FPS, 2020.

Statements	ARa ± (SD)b
BLOCK 1: Preparation, guidance and adaptation to the methodology
I received an explanation about the Flipped Classroom method.	4.27 ± 1.54
I understand the tutor’s role in the Flipped Classroom method.	4.57 ± 1.82
I understand the Flipped Classroom method.	4.55 ± 1.73
I receive reading material before the meetings.	3.11 ± 0.96
The tutor is available to answer questions.	3.95 ± 1.18
I am well adapted to the Flipped Classroom method.	3.43 ± 1.30
I like the Flipped Classroom method.	3.77 ± 1.13
BLOCK 2: Attitude towards study
I study regularly before the classroom meetings.	3.91 ± 1.45
I use the book to study before the meetings.	4.39 ± 1.59
I study through video classes before the meetings.	3.48 ± 1.05
I use the sources sent by the tutor to study.	4.20 ± 1.44
I look for sources other than those indicated by the tutor.	3.66 ± 1.11
I study before the classroom meetings with the monitors.	3.43 ± 1.08
The Flipped Classroom methodology encourages me to study more.	3.70 ± 1.05
I easily develop the practical part of the class due to the previous study.	3.77± 1.22
The Flipped Classroom methodology helps me to consolidate knowledge.	3.70 ± 1.31
BLOCK 3: Use of the pre-test associated with the Flipped Classroom methodology
I use the pre-test to guide my studies.	3.27 ± 0.93
I study before class only because of the pre-test.	2.84 ± 1.05
I use more of my memorization skills to answer the pre-test.	3.50 ± 1.0
I answer the pre-test with my prior knowledge only.	2.07 ± 1.37
The pre-test does not influence the way I study.	2.30 ± 1.29
I consider the pre-test an important element for my academic training.	3.20 ± 0.89
I consider that the pre-test has an adequate amount of questions.	3.43 ± 1.04
I consider that pre-test has an adequate level of difficulty.	3.61 ± 1.29
BLOCK 3: Use of the pre-test associated with the Flipped Classroom methodology
I consider the pre-test questions to be well constructed	3.57 ± 1.24
I consider that the time to answer the pre-test is enough.	3.98 ± 1.29
With the pre-test I feel more encouraged to study before the practical activity	3.89 ± 1.19
If the pre-test did not exist, I would study less for the semiology activities.	3.34 ± 0.94
The pre-test helps me consolidate what I have learned.	3.57 ± 1.14
The feedback after the pre-test helps me build my knowledge better.	3.82 ± 1.19
I consider that the pre-test contributes little to my learning.	2.39 ± 1.18
The pre-test does not make it possible to demonstrate the knowledge I have.	2.11 ± 1.34
I find the pre-test strenuous.	2.27 ± 1.32
I consider that the pre-test does not help in building my knowledge.	2.05 ± 1.38

• ^a
  AR - Average Ranking
• ^b
  SD - Standard deviation

In the first block, students agree that they received explanations about the method (AR 4.27 ± 1.54), understand the tutor’s role (AR 4.57 ± 1.82) and understand the method (AR 4.55 ± 1.73). In the second block, the students affirm they study regularly before classes (AR 3.91 ± 1.45) and, mainly, through the sources sent by the tutor (AR 4.20 ± 1.44). In the third block, the respondents approve the pre-test, as they disagree that the pre-test contributes little to learning (AR 2.39 ± 1.18), it does not make it possible to show the knowledge they have (AR 2.11 ± 1.34), it is strenuous (AR 2.27 ± 1.32) and does not help in the construction of knowledge (AR 2.05 ± 1.38). The students consider that the pre-test guides their studies before the in-person moment, as they disagree that the pre-test does not influence the way they study (2.30 ± 1.29) and that they answer the pre-test with prior knowledge only (2.07 ± 1.37), as well as agreeing that, with the pre-test, they feel more encouraged to study before the practical activity (AR 3.82 ± 1.19).

Grade Evaluation

The results of the grades obtained by students who experienced the teaching of the Clinical Examination before and after the implementation of the FC are shown in Table 3. The evaluation process in both moments was similar, except for the presence of the pre-tests, which were introduced after the implementation of the FC.

Table 3
Mean obtained in the evaluations by the medical students who experienced the learning of the Clinical Examination with the FC methodology, compared to the immediately previous historical control of a class that did not use the FC.

	Flipped classroom		Statistical analysisa
	No (n 142)	Yes (n 66)
Mean (SD)	8.4939 (0.9133)	9.1062 (0.4479)	P= 0.000

• ^a
  Kruskal-Wallis.

The average grade of the 66 students who experienced FC (9.102 - SD 0.4479) was higher when compared to the 142 who did not experience FC (8.4939 - SD 0.9133). The lowest grade obtained by students in the previous methodology was 6.370 and with FC it was 7.660 (Table 3).

DISCUSSION

The use of the FC to learn skills, competences and attitudes in the medical course has been shown to be very effective and well accepted by teachers and students, findings that were also observed in our study^11),(12.

The little previous experience of teachers and students with the FC evidenced in the interview with the teachers and seen in the survey with the students was not an impediment to the implementation of the FC for learning clinical examination. This difficulty was overcome, above all, by the tutors’ theoretical knowledge on the subject and their experience with other active learning methodologies. Jakobsen et al. propose the use of Team-Based Learning (TBL), which is a more widespread methodology, as a way to contribute to the structuring process of the FC. Similarly, de Oliveira et al. used the same strategy in a medical education expansion course in Brazil with good results^17),(18.

The presence of the pre-test was identified as an initial difficulty, mainly regarding its operation, although this difficulty was not verified in the researched literature. The pre-test is frequently used in the experiences of other medical schools that adopt the FC, including in association with a post-test, aiming to measure content retention. The presence of quizzes as a type of pre-test was evaluated in a meta-analysis and verified that, when used at the beginning of classes, they make the application of the FC more effective in the learning process of healthcare professionals^10),(13),(19.

Students report that the pre-test is a useful tool in the learning process and its presence can guide individual study. The teachers pointed out that, initially, the students were more anxious with the presence of the pre-test and that, over time, they overcame this initial difficulty. In contrast to our finding, Uchida et al. compared FC with the traditional methodology for teaching a semiological maneuver and found that students who experienced FC were more self-confident in performing the pre-test in the classroom than students who experienced the traditional methodology²⁰.

The difficulty found in relation to readapting the way student monitors should act in the learning process when retrospectively viewed was overcome, and no similar problem was found in the literature. Our study showed the tutors’ dedication in solving this obstacle with student monitors due to their importance in the learning process, as observed by Nunes et al²¹.

The FC was well accepted by the students, observed both in the research that involved the students and in the teachers’ perception. The literature shows^{9),(10),(18),(22} that the methodology, in general, is well accepted by students, except for Sajid et al., who observed that the assessed students did not have a good perception of the methodology and did not show better content retention²³.

In our study, the students who experienced the FC achieved better grades when compared to those who did not. A meta-analysis published by Chen et al. reported the presence of studies with positive and neutral results. In the study by Martinelli et al., the students preferred the FC to other methodologies, although there was no superiority in terms of knowledge retention^19),(24.

CONCLUSION

Based on the analysis carried out in our study, the implementation of FC for learning Clinical Examination occurred satisfactorily and promoted considerable gains in the construction of knowledge, seen in the assessment by teachers and students, corroborated by the increase of the average grades obtained in the module.

The methodology can be applied to other areas of knowledge that have similar needs in terms of optimizing in-person moments to be dedicated to practical learning and may be the result of new studies of educational intervention and analysis.

REFERENCES

• ¹
  Keiler LS. Teachers’ roles and identities in student-centered classrooms. Int J STEM Educ. 2018;5(1):34.
• ²
  Gillispie V. Using the flipped classroom to bridge the gap to generation Y. Ochsner J. 2016;16(1):32-6.
• ³
  Freeman S, Eddy SL, McDonough M, Smith MK, Okoroafor N, Jordt H, et al. Active learning increases student performance in science, engineering, and mathematics. Proc Natl Acad Sci U S A. 2014;111(23):8410-5.
• ⁴
  Cavanagh AJ, Chen X, Bathgate M, Frederick J, Hanauer DI, Graham MJ. Trust, growth mindset, and student commitment to active learning in a college science course. CBE Life Sci Educ. 2018;17(1):1-8.
• ⁵
  Sivarajah RT, Curci NE, Johnson EM, Lam DL, Lee JT, Richardson ML. A review of innovative teaching methods. Acad Radiol. 2019;26(1):101-13.
• ⁶
  McLean S, Attardi SM, Faden L, Goldszmidt M. Flipped classrooms and student learning: not just surface gains. Adv Physiol Educ. 2016;40(1):47-55.
• ⁷
  Reidsema C, Kavanagh L, Hadgraft R, Smith N, editors. The flipped classroom: practice and practices in higher education. Singapore. Springer; 2017.
• ⁸
  Njie-Carr VPS, Ludeman E, Lee MC, Dordunoo D, Trocky NM, Jenkins LS. An integrative review of flipped classroom teaching models in nursing education. J Prof Nurs. 2017;33(2):133-44.
• ⁹
  Jung H, An J, Park KH. Analysis of satisfaction and academic achievement of medical students in a flipped class. Korean J Med Educ. 2018;30(2):101-7.
• ¹⁰
  Hew KF, Lo CK. Flipped classroom improves student learning in health professions education: a meta-analysis. BMC Med Educ. 2018 Dec 15;18(1):38.
• ¹¹
  Hu X, Zhang H, Song Y, Wu C, Yang Q, Shi Z, et al. Implementation of flipped classroom combined with problem-based learning: an approach to promote learning about hyperthyroidism in the endocrinology internship. BMC Med Educ . 2019;19(1):1-8.
• ¹²
  Cheng X, Lee KKH, Chang EY, Yang X. The “flipped classroom” approach: stimulating positive learning attitudes and improving mastery of histology among medical students. Anat Sci Educ. 2017;10(4):317-27.
• ¹³
  Sezer B, Abay E. Looking at the Impact of the flipped classroom model in medical education. Scand J Educ Res. 2019;63(6):853-68.
• ¹⁴
  Porto CC. Semiologia médica. 8a ed. Rio de Janeiro: Guanabara Koogan; 2019.
• ¹⁵
  Costa GPO, Herculano TB, Gama ALH, Cabral RP, Campos DB, Oliveira DNS. Enfrentamentos do estudante na iniciação da semiologia médica. Rev Bras Educ Med. 2018;42(2):79-88.
• ¹⁶
  Moré CL. A “entrevista em profundidade” ou “semiestruturada” no contexto da saúde. Investigação Qualitativa em Ciências Sociais. 2015;3:126-31.
• ¹⁷
  Jakobsen K, Knetemann M. Putting structure to flipped classrooms using team-based learning. Int J Teach Learn High Educ. 2017;29(1):177-85.
• ¹⁸
  Lucchetti AL, Ezequiel OS, Oliveira IN, Moreira-Almeida A, Lucchetti G. Using traditional or flipped classrooms to teach “Geriatrics and Gerontology”? Investigating the impact of active learning on medical students’ competences. Med Teach. 2018;40(12):1248-56.
• ¹⁹
  Martinelli SM, Chen F, DiLorenzo AN, Mayer DC, Fairbanks S, Moran K, et al. Results of a flipped classroom teaching approach in anesthesiology residents. J Grad Med Educ. 2017;9(4):485-90.
• ²⁰
  Uchida S, Shikino K, Ishizuka K, Yamauchi Y, Yanagita Y, Ikusaka M. The flipped classroom is effective for medical students to improve deep tendon reflex examination skills: a mixed-method study. Res Sq. 2021;(Preprint):1-13.
• ²¹
  Nunes GP, Pirovani BO, Silva HG, Butarelo AV, Rossini JS, Costa JM, et al. The importance of student monitoring in academic learning: a two-year follow-up. Brazilian J Oral Sci. 2018;17:1-7.
• ²²
  Fatima SS, Hashmi S, Rehman R, Akbar R. Teaching “shock pathophysiology” by flipped classroom. Pakistan J Med Sci. 2019;35(6):1631-5.
• ²³
  Sajid M, Shaikh AA, Ikram MF, Cahusac P, Yaqinuddin A, AlKattan W, et al. Comparative analysis of effectiveness between flipped classroom and lecture-based classroom in undergraduate medical education at Alfaisal University. Cureus. 2020;12(11):e11408.
• ²⁴
  Chen F, Lui AM, Martinelli SM. A systematic review of the effectiveness of flipped classrooms in medical education. Med Educ. 2017;51(6):585-97.

• 2
  Evaluated by double blind review process.

• SOURCES OF FUNDING

  The authors declare no sources of funding.

Edited by

Chief Editor: Rosiane Viana Zuza Diniz. Associate Editor: Kristopherson Lustosa Augusto.

Publication Dates

• Publication in this collection
  31 Jan 2022
• Date of issue
  2022

History

• Received
  06 Sept 2021
• Accepted
  17 Nov 2021