Accessibility / Report Error

Lessons learned from COVID-19 pandemic to improve project/service-based learning approaches: findings from Engineering students

Lições aprendidas com a pandemia COVID-19 para a melhoria de abordagens baseadas em projetos e serviço: resultados de estudantes de Engenharia

Lecciones aprendidas de la pandemia de COVID-19 para mejorar los enfoques de aprendizaje basados en proyectos/servicios: hallazgos de estudiantes de ingeniería

Abstract

This paper describes a case study based on the implementation of a project/service-based learning approach in the context of a curricular unit of an Engineering Education programme. It discusses the challenges and pedagogic responses found to adapt the teaching and learning process to the pandemic period, reflecting on the lessons learned to improve project/service-based learning approaches. For data collection, an online questionnaire was applied to students to collect feedback about the effectiveness of the approach adopted. The results from the questionnaire indicate that students considered the alternatives implemented by the professor completely adequate to respond to the COVID-19 constraints. The results from students’ projects revealed that they were able to go beyond what was expected from them, by presenting clear and effective outcomes of the implementation of the project in their own domestic environments. Lessons learned include ways to enhance student learning, drawing on examples from students’ own personal contexts.

COVID-19 Pandemic; Higher Education; Project-Based Learning (PBL; Service-Based Learning (SbL

Resumo

Este artigo descreve um estudo de caso baseado na implementação de uma abordagem de aprendizagem baseada em projeto/serviço no contexto de uma unidade curricular de um programa de Ensino de Engenharia. Discute os desafios e as respostas pedagógicas encontradas para adequar o processo de ensino e aprendizagem à realidade da pandemia, identificando oportunidades de melhoria através das lições aprendidas. Para a recolha de dados, foi aplicado um questionário online aos estudantes, para recolha de feedback. Os resultados do questionário indicam que os estudantes consideraram adequadas as alternativas implementadas pelo professor para responder aos constrangimentos da COVID-19. Os resultados dos projetos revelaram que os estudantes foram além do esperado, em termos da concretização do projeto no seu próprio ambiente doméstico. As lições aprendidas incluem formas de melhorar a aprendizagem, baseando-se em exemplos provenientes da experiência dos próprios estudantes.

Pandemia de COVID-19; Ensino Superior; Aprendizagem Baseada em Projetos (PBL; Aprendizagem e Serviços (ApS

Resumen

Este artículo describe un estudio de caso basado en la implementación de un enfoque de aprendizaje basado en proyectos/servicios en el contexto de una unidad curricular de un programa de Educación en Ingeniería. Discute los desafíos y las respuestas pedagógicas encontradas para adaptar el proceso de enseñanza y aprendizaje al período de la pandemia, reflexionando sobre las lecciones aprendidas para mejorar los enfoques de aprendizaje basados en proyectos/servicios. Para la recopilación de datos, se aplicó un cuestionario en línea a los estudiantes para recopilar comentarios sobre la efectividad del enfoque adoptado. Los resultados del cuestionario indican que los estudiantes consideraron las alternativas implementadas por el docente, completamente adecuadas para responder a las limitaciones del COVID-19. Los resultados de los proyectos de los estudiantes revelaron que fueron capaces de ir más allá de lo que se esperaba de ellos, presentando resultados claros y efectivos de la implementación del proyecto en sus propios entornos nacionales. Las lecciones aprendidas incluyen formas de mejorar el aprendizaje de los estudiantes, basándose en ejemplos de los propios contextos personales de los estudiantes.

Pandemia de COVID-19; Educación Superior; Aprendizaje Basado en Proyectos (ABP; Aprendizaje-servicios (ApS

1 Introduction

The outbreak of COVID-19 has changed many, if not all, sectors of society. One of those sectors is Higher Education. The volume of research on the ways in which professors and students rapidly adapted to this new reality has also increased considerably in the past years ( ASSUNÇÃO FLORES; GAGO, 2020ASSUNÇÃO FLORES, M.; GAGO, M. Teacher education in times of COVID-19 pandemic in Portugal: national, institutional and pedagogical responses. Journal of Education for Teaching, [s. l.], v. 46, n. 4, p. 507-516,July 2020. https://doi.org/10.1080/02607476.2020.1799709
https://doi.org/10.1080/02607476.2020.17...
; CASTIONI et al. , 2021; CAVALCANTI; GUERRA, 2022CAVALCANTI, L. M. R.; GUERRA, M. G. G. V. Os desafios da universidade pública pós-pandemia da Covid-19: o caso brasileiro. Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, v. 30, n. 114, p. 73-93, jan.-mar. 2022. https://doi.org/10.1590/S0104-40362021002903113
https://doi.org/10.1590/S0104-4036202100...
; CHIAPPE; WILLS, 2022CHIAPPE, A.; WILLS, A. E. Crowd-based Open Online Education as an alternative to the Covid-19 educational crisis. Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, v. 30, n. 114, p. 32-51, jan./mar. 2022. https://doi.org/10.1590/S0104-403620210002903341
https://doi.org/10.1590/S0104-4036202100...
; GOMES, et al. , 2021; HATTINGH et al. , 2020; HIDAYATI; SAPUTRA, 2020HIDAYATI, D.; SAPUTRA, W. A. Implementation of online learning during the covid-19 epidemic in Indonesia: assessment of higher education students’ use and implementation of online learning technology. Universal Journal of Educational Research, [s. l.], v. 8, n. 10, p. 4514-4519, 2020. https://doi.org/10.13189/ujer.2020.081019
https://doi.org/10.13189/ujer.2020.08101...
; HIRA; ANDERSON, 2021HIRA, A.; ANDERSON, E. Motivating online learning through project-based learning during the 2020 COVID-19 pandemic. IAFOR Journal of Education, [s. l.], v. 9, n. 2, p. 93-110, 2021. ; OLIVEIRA; GOMES; BARCELLOS, 2020; SCHLEICHER, 2020SCHLEICHER, A. Teacher pofessionalism in the face of COVID-19. Paris: OECD, 2020. ; XU; TANG, 2021XU, Y.; TANG, Q. The reform of modern education during the COVID-19 pandemic. Journal of Physics: Conference Series, [s. l.], v. 1748, 2021. https://doi.org/10.1088/1742-6596/1748/4/042051
https://doi.org/10.1088/1742-6596/1748/4...
), in particular in Engineering Education (AHAG et al. , 2020; ASGARI et al. , 2020; HATTINGH et al. , 2020; KAUR; GARG; KAUR, 2021; KHAN; ABID, 2021KHAN, Z. H.; ABID, M. I. Distance learning in engineering education: Challenges and opportunities during COVID-19 pandemic crisis in Pakistan. International Journal of Electrical Engineering Education, [s. l.], 2021. https://doi.org/10.1177/0020720920988493
https://doi.org/10.1177/0020720920988493...
; LIU et al. , 2021; MU et al. , 2021; MUSA; ALABI, 2020MUSA, A.; ALABI, A. G. F. Disruptive engineering and education in emerging economies: challenges and prospects. In: 2000 IFEES WORLD ENGINEERING EDUCATION FORUM – GLOBAL ENGINNERING DEANS COUNCIL, 2020. [S. l.: s. n.], 2020. ).

In Portugal, schools and universities were closed for the first time in March 2020. Flores and Gago ( ASSUNÇÃO FLORES; GAGO, 2020ASSUNÇÃO FLORES, M.; GAGO, M. Teacher education in times of COVID-19 pandemic in Portugal: national, institutional and pedagogical responses. Journal of Education for Teaching, [s. l.], v. 46, n. 4, p. 507-516,July 2020. https://doi.org/10.1080/02607476.2020.1799709
https://doi.org/10.1080/02607476.2020.17...
) described the national, institutional and pedagogical initiatives and responses to the crisis as well as the difficulties, the challenges and the opportunities. The authors discussed the implications for teaching in such uncertain times, giving particular attention to issues related to the role of practice and the context of students’ practicum as a “real practice” versus “an ideal(ised) practice”. Several constraints were found in this emergency shift to distance learning. In Australia, one of the most successful countries in handling COVID-19, motivating and retaining students was a key challenge for Higher Education Institutions ( SMITH; KAYA, 2021SMITH, E. K.; KAYA, E. Online university teaching at the time of COVID-19 (2020): an Australian perspective. IAFOR Journal of Education, [s. l.], v. 9, n. 2, p. 183-200, 2021. https://doi.org/10.22492/ije.9.2.11
https://doi.org/10.22492/ije.9.2.11...
). The authors sought to overcome these challenges with the application of various digital programs and tools in the online teaching and learning space. In Spain, one of the biggest challenges of COVID-19 pandemic for the final months of the academic year of 2019/2020s was the online assessment of the learning process (GARCÍA-PEÑALVO et al. , 2020). In China, as most universities had to close their campuses and adopt the online teaching model, a study was developed with undergraduate engineering students to investigate the efficacy of various online teaching modes as well as to compare a proposed combined model of online and flipped learning to other online and traditional models (TANG et al. , 2020). The results of the study showed that students were dissatisfied with online learning in general, and they were especially dissatisfied with the communication and Q&A modes. In addition, the combined model of online teaching with the flipped learning improved students’ learning, attention, and evaluation of courses (TANG et al. , 2020).

The pandemic situation created by the virus known as COVID-19 caused an impressive impact in the normal routines in university environments. The most obvious one is the transfer of face-to-face classes to distance classes using video conference platforms but that wasn’t by far the main challenge. A much bigger challenge that we faced was how to deal with the project teamwork in real context in organizations since those organizations were not anymore able to receive students in their facilities.

This paper aims to describe how COVID-19 placed educational demands in a particular project/service-based learning context, within Engineering Education. It discusses the pedagogic responses found to teach lean thinking principles, concepts and practices ( WOMACK; JONES, 1996WOMACK, J.; JONES, D. Lean thinking: banish waste and create wealth in your corporation. New York: Fee Press, 1996. ) and alternative ways for effective student learning and teamwork in an online format. The paper is organized in three main parts. In the first part, the theoretical and conceptual background about project-based learning and service learning are presented. The next section describes the context of the study, followed by the methods for data collection and analysis. The last part of the paper explores the findings based on students’ feedback and satisfaction regarding the challenges and effectiveness of the pedagogical responses implemented. It also discusses the lessons learned from this atypical situation to improve project/service-based learning approaches and identify new ways to enhance student learning, drawing on examples from students’ own personal contexts.

2 Literature review

The roots of Service-based Learning (SbL) or also called Service Learning go back to the beginning of the XX century, when Arthur Dunn incorporated service project in the community as part of his social studies curriculum ( NIEBOER, 2000NIEBOER, R. A. Arthur Dunn: civic visionary from the Heartland. In: ANNUAL MEETING OF NATIONAL COUNCIL FOR THE SOCIAL STUDIES, San Antonio, Nov. 2000. Available from: https://archive.org/details/ERIC_ED448090/page/n1/mode/2up. Access: 2020 Mar. 25.
https://archive.org/details/ERIC_ED44809...
). The term “Service Learning” was introduced in the 1970s (SOUTHERN REGIONAL EDUCATION BOARD, 1970) stating among other things that universities should encourage students to do community service, assist in assuring learning as part of this service as well as stating that students, college faculty and staff cooperate in the administration of programs in which students both serve and learn. Service learning is an educational approach that combines learning processes and community service in a single well-articulated project in which all participants are working on real needs of the environment with the aim of improving it ( CASARES, 2013CASARES, M. G. Service learning and social inclusion. Barcelona: aprenentatgeservei, 2013. Available from: https://www.incorpora.org/documents/ 20181/134822/SERVICE_LEARNING_AND_SOCIAL_INCLUSION/ 037583d2-bebc-45dc-a235-bf5bc75ac501. Access in: 2020 Apr 20.
https://www.incorpora.org/documents/ 201...
). Service learning programs are very similar to Project-based Learning (PBL), a team-based approach where students work together to solve multidisciplinary problems from the real world (BOROCHOVICIUS; CRISTINA; TORTELLA, 2014; LIMA et al. , 2017). The main difference is that the real context in SL are social organizations instead of companies. Learning while developing projects in real context is recognized to be more effective than traditional classroom learning but SL is also believed to hold the potential to broaden and significantly enhance the learning climate for students (LEVESQUE-BRISTOL; KNAPP; FISHER, 2011). It is also an important strategy to prevent students that are at risk in the first years of Higher Education (DONOSO-DÍAZ; ITURRIETA; TRAVERSO, 2018), by motivating them with a challenging and meaningful purpose for their Education. The general learning goals of Service-Learning can be organized in three categories ( FELTEN; CLAYTON, 2011FELTEN, P.; CLAYTON, P. H. Service-learning. New Directions for Teaching and Learning, [s. l.], n. 128, p. 75-84, Dec. 2011. https://doi.org/10.1002/tl.470
https://doi.org/10.1002/tl.470...
): civic learning, academic learning, and personal growth. Service Learning programs can be found in many universities in Europe (CAYUELA; ARAMBURUZABALA; BALLESTEROS, 2020), in many universities in USA, where we stand out Purdue University (EPICS program) (PURDUE UNIVERSITY, 2020), as well as in general across the world.

Service-learning projects imply challenges in the development of community actions that have a high educational component ( ADARLO, 2020ADARLO, G. M. Service-learning as global citizenship education: Acting locally on global challenges and concerns. IAFOR Journal of Education, [s. l.], v. 8, n. 3, p. 7-23, 2020. https://doi.org/10.22492/ije.8.3.01
https://doi.org/10.22492/ije.8.3.01...
). Knowing the needs and realities of other groups and encouraging people to contribute with their input allows stereotypes to be changed and contributes to rebuilding social links. In service-learning projects, people in situations of exclusion are no longer beneficiaries of solidarity-based actions to become agents of change, and service actions are opportunities to incorporate their voices, input, and contributions in the building of inclusive citizenship.

The implementation of lean principles and concepts In social organizations is not completely new. One example comes from Jefferson County ( MURMAN; BAKST, 2017MURMAN, E.; BAKST, J. Growing a Lean community: the Jefferson County WA Lean Journey. [S. l.: s. n.], 2017. ) and another one refers to non-profit organizations discovering lean advantages ( MURRAY; MA, 2015MURRAY, P.; MA, S. The promise of Lean experimentation. StanfordSocial Innovation Review, [s. l.], v. 13, n. 3, p. 24-39, Summer 2015. https://doi.org/10.48558/A24K-3949
https://doi.org/10.48558/A24K-3949...
). Attending to Lean Thinking principles, it is in this kind of organizations and services that lean makes even more sense, due to the scarce resources they have ( SHAFIQ; SORATANA, 2020SHAFIQ, M.; SORATANA, K. Lean readiness assessment model - a tool for Humanitarian Organizations’ social and economic sustainability. Journal of Humanitarian Logistics and Supply Chain Managementv, [s. l.], v.. 10, n. 2, p. 77-99, Apr. 2020. https://doi.org/10.1108/JHLSCM-01-2019-0002
https://doi.org/10.1108/JHLSCM-01-2019-0...
). Some local communities have been benefiting of such services, namely, the local communities where the Lean Sigma Academy of the The University of Texas Rio Grande Valley (UTRGV),Texas Manufacturing Assistance Centre is integrated. One of the few examples about learning Lean Six Sigma with Service-Learning is provided by Braun (BRAUN, 2013).

Higher Education institutions have formal social responsibility expressed in many ways. In Portugal, one of these ways is expressed in the so called “Green book on Social responsibility of HEI” ( ORSIES, 2018OBSERVATÓRIO SOBRE RESPONSABILIDADE SOCIAL E INSTITUIÇÕES DE ENSINO SUPERIOR – ORSIES. Livro verde sobre responsabilidade social e instituições de ensino superior. [S. l.]: Press Forum, 2018. Available from: https://www.uc.pt/sustentabilidade/documents/Livro_Verde_versaoFINAL.pdf. Access: 2023 Mar. 20.
https://www.uc.pt/sustentabilidade/docum...
). By strengthening the community – university partnerships, universities can contribute to bring solutions to the major challenges of the 21st century, such as increasing environmental and socio-economic crises, inequalities of income and wealth and political instabilities ( VON HAUFF; NGUYEN, 2014VON HAUFF, M.; NGUYEN, T. Universities as potential actors for sustainable development. Sustainability, Basel, v. 6, n. 5, p. 3043-3063, 2014. https://doi.org/10.3390/su6053043
https://doi.org/10.3390/su6053043...
). These challenges are also at the heart of the 2030 Agenda for Sustainable Development, adopted by the all United Nations Member States in 2015 (JOHNSTON, 2016). The 17 Sustainable Development Goals (SDG) are an urgent call for action by all countries – developed and developing – in a global partnership. By integrating lean principles and practices into social organizations, through active engagement of engineering students in community projects, universities can lead by example. Furthermore, through curriculum innovation and active learning approaches, future decision makers can learn the competences needed to solve ecological, social, and economic problems in societies, along with the development of human competences and civic awareness (CUMBO; VADEBONCOEUR, 1998; FOLGUEIRAS et al. , 2018). Service Learning plays a crucial role to support HEI to achieve these goals ( ORSIES, 2018OBSERVATÓRIO SOBRE RESPONSABILIDADE SOCIAL E INSTITUIÇÕES DE ENSINO SUPERIOR – ORSIES. Livro verde sobre responsabilidade social e instituições de ensino superior. [S. l.]: Press Forum, 2018. Available from: https://www.uc.pt/sustentabilidade/documents/Livro_Verde_versaoFINAL.pdf. Access: 2023 Mar. 20.
https://www.uc.pt/sustentabilidade/docum...
) and promote the quality of Education for all ( GOMES, 2020GOMES, C. A. Escola de qualidade para todos revisitada: desfolhando as camadas da cebola. Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, v. 28, n. 109, p. 843-862, out./dez. 2020. https://doi.org/10.1590/S0104-40362020002802958
https://doi.org/10.1590/S0104-4036202000...
).

3 Context of the study

The case study reported in this article took place between February and June 2020 in the Master course in Industrial Engineering at the School of Engineering, at the University of Minho, in the north of Portugal. In this study, students were taking the learning unit called “Lean Enterprise”, where the central learning approach adopted is a project/service-based one. This designation is related to the fact that some of the projects were taking place in service organizations. The coordinator of this learning unit is one of the authors of this article. There were 17 students enrolled in this learning unit.

In the initial plan for the course, before the confinement due to COVID, 4 service organizations were selected to host student projects. All selected service organizations (see Table 1 ) are offering their services to the communities in the cities of Braga and Guimarães (north of Portugal), where the two main campus of the University of Minho are located.

Table 1
Host institutions for Service-Based Learning projects

The idea of the projects was to make an initial diagnosis of their processes, using process mapping tools as well as other diagnosing tools and to measure their performance as rigorous as possible. The second phase of the project was to identify problems to be solved as well as improvement opportunities. Finally, improvement actions based on lean concepts, principles, and tools were proposed and implemented (as much as possible) to improve their overall performance and service effectiveness. Given the socio-technical nature of such organizations, students also helped in finding solutions to improve motivation, work satisfaction and team spirit among workers/volunteers. Students were organized in teams with the plan of working in the institution premises one half day per week during the entire semester. Students would be assessed in their presentations and report, by the professor, colleagues and by the institution representative.

That was the initial plan but as student teams were establishing the initial visits to the institutions and starting their projects, the confinement due to COVID was imposed by the government so these projects had to be cancelled. Since the projects had to be cancelled and students as well as professors had to be confined in their homes, the initial plan became impossible to follow. The new challenge was to find ways of keeping the same planned learning outcomes (development of technical and transversal skills established to the curricular unit) in this new unpredictable reality. The planned classroom activities were quite easy to arrange using video conference tools such as Zoom although every planned face-to-face dynamic had to be transformed into digital formats. The project part of the curricular unit was the real challenge. How could we design some kind of real context projects knowing that students were not anymore able to leave their homes?

The idea that came to our minds was for student teams to use the same approach in processes and reality of their homes, applying the principles, concepts and tools that were supposed to be used in the social organizations. At the beginning, students were feeling a bit frustrated because they couldn’t see how they could apply such concepts and tools in their domestic environments.

The assessment system had also to be changed slightly. The initial plan was to have an individual assessment component based on 4 written tests throughout the semester (weight = 35%) and another assessment component based on the team project performance (weight = 65%). The tests had to be taken online and the project performance would be based on the final presentation through videoconference and a final report. The criteria for assessing the presentations were technical content (50%), communication effectiveness (20%), level of innovation (10%), and quality of the discussion (20%). Regarding the final report the assessing criteria were the following: technical content (60%), form and structure (15%), readability (15%), respect for other authors (10%). Peer assessment among students of the same team was also allowed to teams with a weight of 15%.

The availability of professors to assist individual students as well as student teams was based on scheduled videoconference meetings as required by students. Students just had to send an email to the professor to require a videoconference meeting and in average a meeting was set within the next three days. Student teams were also free to require project intermedium presentations to get feedback from the professor and correct possible misalignments. The team meetings, using videoconference tools, were completely managed by each team.

In these new circumstances, the project context changed dramatically, and new challenges were put to students. The first big challenge for student teams was to identify, in the context of their homes, anything where Lean concepts, principles and tools covered in the learning unit could be applied. In the technical content of this learning unit, the ability to identify waste (lean concept), problems and sources of improvement is an important learning outcome. The second challenge is to design possible improvement actions, implement them and then check the results. Unfortunately, the service learning context was lost but the classical project learning approach was maintained.

4 Method

This paper aims to describe how COVID 19 placed educational demands in a particular project/service-based learning context and discuss how the response overcame the challenges.

The following research questions were defined:

  • What were the challenges placed by COVID-19 for the implementation of project/service-based learning?

  • How did professors adapt teaching, learning and assessment in the context of project/service-based learning?

  • What were the pedagogical responses given to overcome the COVID-19 educational demands and the lessons learned by the participants involved?

For data collection, an online questionnaire was applied to students, at the end of the project/service-based learning approach. The questionnaire was organized in five main sections, including topics such as the following: sociodemographic information; changes in teaching, learning and assessment of the curricular unit; challenges for student teamwork and for the development of the project; and the overall evaluation of the project/service-based approach. Data analysis was based on descriptive statistics and a content analysis of the qualitative data from the open-ended questions from the questionnaire.

In total, 14 students answered the questionnaire. Of these, 12 were female and 2 male. The ages ranged from 21 to 30 years old. The age of 22 years old was the most common among the students. Most students (n = 8) were working. All students lived in the district where the university is located.

5 Findings

Findings were organized according to the categories and results from the questionnaire, exploring the main changes employed by the professors in the organization of the curricular unit, the challenges and difficulties identified by students during the development of the project and, finally, students’ perceptions about the adequacy of the pedagogical responses implemented.

5.1 Changes in the organization of the curricular unit

The results were relatively surprising because the restrictions imposed by the pandemic severely limited the expectations of students regarding the real context project. The degree of students’ satisfaction regarding the adaptations is presented in Figure 1 . The question was the following: “Generally speaking, assess your level of satisfaction regarding the adaptation of the teaching, learning and evaluation model of this Learning Unit during the COVID-19 period”. It explored the following aspects: Learning outcomes, Skills developed, Pedagogical strategies used in classes, and Professors’ support and availability. Most of the students were satisfied and very satisfied with the solution adopted.

Figure 1
Student satisfaction with the teaching and learning process

The dimension that students felt the most satisfied about was “professor support/availability”, with half of the respondents (50%) answering “very satisfied”. Students felt that the project they developed was “an excellent alternative” (ST2) and that the professor was “flexible and quick in finding a solution that could be adapted to the new reality, surprising all of us” (ST4). Students referred that “the professor always showed availability and understanding to adjust the course unit according to students’ needs” (ST13). This dimension is also connected to another aspect which is the communication process, also very well classified by students (50% answered “very satisfied”).

Regarding the pedagogical strategies used in classes, students’ answers showed that they were, in general, satisfied (78.6%) with the approaches used by the professor. Students understood and recognized that “considering the current the situation, there is no other way to carry out the project, being the only alternative developing a theoretical work on the subject, but I think that does not bring the same value” (ST14). One student was very specific in pointing out these pedagogical issues when explaining the rating (level 4, on a Likert scale) given to the solution adopted in this curricular unit to respond to the constraints of COVID-19:

I think that the application of concepts at home was a more effective method to learn the tools, than compared to literature reviews and other theoretical works, for example, even though these also have their value. I thought it was great for the professor to send quick and very well explained videos about concepts that would be covered in the next class. It was great, too, to show videos during classes. Most of the students in this class do not have a background in Industry. And it’s okay, we signed up for the master’s degree to learn. Availability to talk about the tests and correct possible flaws in the google forms tool (ST4).

Also, when questioned about their level of satisfaction concerning the adaptation of the curricular unit due to the COVID-19 lockdown and adoption of distance learning at the university, 64.3% and 57.1% of the students inquired were satisfied with the “learning outcomes” and “development of skills”, respectively. This dimension is closely related to one of the open questions included in the questionnaire, where students were asked to identify what they had learned from the project that they considered most important. The answer was common to the majority of the students (10 out of 13 answers), who indicated that they had learned Lean concepts and tools. This confirms that the learning outcomes that are part of the course syllabus were effectively fulfilled. The three students who answered in a different way indicated they had learned to: “think ‘outside the box’ and find new ways of thinking about the same subject” (ST3), “dealing with people, especially when they are resistant to you and/or your project” (ST4), and that “the effort is rewarding” (ST7). The achievements reported from these three students also show the development of other skills by students, beyond the acquisition of technical knowledge in the field.

5.2 Challenges for student teamwork

Students were asked to select the 3 most relevant transversal competences developed with the project, from a set of 6 competences, which focused on the ability to: i) communicate in oral presentations; ii) negotiate and reach consensus; iii) assume leadership role; iv) manage conflicts; v) manage projects and teams; vi) manage time. The results indicated that “time management skills” (14 answers) were the most important transversal skills developed by students in the respective semester, followed by “the ability to manage projects and teams” (11 answers), the “ability to communicate in presentations” (7 answers) and, finally, the “ability to negotiate and reach consensus” (4 answers). Conflict management and leadership skills seemed to be not so important for students and for the development of the project, as no student referred to them as the three most important transversal skills developed by the project.

Other aspects in the questionnaire were included to collect information regarding the way students overcame the challenges presented by the forced lockdown. Since students had to stay at home, several were the challenges for team and project work. Students couldn’t physically meet nor physically access common spaces. In terms of frequency of team meetings, by videoconferencing, 85% of the students had meetings once a week while the others had meetings more than once a week. When students were asked about the online tools used for group communication and project management, Zoom platform (14 answers) was the communication tool most used for synchronous online meetings, followed by Facebook Messenger (10 answers) as the most used tool for asynchronous communication and Google Drive (7 answers) as the most common tool to share documents. According to students’ answers, online tools played an “extremely important” (11 answers) and “very important” (3 answers) role in the effectiveness of the team work and students revealed to be “very satisfied” (11 answers) and “satisfied” (3 answers) towards the online tools used.

5.3 Main difficulties felt during the development of the project and ways to overcome

When asked about the main difficulties felt by students during the project, through an open ended question in the questionnaire, students reported issues related to communication, resistance to change, understanding specific curricular contents, amongst other aspects such as available resources. One of the aspects most referred by students was having difficulties in the communication process. As students were no longer working side by side in the classroom, dealing with this constraint brought several challenges to teamwork and the need to quickly adapt to a new situation. The following quotes from students confirm this:

Because we are at distance, when someone had some difficulty in applying a tool, it was more difficult to understand the problem and help than if we were on site. We overcame this difficulty by improving the way we communicate, with time and also patience (ST2).

Perhaps the distance between the elements of the group. It would be easier to communicate directly, but the group managed to solve it in the best possible way (ST13).

The communication process needed to be adapted. At meetings, we used Zoom and screen sharing to view the ideas and applications of each team member (ST4).

Finding ways to implement the project goals, this is, implementing lean tools within the home environment, was also very demanding and challenging for students as they were able to experience the difficulties in changing people’s mindset and the resistance to change. Some students identified that “convincing someone to do something in a different way” (ST10) or “that it was necessary to show that the gains could be very significant, when the users showed resistance to change” (ST7), as some of the major difficulties in the implementation of the project. As one of the students summarised it:

All difficulties were overcome as a team and, initially, it was a little complicated to understand how this project could be adapted to a more homely environment. Another difficulty was to convince and modify the mentalities of the family members regarding this project (ST1).

Some students also reported difficulties in understanding specific curricular contents, which they tried to overcome by asking for greater support from the professor. In particular, students mentioned difficulties to “understand the differences between SW and OPL, which were explained by the professor, and the attempt to apply different tools in the context of the home” (ST8), “understand some content online” (ST9) and “how to do some of the challenges; [how to overcome] with group discussion and exchange of ideas” (ST3).

5.4 Adequacy of the project approach implemented

When asked to rate (from 1 = not at all adequate to 5 = completely adequate) the adequacy of the project approach implemented in the context of this curricular unit to respond to the constraints of COVID-19, students considered that it was “5 = completely adequate” (5 answers), “4 = very adequate” (6 answers) and “3 = adequate” (2 answers). All students provided a short explanation for their classification, except the two students who rated the solution with “adequate”.

On the one hand, students who considered the project approach completely adequate pointed out the following justifications:

The skills and content taught in class have been assimilated in a much more profound way. The students started to better understand the tools, what are their implementation difficulties and started to have a very critical opinion regarding them. It is, in fact, an incredible solution in the face of current situations (ST1).

Since the risk of working outside the home was significant, adopting this strategy showed students that Lean can be applied everywhere (ST7).

Applying the tools at home, being able to see the improvement of the organization and tidiness, made me want to think about all the things that could and improved, even if they were not presented at work ... it was like it hadn’t been a job, and at the same time very enriching, which made me now see and do some things in a different and more conscious way, because lean was introduced in a very natural way “in our day to day” ... It was not just applying it in a company and say “it’s done”, I feel that it was a much more continuous work, since every day we were faced with the places where tools had been applied and we could see if everything was ok, if not, if it could be done in another way, etc. It surprised me a lot in a positive way. It was the best moment of the whole semester (ST8).

On the other hand, students who considered the project approach very adequate also presented similar arguments to explain their satisfaction with the solution adopted to implement the project:

As for the project, it was an excellent alternative. Regarding the tests, despite understanding that there was no other solution, students with unstable internet, children at home, etc., end up being more affected, since we are not all in the same environment (ST2).

I am satisfied because I think the professor was flexible and quick in proposing a solution that would adapt to the reality that took everyone by surprise (ST4).

The professor has always been understanding and available to adapt the curricular unit according to the students’ difficulties. Considering the situation that we were living, the method proved to be adequate (ST13).

6 Discussion and final remarks

This paper aimed to describe how COVID 19 placed educational demands in a particular project/service-based learning context, within Engineering Education. Based on the three initial research questions posed, the authors discussed the pedagogic responses found to teach lean production practices and concepts and alternative ways for effective student learning and teamwork in an online format.

Regarding the first research question (What were the challenges placed by COVID-19 for the implementation of project/service-based learning?), the main challenges posed by the confinement due to COVID-19 in the beginning of the 2020 to the planned teaching and learning activities, from the professors perspective, were as follows: (1) Students were forced to stay at home having online classes; (2) Students were no longer allowed to be work in the assigned social organizations; (3) Students couldn’t meet the professor to obtain face-to-face support on their projects; and (4) Students couldn’t physically meet in a room to do teamwork with classical team and project team tools such as Kanban or Scrum boards, problem solving templates, project chats, and so on. From the students’ perspective, the main challenges were related to difficulties in the communication process. Dealing with this constraint brought several challenges to teamwork and the need to quickly adapt to a new situation. Besides this, the resistance to change, understanding specific curricular contents, as well as available resources, were also identified as challenges faced by students throughout the development of the project. Although the projects were initially thought as service-learning projects to be carried out in social organizations, which imply challenges in the development of community actions that have a high educational component ( ADARLO, 2020ADARLO, G. M. Service-learning as global citizenship education: Acting locally on global challenges and concerns. IAFOR Journal of Education, [s. l.], v. 8, n. 3, p. 7-23, 2020. https://doi.org/10.22492/ije.8.3.01
https://doi.org/10.22492/ije.8.3.01...
), the students undergone a significant learning experience, both in terms of the development of technical and interpersonal skills. This is one of the most evident outcomes of student engagement in PBL approaches (LIMA et al. , 2017).

About the second question (How did professors adapt teaching, learning and assessment in the context of project/service-based learning?), professors were forced to rethink and adapt their pedagogical approaches. The traditional physical classroom had to be replaced by a virtual classroom and for that reason a large part of the teaching/learning mechanisms used had to be replaced by others (traditional games and simulation that could be carried out in classrooms had to be reinvented for virtual environments). Since the social organizations could no longer be used as real contexts for students to carry out their projects, the professors had to find alternatives ( FERNANDES; DINIS-CARVALHO, 2020FERNANDES, S.; DINIS-CARVALHO, J. A service learning experience with engineering students. Guimarães: PAEE - Project Approaches in Engineering Education Association and Department of Production and Systems, School of Engineering of University of Minho, 2020. p. 227-236. ). All the necessary support required by students (individually or in teams) had to be carried out in virtual environments ( SMITH; KAYA, 2021SMITH, E. K.; KAYA, E. Online university teaching at the time of COVID-19 (2020): an Australian perspective. IAFOR Journal of Education, [s. l.], v. 9, n. 2, p. 183-200, 2021. https://doi.org/10.22492/ije.9.2.11
https://doi.org/10.22492/ije.9.2.11...
). This was the most challenging task finding a solution for the projects. It was necessary to create a new real context for the projects, in which the elements of the team would always be geographically separated. These projects, implemented in a real context and in a team, would have to result in the implementation of Lean principles and concepts in the homes of each of the students since they were confined to their homes ( DINIS-CARVALHO; FERNANDES, 2017DINIS-CARVALHO, J.; FERNANDES, S. Applying lean concepts to teaching and learning in higher education: Findings from a pilot study. International Journal of Engineering Education, [s. l.], v. 33, n. 3, p. 1048-1059, Jan. 2017. ).

Finally, regarding to the third research question (What were the pedagogical responses given to overcome the COVID-19 educational demands and the lessons learned by the participants involved?), the results were very surprising in terms of the effectiveness of the changes implemented. Findings indicate that students were able to go beyond what was expected from them, by presenting clear and effective implementations of the solutions created in their own domestic environments (for example: applying standard work to the dish washer machine with improvement in performance; applying Kanban systems to control the stock of groceries; optimization of grass cutting; 5S in garages, visual management to the swimming pool machinery, and many others). Students were very satisfied with the results of the experience and considered learning to be more effective.

Findings from this study include not only the discussion of the effectiveness of the learning process, but more importantly, the practical benefits and lessons learned from this experience. One important achievement both for professors and students involved in this approach was the awareness of the diversity and flexibility of learning contexts that can be called upon to achieve and assess student learning outcomes. New, flexible and online spaces have emerged with the emergency remote learning and student engagement also requires more attention. This study showed that the learning outcomes of this course, despite the fact that the concepts and tools were originally oriented to companies and other organizations, it was also possible to apply them in a different context (like at home), with great effectiveness.

This work contributes to the developing body of knowledge about the effect of pandemic on engineering Education by investigating the challenges faced by engineering students and bringing forward the positive and practical benefits learned from this experience. Engineering Education and student learning can be enhanced through the creation of more flexible learning settings that consider student ownership as a key component of the learning process, enhanced by active learning approaches such as project and service-based learning.

References

  • ADARLO, G. M. Service-learning as global citizenship education: Acting locally on global challenges and concerns. IAFOR Journal of Education, [s. l.], v. 8, n. 3, p. 7-23, 2020. https://doi.org/10.22492/ije.8.3.01
    » https://doi.org/10.22492/ije.8.3.01
  • AHAG, P. et al. The impact of SARS-CoV-2 on engineering education: student perceptions from three countries. In: IEEE INTERNATIONAL CONFERENCE ON INDUSTRIAL ENGINEERING AND ENGINEERING MANAGEMENT, Singapore, 2020. p. 1266-1270.
  • ASGARI, S. et al. An observational study of engineering online education during the COVID-19 pandemic. Plos One, San Francisco, 2021 Apr;16(4):e0250041. https://doi.org/10.1371/journal.pone.0250041
    » https://doi.org/10.1371/journal.pone.0250041
  • ASSUNÇÃO FLORES, M.; GAGO, M. Teacher education in times of COVID-19 pandemic in Portugal: national, institutional and pedagogical responses. Journal of Education for Teaching, [s. l.], v. 46, n. 4, p. 507-516,July 2020. https://doi.org/10.1080/02607476.2020.1799709
    » https://doi.org/10.1080/02607476.2020.1799709
  • BOROCHOVICIUS, E.; CRISTINA, J.; TORTELLA, B. Aprendizagem baseada em problemas: um metodo de ensino-aprendizagem e suas práticas educativas. Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, v. 22, n. 83, p. 263-294, jun. 2014. https://doi.org/10.1590/S0104-40362014000200002
    » https://doi.org/10.1590/S0104-40362014000200002
  • BRAUN, J. M. Teaching lean Six Sigma with service-learning. Grand Rapids: [s. n.], 2013.
  • CASARES, M. G. Service learning and social inclusion. Barcelona: aprenentatgeservei, 2013. Available from: https://www.incorpora.org/documents/ 20181/134822/SERVICE_LEARNING_AND_SOCIAL_INCLUSION/ 037583d2-bebc-45dc-a235-bf5bc75ac501 Access in: 2020 Apr 20.
    » https://www.incorpora.org/documents/ 20181/134822/SERVICE_LEARNING_AND_SOCIAL_INCLUSION/ 037583d2-bebc-45dc-a235-bf5bc75ac501
  • CASTIONI, R. et al. Universidades federais na pandemia da Covid-19: acesso discente à internet e ensino remoto emergencial. Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, v. 29, n. 111, p. 399-419, jun. 2021. https://doi.org/10.1590/S0104-40362021002903108
    » https://doi.org/10.1590/S0104-40362021002903108
  • CAVALCANTI, L. M. R.; GUERRA, M. G. G. V. Os desafios da universidade pública pós-pandemia da Covid-19: o caso brasileiro. Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, v. 30, n. 114, p. 73-93, jan.-mar. 2022. https://doi.org/10.1590/S0104-40362021002903113
    » https://doi.org/10.1590/S0104-40362021002903113
  • CAYUELA, A.; ARAMBURUZABALA, P.; BALLESTEROS, C. Research report: a review of service-learning in European higher education. [S. l: Unisevitate; 2020.
  • CHIAPPE, A.; WILLS, A. E. Crowd-based Open Online Education as an alternative to the Covid-19 educational crisis. Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, v. 30, n. 114, p. 32-51, jan./mar. 2022. https://doi.org/10.1590/S0104-403620210002903341
    » https://doi.org/10.1590/S0104-403620210002903341
  • CUMBO, K. B.; VADEBONCOEUR, J. A. What are students learning?: assessing service learning and the curriculum. In: ANNUAL MEETING OF THE AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, 1998, San Diego, USA. Proceedings[…] . [S. l.: s. n.}, 1988
  • DINIS-CARVALHO, J.; FERNANDES, S. Applying lean concepts to teaching and learning in higher education: Findings from a pilot study. International Journal of Engineering Education, [s. l.], v. 33, n. 3, p. 1048-1059, Jan. 2017.
  • DONOSO-DÍAZ, S.; ITURRIETA, T. N.; TRAVERSO, G. D. Sistemas de Alerta Temprana para estudiantes en riesgo de abandono de la Educación Superior. Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, v. 26, n. 100, p. 944-967, jul./set. 2018. https://doi.org/10.1590/S0104-40362018002601494
    » https://doi.org/10.1590/S0104-40362018002601494
  • FELTEN, P.; CLAYTON, P. H. Service-learning. New Directions for Teaching and Learning, [s. l.], n. 128, p. 75-84, Dec. 2011. https://doi.org/10.1002/tl.470
    » https://doi.org/10.1002/tl.470
  • FERNANDES, S.; DINIS-CARVALHO, J. A service learning experience with engineering students. Guimarães: PAEE - Project Approaches in Engineering Education Association and Department of Production and Systems, School of Engineering of University of Minho, 2020. p. 227-236.
  • FOLGUEIRAS, P. et al. Service-learning: a survey of experiences in Spain. Education, Citizenship and Social Justice, [s. l.], v. 15, n. 2, Oct. 2018. https://doi.org/10.1177/174619791880385
    » https://doi.org/10.1177/174619791880385
  • GARCÍA-PEÑALVO, F. J. et al. La evaluación online en la educación superior en tiempos de la COVID-19. Education in the Knowledge Society, [s. l.], v. 21, 2020. https://doi.org/10.14201/eks.23086
    » https://doi.org/10.14201/eks.23086
  • GOMES, C. A. Escola de qualidade para todos revisitada: desfolhando as camadas da cebola. Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, v. 28, n. 109, p. 843-862, out./dez. 2020. https://doi.org/10.1590/S0104-40362020002802958
    » https://doi.org/10.1590/S0104-40362020002802958
  • GOMES, C. A. et al. Education during and after the pandemics. Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, v. 29, n. 112, p. 574-594, jul./set. 2021. https://doi.org/10.1590/S0104-40362021002903296
    » https://doi.org/10.1590/S0104-40362021002903296
  • HATTINGH, T. et al. Engineering student experiences of a remotely accessed, online learning environment. 2020 IFEES World Engineering Education Forum - Global Engineering Deans Council, WEEF-GEDC 2020, 2020. Proceedings[…]. [S. n. t.].
  • HIDAYATI, D.; SAPUTRA, W. A. Implementation of online learning during the covid-19 epidemic in Indonesia: assessment of higher education students’ use and implementation of online learning technology. Universal Journal of Educational Research, [s. l.], v. 8, n. 10, p. 4514-4519, 2020. https://doi.org/10.13189/ujer.2020.081019
    » https://doi.org/10.13189/ujer.2020.081019
  • HIRA, A.; ANDERSON, E. Motivating online learning through project-based learning during the 2020 COVID-19 pandemic. IAFOR Journal of Education, [s. l.], v. 9, n. 2, p. 93-110, 2021.
  • JOHNSTON, R. B. Arsenic and the 2030 Agenda for sustainable development. arsenic research and global sustainability. In: INTERNATIONAL CONGRESS ON ARSENIC IN THE ENVIRONMENT, 6., Stockholm, 2016. Proceedings[…], 2016. [S. l.]: CRC, 2020. p. 12-14.
  • KAUR, R.; GARG, A.; KAUR, P. Case study: Student’s response towards online learning in engineering education during COVID-19 pandemic. Journal of Engineering Education Transformations, [s. l.], v. 34, n. 3, p. 62-69, 2021. https://doi.org/10.16920/jeet/2021/v34i3/153917
    » https://doi.org/10.16920/jeet/2021/v34i3/153917
  • KHAN, Z. H.; ABID, M. I. Distance learning in engineering education: Challenges and opportunities during COVID-19 pandemic crisis in Pakistan. International Journal of Electrical Engineering Education, [s. l.], 2021. https://doi.org/10.1177/0020720920988493
    » https://doi.org/10.1177/0020720920988493
  • LEVESQUE-BRISTOL, C.; KNAPP, T.; FISHER, B. The effectiveness of service-learning: it’s not always what you think. Journal of Experiential Education,[s. l.], v. 33, n. 3, p. 208-224, mar. 2011. https://doi.org/10.1177/105382590113300302
    » https://doi.org/10.1177/105382590113300302
  • LIMA, R. M. et al. Ten years of Project-Based Learning (PBL) in Industrial Engineering and Management at the University of Minho. In: GUERRA, A.; ULSETH, R.; KOLMOS, A. (org.). PBL in engineering education. Rotterdam: SensePublishers, 2017. p. 33-51.
  • LIU, Y. et al. Hands-on engineering courses in the COVID-19 pandemic: adapting medical device design for remote learning. Physical and Engineering Sciences in Medicine, [s. l.], v. 44, n. 1, p. 195-200, Mar. 2021. https://doi.org/10.1007/s13246-020-00967-z
    » https://doi.org/10.1007/s13246-020-00967-z
  • MU, H. et al. Discussion on “online hybrid” teaching of engineering drawing course under the background of epidemic situation. In: INTERNATIONAL CONFERENCE ON EDUCATIONAL AND INFORMATIION TECHNOLOGY (ICEIT), 10., 2021. [S ..l: s. n.], 2021. p. 76-82.
  • MURMAN, E.; BAKST, J. Growing a Lean community: the Jefferson County WA Lean Journey. [S. l.: s. n.], 2017.
  • MURRAY, P.; MA, S. The promise of Lean experimentation. StanfordSocial Innovation Review, [s. l.], v. 13, n. 3, p. 24-39, Summer 2015. https://doi.org/10.48558/A24K-3949
    » https://doi.org/10.48558/A24K-3949
  • MUSA, A.; ALABI, A. G. F. Disruptive engineering and education in emerging economies: challenges and prospects. In: 2000 IFEES WORLD ENGINEERING EDUCATION FORUM – GLOBAL ENGINNERING DEANS COUNCIL, 2020. [S. l.: s. n.], 2020.
  • NIEBOER, R. A. Arthur Dunn: civic visionary from the Heartland. In: ANNUAL MEETING OF NATIONAL COUNCIL FOR THE SOCIAL STUDIES, San Antonio, Nov. 2000. Available from: https://archive.org/details/ERIC_ED448090/page/n1/mode/2up Access: 2020 Mar. 25.
    » https://archive.org/details/ERIC_ED448090/page/n1/mode/2up
  • OLIVEIRA, J. B. A. E; GOMES, M.; BARCELLOS, T. A Covid-19 e a volta às aulas: ouvindo as evidências. Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, v. 28, n. 108, p. 555-578, jul./set. 2020. https://doi.org/10.1590/S0104-40362020002802885
    » https://doi.org/10.1590/S0104-40362020002802885
  • OBSERVATÓRIO SOBRE RESPONSABILIDADE SOCIAL E INSTITUIÇÕES DE ENSINO SUPERIOR – ORSIES. Livro verde sobre responsabilidade social e instituições de ensino superior. [S. l.]: Press Forum, 2018. Available from: https://www.uc.pt/sustentabilidade/documents/Livro_Verde_versaoFINAL.pdf Access: 2023 Mar. 20.
    » https://www.uc.pt/sustentabilidade/documents/Livro_Verde_versaoFINAL.pdf
  • PURDUE UNIVERSITY. EPICS. Available from: https://engineering.purdue.edu/EPICS Access: 2020 Apr. 24.
    » https://engineering.purdue.edu/EPICS
  • SCHLEICHER, A. Teacher pofessionalism in the face of COVID-19. Paris: OECD, 2020.
  • SHAFIQ, M.; SORATANA, K. Lean readiness assessment model - a tool for Humanitarian Organizations’ social and economic sustainability. Journal of Humanitarian Logistics and Supply Chain Managementv, [s. l.], v.. 10, n. 2, p. 77-99, Apr. 2020. https://doi.org/10.1108/JHLSCM-01-2019-0002
    » https://doi.org/10.1108/JHLSCM-01-2019-0002
  • SMITH, E. K.; KAYA, E. Online university teaching at the time of COVID-19 (2020): an Australian perspective. IAFOR Journal of Education, [s. l.], v. 9, n. 2, p. 183-200, 2021. https://doi.org/10.22492/ije.9.2.11
    » https://doi.org/10.22492/ije.9.2.11
  • SOUTHERN regional education board. In: ATLANTA SERVICE-LEARNING CONFERENCE REPORT, 10., 1970. Available from: https://digitalcommons.unomaha.edu/slceproceedings/10/ Access: 2020 Mar 25. 2020.
    » https://digitalcommons.unomaha.edu/slceproceedings/10/
  • TANG, T. et al. Efficiency of flipped classroom with online-based teaching under COVID-19. Interactive Learning Environments, London, Sep. 2020. https://doi.org/10.1080/10494820.2020.1817761
    » https://doi.org/10.1080/10494820.2020.1817761
  • VON HAUFF, M.; NGUYEN, T. Universities as potential actors for sustainable development. Sustainability, Basel, v. 6, n. 5, p. 3043-3063, 2014. https://doi.org/10.3390/su6053043
    » https://doi.org/10.3390/su6053043
  • WOMACK, J.; JONES, D. Lean thinking: banish waste and create wealth in your corporation. New York: Fee Press, 1996.
  • XU, Y.; TANG, Q. The reform of modern education during the COVID-19 pandemic. Journal of Physics: Conference Series, [s. l.], v. 1748, 2021. https://doi.org/10.1088/1742-6596/1748/4/042051
    » https://doi.org/10.1088/1742-6596/1748/4/042051

Data availability

Data: The dataset used in this manuscript is available in the Ensaio Scielo open data repository and can be accessed through the DOI: https://doi.org/10.48331/scielodata.GDMOF6

Publication Dates

  • Publication in this collection
    16 June 2023
  • Date of issue
    2023

History

  • Received
    12 Mar 2022
  • Accepted
    24 Mar 2023
Fundação CESGRANRIO Revista Ensaio, Rua Santa Alexandrina 1011, Rio Comprido, 20261-903 , Rio de Janeiro - RJ - Brasil, Tel.: + 55 21 2103 9600 - Rio de Janeiro - RJ - Brazil
E-mail: ensaio@cesgranrio.org.br