Assessment of the quality of digital services provided by an e-learning platform focused on sustainability issues

Abstract

Paper aims

The aim was to adapt and apply a SERVQUAL scale to measure the perceived service quality of an e-learning platform provided by the United Nations for Environment Programme (UNEP) focused on sustainability issues.

Originality

There is a scarce number of publications using the SERVQUAL for digital services evaluation. This study contributes to filling this gap, generating new insights for the service quality literature by proposing an adaptation of the SERVQUAL scale for online sustainability courses.

Research method

The SERVQUAL was carried out with undergraduate and post-graduate students enrolled at a Brazilian higher education institution. A set of 21 questions was set up to evaluate the perceived service quality of the students before/after finishing the online course “Introduction to Life Cycle Thinking (LCT)” provided by UNEP.

Main findings

The results showed that the most important features were related to the empathy, assurance and tangibility quality dimensions with emphasis on the high quality of addressing the user doubts and the audio and video resources that helped students in the learning process. In conclusion, the perceived quality was higher than the expectations for many of the SERVQUAL dimensions.

Implications for theory and practice

Finally, the step-by-step methodological approach used by this paper should be adopted by other e-learning platforms to investigate quality and service management aspects. Empathy, assurance and tangibility may be seen as key components in the service quality management of e-learning platforms.

Keywords
SERVQUAL; Service quality; Gap analysis; Sustainability

1. Introduction

For a long time, the economy of nations depended mainly on the productivity of the First and Second Sectors, i.e. Agriculture and Industry, respectively. However, the Third Sector has emerged in the last century with significant advances in the economy and employment worldwide. According to Rocha et al. (2019), the global economic and financial crisis has a great influence on this, since the demand and investments for the production of tangible goods has been substituted gradually by intangible goods (services). In the new era of Sharing Economies, services providers become crucial, especially regarding digital services (Ferreira et al., 2016).

Currently, it is difficult to think of an activity in which the provision of any type of service is not included. While a product is a set of attributes that result in something physical and tangible, a service is more associated with the value of an activity and the experience of a customer, therefore being interpreted as something more intangible (Kotler & Keller, 2012). Hotel stays, gyms, online shopping, or even essential services such as public transportation or water and electricity supplies, are typical examples of intangible products. Despite being an economic activity, the sale is not linked to the ownership of a good.

With a Gross Domestic Product (GDP) of US$ 1.445 trillion and a population of 212.6 million inhabitants, Brazil is the seventh-largest economy in the world. The Services Sector is the main contributor to the country's employability and GDP, accounting for approximately 72.8% of the Accumulated Added Value in 2020, according to the Annual Service Survey (PAS), conducted by the Brazilian Institute of Geography and Statistics (Instituto Brasileiro de Geografia e Estatística, 2020).

Although the Brazilian growth trend and investments in the Service area does not follow the same speed as observed in developed countries, where this sector represents almost 80% of the Accumulated Added Value, the importance of the tertiary sector as a whole is evident. Services are fundamental inputs for any modern economy and play an important role in spreading innovations and productivity gains to other related sectors (Organização para a Cooperação e Desenvolvimento Econômico, 2018). The PAS document (Instituto Brasileiro de Geografia e Estatística, 2020) was divided into seven categories and the one that draws the most attention is the Services Mainly Provided to Families, which includes the provision of personal services, accommodation, meals, continuing and regular education, and cultural activities. Despite occupying the second position in the ranking of employed people in the country, this category occupies the fourth position in the ranking of the Net Operating Income in Brazil.

Quality management of manufactured goods can be easily handled to detect flaws or breakdowns in parts of a batch that affects the integrity of a product making it out of quality standards. On the other hand, Las Casas (2012) suggests that as the services are more intangible and are present in any commercial offers, what is meaningful is the degree of the provision in the services included in any commercial exchanges. Thus, customer satisfaction becomes more subjective and ineffective, given the lack of standardization.

The quality of a service is based on a general opinion of the customers about a service, consisting of a series of successful or unsuccessful experiences that took place during the delivery of their service, resulting in which the consumer looks for signs or evidence of quality, the perceived quality. According to Stylidis et al. (2020) from the engineering point of view, the perceived quality domain is a place where the product's meaning, form, sensorial properties, and their execution intersect with human experience. For that reason, perceived quality must be controlled during all stages of a service development (Kotler & Keller, 2012).

Intangibility has been recognized as being the key in distinguishing services from products (Zeithaml & Bitner, 1996; Zeithaml et al., 1985). Although essentially intangible, every service may have several tangible and perceptible elements associated. In a gym, for example, the physical environment, the facilities, and the professionals are tangible examples. On the other hand, in an online course module, tangibility elements could be attributed to features in the virtual platform on which the course is made available, such as video and audio performances.

The concept of the intangibility/tangibility in services is further discussed to explore how these two components function as “tangibility” in the minds of consumers (Khang et al., 2014). Chang et al. (2019) and Santos (2002) highlighted the “tangibilization” as a relevant aspect capable of minimizing subjectivity in service quality management. Virtually all service execution involves physical components such as consumables and durables. That can be taken as an indication of a certain quality of service because it could be seen, tasted, and touched (Santos, 2002). In order to properly deal with this issue, the SERVQUAL scale considers tangibility as one of its main quality aspects to be measured (Mano et al., 2013).

The SERVQUAL scale is widely used and important to evaluate service quality (Sumi & Kabir, 2021; Figueiredo et al., 2020). The SERVQUAL can be organized in a model with various criteria (attributes) for analyzing where a quality gap exists and is measured as the difference between customer expectations and customer perceptions (Alam & Mondal, 2019). The first SERVQUAL scale proposed by Parasuraman et al. (1985) identified ten dimensions: reliability, tangibles, responsiveness, credibility, courtesy, competency, security, communication, access, and understanding the customer. Parasuraman et al. (1988), however, highlighted five main dimensions: tangibility, responsiveness, reliability, empathy, and assurance, and Choudhury (2015) explains this template is the most used and adopted version of the SERVQUAL since then.

It is acknowledged that the quality perception is nothing more than the experience of the consumers, based on their expectations. According to Parasuraman et al. (1985) and Kotler & Keller (2012), the expectations are influenced by their personal needs, word of mouth communication, previous shopping experience, advice from acquaintances, information from the service provider, and competitors, as influencing factors. The provision of a service can still be classified by its typology, so this enables the fulfillment of wishes and needs more kindly and effectively (Souza et al., 2016). Based on the Typology of Tukker (2004), the limits between a pure service and a pure product are shown in Figure 1.

Figure 1
Typology of services by Tukker’s classification. Source: Tukker apud Souza et al. (2016).

In summary, Souza et al. (2016) explained the content of service in three ways: Product-Oriented, User-Oriented, Result-Oriented. A product-oriented service refers to service as a factor capable of adding value to the customer, such as offering an exclusive e-book when purchasing a course, for example. User-oriented services include the sale of products whose ownership is not transferred to the customer, even if a user fully uses it, as with the purchase and attending online courses. Finally, result-oriented services are the sale of expertise and are usually linked to professional services, since there is no predefined product such as recovering or mentoring classes, in which the result is variable and slightly uncertain for Souza et al., (2016).

The expansion of the type of services overtime is evident in the context of electronic services (or e-services), accounting for electronic commerce, customer support and the provision of online services (Sumi & Kabir, 2021)), that can be classified as professional services in Figure 1. E-services are characterized by the customer's interaction being mediated by information technologies in which the experience occurs using only two senses, vision and hearing (Rowley, 2006). The key to success in digital services, as in traditional services, lies in the evaluation of the service quality of electronic services (Shankar & Datta, 2020), as according to Kaya et al. (2019), this also affects customer satisfaction and loyalty.

During the COVID-19 pandemic, the participation of information technologies has gained momentum, and the education sector is no exception. From 2020 onward, the outbreak of COVID-19 forced lockdowns across the globe (Sumi & Kabir, 2021). Previously, e-learning, education and correspondence courses were popularly considered as the part of non-formal education, but today, it seems to gradually replace the formal education system if the circumstances enduringly persist in the short-terms (Mishra et al, 2020). At the University level, for example, face-to-face classes were abruptly transferred to virtual/remote environments, which modified teaching and learning routines and experiences (Silva et al, 2021).

Some studies have examined the measurement of service quality in different e-service environments including: retailing (Collier & Bienstock, 2006), shopping (Yang et al., 2004), travel (Ho & Lee, 2007), financial (Sohn & Tadisina, 2008), telecom (Zhou et al., 2019), information technology (Farrapo Junior et al., 2019), government (Stiglingh, 2014), hospital (Lubis & Atin, 2020), universities (Gonzalez Aleu et al., 2021, Ortega-Mohedano et al., 2018), academic libraries (Trivedi et al., 2021), public transportation (Tumsekcali et al., 2021). In these environments, the educational context is very broad and includes e-learning which, according to Udo et al. (2011), requires a corresponding change in its evaluation process and methods, since the traditional service quality evaluation process may be inadequate and not complete. This highlights the relevance of more research efforts exploring the use of the SERVQUAL scale to measure service quality in an e-learning environment.

As discussed until here, the application of the SERVQUAL scale for the evaluation of professional services has been, so far, low explored in the literature, especially for the case of e-services provision. In this sense, the current paper aims to fill this gap by providing a SERVQUAL adapted structure and its application to measure the perceived service quality of an e-learning platform provided by the United Nations Environment Programme (UNEP) focused on sustainability issues.

The novelty of this paper lies in the evaluation of online sustainability courses, since to the best of our knowledge, there are no previous papers on the topic yet. The results achieved could be used to enhance knowledge and feedback mechanisms to develop high-quality oriented e-services in this topic.

2. Theoretical foundation

A brief literature review was carried out to understand the state of the art on the research topic.

Initially, was to define search criteria which were defined considering publications until 2020. The database chosen for the article search was Scopus (Elsevier), considering all document types and the presence of relevant keywords in the article title and, abstract, and/or keywords.

In order to observe the recurrence of the topics covered in the bibliographical survey, this first phase included the following keywords: Life Cycle Thinking, SERVQUAL, Gap Analysis, and Students and Education, which will be better explained in the next paragraphs.

In this way, with the aforementioned search terms and the collection of articles for a brief graphic analysis, showing the evolution of publications over time as well as a content analysis to better support the development of the adaptation of the SERVQUAL instrument. The findings from this step can be summarized below.

“Life Cycle Thinking”: 668 records (Figure 2) presented the term in their titles, abstracts or keywords since 1981. However, the term started to have greater relevance from the year 2013 only. The cumulative percentage shows that 50% of everything that has been produced so far on this issue was covered in the last 6 years. In addition, an increasing number of publications on this topic was observed from 1981 to 2020 in the Scopus database.

Figure 2
Articles about Life Cycle Thinking in the Scopus database. Source: The authors.

“SERVQUAL” AND “Gap Analysis”: the combination of these terms resulted in 1,122 articles (Figure 3), from 1990 until today. As in the previous term, half of the articles were published from 2012 to 2020 and there is a growing trend for more publications in the next few years.

Figure 3
Publications about SERVQUAL and Gap Analysis in the Scopus database. Source: The authors.

“SERVQUAL” AND “Gap Analysis” AND “Students” OR “Education”: Since this research aims to assess the quality of an online course (i.e., a e-service), the words referring to students and education were added, in order to select articles in which students served as an analysis group whose Gap Analysis was applied to improve education provision. In total, only 27 publications (Figure 4) were found and the projection of future articles on this topic is considered very modest, confirming that this can be seen as an emerging area for research. No publication was found about applications in sustainability courses evaluations.

Figure 4
Publications referring to SERVQUAL, Gap Analysis, and Students or Education in the Scopus database. Source: The authors.

The service quality measurement instrument recurrently in the cited articles was the SERVQUAL. It was developed by Parasuraman et al. (1985, 1988) and Parasuraman et al. (1991) and consists of measuring the quality of services by five dimensions: reliability, responsiveness, assurance, empathy and tangibility.

Reliability involves the ability to deliver the promised service fairly and accurately; responsiveness is related to the willingness to help customers and provide prompt service; assurance refers to the knowledge and courtesy of employees and their ability to convey trust and confidence to customers; empathy deals with individualized attention and care to clients; and tangibility involves the appearance of physical facilities, equipment, employees and communication materials (Parasuraman et al., 1988).

The first four dimensions of SERVQUAL relate to human properties, while the last one denotes the relationship with the appearance of employees and the internal atmosphere of the facilities (Roy et al., 2015).

Regarding its structure, SERVQUAL consists of 22 items distributed over the five dimensions of quality and replicated in two parts, one for customer expectations about the services offered and the other for perceptions of the services provided. In addition, he originally uses a 7-point Likert scale ranging from strongly disagree (1) to strongly agree (7).

An adaptation of SERVQUAL for a distance education environment, a purpose in line with the present research, was performed by Bentancourt (2015) as shown in Table 1.

Table 1
SERVQUAL items adapted in Bentancourt (2015) for distance education environment.

DIMENSION	ITEM	QUESTIONS
Tangibility	1	The appearance of the virtual learning environment (VLE) is modern.
	2	Information is easy to find on the VLE.
	3	The teaching material looks nice.
	4	The VLE is generally attractive.
Reliability	5	Events announced on the VLE actually take place.
	6	My requests are provided with a demonstration of sincerity.
	7	I get correct information the first time.
	8	I find information easily.
	9	The information available is correct.
Responsiveness	10	The information is made available within the announced deadline.
	11	My requests are responded to promptly.
	12	I perceive that there is interest in fulfilling my requests.
	13	I find available means to obtain the information.
Assurance	14	The information system inspires confidence.
	15	I feel secure in using the VLE resources.
	16	My requests are dealt with cordially.
	17	There is information needed to answer my questions.
Empathy	18	I get individualized attention.
	19	I can access the course environment at any time.
	20	I can count on personalized service.
	21	Information services are provided responsibly.
	22	My real needs are understood.

• Source: Bentancourt (2015, p. 60).

From Table 1, it is observed that the adaptation by Bentancourt (2015) maintained the five original dimensions of SERVQUAL as well as their items.

3. Methodology

This research characterized the service provided by the evaluated e-learning platform as user-oriented, according to Figure 1. Targeting to identify the specific characteristics inherent to a group and to evaluate different dimensions and behaviors, this paper classifies the quality of the service provided along with the resulting experience through an e-learning platform. Thereby, to accomplish the proposed objective, the research method used to carry out this research was the survey, which was carried out in accordance with the recommendations of Forza (2002) with an exploratory and descriptive approach, as an initial bibliographic survey delivered the variables that were incorporated into a questionnaire, the which was applied to the target population, to characterize it in relation to the dimensions of quality in services derived from the SERVQUAL instrument. The following subsections detail the application of this research method.

3.1. Object under investigation

The United Nations Environment Program (UNEP), also known as the UN environment, is responsible for coordinating responses to environmental issues within the United Nations System. Among them is the achievement of the Sustainable Development Goals (SDGs), which are on the 2030 agenda and include 17 interconnected global goals focused on achieving a better and more sustainable future for all (United Nations Environment Programme, 2021).

In particular, ODS 12, entitled “responsible consumption and production”, establishes that production and consumption practices respect the biophysical capacity of the planet. One approach that can contribute to achieving this goal is Life Cycle Thinking (LCT), which according to Farjana et al. (2021) is a way of thinking that aims to reduce or eliminate the environmental, social and economic consequences of a product or process throughout the stages of its life cycle. Taking that into account, UNEP provides an online e-learning course composed of four modules about LCT, whose quality was assessed in this study using the SERVQUAL scale. In the end, the goal was to generate new insights and quality requirements for future research in the topic of e-services focused on sustainability issues.

The LCT online course is free of charge and can be found in Portuguese language at the UNEP’s webpage (United Nations Environment Programme, 2022), in the following link: https://www.learnlifecycle.com/courses/Introducao-ao-Pensamento-do-Ciclo-de-Vida. The course covers 36 lessons organized in the following modules:

• An overarching introduction to LCT: module 1 is composed of six blocks explaining why LCT is important and its meaningful impacts in terms of tools, criteria and potential applications. In total, there are three videos (29 min), a self-check quiz for students and additional resources for download.

• LCT applications in public policies: module 2 is a drive change in public policies about sustainable public policy making, organized in three main lectures totaling 26 min of videos. Materials for download are available with free access to them and a self-check quiz is provided at the end of this module.

• Applications in businesses: module 3 is a drive change in business focused on LCT applications in businesses. There are three main lectures dedicated to: (1) linking sustainability to the value chain (7 min), (2) how to make sustainability work within organizations (11 min), and (3) communicating LCT results (5 min). Each lecture has a video and downloadable materials for students, followed by a quiz at the end of this module for progress evaluation in the course.

• Sustainable consumption and lifestyles: module 4 is a block focused on sustainable lifestyles topics, as follows: lecture 1: the use of LCT in consumer information (10 min); lecture 2: labeling, declarations, and certified qualifications towards sustainable consumption (9 min); and Lecture 3: sustainable consumption decisions and lifestyles (10 min). A self-check quiz ends this module and the course as a whole.

3.2. Research instrument

The research instrument was based on the original SERVQUAL instrument and Bentancourt (2015) adapted instrument.

The applied questionnaire has 21 questions on aspects related to the provision of the service offered by the e-learning platform of UNEP, in addition to a set of demographic questions. The answers should follow a Likert scale model, in which the scores vary between 1 and 5, which mean (1) “I consider it unimportant” to (5) “I consider it very important” for expectation analysis, and “I totally disagree or totally agree” for perception analysis. The questionnaire with the statements is in Table 2.

Table 2
Questionnaire used for the investigation of Expectations and Perceptions.

DIMENSION	ITEM	QUESTIONS
Tangibility	1	The use of the platform is intuitive
	2	The audio features are satisfactory
	3	Video features are satisfactory
	4	The platform supports questions
	5	Content must be clearly available within the platform
	6	Information about the course on the platform is easy to detect
	7	The teaching material has a pleasant and modern appearance
	8	The platform must offer certification after completing the online course
Reliability	9	The modules comply with the proposed topic
	10	Information is clearly communicated throughout the course
	11	Supporting materials (links, articles) are relevant for in-depth understanding
	12	The material offered is sufficient to understand each module’s content
	13	The content evaluation process must be consistent with the content taught
	14	The idealization of the course (institution, team of people) must be reliable
	15	The data and information must be correct and free of any types of errors (grammatical, typing, conceptual)
Responsiveness	16	Access to the to the platform is quick (login / register)
Assurance	17	Access to the platform is secure (no risk of hacking, loss of access data, content breaches, etc.)
	18	There are a variety of languages available for the course
	19	Questions about the course in general are answered (i.e, FAQ)
Empathy	20	The course should offer feedback on the questionnaires answered in the evaluation process
	21	The course should offer the possibility to make suggestions for improvement on each module

• Source: The authors.

The questionnaire was made available online using google forms tool in two sections (Expectations and Perceptions).

3.3. Data collection

The population comprised two groups of students from the Federal University of São Carlos in Brazil. The completion of the form was optional, emphasizing that only those who intended to take the course offered by UNEP were invited to respond. The first group corresponded to 60 students in the discipline of Special Topics in Production Management offered in the second semester of 2019 for the undergraduate course in Bachelor’s Production Engineering. The second group of 60 students corresponded to students in the 2020 classes during the MBA in Environmental Management and Sustainability. The questionnaire was applied between July and August 2020 for these two groups.

The sample comprised the responding students from these two groups, with a total of 54 responses to the first section (Expectations), and 48 responses for the section about the quality Perception after completing the online course. After the withdrawal of respondents in only one of the two sections and duplicates, a final sample of 43 students was considered, of which 21 represented undergraduate students and 22 those from the MBA students.

It is worth mentioning that the LCT online course was made available into Brazilian Portuguese language only in the first semester of 2020. Thus, the first sample of students accessed the content offered in English, Spanish, French, or Arabic, while the second group attended the course made available in Portuguese. This was the main reason why demographic questions were added to the analysis of the study. This section of questions provided inferences about some personal characteristics of the sample of people, such as gender and age, and personal skills in English, Spanish or other languages. Given that the first group was formed by students of the same age range, with an average of 23 years, the demographic questions assessed only the level of understanding of the English, Spanish and/or other languages and highlighted the gender contribution of the respondent’s data.

Due to the pandemic caused by the COVID-19, the second group was in social isolation and, because of that, the authors decided to explore a little bit more the characteristics of these individuals in a section called Behavior, applied only to these respondents (MBA students). The following descriptions are all about the demographic and behavioral issues covered in Table 3.

Table 3
Description of demographic and behavioral issues.

Sample Group	ITEM	QUESTIONS
Undergraduate students (2019)	1	Which gender do you identify with? (male/female/other)
	2	What is your level of understanding in the English language?
	3	What is your level of understanding in the Spanish language?
MBA Students (2020)	1	Which gender do you identify with? (male/female/other)
	2	What is your level of understanding in the English language?
	3	What is your level of understanding in the Spanish language?
	4	Is this the first time you have taken an online course?
	5	Compared to the period before the spread of the coronavirus, select the alternatives that best fit your reality. During the COVID-19 Pandemic:
		Q1 - The time I was online increased by up to 50%
		Q2 - The time I stay connected more than doubled
		Q3 - I consumed more online services in the purchasing sector (e-commerce, delivery)
		Q4 -I consumed online services in the education sector (online course modules, remote studies)
		Q5 - I consumed more online services for leisure (streaming movies and music, social networks, etc.)
		Q6 - My behavior was not affected in any way
	6	According to your response to the previous item regarding the COVID-19 pandemic, report below on the impact of digital services on your personal / professional life, including taking online courses like the one in question.

• Source: The authors.

3.4. Data analysis

After collecting all the answers obtained in the two sections (expectations and perceptions), the study proceeded to the Gap analysis. This analysis, according to Parasuraman et al. (1985), is part of SERVQUAL and helps to understand how the quality of a service is perceived by the customer when comparing what he expected to receive, expectations, with what he perceived, perceptions. In this way, each statement had its scores compared and the final score resulted from the difference between them (see Equation 1).

$Ga{p}_i=Perceptio{n}_i-Expectatio{n}_i;i=\mathrm{1..}n$ (1)

Depending on the results of this comparison - Equation 1, the perception of the service can be equal, better or worse than the expectations, implying a satisfactory perceived quality (gap = 0), ideal (gap > 0) or unacceptable (gap < 0), respectively (Parasuraman et al., 1985).

The data analysis involved: investigation of data distribution, analyzing asymmetry and kurtosis statistics; analysis of internal consistency of statements, analyzing Cronbach's alpha values; and, finally, the multivariate methods of factor analysis and cluster analysis, to investigate the relationship between the variables under study. The data analysis was considered Hair et al. (2009) and was supported by SPSS® 20.0 software.

The first statistical test performed refers to kurtosis and asymmetry. Kurtosis is an indicator of the concentration of the values of a distribution around its central point, ending up defining the degree of flattening of a curve. Asymmetry, in turn, demonstrates the frequency of asymmetric data in relation to the average, showing which scores were the most relevant to the respondents (Hair et al., 2009).

In the sequence, the Cronbach’s Alpha indicator determines that reliability must be analyzed to ensure the adequacy of each item in a questionnaire. An item is measured by the same concept or performance indicator and if the results are greater than 0.70 then the items were considered reliable (Taber, 2018).

After that, Factor analysis was performed. According to Hamerski & Abreu (2011) is a statistical method capable of reducing data complexity, therefore aiding in their interpretation, enabling the creation of new indicators, and even predicting responses regarding the degree of satisfaction of an online course, for example.

Finally, cluster analysis was carried out. According to Sott et al. (2020), it is possible to make a hierarchical grouping of variables based on a calculation of the distances between all items, so that the closer variables are those with the higher similarities, resulting in a quick visual analysis. The dendrogram analysis shows how the factors analyzed are related to each other in the questionnaire, with a focus on the quality Perceptions.

4. Results and discussion

Having presented all the factors and methods used, this section will demonstrate the analyses and results obtained during the evaluation of the quality of the service provided.

4.1. Overview of the responses

Initially, a simple and superficial analysis of the respondents was carried out, taking into account only the data taken from the questionnaires and the fact that they are minimally grouped (age range, level of education, and place of education).

Figures 5, 6, 7, and 8 show that if the online e-learning modules are not offered to the group of MBA students in the Portuguese version, perhaps the adherence and quality perception would be lower, since most students consider themselves at a basic or intermediary levels in foreign languages, despite being more aligned with the LCT topic, when compared to undergraduate students, given their expertise. On the other hand, language was not the limiting factor for the three respondents (Figure 6) who said never taken any type of online courses. Undergraduate students, in turn, usually took online mini-course modules as a complementary activity in several semesters during the Production Engineering bachelor, and, therefore, they were not asked about this point.

Figure 5
Gender Identity of the respondents. Source: The authors.

Figure 6
Familiarity with e-learning courses. Source: The authors.

Figure 7
English Knowledge. Source: The authors.

Figure 8
Spanish Knowledge. Source: The authors.

Another interesting issue to the group of MBA students compared with literature was the behavioral investigation during the quarantine period due to the COVID-19 pandemic. According to an article published by The New York Times (2020), between March and April 2020, there was a large increase in the access to platforms such as Zoom, Google Classroom, and Microsoft Teams, that enable them to perform home office and to attend online classes. According to the Global Web Index (2020), the consumption of online services was higher for individuals of generations Z and Y, people aged between 16-23 and 24-37 years, respectively, which is precisely the age groups investigated in this study for the two groups of people. The global index shows an increase in the period of connection and highlights an increase of approximately 50% in the consumption of online videos, and this result is in line with Figure 9, where approximately 40% of the MBA students confirmed this situation (Q1, Q2 and Q5).

Figure 9
Results about the Behavioral Section (responses from the MBA Students). Source: The authors.

In order to identify the behaviors generated by the online course under analysis, behavioral questions were asked to understand the answers of the students to this growing trend of online services, mainly in the education sector and the results are shown in Figure 9.

In this section respondents could check more than one answer and the results show a growth in time spent online during the first wave of COVID-19 pandemic. Based on the responses from question 5 from Table 2, it was possible to observe that, compared to the period before the pandemic and the time this survey was taken, there was an increase both in the time spent online (Q1 and Q2) and in the demand for this type of service for educational matters (Q4).

4.2. Statistical analysis of the responses

The first factor analyzed was the relationship between the aspects evaluated in each of the two samples of responses, expressed through dendrograms, which are tree diagrams showing clustered groups according to their levels of similarity. For this purpose, the questions answered by the two samples of students received scores allowing the observation of the relevance of each one, in Figure 10 and Figure 11.

Figure 10
Distribution of scores among the Expectation questions. Source: The authors.

Figure 11
Distribution of scores between the Perception questions. Source: The authors.

When comparing the results for Expectation and Perception questions, it is possible to identify some aspects based on the Likert scale. There is an apparent drop in Score 5 questions (very important/strongly agree) between Expectation and Perception. Further, few questions were evaluated as Score 1 (minor / strongly disagree), indicating there are many requirements when it comes to online courses, since it is already a very vast sector that is explored by students worldwide, resulting in a more intense search for quality. Regarding the remote education platforms already consolidated in the market, such as Coursera and Udemy, for example, essential factors are the offer of certifications and a pleasant interface. These are essential points to any student and, since the perceived quality also influences external factors, these considerations should be borne in mind when offering a similar service.

The data resulting from the evaluations were grouped in the following tree dendrograms (Figures 12 to 13), with the clustering pattern comparing the five quality dimensions of SERVQUAL scale. The dendrograms were built taking into account Perception and Expectation results.

Figure 12
Quality performance dimensions dendrogram. Source: The authors.

Figure 13
Dendrogram on the statement questions. Source: The authors.

This first dendrogram shows that Tangibility is the largest cluster among the quality aspects, standing out as the greatest engagement in this study. This indicates that questions 1 to 9 (Table 2) about audio, video, content resources, presentation, and certification were the easiest points to evaluate. As a result, they require a higher quality level, being less flexible. In addition, Assurance and Reliability have the same relevance and empathy is the result of these two. The Responsiveness factor varies only according to Tangibility.

In more detail, the clusters formed by the grouping of questions are presented in Figure 13.

This dendrogram groups the questions in Figure 13 in a way different from the quality aspects previously shown in Figure 12. If a line is drawn in 15 (x-axis), we have the definition of three large clusters, better described below:

• T9 - Item 9 of the Tangibility Dimension (see Table 03 again): issue with greater prominence (distant from the others) forming a cluster of 'Certification', that is, a key factor that deserves attention, because it is strongly desired by students;

• Groups S2, E1, S3, E2, and T4: statement S2 is highlighted in this cluster and refers to the variety of languages. The others refer to feedback, improvement, FAQ, and support questions. This cluster can be considered as 'Platform Support' and all these issues form a set of aspects with the same quality importance;

• Groups C4, T5, T8: indicates that the statements regarding the availability and effectiveness of the course materials, as well as their appearance (whether it is pleasant and modern) and ease of access in the platform, have the same importance and, therefore, can be classified as a cluster called ‘Platform Resources’. It is worth mentioning that these statements are almost equivalent to the S1 question, which measures secure access to the platform. All the other questions are inserted in this cluster, but with a little less relevance.

This dendrogram shows that questions C1, C2, R2, and T6 presented little relevance in the survey. They deal with the fulfillment of the proposed issue; information transmitted in a clear way; material consistent with the scope of the course; and content made available in a clear manner. This situation is further explored ahead, but the adherence of these statements was so low that they could even be excluded from the study analysis of results.

Inferences like these ones allow working on improvements that are also multivariate, taking into account the type of gap, the assessment of quality performance, and the dependence between factors in the evaluation process.

For data treatment, relevant responses that are delimited by -1 and 1 were considered relevant, and through the analysis performed with the help of the SPSS® 20.0 software, Tables 4 and 5 were built considering Expectation (EX) and Perception (PC), and the following points of attention were identified.

Table 4
Results for the sample of undergraduate students.

Question	Standard deviation	Asymmetry	Kurtosis
EX_6	0.4629	-1.023	-1.064
EX_10	0.5774	-1.595	1.895
EX_11	0.5118	-2.829	7.918
EX_14	0.3008	-2.975	7.562
EX_15	0.7303	-2.775	8.734
PC_06	0.8452	-1.166	1.180
PC_11	0.5071	-.0311	-2.115
PC_14	0.8106	-1.613	2.821
PC_17	0.8106	-1.613	2.821
PC_15	0.8309	-0.496	-1.364

• Source: The authors.

Table 5
Results for the sample of MBA students.

Question	Standard deviation	Asymmetry	Kurtosis
EX_6	0.4629	-1.023	-1.064
EX_10	0.5774	-1.595	1.895
EX_11	0.5118	-2.829	7.918
EX_14	0.3008	-2.975	7.562
EX_15	0.7303	-2.775	8.734
PC_06	0.8452	-1.166	1.180
PC_11	0.5071	-. 311	-2.115
PC_14	0.8106	-1.613	2.821
PC_17	0.8106	-1.613	2.821

• Source: The authors.

Regarding kurtosis, also considering results between -1 and 1, the responses were very similar to each other, reducing the quality of data analysis. In addition, questions regarding asymmetry outside the range of -1 and 1 indicate responses with very low or very high values, which may indicate low adherence of respondents to the question. Likely, these items were poorly formulated, creating doubts for the respondents or inducing them to a specific answer, in such a way the same questions presented discrepant results in both the Perception and Expectation sections. Since statistics are based on comparisons, these questions with low variability become less interesting and, therefore, will be discarded from the rest of the analysis.

The commitment of the respondents to the questionnaire can also be evaluated, and some participants may be excluded from the sample, if necessary. This analysis is done through the standard deviation of the answers. If the set presents values less than 0.5, it indicates little variation, that is, only 5 scores, for example, and this suggests an unrealistic assessment. In Table 6 are listed the students who will be withdrawn from the study due to low adherence to the questionnaire.

Table 6
Standard deviation of the answers of the respondents.

Graduation Students
Expectation		Perception
Student #	Standard deviation	Student #	Standard deviation
3	0	3	0.600
MBA Students
Expectation		Perception
Student #	Standard deviation	Student #	Standard deviation
5	0	5	0

• ^#
  - identification character number for the student's sample. Source: The authors.

There were only two unsatisfying results for this analysis. Student number 3 in the graduation group sample presented no variation in their expectation responses which may be read as lack of commitment to the questionnaire or too high requirements towards the online course. In the MBA sample, the same happened with student number 5.

In order to prove that the variables were reliable, Cronbach's Alpha was calculated in Table 7. Due to the values higher than 0.8, the results can be considered that all items in the questionnaire have a positive relationship with each other and, even if more questions were excluded, the rest would still be valid.

Table 7
Cronbach’s Alpha Analysis.

Reliability statistics - Graduation
Cronbach's Alpha	Cronbach's alpha based on standardized items	Nº of items
0.844	0.862	46
Reliability statistics - MBA
Cronbach's Alpha	Cronbach's alpha based on standardized items	Nº of items
0.882	0.904	45

• Source: The authors.

Therefore, based on these statistical results a final interpretation can be done in terms of the gap analysis based on the SERVQUAL scale interpretation in Section 4.3.

4.3. SERVQUAL analysis

Several authors like Bentancourt (2015) and Oliveira & Ferreira (2008) point out that a negative score indicates that perceptions are below expectations, showing which service generates an unsatisfactory result for the customers. On the other hand, a positive score indicates that a service presents a higher performance than expected. It is worth mentioning that when scoring negative differences between the expected service and the one actually obtained, a Gap type 05 is observed and the identification of all parameters influencing the service operation is essential for the quality improvement of the business and for consumer satisfaction as a whole (Large & König, 2009). Gap 05 is defined as the discrepancy between what the user expects to receive and the perception he has of the service offered (Parasuraman et al., 1985).

Considering the elimination of the questions and respondents highlighted in Table 5 and 6, respectively, based on the statistical results in Section 4.2, and applying the Equation 1, we have the following results in Tables 8 and 9.

Table 8
Gap analysis for the answers of the undergraduate students.

T								Rel							Resp	A			E
1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21
-0.73	-0.20	-0.60	-0.27	1.20	-	-0.40	-3.00	-	-	-	0.33	0.47	-	-	-0.07	-	0.20	-0.33	-1.47	-0.20
-0.57								0.11							-0.07	-0.07			-0.64
General mean								-0.25

• Source: The authors.

Table 9
Gap Analysis for the answers of the MBA students.

T								Rel							Resp	A			E
1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21
-0.27	-0.20	-0.07	-0.47	-	-0.53	-0.53	-0.20	-	-	-0.47	-	-	-0.07	-0.27	-0.27	-0.30	-1.40	-1.20	-0.40	-0.87
-0.28								-0.11							-0.27	-0.97			-0.63
General Mean								-0.45

• Source: The authors.

The factors with the greatest disparity for undergraduate students were related to Empathy (E) and Tangibility (T) (Table 8), while the MBA students put Assurance (A) and E as the most important factors (Table 9). The issues whose Expectations were greater than the Perceptions are highlighted in Table 10.

Table 10
Description of the issues with the greatest quality gaps.

Dimension	Question
Tangibility	The audio features are satisfactory
Assurance	There are a variety of languages ​​available for the course
	Questions about the course in general are answered (FAQ)
Empathy	The course should offer the possibility to make suggestions for improvement

• Source: The authors.

These factors can be linked to Gap Type 02 (Parasuraman et al, 1985), that is, a discrepancy between management perception of customer expectations and service quality specifications, and because this inconsistency exists between expected service and reality, there is also a Gap 05, which is the difference between what the customer expects and what is actually offered.

Such gaps are intrinsically related to the fact that the client associates the present experience with external factors, such as the consumption of similar services in the topic of providing online courses. As previously seen, undergraduate students are often exposed to the offer of online teaching modules, which indicates a kind of broad benchmarking. In addition, according to the Global Web Index (2020), the age range of these students indicates greater consumption of online resources, such as streaming and social networks. They are used to more fluid and slightly customized interfaces and, when faced with the simplicity offered by the UNEP platform, the students can feel slightly frustrated, which explains the lowest ratings. The empathy factor, in this case, is also intrinsically related to the platform itself, whose dissatisfaction refers to the improvement of tangibility and assurance as a whole.

MBA students highlighted assurance as the least satisfactory quality performance. This group was able to take the course in Portuguese language during 2020 year, but even so, it highlighted the language factor as below expectations. Another point was in relation to FAQ availability, which may indicate the lack of clarity regarding the objectives, available resources, course duration, among other basic specifications necessary for enrollment in any teaching online course. Another highlight should be given to Question 18 (see in Table 2), the material offered is consistent with what was promised in the scope, which was assigned a zero gap result, that is, the expectation was completely met by the service provider.

It seems that the quality assessment was quite positive, given that the general results were represented by low general means in Tables 8 and 9. In addition, despite the differences in age, education, and period of application of this study, both the evaluated groups of people achieved similar gap results, indicating a good level of satisfaction.

4.4 Closing remarks about SERVQUAL in educational environment

In Table 11 there are brief summaries of the results obtained by studies similar to this one, for further comparisons. A first observation is that no previous study was performed before for investigating e-service in sustainability topics. The publications in Table 11 were those extracted from the 27 publications in the topic SERVQUAL applications in educational systems, however, only 4 studies were considered closer to the LCT online e-learning course under investigation for the comparisons.

Table 11
Publications dealing with SERVQUAL analysis in the educational environment.

Publication	Results and Discussions - Brief Summary
Application of the SERVQUAL Scale in the undergraduate course in Agroindustrial Production Engineering. (Ferreira et al., 2017)	It was noted that academics value and have their perception of quality focused on tangible aspects, such as appearance of physical facilities, equipment, and hygiene of the places.
Online education in the Production Engineering course: the case of a Private University. (Bernini, 2008)	Tangibility was the aspect with the greatest disparity between expectations and perceptions in the gap analysis. Some factors such as location of the discipline URL, web navigation, appearance of the virtual environment, and resources available on the platform showed the greatest quality gaps.
SERVQUAL as an information quality management tool in the distance learning environment. (Bentancourt, 2015)	In this study, the Reliability and Tangibility aspects stand out as the most important aspects and the suggestion for improvement is linked to the availability of information and data within the promised deadline and a better use for chat resources.
Use of the SERVQUAL Scale adapted in the Assessment of Perception of Learning in Distance Education in an Higher Educational Institution. (Alves, 2011)	Two groups were submitted to evaluation. The first, had a concentration of scores 3 and 4 in the analysis of perceptions and expectations, indicating a provision of services in accordance with the expectations. The other group, of a more technical nature, scored lower values in the questions related to the online teaching environment (linked to tangibility issues). The empathy aspect, however, was the worst evaluated by both groups, a criticism in relation to the service that the student receives from teachers and course coordinators.

• Source: The authors.

Despite using the same quality assessment methodology (SERVQUAL scale), these studies presented their results a little differently than the present research. Therefore, the comparison between general averages is not very accurate. However, it is possible to highlight that the tangibility factor is the most repeated parameter during the quality assessment in educational systems, as expected, since it is the “most tangible” one and, therefore, the easiest to relate to personal desires and past experiences.

Also, the current literature can not be directly compared with the gap analysis on this paper because there was no published article in the topic of e-service and sustainability issues, for example. The current literature is more focused on e-services in education in general (Bentancourt, 2015) or more focused in specific courses not covering sustainability content (Bernini, 2008; Ferreira et al., 2017). Based on the gap analysis, some recommendations can be done for the LCT online course studied and the service provider as well (i.e., UNEP):

• According to the results, the hotspots to give more attention are empathy, tangibility and assurance in the SERVQUAL scale, allowing the client to have a more satisfactory experience based on some front-end reformulations using design-thinking principles, for example;

• To continue improving tangibility and assurance aspects, more investment in technology resources are needed, so UNEP needs to look for updating some relevant issues, such as: a more interactive and inviting appearance in the interface for each module. It is necessary to continue investing in language resources, making the course available in as many languages ​​as possible, including visual and audio resources available in the local language. And to enhance empathy, maybe an invitation call for the course attendees could be organized yearly by UNEP, as a more personal approach to contact the students and answer their questions and suggestions of improvement for the platform as a whole;

• There were some specific relevant comments from the student’s perceptions on the four modules of this course: many download materials (brochures, manuals, tools, etc.) are available at the end of each lecture from each course module, however, the number of available resources could be reduced to avoid excess of resources in the learning process. In addition, these download materials could be all translated to the local language as well because currently, most of them are all available only in English;

• As sustainability topics are necessary for modern society and are increasing fast in the last decades, more courses are desired in addition to this LCT online course. UNEP should continue to give attention to make this knowledge available for free and for more people across the world.

Recently Brazil approved the General Data Protection Law (GDPL) and its establishment may affect how web developers will provide their services online. As a law, there are now clear regulations that must be strictly followed and points regarding access to the platform and protection are probably the main factors that need to be reviewed and redesigned since data reliability is critical and is strongly related to business efficiency and competition (Serviço Federal de Processamento de Dados, 2020). Both the tangibility and assurance performance indicators should be enhanced, also in terms of a design thinking perspective such as UI (user interface) and UX (user experience), in which online satisfaction is defined by the ease of efficient learning and avoiding frustrations during its use. Therefore, the more personalized the user experience, the greater are the chances that a user will be positively surprised and satisfied (Nielsen, 1995).

5. Conclusions

The Third Sector is increasingly establishing itself as the main sector among the developed economies and the trend is that the supply of pure products decreases more and more. Quality analysis for all types of services must be disseminated among providers since the identification of their strengths and weaknesses is the easiest way to stand out among competitors. Further, the concept of shared economy, in which just one product is developed to fulfill a group of customers, continues to grow. With this in mind, this study aimed to adapt and apply a SERVQUAL scale to measure the perceived service quality of an e-learning platform provided by the United Nations for Environment Programme (UNEP) focused on sustainability issues.

For the last few years, educational institutions have been offering e-learning course modules, but in 2020, during the COVID-19 pandemic, the provision of virtual services increased significantly, especially the educational ones. Even after containing the effects of the pandemic, it is estimated that remote service solutions will continue to be offered following an increasing demand.

To understand the behavior of the students in response to remote activities, the evaluated students were asked about the impact of digital services on their personal and professional lives. Most students reported having increased their connection time during the pandemic and some reported that the impact on information consumption was very large, via social networks and online teaching platforms. Another highlight is the convenience offered by remote education, in addition to unique opportunities, such as the emergence of several courses that would face difficulties in being offered in-person, a barrier completely overcome by the digital services.

For the e-learning modules offered by UNEP, the case study for the SERVQUAL application revealed few suggestions for improvements. According to the results, as the hotspots were more focused on empathy, tangibility and assurance aspects, the platform must undergo some front-end reformulations. This solution requires an investment in technology resources, so that the platform receives an updated version, with a more interactive, inviting appearance and with clearer information regarding the characteristics of the course (FAQ). In addition, it is necessary to continue investing in language resources, making the course available in as many languages ​​as possible, especially because UNEP is a worldwide reference authority, which preaches for diversity and inclusion, among many other social values. In fact, the LCT online course proposed by UNEP is an attempt to spread some of these values involved in sustainable development. Raising awareness on this topic among the population is the initial step towards spreading knowledge about this issue. Then, as a suggestion to enhance empathy on this course, maybe an annual invitation call for the attendees could be organized by UNEP, as a more personal strategy to contact the students and receive their relevant feedback for improvements in the platform and the sustainability courses provided as a whole.

This study is not free of limitations, and the most one is a low number of respondents for the statistical and SERVQUAL application. However, as discussed, the statistics proved to be relevant for the small sample of responses analyzed, and also the results achieved were valuable as the step-by-step methodological approach used in this paper should be adopted by other e-learning platforms to investigate quality and service management aspects focused on sustainability issues.

Acknowledgements

This research was supported by PROEX grant number 23112.002880/2019-13. The authors are grateful by the responses of each student on this survey and thanks the UNEP Life Cycle Initiative’s team for the support on this research.

• How to cite this article: Delgado, B. C., Farrapo Júnior, A. C., Mergulhão, R. C., & Silva, D. A. L. (2022). Assessment of the quality of digital services provided by an e-learning platform focused on sustainability issues. Production, 32, e20210110. https://doi.org/10.1590/0103-6513.20210110

References

• Alam, M. S., & Mondal, M. (2019). Assessment of sanitation service quality in urban slums of Khulna city based on SERVQUAL and AHP model: a case study of railway slum, Khulna, Bangladesh. Journal of Urban Management, 8(1), 20-27.. http://dx.doi.org/10.1016/j.jum.2018.08.002
  » http://dx.doi.org/10.1016/j.jum.2018.08.002
• Alves, A. M. (2011). Utilização da escala SERVQUAL adaptada para a avaliação da percepção de aprendizagem em Educação à Distância (Dissertação de mestrado). Centro Estadual de Educação Tecnológica Paula Souza, São Paulo.
• Bentancourt, S. M. P. (2015). Servqual como instrumento de gestão da qualidade da informação em ambiente EaD. (Dissertação de mestrado). Universidade Federal de Santa Catarina, Florianópolis.
• Bernini, D. S. D. (2008). Educação online no curso de engenharia de produção: uma universidade privada (Dissertação de mestrado). Universidade Estadual do Norte Fluminense, Campos dos Goytacazes.
• Chang, K.-C., Hsu, Y.-T., Hsu, C.-L., & Sung, Y.-K. (2019). Effect of tangibilization cues on consumer purchase intention in the social media context: regulatory focus perspective and the moderating role of perceived trust. Telematics and Informatics, 44, 101265. http://dx.doi.org/10.1016/j.tele.2019.101265
  » http://dx.doi.org/10.1016/j.tele.2019.101265
• Choudhury, K. (2015). Evaluating customer-perceived service quality in business management education in India: a study in topsis modeling. Asia Pacific Journal of Marketing and Logistics, 27(2), 208-225. https://doi.org/10.1108/APJML-04-2014-0065
  » https://doi.org/10.1108/APJML-04-2014-0065
• Collier, J. E., & Bienstock, C. C. (2006). Measuring service quality in e-retailing. Journal of Service Research, 8(3), 260-275. http://dx.doi.org/10.1177/1094670505278867
  » http://dx.doi.org/10.1177/1094670505278867
• Farjana, S. H., Mahmud, M. P., & Huda, N. (2021). Life cycle assessment for sustainable mining USA: Elsevier.
• Farrapo Junior, A. C., Mergulhão, R. C., Piato, E. L., Silva, J. E. A. R., & Pimenta, M. L. (2019). Quality assessment of Information Technology support services in a Federal Higher Education Institution. Práticas Em Gestão Pública Universitária, 3(2), 31-57. Retrieved in 2021, September 21, from https://revistas.ufrj.br/index.php/pgpu/article/view/29523/19842
  » https://revistas.ufrj.br/index.php/pgpu/article/view/29523/19842
• Ferreira, J. L., Ferreira, L. E. M., Junior, P. D. S., Oliveira, C. C., & Rocha, R. P. (2017). Aplicação da Escala SERVQUAL no curso de graduação em Engenharia de Produção. In Anais do VII Congresso Brasileiro de Engenharia de Produção (pp. 12).
• Ferreira, K. M., Méxas, M. P., Abreu, W., & Mello, P. (2016). Economia compartilhada e consumo colaborativo: uma revisão da literatura. In Anais do Congresso Nacional de Excelência em Gestão: INOVARSE (Vol. 29). Rio de Janeiro: Universidade Federal Fluminense.
• Figueiredo, R. A., Mergulhão, R. C., Prancic, E., & Mendes, G. H. S. (2020). Impacto de intervenções na qualidade dos serviços de uma biblioteca universitária. Ciência da Informação, 49(1), 48-65.
• Forza, C. (2002). Survey research in operations management: a process-based perspective. International Journal of Operations & Production Management, 22(2), 152-194. http://dx.doi.org/10.1108/01443570210414310
  » http://dx.doi.org/10.1108/01443570210414310
• Global Web Index. (2020). Coronavirus Research. April 2020 Series 4: Media Consumption and Sport (28 p.).
• Gonzalez Aleu, F., Granda Gutierrez, E. M. A., Garza-Reyes, J. A., Garza Villegas, J. B., & Vazquez Hernandez, J. (2021). Increasing service quality at a university: a continuous improvement project. Quality Assurance in Education, 29(2/3), 209-224. http://dx.doi.org/10.1108/QAE-02-2021-0020
  » http://dx.doi.org/10.1108/QAE-02-2021-0020
• Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2009). Análise multivariada de dados Porto Alegre: Bookman Editora.
• Hamerski, H., & Abreu, J. A. P. (2011). O uso de ferramentas estatísticas para avaliação da aplicabilidade do modelo SERVQUAL. In Anais do XXXI Encontro Nacional de Engenharia de Produção Belo Horizonte, MG.
• Ho, C. I., & Lee, Y. L. (2007). The development of an e-travel service quality scale. Tourism Management, 28(6), 1434-1449. http://dx.doi.org/10.1016/j.tourman.2006.12.002
  » http://dx.doi.org/10.1016/j.tourman.2006.12.002
• Instituto Brasileiro de Geografia e Estatística – IBGE. (2020). Pesquisa Anual de Serviços Rio de Janeiro: IBGE. Retrieved in 2021, September 21, from https://www.ibge.gov.br/
  » https://www.ibge.gov.br/
• Kaya, B., Behravesh, E., Abubakar, A. M., Kaya, O. S., & Orús, C. (2019). The moderating role of website familiarity in the relationships between e-service quality, e-satisfaction and e-loyalty. Journal of Internet Commerce, 18(4), 369-394. http://dx.doi.org/10.1080/15332861.2019.1668658
  » http://dx.doi.org/10.1080/15332861.2019.1668658
• Khang, J., Yu, Y.-M., & Lee, H.-H. (2014). Moderating effects of the fit between service tangibilization and organizational performance. Service Business, 8(2), 239-266. https://doi.org/10.1007/s11628-013-0195-9
  » https://doi.org/10.1007/s11628-013-0195-9
• Kotler, P., & Keller, K. L. (2012). Administração de marketing (14. ed.). São Paulo: Pearson.
• Large, R. O., & König, T. (2009). A gap model of purchasing’s internal service quality: Concept, case study and internal survey. Journal of Purchasing and Supply Management, 15(1), 24-32. http://dx.doi.org/10.1016/j.pursup.2008.10.001
  » http://dx.doi.org/10.1016/j.pursup.2008.10.001
• Las Casas, A. L. (2012). Administração de marketing: conceitos, planejamento e aplicações à realidade brasileira São Paulo: Atlas.
• Lubis, R., & Atin, S. (2020). E-service quality of hospital to measure patient satisfaction. Journal of Engineering Science and Technology, 1, 21-27.
• Mano, R. F., de Medeiros, F. G., Diniz, I. S. F. N., & de Abreu, N. R. (2013). Transporte Público e Acessibilidade: um estudo comparativo da qualidade percebida e esperada por usuários com deficiências em João Pessoa. Revista Gestão da Produção Operações e Sistemas, 9(3), 71-84. http://dx.doi.org/10.15675/gepros.v9i3.1007
  » http://dx.doi.org/10.15675/gepros.v9i3.1007
• Mishra, L., Gupta, T., & Shree, A. (2020). Online teaching-learning in higher education during lockdown period of COVID-19 pandemic. International Journal of Educational Research Open, 1, 100012. http://dx.doi.org/10.1016/j.ijedro.2020.100012 PMid:35059663.
  » http://dx.doi.org/10.1016/j.ijedro.2020.100012
• Nielsen, J. (1995, June). Technology transfer of heuristic evaluation and usability inspection. In Proceedings of the International Conference on Human-Computer Interaction Tokyo, Japan.
• Oliveira, O. J., & Ferreira, E. C. (2008). Adaptação e aplicação da escala SERVQUAL na educação superior. Revista Gestão da Produção Operações e Sistemas, 3(3), 133-146. Retrieved in 2021, September 21, from https://revista.feb.unesp.br/index.php/gepros/article/viewFile/473/193
  » https://revista.feb.unesp.br/index.php/gepros/article/viewFile/473/193
• Organização para a Cooperação e Desenvolvimento Econômico – OECD. (2018). Relatórios Econômicos OCDE: Brasil 2018 (pp. 43-44). Retrieved in 2021, September 21, from https://www.oecd.org/economy/surveys/Brazil-2018-OECD-economic-survey-overview-Portuguese.pdf
  » https://www.oecd.org/economy/surveys/Brazil-2018-OECD-economic-survey-overview-Portuguese.pdf
• Ortega-Mohedano, J., Rodríguez-Conde, M., & Muñoz-Gallego, P. (2018). Service and e-service quality and satisfaction of students from the european higher education area: a measurement scale of quality of higher education from a student’s perspective. In Proceedings of the 11th Annual International Conference of Education, Research and Innovation (ICERI) Seville, Spain. http://dx.doi.org/10.21125/iceri.2018.1914.
• Parasuraman, A., Zeithaml, V. A., & Berry, L. L. (1985). A conceptual model of service quality and its implications for future research. Journal of Marketing, 49(4), 41-50. http://dx.doi.org/10.1177/002224298504900403
  » http://dx.doi.org/10.1177/002224298504900403
• Parasuraman, A., Zeithaml, V. A., & Berry, L. L. (1988). SERVQUAL: a multiple item scale for measuring consumer perceptions of service quality. Journal of Retailing, 64(1), 12-40.
• Parasuraman, A., Berry, L. L., & Zeithaml, V. A. (1991). Understanding customer expectations of service. Sloan Management Review, 32(3), 39-48.
• Rocha, C. S. C., Tatsch, A. L., & Cário, S. A. F. (2019). Mudança estrutural e seu impacto na produtividade: uma análise da ascensão do setor de serviços na economia brasileira. Revista Economia Ensaios, 33(No. esp.):26-45. http://dx.doi.org/10.14393/REE-v33n0a2019-50413
  » http://dx.doi.org/10.14393/REE-v33n0a2019-50413
• Rowley, J. (2006). An analysis of the e‐service literature: towards a research agenda. Internet Research, 16(3), 339-359. http://dx.doi.org/10.1108/10662240610673736
  » http://dx.doi.org/10.1108/10662240610673736
• Roy, S. K., Lassar, W. M., Ganguli, S., Nguyen, B., & Yu, X. (2015). Measuring service quality: a systematic review of literature. International Journal of Services, Economics and Management, 7(1), 24-52.
• Santos, J. (2002). From intangibility to tangibility on service quality perceptions: a comparison study between consumers and service providers in four service industries. Managing Service Quality, 12(5), 292-302. http://dx.doi.org/10.1108/09604520210442083
  » http://dx.doi.org/10.1108/09604520210442083
• Serviço Federal de Processamento de Dados – SERPRO. (2020). A LGPD e seus desafios no setor público Retrieved in 2022, July 26, from https://www.serpro.gov.br/lgpd/noticias/2020/lgpd-desafios-setor-publico-serpro
  » https://www.serpro.gov.br/lgpd/noticias/2020/lgpd-desafios-setor-publico-serpro
• Shankar, A., & Datta, B. (2020). Measuring e-service quality: a review of literature. International Journal of Services Technology and Management, 26(1), 77-100. http://dx.doi.org/10.1504/IJSTM.2020.105398
  » http://dx.doi.org/10.1504/IJSTM.2020.105398
• Silva, D. A. L., Giusti, G., Rampasso, I. S., Farrapo Junior, A. C., Marins, M. A. S., & Anholon, R. (2021). The environmental impacts of face-to-face and remote university classes during the COVID-19 pandemic. Sustainable Production and Consumption, 27, 1975-1988. http://dx.doi.org/10.1016/j.spc.2021.05.002 PMid:34778504.
  » http://dx.doi.org/10.1016/j.spc.2021.05.002
• Sohn, C., & Tadisina, S. K. (2008). Development of e-service quality measure for internet-based financial institutions. Total Quality Management, 19(9), 903-918. http://dx.doi.org/10.1080/14783360802224412
  » http://dx.doi.org/10.1080/14783360802224412
• Sott, M. K., Bender, M. S., Furstenau, L. B., & dos Santos, R. B. (2020). Análise da satisfação do cliente através de uma abordagem multivariada: um estudo de caso. In Anais do Congresso Internacional de Administração Ponta Grossa, PR.
• Souza, L. H. B., Braga, F. A. S., & Mendes, G. H. S. (2016). Análise de Tipologias de Sistema Produto Serviço (PSS). In Anais do XXXVI Encontro Nacional de Engenharia de Producão São José dos Campos: ABEPRO. Retrieved in 2021, September 21, from http://www.abepro.org.br/biblioteca/TN_STO_227_329_29740.pdf
  » http://www.abepro.org.br/biblioteca/TN_STO_227_329_29740.pdf
• Stiglingh, M. (2014). A measuring instrument to evaluate e-service quality in a revenue authority setting. Public Relations Review, 40(2), 216-225. http://dx.doi.org/10.1016/j.pubrev.2013.12.001
  » http://dx.doi.org/10.1016/j.pubrev.2013.12.001
• Stylidis, K., Wickman, C., & Söderberg, R. (2020). Perceived quality of products: a framework and attributes ranking method. Journal of Engineering Design, 31(1), 37-67. http://dx.doi.org/10.1080/09544828.2019.1669769
  » http://dx.doi.org/10.1080/09544828.2019.1669769
• Sumi, R. S., & Kabir, G. (2021). Satisfaction of E-Learners with Electronic Learning Service Quality Using the SERVQUAL Model. Journal of Open Innovation: Technology, Market, and Complexity, 7(4), 227. http://dx.doi.org/10.3390/joitmc7040227
  » http://dx.doi.org/10.3390/joitmc7040227
• Taber, K. S. (2018). The Use of Cronbach’s Alpha when developing and reporting research instruments in science education. Research in Science Education, 48(6), 1273-1296. http://dx.doi.org/10.1007/s11165-016-9602-2
  » http://dx.doi.org/10.1007/s11165-016-9602-2
• The New York Times. (2020). The office will never be the same Retrieved in 2022, July 26, from https://www.nytimes.com/2020/08/20/style/office-culture.html
  » https://www.nytimes.com/2020/08/20/style/office-culture.html
• Trivedi, D., Bhatt, A., Trivedi, M., & Patel, P. V. (2021). Assessment of e-service quality performance of university libraries. Digital Library Perspectives, 37(4), 384-400. https://doi.org/10.1108/DLP-07-2020-0072
  » https://doi.org/10.1108/DLP-07-2020-0072
• Tukker, A. (2004). Eight types of product–service system: eight ways to sustainability? Experiences from SusProNet. Business Strategy and the Environment, 13(4), 246-260. https://doi.org/10.1002/bse.414
  » https://doi.org/10.1002/bse.414
• Tumsekcali, E., Ayyildiz, E., & Taskin, A. (2021). Interval valued intuitionistic fuzzy AHP-WASPAS based public transportation service quality evaluation by a new extension of SERVQUAL Model: P-SERVQUAL 4.0. Expert Systems with Applications, 186, 115757. http://dx.doi.org/10.1016/j.eswa.2021.115757
  » http://dx.doi.org/10.1016/j.eswa.2021.115757
• Udo, G. J., Bagchi, K. K., & Kirs, P. J. (2011). Using SERVQUAL to assess the quality of e-learning experience. Computers in Human Behavior, 27(3), 1272-1283. http://dx.doi.org/10.1016/j.chb.2011.01.009
  » http://dx.doi.org/10.1016/j.chb.2011.01.009
• United Nations Environment Programme – UNEP. (2022). Life Cycle Thinking. United Nations Environment Programme Retrieved in 2022, July 26, from https://www.learnlifecycle.com/courses/Introducao-ao-Pensamento-do-Ciclo-de-Vida
  » https://www.learnlifecycle.com/courses/Introducao-ao-Pensamento-do-Ciclo-de-Vida
• United Nations Environment Programme – UNEP. (2021). About UN Environment Programme United Nations Environment Programme. Retrieved in 2021, September 21, from https://www.unep.org/about-un-environment
  » https://www.unep.org/about-un-environment
• Yang, Z., Jun, M., & Peterson, R. T. (2004). Measuring customer perceived online service quality: scale development and managerial implications. International Journal of Operations & Production Management, 24(11), 1149-1174. http://dx.doi.org/10.1108/01443570410563278
  » http://dx.doi.org/10.1108/01443570410563278
• Zeithaml, V., & Bitner, M. (1996). Services management New York: McGraw-Hill.
• Zeithaml, V. A., Parasuraman, A., & Berry, L. L. (1985). Problems and strategies in services marketing. Journal of Marketing, 49(2), 33-46. https://doi.org/10.2307/1251563
  » https://doi.org/10.2307/1251563
• Zhou, R., Wang, X., Shi, Y., Zhang, R., Zhang, L., & Guo, H. (2019). Measuring e-service quality and its importance to customer satisfaction and loyalty: an empirical study in a telecom setting. Electronic Commerce Research, 19(3), 477-499. http://dx.doi.org/10.1007/s10660-018-9301-3
  » http://dx.doi.org/10.1007/s10660-018-9301-3

Publication Dates

• Publication in this collection
  08 Aug 2022
• Date of issue
  2022

History

• Received
  02 Sept 2021
• Accepted
  19 July 2022