EVALUATION OF E-COMMERCE SYSTEMS: COMPARATIVE STUDY BEFORE AND DURING THE COVID-19 PANDEMIC IN BRAZIL

ABSTRACT

The increasing use of e-commerce, stimulated by the period of social isolation given the pandemic of COVID-19, requires accompanying measures, among them, the evaluation of the success of the Information System (IS). A stream of research has been dedicated for over 30 years to evolving / adapting IS evaluation models. This investigation empirically tests the theoretical model of DeLone & McLean (2004); in two distinct periods, before (model 1) and during (model 2) the COVID-19 pandemic. A survey was conducted with 329 consumers of business to consumer e-commerce in Brazil. This research is quantitative and used structural equation modeling to analyze the data. The results showed that, before the pandemic, e-commerce users did not perceive value in the quality of the system's information while, during the pandemic, perceived this aspect as important. That is, they also started to recognize more value in the quality of the system during the pandemic. This was the result of the increased use of the e-commerce system during the period of social detachment. This quantitative study has some limitations. First, the survey was conducted twice, before and during the Covid-19 pandemic specific on the Brazilian e-commerce consumption context. Second, the sample size is limited. The study contributes academically by applying the model quantitatively in a Western context. Furthermore, it has strategic potential, as positive experiences in the IS lead the consumer to reuse it, generating revenue.

Keywords
IS Success Model; E-commerce; B2C; COVID-19; DeLone McLean, Brazil

INTRODUCTION

Social distancing, resulting from the coronavirus pandemic, changed many routines around the world. A historical moment which impacts many face-to-face businesses. Consequently, those changes imply a direct impact and modification on enterprises (i.e education, technological, e-commerces and other businesses) ( Hasanat et al., 2020; Donthu and Gustafsson, 2020) and in consumption patterns. Those changes occurred to stop the spread of the virus. Therefore, online services have become a convenient alternative to supply consumers necessities (Seetharaman, 2020).

E-commerce gained popularity in the last decade. It has been used as the main channel of relationship between businesses and consumers. In addition, it can significantly impact economic results ( Subaeki, Rahman, Putra , Alam, 2019). As the internet advances, e-commerce businesses gain market positions. Moreover, e-commerce makes it possible to carry out transactions with consumers anywhere in the world, thus increasing competitiveness (Ali et al., 2018). Therefore, from the adversities caused by the coronavirus pandemic, e-commerce modified consumption habits. It can provide a convenient alternative to supply social distancing necessities. Thus, it can be an important economical factor for many countries (Watanabe & Omori, 2020).

In the Brazilian context, a research ordered by Associação Brasileira de E-Commerce ( ABComm, 2020), has identified that from 01/03/2020 to 08/04/2020 online sales reduced in the beginning of March. Therefore, the sales raised in April occasioned directly by online shopping. The impact of such reduction was perceived in the tourism, technology, and fashion industries. ( ABComm, 2020). In addition, the association predicted a 23% growth in sales in August, intensified by competition in celebrative dates (i.e Father’s Day).

Thus, it becomes necessary for companies to manage the success of the IS more actively, to remain competitive even during the period of social distance, given the increase in demand for online consumption. Further, the use of these platforms provides convenience for people to consume in the comfort of their homes. B2C commerce also manages to promote the necessary subsidies for organizations to trade, even in turbulent contexts. Salespeople can review key metrics, identify strengths and weaknesses, examine platform performance to guarantee marketing decisions more assertively (Song, Zheng , Zhang, 2018).

Therefore, one factor accounting for organizational success is the performance evaluation of these information systems. Recognize which factors of this system are essential to provide an effective IS ( Angelina, Hermawan , Suroso, 2019), mainly at the pandemic period when IS use is intensified and indispensable for consumption. In this context, considering the importance of measuring IS performance and considering that e-commerce sales during COVID-19 should grow due to social distancing measures, this study seeks to evaluate the success of e-commerce IS of Brazilian users, comparing satisfaction before and during the period of social distance. The study was conducted in the Brazilian context and contributes to reducing the gap of the literature applied to the reality of South America, as observed by the study by Al-Kofari, Hassan & Mohamad (2020).

Therefore, this research has as objective, to evaluate the success of information systems by Brazilians who use Business to Consumer (B2C) e-commerce before and during the period of social isolation caused by the coronavirus pandemic (COVID-19). In addition to comparing the data collected in different social contexts, this study makes use of the taxonomy of DeLone & McLean (1992), reproducing the updated model of Wang (2008). The DeLone & McLean (1992) model is regarded as a performance measurement framework that provides a comprehensive understanding of IS success. This performance measurement framework has been updated for use in e-commerce (DeLone & McLean, 2003), including adjustments and contributions noted for its applications and criticisms that have been received (Al-Kofari, Hassan, Mohamad, 2020).

However, Wang (2008) observed this update has not been evaluated empirically enough in the context of e-commerce ( Wang, 2008), while Al-Kofari, Hassan , Mohamad (2020) reiterated this observation when surveying studies from 2012 to 2018 that used DeLone & McLean's IS evaluation, found that 9% of these surveys applied to e-commerce. A contribution made by argues Al-Kofari, Hassan , Mohamad (2020) the model has been recurrent in studies evaluating the IS success measure, because of its applicability at the level of: individual or organization; systems impact; different technologies and applications; IS performance, in addition to demonstrating a certain ability to explain the success or failure of these SI.

This research can contribute to companies’ management of IS and provides means to assess the use of these systems in the current context. E-commerce consumes a large part of marketing resources through IS ( Sharkey et al., 2010; Subaeki et al., 2019; Zheng et al., 2020), which demands great attention from managers in evaluating its performance. Likewise, the increase in the number of households with computers; the spread of broadband access and the use of mobile devices; the maturation and consolidation of the online retail structure; greater security in payment transactions and increased reliability in delivery (), Oliveira et al., 2017also provide a favorable context for the use of e-commerce IS. This situation is extremely convenient, especially in times of social isolation.

The research is divided into five sections, including this introduction. In the next section, the model is presented along a discussion of previous studies on the topic. Then, the methodology in which the structural equations were used for data analysis is described - collected through a questionnaire. Following, the analysis and discussion of the results are presented. Finally, the final considerations, limitations and suggestions for future research are presented.

E-COMMERCE AND ADVERSE CONTEXT

The coronavirus pandemic (COVID-19) began in 2019 in China. It has been considered as a ‘great depression’ ( Hirsch, 2020), and a ‘global crisis’ ( Wang et al., 2020) which changed routines around the world. Consequently, those changes affect worldwide business (e.g., e-commerces, technologies, higher education, and other businesses ( Donthu& Gustafsson, 2020; Hasanat et al., 2020). The economic consequences stemming from business closures and people's impossibility to work were difficult to measure. These events called for adaptations from organizations to maintain their commercial activities and minimize losses.

Marketing strategies’ innovations can be considered mechanisms to help enterprises to manage crises and overcome environmental risks ( Naidoo, 2010). Digital transformation is a marketing strategy adopted by companies to handle environmental uncertainty.

Several studies have recently focused on e-commerce business based on turbulence and uncertainty caused by Covid-19. The findings indicate the coronavirus pandemic infections have increased the use and attractiveness of e-commerce. That is because online consumption allows customers to avoid face-to-face contact while making purchases ensuring safety ( Watanabe & Omori, 2020).

Moreover, the digital culture has changed the market structure, providing an opportunity for companies to continue to negotiate their products in this adverse context caused by Covid-19. Thus, it is believed that e-commerce will continue to increase after the pandemic. Also, the coronavirus pandemic has required organizations to develop a digital mindset and deliver their products/services while maintaining social distancing ( Seetharaman, 2020 ).

Therefore, companies need to develop marketing strategies to become more efficient on online platforms to promote consumer behavior and understand it ( Nguyen et al, 2020), especially in adverse contexts. Critical factors that motivate and affect consumers' satisfaction when buying online should be considered, since customers depend exclusively on e-commerce to satisfy their needs in a pandemic. In this adverse and uncertain moment, analyzing the benefits of e-commerce platforms’ information systems should be prioritized.

Thus, it's marketing managers' responsibility to understand consumers' demands and guarantee e-commerce sustainability ( Wie & Widjaja, 2017). Most discussions in performance and results of e-commerce are directly related to IS success ( Brown & Jayakody, 2008). In line with this, this study considers the DeLone & McLean (2003) model to evaluate e-commerce systems before and after the pandemic. Those authors affirm the online shopping process fits into the IS success model in the six dimensions of success. Thus, the authors encourage the development of further research that continues to test and challenge your model ( Subaeki; Rahman & Putra , Alam, 2019; Al-Kofari, Hassan & Mohamad, 2020). Additionally, by comparing the model in two distinct moments (e.g after and during the social distancing period) this study contributes to validating the IS success model into an adverse context.

DELONE & MCLEAN MODEL (D&M,2003)

With the advancement of the internet, the growth of Business to Consumer (B2C) e-commerce is constant. It makes possible to carry out transactions with people anywhere in the world, which means working with increasing competitiveness ( Ali et al., 2018). The use of the B2C platform provides the necessary services for organizations, including data analysis. Salespeople can review key metrics, identify strengths and weaknesses, and examine platform performance ( Wie & Widjaja, 2017) to make more assertive marketing decisions and maximize their functioning ( Song, Zheng , Zhang, 2018).

Thus, it is necessary tto evaluate the performance of these information systems (IS), identifying which factors, as well as the relationship between them, are essential to providing an effective IS ( Angelina, Hermawan , Suroso, 2019). Sharkey, Scott and Acton (2010) and Wie and Widjaja (2017) also defend this position when stating that measuring the success of ISs is fundamental in the process of evaluating marketing investment strategies.

In this sense, it is evident that investments in Information Technology (IT) and data analysis are fundamental for performance evaluation of e-commerce systems ( Subaeki, Rahman & Putra , Alam, 2019; (Zheng, Zhang , Song, 2020). The term performance generally refers to effectiveness and productivity ( Tam & Oliveira, 2016). Thus, a bad experience with IS can result in loss of revenue, as the website is the customer's first interaction with the product ( Wani et al., 2017), which demonstrates the importance of monitoring the success of ISs.

DeLone and McLean (1992) conducted a comprehensive review of the literature on Information Systems (IS) and proposed a measurement of IS success. The identification of what is considered a success in systems is a relevant topic in the field ofISstudies ( Al-Kofari, Hassan & Mohamad, 2020; Brown & Jayakody, 2008). The idea underlying this theme considers that systems must support processes and meet users' expectations ( DeLone & McLean, 1992). This model, over time, received criticism, which resulted in the updated of the model by DeLone and McLean (2003), including the adaption to the e-commerce context ( Wang, 2008). This model is recognized for being among the most influential theories in modern IS research ( Al-Kofari, Hassan , Mohamad, 2020).

The updated model from D&M (2003) identified six dimensions interrelated to IS success. Such dimensions suggested the quality of the content, as well as the quality of the IS system and service, determine the customer's intention to use and their real satisfaction with the IS. The more satisfied users are with the SI, the more they will use it, which, consequently, determines the benefits they receive from the system. The benefits reinforce not only the customers' intention to use but also the actual use and their satisfaction with the IS ( DeLone & McLean, 2003).

According to Sharkey et al. (2010), the success of the system must necessarily be based on the objectives of what it seeks to manage, from which its use derives from the possibility of measuring both the benefits from the consumer's point of view and the business’ ( Ali et al., 2018). The hypotheses of this study are supported by the study of Wang (2008) and were developed from the taxonomy of DeLone and McLean (1992, 2003); they can be seen in Figure 1.

Figure 1
Research Design, Hypothesis and Constructs.Source: Retrieved from “Assessing e-commerce systems success: a re-specification and validation of the DeLone and McLean model of IS success”, by Y. S. Wang, 2008, Information Systems Journal, 18 (5), 529-557, p. 537.

The model is based on two theories, the Mathematical Theory of Communication (MTC) and the Theory of Influence of Information ( Tam & Oliveira, 2016; Angelina et al., 2019 ). MTC was created by Shannon and Waever (1949), whose object is information. Information is a measure of one's freedom of choice when one selects a message. The concept of information applies not to the individual messages, but rather to the situation itself. The unit of information indicating that in this situation one has certain freedom of choice, in selecting a message, which is convenient to regard as a standard or unit amount ( Shannon & Waever, 1949, p. 9).

The theory of communication analyzes information at three levels. The first, called technical, is concerned with how the symbols of communication can be transmitted. Mathematically, they relate to the precision of the transference of the sender to the receiver of a finite number of symbols. The second level is semantics which concerns the precision with which the symbols sent are interpreted satisfactorily by the recipient. The third level is effectiveness, which refers to understanding how effectively the meaning received affects the conduct in the desired way. Problems with effectiveness concern the success with which the meaning conveyed to the recipient leads to the desired conduct on his part. At first glance, it may seem undesirable to suggest that the purpose of all communication is to influence the conduct of the recipient. However, communication affects the receiver's conduct to some extent ( Shannon & Waever, 1949).

Quality of the information is presented as one of the items of IS success model in the D&M model. Further, it relates to the provision of an informational base, to potential buyers and suppliers of transactions, which may be useful in driving their decision to use the IS. The information underlies the objectives of serving the user, since the experience is somewhat individualized, and therefore, the interpretations can be diverse ( DeLone & McLean, 2003 ).

Angelina et al. (2019) states the MTC technical level was used in the D&M model to explain how information can be produced based on the accuracy and efficiency of the system, while the semantic level was in charge of measuring the success in delivering the information. Thus, H1: states the quality of information has a significant effect on the use of the IS, while H4: Quality of information positively affects the Perceived Value in the context of e-commerce, as stated by Wang (2008).

The Theory of Influence of Information introduced this emphasis, which contemplates the sequence of events that interfere with the receipt of information, the recipient, and the system ( Angelina et al., 2019 ). The quality of the information encompasses content, such as texts, photos, graphics and videos ( DeLone & McLean, 2003; Wie & Widjaja, 2017).

Both the quality of the system and the quality of the information indirectly affect the success of the IS, predominantly, through user satisfaction. To this end, it is important to provide customers with updated, relevant data information through an interactive and user-friendly system. Among the three constructs (quality, use and satisfaction), user satisfaction weighs more significantly on net benefits than using the system ( Angelina et al., 2019 ). In this direction, based on the study by Wang (2008), H2 was outlined: The quality of the system positively affects the satisfaction of users in the context of e-commerce, and H5: Quality of the system positively affects the Perceived Value in the context of e-commerce.

Service quality is defined as the general support provided by the IS service provider ( DeLone & McLean, 2003). This dimension reveals that the efficiency and availability of the system are related to the website system, while the dimensions of compliance and privacy refer to the commitment of the e-commerce company to better serve its consumers. Like Wang (2008) H3 was outlined: Service quality positively affects user satisfaction in the context of e-commerce, and H6: Service quality positively affects perceived value in the context of e-commerce.

The other hypotheses also used Wang (2008), H7: Perceived Value positively affects the Reuse Intention in the context of e-commerce. However, the perceived value also partly explains user satisfaction. Thus, H8 seeks to verify whether Perceived Value positively affects user satisfaction in the context of e-commerce. And finally, H9: User Satisfaction positively affects the Intention to Reuse in the context of e-commerce.

After the publication of DeLone & McLean’s model (2003), many IS researchers modified it ( Angelina et al., 2019). The adaptations were generated due to contextual or business specificities, including constructs from other models, reinforcing acertain aspect considered little explored by the original model.

The literature presents some D and M model application. Accordingly, Yel,Sferianto and Anugrah (2020), the original model (D&M) was applied for e-commerce in Indonesia. Ali et al. (2018) criticized Wang's adaptation (2008) for not including trust in his model of IS’s success evaluation. The authors considered this item to be important, as they applied the study to the context of tourism in India. Trust was a significant factor in the success of e-commerce since buyer and seller do not transact in person, and customers cannot physically touch or feel the product. Therefore, privacy and trust are valuable constructions in this environment.

Jusop et al. (2018) applied the model in the context of Halal e-commerce including, in the IS model of success, the prerequisites of conformity with Sharia, the Islamic law of the contract. To do so, they considered form (offer and acceptance), contracting parties (buyer and seller) and the object (object and price), to avoid any prohibited elements, such as usury or interest, uncertainty, illegality, and games.

Wani et al. (2017) also applied the DeLone & McLean (2003) model to tourism e-commerce in India. The authors consider that the model offers a well-organized basis for categorizing the vast number of IS success measures developed in the IS literature and believe it is relevant to add the variable - user experience - to attribute emotions, perceptions, attitudes and thoughts evoked as a result of interactions with a system.

Wie & Widjaja (2017) used the DeLone & McLean (2003) model in B2C e-commerce comparatively in Indonesia, Japan, and South Korea. They included moderators: national identity; prevention of uncertainty; and attitude towards online shopping. The results indicated that all quality factors influence user satisfaction towards the website, in addition to cultural factors and attitudes towards these purchases.

Tam & Oliveira (2016) added the Task Technology Fit (TTF) applied to the context of banking services via mobile devices. For the authors, usage decisions can be influenced by the quality of the service and the interaction of the staff. Thus, the characteristics of the task must be considered among the factors to broaden the customer's assessment of banking services via mobile devices.

METHODOLOGY

The present study adopts a quantitative approach, which main goal is to test, verify and measure variables ( Creswell, 2017). Also, the selected method for this study was a survey, which can provide a quantitative description, trends and opinions of a population from a sample ( Creswell, 2017).

We conducted a search on Ebscohost base, using terms “DeLone & McLean” and “e-commerce”. Initially, 142 studies were found that used the model. When refining the analysis of these results, studies that did not apply to the B2C e-commerce model were excluded. These results allowed the construction of the theoretical background and to identify a validate a questionnaire that could be replicated. Furthermore, other techniques used in this paper are shown in Table 1.

Table 1.
Summary of research design.

Method	Disposition	Approach		Data Collection		Temporality
Survey	Conclusive	Descriptive	Quantitative	Survey	Online structured questionnaire	Transversal

To obtain consumer data in the Brazilian context, an online self-administered questionnaire was developed using Qualtrics. The population in this study was composed of e-commerce users, following Wang's research (2008). We used non-probabilistic and convenient samples (i.e voluntary peoples contacted by social media). Thus, participants answered the questionnaire according to their availability.

Data collection was performed using the structured research instrument, developed and validated by Wang (2008), containing 21 questions and divided into six blocks. The questions were structured on a five-point Likert scale, ranging from stronglydisagree (1) to strongly agree (5). In addition, we conducted both a translation and reverse translation to ensure the linguistic integrity of the questionnaire adaptable to the context. Thus, it had to under go tests and adaptations before its application.

We divided the research into two stages of collection; the period before and during the social isolation generated by the pandemic of COVID-19. For the second collection (during the period of social isolation), a critical question was created to ensure the respondent had made online purchases, so that he or she could evaluate the e-commerce system.

The number of responses followed the recommendation of Hair Jr et al., (2014) of using a minimum sample of five times the total number of indicators. Thus, this resulted in 152 responses collected in August 2019 and 177 responses between April and May 2020. The collected data was purified by eliminating missing values and outliers, as shown in Table 2.

Table 2.
Data collection

Period	Before the Social Isolaion Period	During Social Isolation
Sample	152	177
Missing Values	3	2
Outliers	7	5
Total	142	170

After the purification procedures, the resulting sample consisted of 61.8% of female respondents, and 32.2% of male’s respondents. Furthermore, 61,2% of the respondents represents the range of ages between 21 to 40 years, 40.6% are in the 21 to 30 years old, and 20.6% are in 31 to 40 years range.

To analyze the data, structural equation modeling (SEM) method was used; the method is considered adequate for the simultaneous analysis of relationships between constructs, in which dependent variables become independent ( Hair Jr et al., 2014). Following, the analysis of the results is demonstrated, starting with the evaluation of the measurement model, to verify its consistency through the reliability of the indicators and, later, of the analyzed constructs.

RESULTS

Initially, we present the demographic data of the sample. It is worth mentioning that one of the requisites for participants, was to have engaged in an e-commerce transaction in the last 60 days from the date of data collection. These criteria are especially important for the development of future studies, considering that characteristics of the sample may prove to be a limitation of the research.

In both phases of application of the study (2019-2020), respondents are concentrated in the same age groups. Before the pandemic, 75.4% belonged to the 21 to 40 range of age group and during the pandemic, the concentration in these groups is 61.2%. There was a small dispersion among the other age groups. This data conforms the e-commerce consumer profile surveys, which reiterates the respondent's qualification. Data from 2018 shows that the age group with the highest consumption is between 35 and 49 years old (37%), followed by the 25 to 34 age group (24%); while those above 50 years old account for 29% of consumption, with other groups being less expressive (Mendes, 2019). In this sense, the sample is partially conforms the profile of Brazilian consumers.

The profile of respondents before and during the period of social isolation has changed. In the first phase of the research, it was predominantly male (52.8%), while during the pandemic, it was female (61.8%). The Social Miner report (2020) reported that 60.8% of e-commerce orders were made by women, in the beauty, fashion and accessories, and multicategories categories. This indicator is related to the sales category volumes. In this investigation, most purchases were on fashion items, followed by perfume and cosmetics, and food and beverages. Meanwhile, the lowest indicator was consumption in the tourism area, as expected, given mobility restrictions imposed by the need for social isolation. The data not only illustrates the profile of respondents, but also positions them in terms of characteristics similar to consumers pointed out by the statistics according to a report by Social Miner (2020) .

Underlying the objective of this study, it was expected that there would be changes in the level of consumption of e-commerce, due to social isolation and the need to resort to remote purchasing. However, it must be considered that data was collected in the first 60 days of isolation, and many businesses were working normally and decrees regulating economic activities were managed by each state. However, not all states have enacted restrictive circulation measures. The results showed 54.7% of consumers/respondents did not change their e-commerce consumption routine due to the pandemic. Moreover, of the consumers who changed, 23.5% increased consumption from 1 to 20%, and 11.2% increased from 21 to 40%. Finally, 4.1% declared a reduction in consumption pattern.

It is worth remembering that the categories with the highest sales in e-commerce in Brazil, are fashion, cosmetics, and perfume articles ( Ebit, 2019), which are considered non-essential. ABComm data (2020) indicates that tourism and fashion will be the sectors most impacted by the change in consumption patterns. This reduction may, therefore, point to a more cautious stance in the consumption pattern, given the market uncertainties and prospects of loss of purchasing power for the general consumer in this context of social isolation.

Result of the Measurement Model

To present consistent scale results and ensure reliability and acceptable validity, we analyzed the measurement model to identify psychometric evidence. For the analysis, we used the statistical software SmartPLS 3^®, the program was used to verify the model validity and run the hypothesis tests.

Therefore, the level of reliability should present a minimum threshold to differentiate a systematic error and a random error ( Hair Jr. et al., 2014). If the scale is reliable, it is assumed the data instrument is valid. To measure the reliability, the Cronbach coefficient was used. Further, to ensure indicators and construct reliability, composite reliability is the most suitable measure. Therefore, to guarantee the reliability of the construct, both Cronbach's alpha and composite reliability must present coefficients equal to or above the appropriate value of 0.7 ( Hair Jr. et al., 2014).

The discriminant validity, according to Fornell and Larcker (1981), verifies whether the diagonal values are greater than any other correlation, which indicates the existence of discriminant validity ( Hair Jr. et al., 2014). Convergent validity defined as “the extent to which a measure relates positively to alternative measures of the same construct” ( Hair Jr. et al., 2014, p. 102) was also considered. It was measured using the Average Variance Extracted (AVE) coefficients and the indicator loads, the reliability of the indicators is given from their outer loadings.

The AVE measures how much of the variance is explained by the latent construct ( Hair Jr. et al., 2014). Through the average of the squares of the loading of the indicators, the AVE obtained must reach at least a coefficient of 0.5, which means the commonality between the items is at least 50%. The tests were performed for the two samples studied and the results are highlighted Tables 3 and 4.

Table 3.
Results before the period of social isolation

	AVE	Cr	1	2	3	4	5	6
Reuse intention	0,873	0,954	0,935
Quality information	0,537	0,821	0,380	0,733
Quality service	0,549	0,879	0,366	0,454	0,741
Quality system	0,773	0,872	0,300	0,406	0,379	0,879
User satisfaction	0,763	0,906	0,566	0,540	0,590	0,505	0,873
Perceived value	0,784	0,916	0,514	0,364	0,410	0,413	0,652	0,885

• Notes - Cr = composite reliability and AVE = average variance extracted& (a) evaluated in the general measurement model, with the constructs correlated with each other. Diagonal italic numbers show the square root of the stroke.

Table 4.
Results during the period of social isolation

	AVE	Cr	1	2	3	4	5	6
Reuse intention	0,844	0,942	0,919
Quality information	0,607	0,860	0,429	0,779
Quality service	0,562	0,885	0,335	0,603	0,750
Quality system	0,718	0,836	0,496	0,574	0,527	0,847
User satisfaction	0,777	0,913	0,598	0,685	0,625	0,690	0,882
Perceived value	0,753	0,901	0,529	0,501	0,454	0,557	0,683	0,868

• Notes - Cr = composite reliability and AVE = average variance extracted& (a) was evaluated in the general measurement model, with the constructs correlated with each other. Diagonal italic numbers show the square root of the stroke

We demonstrate in Tables 1 and 2 that shared variance values (below the diagonal, in italics) are smaller than the average variance extracted in each construct pair. Thus, since the correlations started to be lower than the square root of the AVE, it is possible to infer a discriminant validity between the constructs. Likewise, all constructs analyzed showed AVE values greater than 0.5 and Cr above 0.07 ( Hair Jr. et al., 2014).

Regarding the analysis of convergent validity, we verified the loading of the indicators (outer loadings), which determines how much that variable contributes to the measurement of the construct. According to Hair Jr. et al. (2014), values above 0.708 are satisfactory. However, according to the authors, indicators that present the values of outer loadings between 0.4 and 0.708 should be analyzed with additional caution, so they do not harm the explanation of the construct. Since this is a replication of a study and considering it does not harm the AVE and Cronbach's Alpha, we decided to keep the indicators within this range. The results are shown in Table 5.

Table 5.
Results before and during the pandemic

Construct	Cronbach’s alpha before	Cronbach’s alpha during	Variable	Outer loadings before	Outer loadings during
Reuse intention	0,927	0,908	RI1 Assuming you have access to e-commerce, do you intend to reuse it?	0,918	0,929
			RI2 Are you going to use e-commerce again in the future?	0,967	0,946
			RI3 Do you intend to use e-commerce frequently in the future?	0,917	0,881
Quality information	0,704	0,783	QI1 Does the e-commerce system provide the information you are looking for?	0,738	0,809
			QI2 Does the content of the information available in the system meet your needs?	0,863	0,776
			QI3 Do you trust the transactions carried out on that system?	0,609	0,712
			QI4 Does the e-commerce system offer updated information?	0,700	0,816
Quality service	0,836	0,845	QV1 When you have a problem, does the e-commerce system show a sincere interest in solving it?	0,731	0,713
			QV2 Is the e-commerce system always ready to help you?	0,767	0,747
			QV3 Do you feel secure in your transactions with the e-commerce system in terms of security and privacy protection?	0,644	0,666
			QV4 Does the e-commerce system have the knowledge to answer your questions?	0,793	0,782
			QV5 Does the e-commerce system give you individual attention?	0,716	0,777
			QV6 Does the e-commerce system understand your specific needs?	0,784	0,805
Quality system	0,708	0,609*	QS1 Is the e-commerce system friendly?	0,899	0,871
			QS2 Is the e-commerce system easy to use?	0,859	0,822
User satisfaction	0,844	0,857	US1 Are you satisfied with the use of this e-commerce system?	0,897	0,880
			US2 Is this e-commerce system of high quality?	0,880	0,855
			US3 Did this e-commerce system meet your expectations?	0,842	0,909
Perceived value	0,863	0,837	PV1 Does the product / service offered by e-commerce have an advantageous price (monetary value)?	0,866	0,833
			PV2 Is the price of the product/service offered by e-commerce acceptable?	0,907	0,886
			PV3 Can the product/service offered by e-commerce be considered a good purchase?	0,883	0,883

• * It is given that the alpha value is affected by the number of items that make up a scale (KRUS & HELMSTADTER, 1993). The seminal presents only 2 items for construct measurement. Therefore, it is close to the acceptable value (0.7), Since it is a psychometric measure and the study is a replication, it was decided to keep the construct in the model.

The results indicate that all the outer loadings of these constructs had sufficient indexes to reach the values suggested by Hair Jr. et al. (2014). Likewise, the second column of the tables shows that Cronbach's Alpha values for all constructs are satisfactory (Hair Jr. et al., 2014). Thus, after verifying the reliability of the measurement instrument, we present the test of the proposed hypotheses.

Result of the structural model

We perform the structural model test, in which proposed relationships between the constructs (hypotheses) are tested, considering previous criteria. For this, we used the technique of structural equations since it sought to verify the simultaneity of the relationships between variables in the same theoretical model. The SmartPLS 3® software was used to calculate Partial Least Squares Structural Equation Modeling (PLS-SEM). This method maximizes the explanation of the variation of the dependent variable through its predictors, minimizing the residual variables (errors)( Hair Jr. et al., 2014).

For the calculation of path coefficients, the null hypothesis is rejected, in which the values must be different from 0, demonstrating the existence of a relationship. Thus, through the bootstrapping procedure (resampling) in SmartPLS 3®, we considered 5000 resamples, expecting t-statistic values above 1.96, for a significance level of 5%. Likewise, p-values below 0.05 are expected, which demonstrates the significance of the relationship.

We also present the standard deviation, as it provides information on the dispersion of responses concerning the average obtained, which makes the values vary more or less in relation to the midpoint ( Freund and Wilson, 2006). The results of the hypothesis test are shown in Tables 6 and 7.

Table 6.
Results before the period of social isolation

Path	Coefficient (β)	SD	t-value	p-value	result
H1 - Quality Information -> User Satisfaction	0,209	0.064	3.259	0.001	Accept
H2 - Quality System -> User Satisfaction	0,151	0.060	2.519	0.012	Accept
H3 - Quality Service -> User Satisfaction	0,273	0.057	4.780	0.000	Accept
H4 - Quality Information -> Perceived Value	0,146	0.087	1.681	0.093	Rejects
H5 - Quality System -> Perceived Value	0,261	0.079	3.301	0.001	Accept
H6 - Quality Service -> Perceived Value	0,245	0.076	3.208	0.001	Accept
H7 - Perceived Value -> Intention Reuse	0,252	0.083	3.040	0.002	Accept
H8 - Perceived Value -> User Satisfaction	0,402	0.068	5.931	0.000	Accept
H9 - User Satisfaction -> Reuse Intent	0,401	0.096	4.202	0.000	Accept

• SD-standard deviation& t-value>1,96; p-value<0,05

Table 7.
Results during the period of social isolation

Path	Coefficient (β)	SD	t-value	p-value	result
H1 - Quality Information -> User Satisfaction	0,262	0.066	3.949	0.000	Accept
H2 - Quality System -> User Satisfaction	0,264	0.064	4.099	0.000	Accept
H3 - Quality Service -> User Satisfaction	0,182	0.057	3.176	0.002	Accept
H4 - Quality Information -> Perceived Value	0,211	0.091	2.321	0.021	Accept
H5 - Quality System -> Perceived Value	0,365	0.072	5.067	0.000	Accept
H6 - Quality Service -> Perceived Value	0,134	0.092	1.458	0.145	Rejects
H7 - Perceived Value -> Intention Reuse	0,226	0.071	3.190	0.002	Accept
H8 - Perceived Value -> User Satisfaction	0,322	0.057	5.689	0.000	Accept
H9 - User Satisfaction -> Reuse Intent	0,443	0.086	5.164	0.000	Accept

• SD-standard deviation& t-value>1,96; p-value<0,05

Table 6 reveals the results of the test performed on the data collected from the period of social isolation caused by the Coronavirus pandemic. Path coefficients other than 0 and t and p statistics below the expected values reveal the acceptance of the proposed hypotheses, except for H4 (β = 0.146; t = 1,681; p = 0.093). This result demonstrates that Information Quality of an e-commerce system does not lead to a Perceived Value by the user. The results for the data collected during the pandemic share the same results, except for H4, which was accepted (β = 0.211; t = 2.321; p = 0.021) and H6, which was rejected (β = 0.134; t = 1,458; p = 0.145). We also evaluated the R² which shows the predictive precision of the structural model ( Ringle, Silva & Bido, 2014).

Thus, in terms of the degree of explanation, the Intent to Reuse construct obtained an R² of 0.347 in the first model (before isolation), indicating that 34.7% of the variations in this construct can be explained by the other constructs of the same model. In the second model, the R² was 0.377 (during isolation), showing that 37.7% of the construct's variations are explained by its precedents.

DISCUSSION OF RESULTS

As already mentioned, the updated D and M model identified six dimensions interrelated to IS success. This study was organized based on the six dimensions of an updated D and M model, replicating the study by Wang (2008) , applied to e-commerce. The quality of information, system and service led to satisfaction and value perceived by the user, which subsequently generates an intention to reuse e-commerce. Tam and Oliveira (2016) reinforce that the quality of information plays a critical role in the development of a favorable position relative to the benefits of using specific information. The Social Miner report ( Social Miner, 2019) reinforces the need to monitor the buyback rate of consumers, as an indicator of the result of the activity

The study was conducted in two different periods of the first, in August 2019 - called model 1 – considered a period of regular transactions, and the second, in April 2020 - called model 2 – considered a period of atypical e-commerce transactions, given the need for social isolation, and the closing of physical stores considered as non-essential due to the coronavirus pandemic. The reasoning for working with these periods comparatively was to verify whether there was an impact on the evaluation of e-commerce systems due to this period of social isolation generated by the COVID-19 pandemic.

The first part of the questionnaire refers to Quality of information, in which four questions assess whether the e-commerce system provides the necessary information, content of the information meets the needs, transactions are considered reliable, and the e-commerce offers up-to-date information.

H1 proposed the impact of information quality on user satisfaction with the system. The hypothesis was accepted in both models, demonstrating that the quality of information enables the user to be satisfied with the e-commerce system. These resultsconfirm the findings of Wang (2008), Wie & Widjaja (2017) and Yel, Sferianto and Anugrah (2020), while diverging from the findings of Angelina et al., (2019) who found that the quality of the information did not affect the use of the system, albeit, affecting user satisfaction, which in turn affects usage.

Model 1 shows how a variation of the standard deviation upwards or downwards in the information quality score would lead to a 20.9% variation in user satisfaction (β = 0.209; t-value = 3.259; p-value = 0.001). Model 2 shows that the variation isslightly greater, since the variation in the standard deviation of the previous construct, would lead to a 26.2% variation in user satisfaction (β = 0.262; t-value = 3,949; p-value = 0.000). These results show an increase in the importance of quality of information as a predictor of satisfaction from model 1 to model 2.

H4 proposes the influence of information quality on the perceived value. Model 1 did not show significance in the relationship (β = 0.146; t-value = 1.681; p-value = 0.093) and model 2 shows that this hypothesis was accepted (β = 0.211; t-value = 2.321; p-value = 0.021). This result converges with the findings of Angelina et al. (2019). Specifically, the result shows that, before social isolation, e-commerce users did not perceive value in the quality of the system's information and, during the pandemic, they already perceived this aspect as important. The issue of social isolation can be a justification for the results found since users do not physically have access to products and need a wealth of information to purchase e-commerce systems.

The second part of the questionnaire used two items to measure Quality of the system, identifying whether the e-commerce system is user friendly and whether it is easy to use. The variation of the standard deviation of the Quality of the system accounts for the variation of the user satisfaction in 15.1% and 26.4%, before (β = 0.151; t-value = 2.519; p-value = 0.012) and during (β = 0.264; t-value = 4.099; p-value = 0.000) the pandemic, respectively. Thus, H2’s hypotheses are accepted for user satisfaction. These results corroborate the results of Tam and Oliveira (2016), Wang (2008), Wie and Widjaja (2017), and Yel,Sferianto and Anugrah (2020) who verified that quality of the system increases user satisfaction. Furthermore, evidence from the study by Angelina et al. (2019) found no significant effect of the quality of the system on user satisfaction and attributed this result to factors related to trust.

H5 measures the relationship between the quality of the system and the perceived value. In both models, the hypothesis was accepted, with a small variation from before (β = 0.261; t-value = 3,301; p-value = 0.001) and during (β = 0.365; t-value = 5.067; p-value = 0.000) the pandemic. As the results show, users also started to recognize more value in the quality of the system during the pandemic; such results can be attributed to the increased use of the e-commerce system during the period of social distancing. The quality of the system, as highlighted by Wie and Widjaja (2017), is for marketers to ensure that hardware and software are always up to date. In addition, the results show how connectivity is a problem in developing countries.

The third part measures Quality of Service. A total of six questions were used to find out whether, in the presence of problems, the e-commerce system service shows a sincere interest in solving it, whether it is willing to help the user, whether the user feels safe in terms of security and privacy protection, whether the e-commerce system service has the knowledge to answer questions, whether there is individual attention and whether specific needs are understood. Jusop et al. (2018) reinforce that the quality of service is a characteristic valued by the consumer who uses the systems because if there is a failure in support, guidance, or responsiveness to the customer, it can lead the company to lose customers.

The results show that for H3, in model 1, the variation of one standard deviation upwards or downwards in the Service quality score would lead to a 27.3% variation in user satisfaction (β = 0.273; t-value = 4,780; p-value = 0.000), and in model 2 the variation is 18.2% (β = 0.182; t-value = 3.176; p-value = 0.002). H6, in turn, verified the relationship between the quality of service and the perceived value, obtaining a variation of 24.5% in the first model (β = 0.245; t-value = 3,208; p-value = 0.001) and 13.4% in the second model, but showing no significance (p-value <0.05) (β = 0.134; t-value = 1,458; p-value = 0.145).

The quality of service has a significant effect on user satisfaction ( Angelina et al., 2019; Wang, 2008; Wani et al. (2017); . . Widjaja (2017). The results of Wang (2008) reveal that quality of the service has a stronger effect than quality of the information and quality of the system, both of which influence perceived value and user satisfaction. Wie and Widjaja (2017), place trust and empathy as the primary challenges for working with Indonesian clients.

Wang (2008) points out that the quality of service dimension was added in the updated D and M model, and since then, it has received little empirical investigation. The findings by Wani et al. (2017) applied to travel websites highlighted that the quality of service has a significant and active position in the user's experience with the system.

Wang (2008) highlights, as a limitation of his study, the characteristics of the consumers who participated in the investigation. Specifically, it was found they are from collectivist oriental culture, which points to a possible explanation for these changes in results. Consequently, the findings may be caused by the culture of the Brazilian consumer, who is western and individualistic. Likewise, due to the immediacy of social distance, problems in e-commerce may not have been noticed yet, which leads the user to not perceive value in the quality of the service.

The Perceived value dimension sought answers to assess whether the price of the e-commerce product/service has an advantageous price, whether it is an acceptable price and whether the product/service purchased through e-commerce can be considered a good purchase. H7 considered the relationship between perceived value and the intention to reuse, with an effect of β = 0.252 in model 1 (t-value = 3.040; p-value = 0.002) and β = 0.226 in model 2 (t-value = 3.190; p-value = 0.002). H8 verified the impact of the perceived value on user satisfaction, with an effect of β = 0.402 on model 1 (t-value = 5,931; p-value = 0.000) and β = 0.322 on model 2 (t-value = 5,589; p-value = 0.000).

As seen previously, the quality of the system is not a significant element for the perceived value of the user during the pandemic, which may be the basis for justifying the reduction of the H7 path coefficients in model 2. By not promoting the perceived value, variations in service quality are accounted from quality of information and quality of the system, which makes the perceived value less intense. Likewise, before the pandemic, the relationship between perceived value and user satisfaction is the one with the strongest path coefficient, however, this is not maintained during the pandemic. Due to social isolation, the perceived value may not be a necessary attribute for user satisfaction since it is a convenient alternative to meet the immediate needs of consumers.

The User Satisfaction dimension has three questions that aim to assess whether the user is satisfied with the e-commerce system, as well as whether it is of a high quality and meets the respondent's expectations. H9 sought to verify the relationship between this variable and the intention to reuse, showing that a variation in the standard deviation of the first justifies a variation in the standard deviation of the second by 40.1% in model 1 (β = 0.401; t-value = 4.202; p-value = 0.000) and 44.3% in model 2 (β = 0.443; t-value = 5.164; p-value = 0.000). The hypotheses are accepted in both models. Wang (2008) noted that user satisfaction has a significant determining effect on the intention to use the system.

During the pandemic, this effect is stronger, which demonstrates the need for managers to be determined to satisfy IS users in online purchase transactions during the period of social distance. DeLone and McLean (2003) point to the importance of user satisfaction throughout the purchase process; a fact corroborated by Sharma and Agarwall (2019), in which positive opinions of a customer can influence other customers regarding the purchase of a certain product/service. Angelina et al. (2019) found the convenience provided by e-commerce makes life easier for users, thus saving time, such that in the adverse context generated by the social isolation imposed by the COVID-19 pandemic, it was fundamental to reduce the circulation of people in the trade.

CONCLUSION AND FUTURE DIRECTIONS

The purpose of this article was to evaluate the success of information systems by Brazilians who use e-commerce in the Business to Consumer (B2C) before and during the COVID-19 pandemic. therein line with this objective, it was expected that the pandemic period of COVID-19 and the period of social isolation and the closing of physical stores businesses, would result in an increase in the online purchases, which was not identified in the early days of social isolation. However, the study included two phases: before and during the pandemic. This is interesting for assessing the success of information systems since the formation of the sample by accessibility was not necessarily applied to the same target audience, that is, two different studies were carried out validating the D and M model, adapted for e-commerce, according to Wang (2008), with some caveats.

Concerning results, model 1 (data before the pandemic) indicated that the Information Quality of an e-commerce system does not lead to a Perceived Value by the user. In the data collected during the pandemic, the results are almost the same. However, there were some changes: H4 was accepted and H6 was rejected. These results clearly highlight the context of the coronavirus pandemic has shown some peculiarities in the use of e-commerce systems since consumers do not perceive value in the quality of the service. Likewise, user satisfaction is found to be the most important element for predicting intention of reusing the system. This situation reinforces the fact that the intention to reuse is related to a cognitive perspective attributed as the 'perception of value' but, mainly, in the individual satisfaction related to the individuals' attitudes/actions ( Wang, 2008), since it provides the intention of reuse in both models.

In addition, we calculated the R² in terms of the degree of explanation, in which the construct Reuse Intention obtained an R² of 0.347 in the first model (before the pandemic), indicating that 34.7% of the variations in this construct can be explained by the other constructs of the model. While in the second model, the R² was 0.377 (during the pandemic), showing that 37.7% of the variations in the construct are explained by its precedents. These results demonstrate that, despite the reduction in the explanatory model concerning that proposed by Wang (2008) (R²: 64%), the model that measures satisfaction with the IS is still appropriate to capture the intention of reusing the user through precedents that characterize the tool.

It should be noted that Wang (2008) warned there could be disagreements considering a sample of western and individualistic consumers. In this sense, the limitations of the research refer to the application that was restricted only to the experiences of Brazilian users who made online purchases, ignoring those who may have abandoned the e-commerce sites for some reason related to information systems. The sample size may also have been a limiting factor.

As future research opportunities, we suggested specifically investigating the items that presented low contribution as an explanatory factor, to investigate users who have abandoned the online shopping experience, and other factors that would be likely to contribute to the success of ISs that were not covered in this model. Furthermore, this research can be applied after the coronavirus pandemic to compare results among the virus life cycle and its impacts on e-commerce business.

Acknowledgment

The authors thank the financial support to this project provided by : Coordenação de Aperfeiçoamento de Pessoal de Nível Superior – CAPES - Brazil

• ABComm. (2020). Associação Brasileira de E-Commerce. Retrieved from: https://bit.ly/2zreQLM
• Ali, N. I., Samsuri, S., Brohi, I. A., Soomro, A. B., Soomro, S.Shah, A. (2018, July). Preliminary Study on Factors Affecting E-Commerce Success: A Modified DeLone and Mclean Model. In 2018 International Conference on Information and Communication Technology for the Muslim World (ICT4M) (pp. 120-125). https://doi.org/10.1109/ICT4M.2018.00031
• Al-Kofahi, M. K. (2020). Information Systems Success Model: A Review of Literature. International Journal of Innovation, Creativity and Change. Retrieved from: https://www.ijicc.net/images/vol12/ihttps://www.ijicc.net/images/vol12/iss10/121013_Kofahi_2020_E_R.pdf
  » https://www.ijicc.net/images/vol12/iss10/121013_Kofahi_2020_E_R.pdf
• Angelina, R. J., Hermawan, A. & Suroso, A. I. (2019). Analyzing E-Commerce Success using DeLone and McLean Model. Journal of Information Systems Engineering and Business Intelligence, 5(2), 156-162. https://doi.org/10.20473/jisebi.5.2.156-162
  » https://doi.org/10.20473/jisebi.5.2.156-162
• Brown, I.& Jayakody, R. (2008). B2C e-commerce success: A test and validation of a revised conceptual model. The Electronic Journal Information Systems Evaluation, 11(3), 167-184.
• DeLone, W. H.& McLean, E. R. (1992). Information systems success: The quest for the dependent variable. Information systems research, 3(1), 60-95. https://doi.org/10.1287/isre.3.1.60
  » https://doi.org/10.1287/isre.3.1.60
• DeLone, W. H.& McLean, E. R. (2003). The DeLone and McLean model of information systems success: a ten-year update. Journal of management information systems, 19(4), 9-30. https://doi.org/10.1080/07421222.2003.11045748
  » https://doi.org/10.1080/07421222.2003.11045748
• DeLone, W. H.& Mclean, E. R. (2004). Measuring e-commerce success: Applying the DeLone, McLean information systems success model. International Journal of electronic commerce, 9(1), 31-47. https://doi.org/10.2307/27751130
  » https://doi.org/10.2307/27751130
• Creswell, J. W.& Creswell, J. D. (2017). Research design: Qualitative, quantitative, and mixed methods approaches. Sage Publications.
• Donthu, N.& Gustafsson, A. (2020). Effects of COVID-19 on business and research. Journal of Business Research, 117, 284-289. https://doi.org/10.1016/j.jbusres.2020.06.008
  » https://doi.org/10.1016/j.jbusres.2020.06.008
• EBIT. Webshoppers 39a Edição. 2019. Retrieved from: https://bit.ly/2TzBf0h
• Fornell, C.& Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of marketing research, 18(1), 39-50. https://doi.org/10.2307/3151312
  » https://doi.org/10.2307/3151312
• Freund, R. J., Wilson, W. J.& Sa, P. (2006). Regression analysis. Elsevier.
• Hair Jr, J. F., Gabriel, M. L. D. D. S.& Patel, V. K. (2014). Modelagem de Equações Estruturais Baseada em Covariância (CB-SEM) com o AMOS: Orientações sobre a sua aplicação como uma Ferramenta de Pesquisa de Marketing. Revista Brasileira de Marketing, 13(2), 44-55. https://doi.org/10.5585
  » https://doi.org/10.5585
• Hasanat, M. W., Hoque, A., Shikha, F. A., Anwar, M., Hamid, A. B. A., &Tat, H. H. (2020). The Impact of Coronavirus (Covid-19) on E-Business in Malaysia. Asian Journal of Multidisciplinary Studies, 3(1), 85-90.
• Hirsch, P.B. (2020). A breakfast of champions: brand marketing lessons from the Great Depression. Journal of Business Strategy, 41(4), 63-67. https://doi.org/10.1108/JBS-04-2020-0081
  » https://doi.org/10.1108/JBS-04-2020-0081
• Jusop, M., Ismail, R. M.& Ismail, N. A. (2018). Understanding Halal E-Commerce Practice: A Proposed Conceptual Framework. Global Business &Management Research, 10(3).
• Krus, D. J.& Helmstadter, G. C. (1993). The problem of negative reliabilities. Educational and Psychological Measurement, 53(3), 643-650. https://doi.org/10.1177/0013164493053003005
  » https://doi.org/10.1177/0013164493053003005
• Naidoo, V. (2010). Firm survival through a crisis: The influence of market orientation, marketing innovation and business strategy. Industrial Marketing Management, 39(8), 1311-1320. https://doi.org/10.1016/j.indmarman.2010.02.005
  » https://doi.org/10.1016/j.indmarman.2010.02.005
• Nguyen, H. V., Tran, H. X., Van Huy, L., Nguyen, X. N., Do, M. T., Nguyen, N. Online Book Shopping in Vietnam: The Impact of the COVID-19 Pandemic Situation. Publishing Research Quarterly, 36(3), 437-445, 2020. https://doi.org/10.1007/s12109-020-09732-2
  » https://doi.org/10.1007/s12109-020-09732-2
• Oliveira, L. K., Morganti, E., Dablanc, L.& Oliveira, R. L. M. (2017). Analysis of the potential demand of automated delivery stations for e-commerce deliveries in Belo Horizonte, Brazil. Research in Transportation Economics, 65, 34-43. https://doi.org/10.1016/j.retrec.2017.09.003
  » https://doi.org/10.1016/j.retrec.2017.09.003
• Ringle, C. M., Da Silva, D.& Bido, D. D. S. (2014). Modelagem de equações estruturais com utilização do SmartPLS. Revista Brasileira de Marketing, 13(2), 56-73. https://doi.org/10.5585/remark v.13i2.2717
  » https://doi.org/10.5585/remark
• Seetharaman, P. (2020). Business models shifts: Impact of Covid-19. International Journal of Information Management, 54, 102173.
• Shannon, C. E.& Weaver, W. (1949). The mathematical theory of information. Urbana: University of Illinois Press.
• Sharkey, U.,Scott, M.& Acton, T. (2010). The influence of quality on e-commerce success: an empirical application of the DeLone and Mclean IS success model. International Journal of E-Business Research (IJEBR), 6(1), 68-84. https://doi.org/10.4018/jebr.2010100905
  » https://doi.org/10.4018/jebr.2010100905
• Sharma, H.& Aggarwal, A. G. (2019). Finding determinants of e-commerce success: a PLS-SEM approach. Journal of Advances in Management Research. https://doi.org/10.1108/JAMR
  » https://doi.org/10.1108/JAMR
• Social Miner. (2019). Retrieved from https://bit.ly/2A0Hh33
  » https://bit.ly/2A
• Song, P., Zheng, C., Zhang, C.& Yu, X. (2018). Data analytics and firm performance: An empirical study in an online B2C platform. Information &Management, 55(5), 633-642. 10.1016/j.im.2018.01.004
• Subaeki, B., Rahman, A. A., Putra, S. J.& Alam, C. N. (2019). Success model for measuring information system implementation: Literature review. Journal of Physics: Conference Series. IOP Publishing, 1402(7), 77015). Retrieved from: https://iopscience.iop.org/article/10.1088/1742-6596/1402/7/077015
  » https://iopscience.iop.org/article/10.1088/1742-6596/1402/7/077015
• Tam, C..& Oliveira, T. (2016). Understanding the impact of m-banking on individual performance: DeLone & McLean and TTF perspective. Computers in Human Behavior, 61, 233-244. https://doi.org/10.1108/IntR-05-2016-0117
  » https://doi.org/10.1108/IntR-05-2016-0117
• Wang, Y. S. (2008). Assessing e-commerce systems success: a respecification and validation of the DeLone and McLean model of IS success. Information Systems Journal, 18(5), 529-557. https://doi.org/10.1108/IntR-05-2016-0117
  » https://doi.org/10.1108/IntR-05-2016-0117
• Wang, Y., Hong, A., Li, X.& Gao, J. (2020). Marketing innovations during a global crisis: A study of China firms’ response to COVID-19. Journal of Business Research, 116, 214-220.https://doi.org/10.1016/j.jbusres.2020.05.029
  » https://doi.org/10.1016/j.jbusres.2020.05.029
• Wani, M., Raghavan, V., Abraham, D.& Kleist, V. (2017). Beyond utilitarian factors: User experience and travel company website successes. Information Systems Frontiers, 19(4), 769-785. https://doi.org/10.1007/s10796-017-9747-1
  » https://doi.org/10.1007/s10796-017-9747-1
• Watanabe, T.& Omori, Y. (2020). Online consumption during the covid-19 crisis: Evidence from Japan. Covid Economics, 32, 208-241.
• Wie, L. A. Y. P.& Widjaja, A. W. (2017). B2C E-commerce Site Success Factors: A Comparison between Indonesia, Japan, and South Korea. International Journal of Economics &Management, 11(S2), 505-527.
• Yel, M. B., Sfenrianto, S..& Anugrah, R. D. (2020). Using DeLone and McLean model for evaluating an e-commerce website. IOP Conference Series: Materials Science and Engineering. IOP Publishing. 725(1), 012108. Retrieved from: https://iopscience.iop.org/article/10.1088/1757-899X/725/1/012108/meta
  » https://iopscience.iop.org/article/10.1088/1757-899X/725/1/012108/meta
• Zheng, K., Zhang, Z.& Song, B. (2020). E-commerce logistics distribution mode in big-data context: A case analysis of JD. COM. Industrial Marketing Management. 86, 154-162. https://doi.org/10.1016/j.indmarman2019.10.009
  » https://doi.org/10.1016/j.indmarman

APPENDIX

To access the form used for data collection, access: https://drive.google.com/drive/folders/1xUh0cbyoNgtdoEBQpMA4YCBCTCJOar-o?usp=sharing

The databases were made available in link format on Google Drive. Therefore, to analyze the database as, access:https://drive.google.com/drive/folders/1_1SwQXApKjYzdz3TohFjFCufogduZ89o?usp=sharing

To view descriptive analyzes of data collected before and during the pandemic, access: https://drive.google.com/drive/folders/1zhp3VRDQStfJdwib2zOr_7OI97Z7ZzHF?usp=sharing

Publication Dates

• Publication in this collection
  22 Sept 2023
• Date of issue
  2023

History

• Received
  03 Mar 2022
• Accepted
  30 Dec 2022