A model for evaluating efficiency - an application in information technology and systems investments

A model for evaluating efficiency - an application in information technology and systems investments

Ana Paula Henriques de GusmãoI,^*; Ana Paula Cabral Seixas Costa^II

^INucleus of Technology / CAA, Universidade Federal de Pernambuco (UFPE). E-mails: [e-mail] / [e-mail]

^IIDepartment of Production Enginering / DEP, Universidade Federal de Pernambuco (UFPE). E-mail: [e-mail]

ABSTRACT

Despite being widely applied in real problems that tackle evaluating efficiency, Data Envelopment Analysis (DEA) models are frequently criticized on account of the weights of evaluation criteria often being defined loosely. Thus, approaches to incorporating value judgments in DEA models have been used in order to obtain more consistent results with managerial reality. It is against this background that this paper proposes a DEA model for evaluating efficiency, where the value judgments of those responsible for evaluation, regarding the criteria, are defined based on the philosophy of the SMARTS method and incorporated into the model by the Assurance Region (AR) method. The model proposed is applied using information about the investments made in the area of Information Technology and Information Systems by Brazilian banks The aim is to exemplify the application of the model and raise points for discussion with regard to its merits.

Keywords: Data Envelopment Analysis, SMARTS, Information Systems and Technology.

1 INTRODUCTION AND MOTIVATION

Data Envelopment Analysis (DEA) consists of solving a linear programming (LP) problem on the basis of the relative efficiency of the DMUs (Decision Making Units) observed such as banks, schools, industries, etc. What these DMUs have in common is that they convert the same set of inputs (resources) into the same set of outputs (results, expected outputs).

Classic DEA models allow complete freedom regarding the selection of weights that give the maximum efficiency value of the DMU. As is highlighted by Allen et al. (1997), this freedom is important in identifying inefficient units, i.e., those DMUs that have a poor performance, even with their own set of weights. Having the flexibility to choose weights is one of the advantages offered by the DEA model. However, the calculated weights may be inconsistent with a priori knowledge about the relative values of inputs and outputs.

The evolution of DEA models, so that they may incorporate value judgments in the definition of efficiency, can be understood as a result of applications of this method in real life organizations, as set out in studies such as those by Charnes et al. (1990), Bernroider & Stix (2006), Oliveira et al. (2008) and Gomes et al. (2009a). According to Allen et al. (1997), value judgments correspond to "logical constructs, incorporated within an efficiency assessment study, reflecting the Decision Makers' (DM) preferences in the process of assessing efficiency".

Some papers, with the aim of overcoming the limitation of DEA regarding the choice of weights, have suggested modifications of the constraints used in the LP problem. Angulo-Meza & Lins (2002) present a review of the methods used when incorporating DMs' value judgments in DEA and which, therefore, presents a greater discriminatory strength vis-à-vis the definition of efficient DMUs. This review is based on the work of Allen et al. (1997), which identifies the following approaches to incorporating value judgments in DEA:

- Direct restrictions on the weights.

- Adjusting the observed input-output levels.

- Restricting the virtual inputs and outputs.

The model proposed in this paper uses the approach of direct restrictions on the weights, in order to take into consideration relevant information in the context of evaluating efficiency. To do so, the weights restrictions, defined using the procedure Swing Weights of the SMARTS method (Simple Multi-attribute Rating Technique using Swing Weights) will be incorporated into the DEA model using the Assurance Region (AR) method.

It is worth mentioning that the use of DEA models, to measure the relative efficiency of production units, has been quite attractive in various application sectors. In this paper, the DEA model, with appropriate considerations, will be applied in the area of Information Technology (IT) and Information Systems (IS). The use of the proposed model to evaluate investments in this area aims to support the decisions of investors and organizations as a whole, by pointing out the sources of inefficiency and units that can serve as a benchmark for the practices adopted.

In the next section, a brief review of the literature on DEA is given. Then, the SMARTS method is presented. Section 4 discusses the proposition of this article, from a discussion regarding the integration of the two approaches presented in previous sections. In Section 5, there is discussion of the evaluation of investments in IS and IT in order to contextualize the application of the model. Finally, there is a presentation of the results obtained from applying the model with data from the 2003 ranking of the companies most computerized in 2002, according to the magazine Info Exame (2003).

2 REVIEW OF THE LITERATURE ABOUT DEA

The original DEA model was proposed by Charnes et al. (1978) and received important contributions from Cooper et al. (2000) and Thanassoulis (2001). The definition of efficiency in DEA is specified as the ratio of the weighted sum of outputs to the weighted sum of inputs of a DMU.

Although various forms of the efficiency frontier can be determined, there are two models of data envelopment analysis that are considered classics: that of the CCR or CRS (constant returns to scale) and that of BCC or VRS (variable returns to scale). The CCR model, originally presented by Charnes et al. (1978), works with constant returns to scale, that is, any change in inputs produces a proportional variation in the outputs. The BCC model, as per Banker et al. (1984), works with variable returns to scale, that is, it replaces the axiom of proportionality between input and output by the axiom of convexity. In a nonmathematical way, it can be argued that in the BCC a DMU is efficient if, on the scale in which it operates, it is the one that best takes advantage of the inputs available to it. In the CCR model, a DMU is efficient when it presents the best ratio of outputs relative to inputs, i.e. makes the best use of inputs without considering the operating scale of the DMU (Angulo-Meza et al., 2007).

The DEA model can be divided into: input oriented and output oriented versions. In the input oriented model, the goal is to minimize inputs, while keeping the output unchanged, whereas the output oriented model seeks to maximize results, while the inputs remaining unchanged (Soares de Mello et al., 2005). Independent of the orientation, the DEA model consists of an LP problem for each DMU, in which the decision variables are the weights for the inputs and outputs chosen.

For reasons that will be discussed below, the BCC model input oriented is used in this article for evaluating the efficiency of companies investing in IT/IS. This model can be represented as follows:

In this model, there are n DMUs (o = 1,..., n) to be evaluated, each of these DMUs consuming different quantities, x_ik, of the r (i = 1,..., r) inputs and producing different quantities, y_jk, of the s (j = 1,..., s) outputs. The weight of input i is represented by ν_i and u_j represents the weight of output j.

As already mentioned, the classic DEA models do not take into account the value judgments in the process of defining the weights of inputs and outputs. According to Allen et al. (1997), the reasons that prompt the incorporation of value judgments into a DEA are the needs of:

- incorporating prior views on the value of individual inputs and outputs;

- setting the ratios between the values of certain inputs and/or outputs;

- incorporating prior views on efficient and inefficient DMUs;

- the assessed efficiency needs to respect the economic notion of input/output substitution;

- enabling discrimination between efficient units.

The restrictions that will be incorporated into the DEA model in this article are categorized as direct restrictions on the weights. These, in turn, are subdivided into: Assurance Regions of type I (ARI), Assurance Regions type II (ARII) and absolute weights restrictions.

The absolute weight restrictions are mainly introduced to prevent the inputs or outputs from being over emphasized or under-estimated in the analysis. To do so, numerical bounds are imposed on these multipliers. On the other hand, neither ARI nor ARII works with absolute values but rather with ratio values between inputs or between outputs, or between inputs and outputs. The methods of Assurance Regions - developed by Thompson et al. (1990) - receive this name because they limit the variation of the weights to a particular region by adding restrictions, which have upper and lower bounds for each multiplier, to the classic DEA models.

In this paper, the proposed model adds the value judgments of DMs - the people responsible for investing in IT and IS - using the ARI method. These restrictions are presented as follows:

As already presented, ν_i represents the weight of input i. It is also possible to construct constraints for the weights of outputs. To do so u_j is used instead of ν_i in order to represent the weight of output j. The parameters κ_i, α_i and β_i are constants specified by DMs, which reflect their value judgments regarding the relative importance of the weights of inputs or outputs (Allen et al., 1997).

The type (3) restriction is more prevalent, and reflects marginal rates of substitution in the context of Charnes et al. (1978). The limits for ARI depend on the value scale of inputs and outputs, i.e., they are sensitive to the units of measurement for these factors (Soares de Mello et al., 2005).

Charnes et al. (1990) and Thompson et al. (1990) note that when imposing ARI there will always be at least one efficient DMU. Although this concept is easily understood, the ARI method requires upper and lower bounds to be defined for constructing weights restrictions on inputs and outputs. The choice of these bounds depends on the data and in some cases, requires the judgments of experts (Amatatsu & Ueda, 2009). In this sense, there is not a method that addresses all cases, and different approaches may be appropriate in different contexts (Allen et al., 1997). There is not only one process capable of determining the numerical values of the limits (Thanassoulis & Dyson, 1988). Many researchers have proposed various methods for determining the limits of the ratio between the weights of the criteria. Thanassoulis and Dyson (1988) discuss the use of regression analysis, Ueda (2000) and Ueda (2007) apud Amatatsu & Ueda (2009) discuss the canonical correlation analysis, Roll et al. (1991) and Golany & Roll (1993) use weights that are obtained from solving DEA models without restrictions. Beasley (1990) and Kornbluth (1991) suggest setting limits based on expert judgments.

The next section discusses the use of the procedure of Swing Weights of the SMARTS method for setting the upper and lower limits for the ratios between the criteria weights. This procedure, therefore, arises as an alternative to setting restrictions of the Equation (3) type.

A similar proposal is presented by Oliveira et al. (2008), who discuss the use of the multicriteria decision aiding method MACBETH for constructing restrictions for the multipliers, which are incorporated into the DEA model through the ARI method. This paper presents the mathematical equivalence that can exist between the MACBETH weights and the values used in ARI type weight restrictions. Other relevant papers in this context (Soares de Mello et al., 2002; Vieira Junior, 2008) have discussed the semantic equivalence from the use of bounds of the weights, obtained by the MACBETH method, in a weights restrictions approach named Virtual Weight Restrictions.

3 SMARTS METHOD

Before discussing the SMARTS method, it is important to situate it within the multicriteria approach. MCDM (Multiple Criteria Decision Making) methods have gained much attention in operational research due to the fact that many practical problems are represented by multiple and conflicting criteria.

A multicriteria decision problem can be expressed by a decision matrix D (n × s), where the elements d_kj represent the assessment or utility of alternative k, A_k (for k = 1,..., n), with respect to a set of criteria (or attributes) j, C_j (for j = 1,..., s). Thus, the alternative A_k can be represented by the vector A_k = (d_k1,..., d_ks).

To represent the importance of the criteria, a weight w_j for each criterion can be determined resulting in a vector w = (w₁,..., w_s).

The process of establishing weights for the criteria is one of the most important and difficult steps in MCDM methods. In SMARTS, the weight does not only depend on the degree of importance of the criterion, but also on established reference values. Therefore, it is more appropriate to talk about constants of scale than about weights when dealing with compensatory MCDM methods, like SMARTS. Scale constants are associated with substitution rates, which suggests that the disadvantage of an alternative in relation to a criterion can be balanced by the advantage of this alternative in relation to other criteria (Almeida, 2010).

The SMARTS method, developed by Edwards & Barron (1994), is a method of multi-attribute utility measurement, which presents a procedure for eliciting weights, known as Swing Weights. The advantage of this procedure is the simplicity of both the responses required from the DM as well as the analysis that is made of these responses.

SMARTS does not require judgment of preference or indifference between the alternatives, as required in most MCDM methods. In this method, the definition of the weight (scale constant) of a criterion is performed by analyzing the assessments of all alternatives to this criterion and not only its relevance to the DM. If all alternatives have similar assessments for the same criterion, the latter becomes meaningless in the decision process, although it can still be extremely important for decision makers when considering other alternatives (Gusmão, 2009).

The Swing Weights procedure is divided into two stages. In the first step, an ordering of criteria from the DM's value judgments is obtained, by taking into account the evaluations of alternatives for each criterion. Then, the scale constants are defined for each criterion.

In the first step, to obtain a ranking of the criteria, a hypothetical alternative (imaginary) is set that has the worst assessments in all criteria involved in the decision process, by taking into consideration the evaluations of real alternatives. The DM is then asked to state for which criterion he/she would first like to increase the assessment from worst to best. The criterion chosen is then deemed to be the most important one. Then, the DM reviews the remaining criteria and again chooses one at a time the criteria for which he/she would like to improve the assessment. Thus, a ranking of the evaluation criteria is achieved.

In the next stage of Swing Weights, the constants of scale of the criteria are obtained. To do so, a scale of 0 to 100 points is defined, where the criterion deemed the most important one receives the highest score. Then, the decision maker is asked about the weight of the second most important criterion, given the point on the scale of 0 to 100 points of the most important one. This idea is most easily understood when working with monetary scales and asks how much it costs, i.e., what is the value, on a monetary scale, of the best evaluation of all other criteria.

When considering, for example, the criterion C₂ as the most important in a set of criteria C_j (j = 1,..., s), the decision maker must determine the value S_j (j = 1,..., s) such that:

where S_j is defined by the DM within the range of 0 to 100 points of the criterion C₂. The term represents the amount of C₂ required to equal the value of this criterion to criterion C_j. Of course, since C₂ is preferable to all other criteria, S_j will always be less than or equal to 100∀j ≠ 2. For j = 2, S_j will be equal to 100.

The representation of Equation (4) in the form w_j/w₂ = S_j/100 gives the notion of a ratio between the scale constants of criteria j and 2. It is clear that both the scale constants obtained from the Swing Weights procedure and the values α_i and β_i of Equation (3), reflect the marginal rates of substitution. Thus, in a way that is similar to obtaining ratios between the scale constants in the Swing Weights procedure, the values α_i and β_i of Equation (3) can be set.

It is important to distinguish the concepts of the limits α_i and β_i from the terms ν_i and u_j addressed in the ARI method. While the former reflects substitution rates (the ratio between the weights ν_i and u_j) of inputs or outputs, the terms ν_i and u_j are scale constants (weights) attached to the ith input and the jth output, respectively, and are the variables of the model. Thus, the purpose of this paper is not about running the SMARTS method with the data from the problem and using the scale constants obtained from this method as restrictions on the weights of DEA model. That would be a mistake. The idea here is to build the bounds of the equation of the ARI method, following the philosophy of ordering and obtaining the ratios between the weights - substitution rates - of the Swing Weights procedure.

4 THE MODEL PROPOSED

As presented previously, the use of DEA to evaluate efficiency is common in the business environment and there has been much discussion on its features and improvements in the academic environment. This article seeks to add the value judgments of those responsible for the efficiency assessment on combining a DEA model with other methods.

In a similar way to calculating S_i in the Swing Weights procedure, the values α_i and β_i of Equation (3) can be set.

From the SMARTS Swing Weights procedure, it is possible to set a ratio between the weight of a certain criterion C_j and the weight of another (e.g. C_l seen as the most important criterion by DM), using the Equation:

Thus, if in the Swing Weights procedure, instead of the DM being questioned about the exact value of S_j, he had to define maximum and minimum values, besides this requiring less from the DM, as this offers more flexibility with respect to his/her judgments, equations would be obtained for each C_j of the type:

where I_j/100 represents the minimum value for the ratio between the weight of criterion C_j and the weight of criterion C_l. So this value is comparable to the parameter α_i of restrictions of the ARI type. Similarly, the maximum value for the ratio between the weight of criterion C_j and the weight of criterion C_l is comparable to the parameter β_i of restrictions of the ARI type. The value w_j can represent both the weights of inputs (ν_i) and the weights of outputs (u_j). Thus, Equation (6) has the same interpretation as Equation (3).

Therefore, we can incorporate into the DEA the restrictions of the Equation (6) type in order to effectively differentiate the alternatives that are efficient when considering the DM's value judgments on the relationships between the weights of the model.

Before applying the proposed model, it is important to contextualize the problem of evaluating investments in IT/IS.

5 EVALUATION OF IT/IS INVESTMENTS

Ballantine & Stray (1998) argue that there are two main reasons why evaluating IT/IS investments has become important. The first is that organizations have dedicated large amounts oftheir resources to such investment, as shown above. According to the International Data Corporation (IDC), the global IT market grew by 8% year on year to more than 1.5 trillion US dollars in 2010. The expectation for this market in 2011 is a growth rate of 7% (IDC 2011). Within this context, Brazil has stood out. It is the country in the BRIC block (Brazil, Russia, India and China), which ranks second in investing in this sector (30 billion US dollars in 2010), according to IDC (2010). The second reason is that, given the increasing levels of spending on IT/IS and the competitive environment faced by companies, assessing the effectiveness of these technologies has become of ever greater concern to them.

A review of the literature reveals there is constant use of traditional financial techniques such as tools to support decision making in the process of prioritizing IT/IS. Two of the most sophisticated of these techniques are NPV (Net Present Value) and IRR (Internal Rate of Return). However, IT/IS investments are not of the same nature as other capital investments. Traditional financial evaluation techniques do not allow the inclusion of important and intangible costs and benefits of investing in this segment (Ballatine & Stray 1998; Adelakun & Jennex 2002).

Thus, measuring the success of investments in systems and technologies remains a top concern for both practitioners and researchers due to the high investments made, the number of IS/IT failures, and the paradox of high investments and low productivity returns (Urbach et al., 2008). According to Brynjolfsson & Hitt (1998) this paradox, known as the "productivity paradox", deals with the apparent contradiction between the remarkable advances in computer power and the relatively low growth of productivity in various sectors of the economy. According to these authors, the problem very often lies in the way that productivity is measured. There are two aspects of productivity that are increasingly important if very accurate measurements are to be taken: outputs (results, expected products) and inputs (investment, consumables). Appropriate measures for outputs should include not only the quantity of products manufactured, but especially the value created - in terms of quality, customization, etc - to customers. Likewise, appropriate measures of inputs include not only hours of work, but also the amount and quality of the equipment used, materials and other resources that are consumed, training the workforce, besides the amount of organizational capital required.

According to Brynjolfsson (1993), productivity is the fundamental measure of the contribution of technology. Therefore, productivity and, consequently efficiency, are important and appropriate performance measures with which to evaluate investments in IT/IS.

5.1 Using the DEA model to evaluate IT/IS investments

Within the context of evaluating investments in IT/IS, DEA has been widely used. Shafer & Byrd (2000), Sowlati et al. (2005) and Bernroider and Stix (2006) make contributions in this context.

Some of the advantages of using DEA, in relation to the traditional models used in evaluating investments in technology and systems, are the facts of being able to consider various aspects in the evaluation, in addition to using the idea of better utilization of the resources (investments made). The evaluation criteria can be strategic, operational, financial, etc. Another advantage is the possibility of conducting this evaluation so as to consider the impacts of the investments in different periods, which will permit a time analysis to be made.

One of the greatest limitations when using the DEA model is that the results can only be interpreted in relation to the inputs and outputs included in the model. Different inputs and outputs result in different values for the efficiencies of the alternatives under study. Therefore, inputsand outputs aligned with the strategic objectives of the organization should be used in the model.

6 APPLICATION AND RESULTS FROM THE MODEL PROPOSED

The purpose of this section is to conduct an implementation of the model proposed. To do so, we used the data presented in the annual editions of Info Exame which list the major companies that invest in IT in Brazil.

Despite Info Exame annually publishing the ranking of the "100 most connected companies in Brazil", since 2005 the evaluation of companies, made by the magazine, has focused more on operational and financial factors than on strategic criteria. Thus, because the 2003 rankings also show information about the strategic benefits from the investments in IT/IS, it was decided to work with the data published in that year.

Before applying the model per se, the problem should be structured, which means defining: the units to be assessed (DMUs), the evaluation criteria (inputs and outputs) and the type of DEA model (CCR, BCC, and so forth; input oriented, output oriented, etc.).

6.1 DMUs

Companies from various sectors of the economy are included in the Info Exame survey. Due to the proposed model carrying out comparative assessments, we tried to work with companies operating in the same industry sector. From this premise and with a view to working with a reasonable number of alternatives, a set of DMUs was defined on which the assessment will be conducted. The companies selected are companies that operate in the banking sector. The application used the data presented in 2003 which referred to data from 2002.

Another relevant factor considered is that the banking sector has been the business sector which most invests in software, hardware, networking technologies and telecommunications, among others. In fact, banks have always occupied the top positions in the Info Exame ranking. Investments by this sector in IT/IS totaled R$ 19.4 billion in 2009, according to the Febraban (2010), which is justified primarily due to great dependence of banking products and services on these technologies.

The work of Becker et al. (2003) assesses the efficiency of investments made by the banking sector using the DEA model, without, however, considering DMs' value judgments.

6.2 Inputs and outputs

Having defined the DMUs, we tried to identify the inputs and outputs that would be used toassess them from the model proposed. The definition of inputs and outputs considered the objective of the model proposed: assessing the efficiency of investments in IT/IS made by organizations, with respect to financial, operational and strategic aspects. To do so, the input considered was input the investment in IT, in 2002 (in thousands of US$). The outputs used were: turnover in 2002 (in thousands of US$), storage capacity (in terabytes), the number of business transactions that can be performed through the Internet (purchase and/or sale) and the % of the management carried out using Intranet resources.

It is worth noting that the input of Investment in IT, in 2002, does not take into account either the maintenance costs of the IT infrastructure or the payroll of professionals who work in this area, as happens in most companies where these costs are incorporated into the budget but not into organizational investments.

Table 1 lists the DMUs and shows their performance in relation to the inputs and outputs.

If on the one hand, a model with many variables can result in an extremely benevolent assessment with several efficient DMUs (Gomes et al., 2009), on the other, the values of the efficiencies cannot be estimated correctly if the data does not contain a sufficient number of DMUs(Bendoly et al., 2009). At the stage of defining and selecting the DMUs and the input and output variables, some authors suggest that the number of DMUs should be greater than the product of the number of inputs and outputs (Soteriou & Zenios 1998), or three times the sum of inputs and outputs (Chen & Zhu 2004). For Dyson et al. (2001), the number of DMUs should be at least twice the product of the number of inputs and the number of outputs, in order to reach a reasonable level of discrimination. Therefore, having a set of twelve DMUs, one input and four outputs, meets the rules proposed by these authors.

Although some authors (Byrd & Shafer, 2000; Brynjolfsson & Hitt, 1998) claim that IT investments typically involve benefits over the years that follow the investment, it is possible in some situations, to realize the benefits soon after these investments. We must therefore consider the type of organization that is being analyzed, the focus of investments - the trend is that investments in hardware have a more immediate effect than investments in software like ERP - and the market in which the organization operates. The banks, although they invest in management systems in order to improve management efficiency, have other much more latent purposes in mind: those of making their processes more agile and safer and of enabling an increasing amount of data to be processed. Since this is a service that involves interaction with a lot of customers, the automation of their processes becomes important. Due to these characteristics, banks target most of their investments on acquiring hardware. Thus, investments in IT/IS tend to involve more immediate benefits for the banking sector. This can be seen in Figures 2 and 3, which relates the investments made in 2002 with turnover in 2002 and 2003, respectively.

It can be concluded, from Figures 2 and 3 that investments in IT/IS, made by organizations in 2002, present a linear relationship with the 2002 turnover (R = 0.8345), higher than that of the 2003 turnover (R = 0.7798). This conclusion can also be interpreted as follows: in general, the financial benefits from investing in IT/IS in 2002 were perceived in the same year more than in the following year. Although the analysis has focused on the linear relationship between these variables, higher values are obtained for the correlation coefficient when polynomial terms are added, yet the greater fitness of the model is preserved when the variables of investment and turnover correspond to those of 2002.

As recommended by Gomes et al. (2009b), in this stage, it is important to check whether or not there is a causal relationship between the variables of input and output. To do so, non-parametric and parametric regression model can be used. Figure 3 presents the graph of linear regression analysis for the output "Storage capacity" in relation to the input "Investment in IT in 2002".

It can be concluded, from the coefficients of determination (R²), calculated for the regression, that there is a significant linear relationship between this output and the input. Similar analysis was carried out for the other outputs and the results indicate a strong relationship (linear or polynomial) between the variables of input and output.

Regarding the use of inputs and outputs expressed as a ratio or percentage, Dyson et al. (2001) argue that this may be acceptable if all the inputs and outputs are of this kind. What is not possible is to mix variables expressed as a percentage or ratio with variables of absolute values. It is observed that in the data presented in Figure 1, an output (% of the management carried out using intranet resources) is expressed in percentages. To resolve this problem, Dyson et al. (2001) suggest several possible approaches, from which putting all data on the ratio scale through normalization was chosen for this. Normalization of data still becomes necessary when weight restrictions of the ARI type are used, if the intention is to measure the importance attached by each DMU to each variable (Gomes et al., 2009a). Thus, what is avoided is that the resulting values of weights will have been influenced by the scale of the data.

6.3 DEA model

The BCC DEA model will be used for the application proposed, since in this case there is no evidence that there is proportionality between inputs and outputs which allow the CCR model to be used. Hollingsworth and Smith (2003) further explain that the BCC formulation is more appropriate when inputs or outputs include ratio variables.

Starting from the assumption that organizations are increasingly seeking efficiency from investments made in IT/IS, by achieving benefits which incur the least expenditure of resources, this paper will work with the orientation to inputs.

6.4 Application of the traditional DEA model and the proposed model

When applying the DEA BCC input oriented model, with no restriction on the input and output weights, the efficiencies of each DMU are achieved. These are presented in Table 2.

Then, it is observed that from the twelve DMUs, eight were evaluated as having maximum efficiency. As already discussed, this usually occurs due to the flexibility of the LP problem, for each DMU, in determining the value of the weights of inputs and outputs to maximize the weighted sum of the results and thus maximizing efficiency.

Now, applying the model proposed here will require that a decision maker - who in this case can be the person responsible for IT investments in a particular company and who desires to compare the performance of his/her investments with other companies - expresses his/her preferences regarding the outputs. This judgment is not necessary for the input, because, in this case study, there is only one input.

Then, an imaginary DMU is built which presents the worst ratings for the outputs. The performance of the imaginary DMU is presented in Table 3.

At this point, the decision maker is asked to choose for which criterion (output) he would like to improve the performance of the imaginary DMU from the worst assessment for the best. For the output "IT Investment in 2002 (in thousands of US$)" the improvement would be of US$ 88 million to US$ 3,070 million. In the case of output "storage capacity (in terabytes)", the increase would be from 0.6 terabytes to 185 terabytes. Following this same logic, the decision maker looks at the possible increments of the other criteria and sets the criteria (output) for which the imaginary DMU should first have its performance improved.

Supposing that the DM considers the output "% of the management carried out using intranet resources" as the most important output, he/she gives the imaginary DMU the best score of 100 for that output. Once again, the DM, after this change, re-analyzes the imaginary alternative and defines the output for which he/she would next like to improve the performance of thisalternative. After this procedure is completed, an order of preference with respect to outputs is obtained. This ordering can be defined by the DM as follows: the "% of the management carried out using intranet resources" output is preferable to "the number of business transactions performed through the internet" output, which is preferable to that of "storage capacity (in terabytes)" which finally is preferable to that of "turnover in 2002 (in thousands of US$)".

Since the output of "% of the management carried out using intranet resources" is considered the most relevant output by the DM in this problem, the next step is to evaluate the other outputs on the scale 0 to 100 points of the output of "% of the management carried out using intranet resources" where 100 points is equivalent to 100% performance and 0 is equivalent to 10% performance. Then, the DM must set a maximum and minimum value which would take the best performance of each remaining output on this scale.

Thus, calling u₄ the weight of the output "% of the management carried out using intranet resources" and u₃ the weight of the output "the number of business transactions performedthrough the internet". The decision maker should set the values I_j and S_j such that:

Suppose that the DM has determined that the best performance of the output "the number of business transactions performed through the internet" has a maximum value which is equivalent to 90 points on a scale from 0 to 100 points in the "% of the management carried out using intranet resources" and at least 80 points, the following restrictions on the weights of outputs ν₄ and ν₃ could be established:

Similarly, other restrictions are set for the weights of outputs "storage capacity (in terabytes)" and "Turnover in 2002 ($ thousands U.S. $)" respectively:

Such restrictions, built with respect to the weight of the output considered most important, are aggregated to the LP problems for each DMU. These restrictions are of the ARI type and, when inserted into the BCC DEA input oriented model, generate the efficiency results presented in Table 4.

It can be seen that with the inclusion of weight restrictions in DEA, some DMUs have had the efficiency value of their investments modified. The DMUs Citibank, Nordeste Bank and Banco do Brasil, once considered 100% efficient, display a lower efficiency value, since the restrictions with respect to the weights of the criteria limited the choice of weights that would result in greater efficiency. The DMUs that have kept the value of efficiency calculated earlier - Fibra Bank, LIoyds TSB Bank, Unibanco, Itaú and Bradesco - are those that have already met the decision maker's preferences even before using weight restrictions.

By using the DM's value judgments, obtained by the proposed model, and incorporated into the DEA model through the ARI method, it is possible to distinguish the banking companies that have invested in IT/IS efficiently from the banks which were not as efficient.

The ranking of the companies by the criterion of efficiency value resulted in a ranking that is not very similar to the ranking of the Info Exame in 2003. Of course, similarity between these rankings could not be expected, since the evaluation method in this article is concerned not only with how the company is computerized or with the volume required to achieve the current level of computerization, but with the cost/benefit relationship. An evaluation model based on this relationship is very suitable for the current needs of companies with respect to the identification of technologies (software, hardware, communications, etc.) that provide the best returns. It also assists companies in identifying practices that can lead to more efficient IT investments, for example: restructuring the companies by adopting new systems, continuous training of the users involved in the IT area, use of monitoring and control mechanisms of the system's performance, etc. So it is worth the question: why is a small bank like Fibra considered more efficient with respect to its investments in IT/IS, than the Banco do Brasil? What are the practices adopted by the first company that led it to a greater exploitation of inputs?

Thus, the reading that should be done of the results obtained from applying the model is associated with the exploitation that each company gets from investments in IT/IS. This model allows any one company to carry out an assessment of its performance against the performance of other companies, which operate in same market segment in terms of efficiency of investments. It is important that firms make such an assessment not only to become increasingly competitive, thus maximizing the benefits from IT and IS, but also to guide future investment decisions in this area.

Companies like Banco do Brasil, whose investment in IT/IS, in 2002, was one of the largest considering all sectors of the economy, despite being among the 10 most highly computerized businesses according to the 2003 ranking of Info Exame, must consider other practices thatshould be adopted in order for its investments to have more beneficial effects for the business. After applying the model, it can be seen that, according to the value judgments of some DMs, this bank is not doing well in the question of making efficiency investments in IT/IS.

As a final consideration, it is important to notice that if this application was made based on data obtained directly from research conducted in organizations, it would be possible to test hypotheses and reach conclusions that better adhere to the reality of investments in IT/IS. However, the purpose of this application is only to illustrate how to use the proposed model and the reading that should be made from the results obtained.

7 FINAL CONSIDERATIONS

This paper presents a model for evaluating the efficiency of investments in IT/IS that uses the Data Envelopment Analysis approach. In order to aggregate users' and DMs' judgments and preferences, the philosophy of the SMARTS Swing Weights procedure was used to determine the limits of the ratios between the weights of criteria (inputs or outputs) and these were incorporated into the model by the method of Assurance Region of type I - ARI.

A limitation of this model is the fact that restrictions on the weights of the criteria (inputs and/or outputs) are constructed in relation to the criteria considered most relevant. However, it is not necessary to establish restrictions on the weights of all inputs and/or outputs. The restrictions should reflect the DM's value judgments, which may not present restrictions to the weight of one or more criteria. Future studies could also address the consideration of value judgments of more than one DM, thus studying the integration of group decisions with the proposed model.

Although the DEA model used in this paper was the BCC, the weight restrictions that it defines can be incorporated into the formulations CCR and BCC of the DEA. This is because the ARI method can be used in both formulations.

As an example of the use of the proposed model in real cases, one case study was conducted. The model was used to evaluate the efficiency of companies in the Brazilian banking sector, which invest heavily in IT/IS. This application becomes relevant because there is still much discussion in academic circles about the methods used to assess the contribution of computers and information systems make to companies and, in a broader sense, to the economy of a nation. The use of traditional financial models in this assessment has the disadvantage of not taking into account the strategic aspects of investments in IT/IS. Still, it must be taken into account that the benefits from adopting new systems and technologies are not immediately perceived by firms, and changes are often required in organizational structure and in the actual operation of the company if it is to achieve its purpose.

The data used in the case study were taken from the 2003 ranking of the "100 most connected companies of Brazil" to illustrate the application of the model and to discuss some differences from other forms of evaluating IT investments. Due to some assumptions, the quantities of DMUs, inputs and outputs were limited. Possibly a more significant number of DMUs and criteria could have been considered in the analysis, which obviously would lead to different results. For purposes of illustration of the application of the proposed model, however, the data used were satisfactory.

Although the DEA model used in the case study is input oriented, it is possible to work with output orientation, depending on the users' objectives. Restrictions on the weights of criteria focused more on the outputs, although it was shown that restrictions on both the weights of inputs, as on the weights of outputs, could be used by the proposed model.

Finally, it can be seen that the model is easily applicable and deals with concepts that are part of everyday business life. Yet the model presents another focus on comparing the performance of companies investing in IT and IS, by addressing not only the potential of each one in relation to these technologies, but the exploitation that they make of the technologies. From this, it was seen that the companies which are the most computerized are not always the ones that can extract the best that technology can offer.

• [1] Adelakun O & Jennex M. 2002. Stakeholder process approach to information system evaluation in: Proceedings of the 8th Americas Conference on Information Systems. AMCIS 2002, August 9-11. Dallas, Texas, USA.
• [2] Almeida AT. 2010. O conhecimento e o uso de métodos multicritério de apoio a decisăo. Recife: Editora Universitária - UFPE.
• [3] Allen R, Athanassopoulos A, Dyson RG & Thanassoulis E. 1997. Weights restrictions and value judgments in data envelopment analysis: evolution, development and future directions. Annals of Operations Research, 73: 13-34.
• [4] Angulo-Meza L, Soares de Mello JCCB, Gomes EG & Fernandes AJS. 2007. Selecçăo de variáveis em DEA aplicada a uma análise do mercado de energia eléctrica. Investigaçăo Operacional, 27: 21-36.
• [5] Angulo-Meza L & Lins MPE. 2002. Review of methods for increasing discrimination in data envelopment analysis. Annals of Operations Research, 116: 225-242.
• [6] Revista Info Exame. 2003. As 100 empresas mais ligadas do Brasil, Săo Paulo, 205: 69-78.
• [7] Banker RD, Charnes A & Cooper WW. 1984. Some models for estimating technical scale inefficiencies in data envelopment analysis. Management Science, 30: 1078-1092.
• [8] Ballatine J & Stray S. 1998. Financial appraisal and the IS/IT investment decision makingprocess. Journal of Information Technology, 13: 3-14.
• [9] Beasley J. 1990. Comparing university departments. Omega, 18(2): 171-183.
• [10] Becker JL, Lunardi GL & Maçada ACG. 2003. Análise de eficięncia dos Bancos Brasileiros: um enfoque nos investimentos realizados em Tecnologia de Informaçăo (TI). Revista Produçăo, 13(2): 70-81.
• [11] Bendoly E, Rosenzweig ED & Stratman JK. 2009. The efficient use of enterprise information for strategic advantage: a data development analysis. Journal of Operations Management, 27: 310-323.
• [12] Bernroider EWN & Stix V. 2006. Profile distance method - a multi-attribute decision making approach for information system investments. Decision Support Systems, 42: 988-998.
• [13] Brynjolfsson E. 1993. The productivity paradox of information techonology. Communications of the ACM, 35: 66-77.
• [14] Brynjolfsson E & Hitt LM. 1998. Beyond the Productivity Paradox: Computers are the Catalyst for Bigger Changes. Communications of the ACM, 41: 49-55.
• [15] Charnes A, Cooper WW & Rhodes E. 1978. Measuring the efficiency of decision making units. European Journal of Operation Research, 2: 429-444.
• [16] Charnes A, Cooper W, Huang Z & Sun D. 1990. Polyhedral cone-ratio DEA models with an illustrative application to large commercial banks. Journal of Econometrics, 46: 73-91.
• [17] Chen Y & Zhu J. 2004. Measuring information technology's indirect impact on firm performance. Information Technology and Management, 5(1): 9-22.
• [18] Cooper WW, Seiford LM & Tone K. 2000. Data Envelopment Analysis: A Comprehensive Text with Models, Applications, References and DEA-Solver Software Boston: Kluwer Academic Publishers, USA.
• [19] Dyson RG, Allen R, Camanho AS, Podinovski VV & Sarrico CS. 2001. Pitfalls and protocols in DEA. European Journal of Operational Research, 132(2): 245-259.
• [20] Dyson RG & Thanassoulis E. 1988. Reducing weight flexibility in data envelopment analysis. Journal of the Operational Research Society, 39(6): 563-576.
• [21] Edwards W & Barron FH. 1994. Smarts and Smarter: Improved Simple Methods for Multi-attribute Utility Measurement. Organizational Behavior and Human Decision Processes, 60: 306-325.
• [22] FEBRABAN. 2010. O setor bancário em números.
• [23] Gomes EG, Soares de Mello JCCB & Mangabeira JAC. 2009a. Estudo da sustentabilidade agrícola em município amazônico com análise envoltória de dados. Pesquisa Operacional, 29(1): 23-42.
• [24] Gomes EG, Soares de Mello JCCB, Souza GDS, Angulo-Meza L & Mangabeira JADC. 2009b. Efficiency and sustainability assessment for a group of farmers in the BrazilianAmazon. Annals of Operations Research, 169(1): 167-181.
• [25] Gusmăo APH. 2009. Modelo para avaliaçăo de eficięncia de investimentos em Sistemas de Informaçăo. Tese (Doutorado em Engenharia de Produçăo) - Universidade Federal de Pernambuco. 119 p.
• [26] Hollingsworth B & Smith P. 2003. Use of ratios in data envelopment analysis. Applied Economics Letters, 10(11): 733-735.
• [27] IDC. 2010. Brazil IT Spending by State 2010, Săo Paulo, Săo Paulo.
• [28] IDC. 2011. Worldwide IT Spending Outperformed Expectations in 2010, Reaching $1.5 Trillion, According to IDC's Worldwide Black Book. IDC Press Release, Framingham, Massachusetts.
• [29] Kornbluth JSH. 1991. Analyzing policy effectiveness using cone restricted Data Envelopment Analysis. Journal of the Operational Research Society, 42(12): 1097-1104.
• [30] Oliveira LSM, Correia TCVD & Soares de Mello JCCB. 2008. Uso do método macbeth para gerar intervalos para os multiplicadores em modelos dea com restriçőes cone ratio in: XL Simpósio Brasileiro de Pesquisa Operacional - SBPO Joăo Pessoa.
• [31] Roll Y, Cook WD & Golany B. 1991. Controlling factor weights in Data Envelopment Analysis. IIE Transactions, 23(1): 2-9.
• [32] Roll Y & Golany B. 1993. Alternate methods of treating factor weights in DEA. Omega, 21(1): 99-109.
• [33] Shafer SM & Byrd TA. 2000. A framework for measuring the efficiency of organizational investments in information technology using data envelopment analysis. Omega, 28: 125-141.
• [34] Soares de Mello JCCB, Ângulo-Meza L, Gomes EG & Biondi Neto L. 2005. Curso de análise envoltória de dados in: Simpósio Brasileiro de Pesquisa Operacional, Gramado, 2521-2547.
• [35] Soares de Mello JCCB, Lins MPE, Soares de Mello MHC & Gomes EG. 2002. Evaluating the performance of calculus classes using operational research tools. European Journal of Engineering Education, 27(2): 209-218.
• [36] Soteriou AC & Zenios SA. 1998. Data envelopment analysis: an introduction and an application to bank branch performance assessment in: Marcoulides, G.A. (Ed.), Modern Methods for Business Research, Lawrence Erlbaum, Mahwah, NJ.
• [37] Sowlati T, Paradi JC & Suld C. 2005. Information Systems Project Prioritization using Data Envelopment analysis. Mathematical and Computer Modelling, 41: 1279-1298.
• [38] Thanassoulis E. 2001. Introduction to the Theory and Application of Data Envelopment Analysis: A Foundation Text with Integrated Software Boston: Kluwer Academic Publishers, USA.
• [39] Thompson RG, Langemeier LN, Lee C, Lee E & Thrall R. 1990. The Role of Multiplier Bounds in Efficiency Analysis with Application to Kansas Farming. Journal of Econometrics, 46: 93-108.
• [40] Ueda T & Amatatsu H. 2009. Determination of bounds in DEA Assurance Region Method - Its application to evaluation of baseball players and chemical companies. Journal of the Operations Research Society of Japan, 52(4): 453-467.
• [41] Urbach N, Smolnik S & Riempp G. 2008. A Methodological Examination of Empirical Research on Information Systems Success: 2003 to 2007 in: Proceedings of the 14^th Americas Conference on Information Systems. AMCIS 2008, August 14-17. Toronto, Ontario Canada.
• [42] Vieira Jr H. 2008. Multicriteria approach to data envelopment analysis. Pesquisa Operacional, 28(2): 231-242.

*

Corresponding author

Publication Dates

• Publication in this collection
  04 Dec 2012
• Date of issue
  Dec 2012