Efficiency evaluation of a small number of DMUs: an approach based on Li and Reeves's model

Efficiency evaluation of a small number of DMUs: an approach based on Li and Reeves's model

João Carlos Correia Baptista Soares de Mello^{I, *}; João Carlos Namorado Clímaco^II; Lidia Angulo Meza^III

^IDepartamento de Engenharia de Produção, Universidade Federal Fluminense (UFF), Niterói – RJ, Brasil, [e-mail], [e-mail]

^IIFaculdade de Economia da Universidade de Coimbra & INESC-Coimbra, Coimbra - Portugal, [e-mail]

^IIIDepartamento de Ciência dos Materiais, Universidade Federal Fluminense (UFF), Volta Redonda – RJ, Brasil, [e-mail]

ABSTRACT

This paper deals with the evaluation of Decision Making Units (DMU) when their number is not large enough to allow the use of classic Data Envelopment Analysis (DEA) models. To do so, we take advantage of the TRIMAP software when used to study the Li and Reeves MultiCriteria DEA (MCDEA) model. We introduce an evaluation measure obtained with the integration of one of the objective functions along the weight space. This measure allows the DMUs joint evaluation. This approach is exemplified with numerical data from some Brazilian electrical companies.

Keywords: data envelopment analysis; multiobjective linear programming; TRIMAP.

RESUMO

Este artigo trata da avaliação de Unidades Produtivas (Decision Making Units – DMUs) quando seu número é inferior ao recomendado na Análise Envoltória de Dados (Data Envelopment Analysis – DEA). Para isso é explorado o uso do software TRIMAP no modelo MCDEA (MultiCriteria DEA) de Li e Reeves. É proposto um índice de avaliação de desempenho baseado nos valores assumidos por uma das funções objetivo do modelo MCDEA. Estes valores, obtidos pelo TRIMAP, são integrados ao longo de todo o espaço dos pesos. O índice obtido permite uma avaliação de conjunto das DMUs avaliadas. O modelo é ilustrado com um exemplo numérico de avaliação de empresas distribuidoras de energia elétrica.

Palavras-chave: DEA; programação linear multiobjetivo; TRIMAP.

1. Introduction

For a number of years, several researchers have noticed that Decision MultiCriteria Aid Methods – MCDA and Data Envelopment Analysis – DEA have points in common. The former, MCDA, include those connected with Multiobjective Linear Programming – MOLP. So, many works have taken advantage of those common points from a different approach to improve on the one being used. The idea to use MOLP, in a DEA context, comes from the Pareto efficiency shared by both approaches. Clímaco et al. (2008) have revised and classified the works that follow this approach.

A class of works that use simultaneously MOLP and DEA is one that simply uses MOLP in DEA problems. Among those, the Li and Reeves's (1999) MCDEA model uses 3 objective functions to restrain the freedom to optimise the DEA-CCR (Charnes et al., 1978). The authors claim the model solves two common DEA problems: the excessive number of efficient DMUs and zero multipliers. However, the computational tool used for the Li and Reeves's work has partially limited the potentialities of the proposed model. As the model has 3 objective functions, the computational interactive environment of the TRIMAP software (Clímaco & Antunes, 1987; Clímaco & Antunes, 1989; Clímaco & Antunes, 2000) provides an adequate tool for a more comprehensive analysis and lays the ground to propose an evaluation index based on the MCDEA model.

MOLP and DEA share several concepts. To avoid naming confusions, the word "weights" will be used for the objective function weighing coefficients in the multiobjective problem, whereas the word "multipliers" will be used for the input and output coefficients in DEA models. Furthermore, the word "efficient" will always be used in a DEA context, whereas optimal Pareto solutions will be named "non dominated" for MOLP problems.

The Li and Reeves model will be reviewed in the next section. Next to it, TRIMAP will be briefly condensed and in section 4 it will be shown how TRIMAP can be used for the MCDEA model. Section 5 proposes an index based on this model, which is numerically exemplified in section 6. Finally, the conclusions of this paper are presented.

2. Li and Reeves's model

Researchers Xiao-Bai Li and Gary R. Reeves (1999) have introduced a multiobjective model aiming at solving two frequent problems in DEA: to increase and improve DMUs discrimination and promote a better multiplier distribution among variables. The first difficulty crops up when the number of variables is not much larger than that of inputs and outputs as standard models class too many DMUs as efficient. The second difficulty arises when a DMU is efficient with non-zero multipliers for only some of the variables thus benefiting those in which they have a good performance and ignoring those with a low performance. These two difficulties are closely interwoven.

So, to get around these difficulties, the authors propose a DEA multicriteria approach in which additional objective functions are included to optimise other relative efficiency measurements. These will be subject to the same constraints of the classic multipliers CCR model introduced by Charnes et al. (1978). The additional objective functions restrict the variable multipliers flexibility.

In DEA, a DMU is efficient when h₀ = 1, meaning that the constraint for that DMU is active and, thus, its slack is zero. The model basic assumption is to use the slack as an efficiencymeasurement instead of h. The slack is named d. So, the CCR model in (1a) can be reformulated as in (1b):

In these, v_i and u_r are respectively input i, = m , and output r, r = 1,...,s , multipliers; x_ij and y_rj are DMU j, j = 1,...,n inputs i and outputs r; x_io and y_ro are DMU 0 inputs i and outputs r. Conversely, d₀ is DMU 0 and d_j is DMUj deviation variable, i.e., the amount a DMU deviates from efficiency. In this model, DMU 0 is efficient if, and only if, d₀ = 0, i.e., h₀= 1. If DMU 0 is not efficient its efficiency score is h₀ = 1 – d₀. So, the more efficient a given DMU is, the smaller is d₀. As d₀∈ [0, 1] it can be seen as a measure of inefficiency, i.e., the lesser the value of d₀, the lesser the inefficiency between 0 and 1 or, in other words the more efficient is DMU₀. It should be emphasised that, in the Li and Reeves's model, the only deviation that varies between 0 and 1 is the one for the DMU being studied. Deviations for the other DMUs can be greater than one.

Therefore, it can be said that this formulation of the classic DEA method minimises the DMU inefficiency, as measured by d₀, with the constraint that the weighted sum of outputs be less than or equal to the weighted sum of inputs for each DMU.

The MCDEA model uses two other objective functions to restrain the freedom of choice of multipliers. One is the "equity" objective function to minimise maximum deviation that attempts to make all DMUs efficiencies homogeneous. The other is the "generalised benevolence" objective function that minimises the sum of deviations. At this point it should be emphasised that the "generalised benevolence" function tries to optimise the evaluation of all DMUs from the point of view of the DMU being studied. This is an evaluation procedure the bases of which are the same of the Doyle & Green (1994) benevolent cross evaluation. It should be emphasised that the term "benevolent" refers to how the DMU being studied evaluates the others. This evaluation ends up by restraining more the multipliers of the DMU being studied than the so-called "aggressive" evaluation. So, the MCDEA succeeds in limiting the multipliers with minisum objective function, as well.

Therefore, with these two additional objective functions, the multiobjective programming problem called Multiple Criteria Data Envelopment Analysis – MCDEA is formulated as in (2), in which the first objective function is the classic efficiency maximisation, whereas the second and third are respectively the "equity" and "generalised benevolence" objective functions.

As mentioned before, the classic efficiency of the DMU being studied is d₀ in the first objective function. We can also define the efficiency of a given DMU in respect of the second and third objectives: a DMU 0 is minimax efficient if and only if the d₀value corresponding to the solution that minimises the second objective function of model (2) is zero. Similarly, a given DMU 0 is minisum efficient if, and only if; the d₀value corresponding to the solution that minimises the third objective function of model (2) is zero.

The minimax and minisum objectives do not in general favour the classic efficiency of the DMU being studied, as opposed to the DEA classic objective. So, both objectives tend to restrain the classic efficiency (if DMU 0 is either minimax or minisum efficient it must also be DEA efficient as, by definition, both the minimax and minisum efficiencies require d₀= 0). On the other hand, if DMU 0 is DEA efficient, it may ot may not be minimax or n minisum efficient as d₀= 0 does not necessarily imply that M and/or be minima).

However in some cases, the minimax objective may not restrain much the multipliers combination that optimises the classic objective function. This tends to occur whenever the worst evaluated DMU has similar strong and weak points to the DMU being studied. In other words, when it has a similar distribution of multipliers. In these cases, the minisum objective becomes important to restrain the freedom to allocate multipliers because the minimax objective function did not do it.

It must be emphasised that the MCDEA model does not abandon the concept of efficient frontier as happens in the Cross Evaluation model (Sexton et al., 1986). The first objective function makes sure that analyses are still carried out near the efficiency frontier. Therefore the interpretation of the productivity frontier is not lost.

The authors mention in their original work that other objective functions may be used but no further works have been found on this subject. Further details on this theme can be found at Li and Reeve's (1999) original work or in the revisions carried out by Angulo-Meza & Lins (2002) or Clímaco et al. (2008).

In their original work the authors have only used, as a means to find a solution, the weighed sum and comment on the behaviour of one of the functions when the other is optimised. They have used the ADBASE software and claim there are no adequate computational tools, whether in performance or ease of use, to study the model.

There is indeed one such tool. The TRIMAP, which works with three objective functions is an adequate analysis tool and will be condensed forthwith.

3. TRIMAP

The TRIMAP method, developed by Clímaco & Antunes (1987, 1989), is a free search method that provides decision makers with a non dominated solutions learning tool. TRIMAP combines the reduction of the admissible region with the reduction of weights. Decision makers can specify their preferences by putting a floor to the objective function values and imposing restraints to the weights space. Making use of Roy's (1987) terminology, "convergence" (for whatever utility function) in TRIMAP should be replaced by "creation" so the interactive process become a construction and not the discovery of something pre-existing. Although this is the subjacent philosophy to TRIMAP, in this particular paper it will be used mostly as a generator of solutions and weights space analysis tool.

TRIMAP allows the generation of non-dominated solutions in tri-criteria linear programming problems. It comprehends a set of procedures of a set of non-dominated solutions free search, based on a progressive and selective learning process. Although the restraint of working with three objective functions generally limits usage, its graphic representation is very helpful. One of its representations is of special interest to study the Li and Reeves's model: the representation of the weights space. This paper is focused on the study of Li and Reeves's model so the TRIMAP description will emphasise its more useful features for that study. The reader who wishes to deepen the study of TRIMAP should read Clímaco et al. (2003). A typical use of TRIMAP in multiobjective decision problems is sketched on Figure 1.

TRIMAP starts by computing some of the objective functions values so as to obtain the first assessment of its values variation range in the convex polyhedron non-dominated region. At the beginning only non-dominated basic solutions that optimise each of the objective functions are computed. The user may request the calculation of other non-dominated basic solutions, which is done by the optimisation of a mono objective problem: the optimisation of the weighed sum of the three initial objective functions. Here, the system of weights cannot be part of any indifference region of the non-dominated solutions previously computed.

Some graphic tools are at the disposal of the user to study the problem. For the purposes of this paper, the most important is the graphic decomposition of the weights space in indifference regions corresponding to the non dominated basic solutions obtained so far. Indifference regions are those in which the objective functions weights can vary without alterations the solution obtained. Weight direct constraints can also be shown on this graphic. The same applies to the objective functions admissible values, which TRIMAP automatically conveys to the weights space.

TRIMAP will show a two dimensional projection for the objective functions space. All non-dominated solutions that have been computed so far are shown and non-dominated surfaces and their intersecting lines can be identified as well. The final graphic tool is the "spider's web" that shows for each objective function the difference between its current value and the reference point established by the user (the ideal point, eventually).

This set of graphs creates an understanding of the non-dominated region shape of the admissible polyhedron that can be improved as the "learning process" advances.

To acquire a better knowledge of the non-dominated area and, eventually, to generate all of some well chosen non-dominated basic solutions the user can proceed according to one of the following alternatives:

i) Selecting sets of weights corresponding to areas of the weights space not yet filled by indifference regions. Compute new non-dominated basic solutions and their indifference regions based on the selected weights.

ii) Selecting three basic solutions already known. A new non-dominated basic solution can be obtained from the optimisation of weighed sum of the three objective functions their gradient being perpendicular to the plane defined by the selected solutions. Should any of the weights be negative, the programme will automatically correct the gradient.

One can still sweep non-dominated faces with a similar interface to the "Pareto Race" method between two previously computed of its farthest points. The programme shows a line that goes forward or backwards over the face projection, its velocity being controlled by the user. The objective function values that correspond to the swept points are dynamically shown on a bar graph.

Beyond the graphic aspect, TRIMAP shows a text that condenses all the numeric results so far. Among other data, the objective functions basic variable values, the area share occupied by the indifference region and the Chebychev distance to the ideal solution can be shown for each non-dominated basic solution that has been computed.

4. Li and Reeves's model in TRIMAP

TRIMAP is a powerful tool to study the MCDEA model, thanks to its graphic representation of the weights space decomposition and the numeric values it offers. TRIMAP computes at once all optimal solutions for the objective function and identifies the non-dominated ones in the tri-criteria problem. Even for classic DEA this is a result of import, as traditional DEA programmes do not go any further than the first optimal result. The simple fact of being able to know when alternative multipliers sets exist and which of them correspond to a basic solution is a substantial improvement on the quality of the performed analyses.

Knowing the weights space decomposition makes possible to evaluate whether the DEA efficient solutions are stable. Indifference regions dimensions and shape do indeed give a clear idea of the each objective function (smaller or larger) weight combinations. These lead to the multipliers corresponding to the basic solution shown in that indifference region. On the other hand, large indifference regions allow the evaluation not to alter with objective functions formulations moderate changes. The existence or otherwise of alternative optima allows to check whether the DMU optimal evaluation depends on a single multipliers vector.

A simple check will immediately show whether a given DMU is minimax or minisum efficient: it suffices for an indifference region to contain simultaneously the triangle corners corresponding to both the optimisation of the classic objective function and one of the other two as the case may be.

The weights space decomposition further allows to take into account weight combinations that, despite not conferring to the DMU maximum efficiency, can be classed as good solutions as they do not reduce efficiency too much yet they confer better values to the other objective functions.

One can also choose a solution with a multipliers distribution closer to the decision maker's preferences because TRIMAP gives the multipliers values for each indifference region.

Next an evaluation index is proposed based on the conjunction of all these analyses.

5. MCDEA-TRIMAP efficiency index

The MCDEA model does not fully allow multipliers freedom of choice for the DMU being studied as it takes into account 3 objective functions. However, the imposed constraint depends on the chosen weights for each objective function in the multi objective model. It is even possible to hand back multipliers freedom of choice if one allows that the weights belong to any one of the indifference region that minimise the first objective function.

The weights assigned to each objective function lead to different DMU rankings and, of course, to different sets of the multipliers. As those weights vary within the indifference region, different solutions are created and a single ranking unobtainable. Nor is it easy to ask a decision maker to allocate weights because, despite the mathematical interpretations of the minimax and minisum objective functions being evident in MOLP, they may not be all that clear to a DEA trained decision agent.

As Li and Reeves state that there should not be any prevalence of an objective function over the others, it still seems clear that one must take into account the possible infinite combinations of the objective functions weights. Their values variation is continuous and so it seems adequate to integrate the classic objective function assumed value when the weighed sum of the three objective functions is optimised. This integration should cover the whole possible weights space meaning that the weight sum should be restrained to be one. Obviously, if the integration result is divided by the space size, the classic objective function average value in that space will be obtained. With a few adjustments we propose to use the complement of this value as the new efficiency index. Formally, the new index will be obtained from:

This equation includes FO1 that is the value of the first objective function of model (2).

To proceed with the calculation it suffices to note that the integrating expression is sectionally continuous in the weights space and that in each continuity region it is constant. So all that need to be done is multiply the classical objective function value times the area in which it has that value, add all the results and divide by the total area. An alternative is to calculate the weighted sum of the first objective function keeping as weights the area percentages for which each solution is valid. It should be noted that all these values are easily obtained from the TRIMAP. However, a small modification should be made to the original MCDEA model: as two of the objective functions measure the efficiency deviation of a single DMU and the formula contains the sum of several deviations, it is divided by the number of DMUs (this does not change their optimal point) to avoid distortions in the integration within the weights space.

It should also be noted that in accordance with the model's characteristics described by Li and Reeves this new index would be less, or equal, to the standard DEA efficiency.

6. Numerical example

A case study to evaluate the efficiency of electrical utilities will be carried out to illustrate the methodology proposed in this article. This study is an adaptation of the Senra & Soares de Mello (2004) work. The study assumes that the utilities' priority is social coverage meaning that they should provide the maximum possible number of consumers with the power they need. This approach is particularly important in regions where power distribution is not yet universal. For another approach to evaluate utilities please see (Lins et al., 2007).

Data provided by Sistema Integrado de Acompanhamento e Gestão Empresarial – SIAGE (Follow up and Entrepreneurial Management Integrated System) were used. The data correspond to the month of July 2002. Five non-network member utilities located in areas where serious electrification deficiencies still exist were selected. Those utilities are: Eletro Acre – EA, Manaus Energia – MESA, Boa Vista Energia – BOVESA, Companhia Elétrica do Amazonas – CEAM and Companhia Elétrica de Rondônia – CERON.

Inputs and outputs were chosen from a wide set of variables by Variable System methods (Senra et al., 2007). Each utility Operational Expenses and Fixed Assets were chosen as inputs whereas number of consumers and satisfied demand were chosen as outputs. For confidentiality reasons, data are presented in normal form. The DEA-CCR model was used to calculate these DMUs classical efficiency. It should be borne in mind that with four variables, five DMUs is a number well below the ideal for classic DEA models. As a matter of fact, of these five three were classed as efficient. However, this small number of DMUs, which is inconvenient for classic DEA, is ideal for the MCDEA index. Table 1 shows the normalised data and the DEA-CCR index obtained from the SIAD (Angulo-Meza et al., 2005).

Model (4) corresponds to the DMU EA evaluation. In his model x₁ and x₂ are respectively the multipliers values for inputs Operational Expenses and Fixed Assets. Variables y₁ and y₂ are the multipliers for Number of Consumers and Demand. Variables (d_i) represent deviation variables as explained hereabove.

Figures 2 show the weights space decomposition for the five DMUs: respectively EA, MESA, BOVESA, CEAM and CERON. Each painted area represents a weights indifference region for each objective function. This means that in each of these regions the values obtained for the decision variables are the same irrespectively of the weight used for each objective function. For further details about the interpretation of these pictures kindly read Clímaco & Antunes (1989).

Before calculating the index a qualitative analysis of the figures is needed as done by Clímaco et al. (2008). The triangle right hand lower corner corresponds to the first objective function or, in other words to the complement of the classic DEA efficiency index d0. The upper corner stands for the second objective function, the equity (minimax) function. Finally, the lower left hand corner corresponds to the third objective function, the generalized benevolent function or minisum.

Figures 2.a., 2.c. and 2.e show the efficient DMUs. Figures 2.a and 2.c indicate that there two solutions to increase the classical efficiency. This is an expected result given the existence of multiple optimal solutions for efficient DMUs multipliers problem. As examples consult Rosen et al. (1998), Soares de Mello et al. (2002, 2004, 2008a, 2009), Cooper (2007). On the other hand, Figure 2e represents an also efficient DMU that displays one single classic DEA objective function solution. This does not mean that the DMU has no further optimal solutions, only that TRIMAP has identified only of them because all the others are dominated in terms of the multiobjective formulation.

Figure 2.a. shows that one of the regions that correspond to one of the objective function 1 optimises as well objective function 3. So, in the Li and Reeves's terminology it is said that this DMU is minisum efficient besides being efficient. Figure 2.c shows DMU BOVESA, which is best evaluated by MCDEA model. It is both minisum and minimax efficient and the solutions that optimise objective function 1 (the classic DEA objective function) fully cover the triangle. On the other hand, the DMU shown in Figure 2.e. is just efficient. It is neither minisum nor minimax efficient.

Table 2 shows the MCDEA-TRIMAP efficiency index calculation for every DMU. The table was obtained from the numerical values given by TRIMAP that are numerical representations of the areas shown in Figure 2. TRIMAP indicates each region percentage of the triangle area as well as decision variable and objective functions values in each area. The objective function value is the only one of interest according to equation (3). The index is calculated by the sum of the products of the values of this objective function in each area times the area percentage.

The table above shows that this index has succeeded in establishing a strict ranking (Barba-Romero & Pomerol, 1997) for all DMUs. The most efficient utility is Boa Vista Energia – BOVESA. This means that this utility used best expenses and fixed assets to supply more power to the largest number of people. As opposed to the classic DEA model, in this model it is not possible to attain efficiency just by having the best ratio between a given input and output. In other words, variables cannot be discarded to attain efficiency. Multipliers choice is conditioned by the properties of the other DMUs and so the evaluation can be classed as an integrated evaluation (Angulo-Meza et al., 2003), in which decision makers are the DMUs themselves. The method used here has the advantage of not depending on external opinions (as happens in the multipliers constraint method). When compared to the Cross Evaluation method, this method makes for faster quantitative and qualitative analyses as well as maintaining the interpretation of the efficient frontier. On the other hand, the proposed method is not as simplistic as the composed efficiency calculation based on a inverted frontier (Soares de Mello et al., 2008b). This method has the disadvantage of not offering benchmarks for the inefficient units but it should be pointed that neither the Cross Evaluation nor the Inverted Frontier methods do it, either.

7. Conclusions

It has been shown in this article that the MCDEA model potentialities can be better used with he use of TRIMAP. Hereabove a MCDEA-TRIMAP efficiency index has been proposed. It will be able to evaluate the DMUs performance if the model behaviour on the whole objective functions weights space is taken into account. Similarly to the cross evaluation, the index value depends as much on the inefficient DMUs as on the efficient ones. Thus, it can be classed as an "joint evaluation system" (Angulo-Meza et al., 2003).

It has been shown throughout this paper that this index keeps one of the DEA basic tenets: a relative freedom of choice for the multipliers. This freedom is necessarily lesser than in the original DEA model but, on the other hand, it is not the fixed weight system that may occur in some important cases of cross-evaluation as Anderson et al. (2002) have shown.

This paper has shown how the MCDEA model can be adapted to improve the DEA power of discrimination.

The methods based on Multiobjective Linear Programming can be separated in two groups (Angulo-Meza & Lins, 2002): those that incorporate a priori information provided by a decision-maker and those that do not require any a priori information. The method shown here belongs to the latter group.

As in any other work that uses DEA the results are relative, i.e., the most efficient DMU is not necessarily an efficient DMU in absolute terms. In the particular case of the DMUs in the numeric example, the sector specialists had a poor evaluation of them all.

On the other hand, this work was focused on the creation of an evaluation index with no benchmark comparison. Calculating benchmarks based on multiobjective problems can be found, amongst others, in Golany (1988), Soares de Mello et al. (2003), Lins et al. (2004), Quariguasi-Frota-Neto & Angulo-Meza (2007).

Future studies intend to study the possibility to obtain benchmarks in the MCDEA model as well as to introduce the typical interactivity of the MOLP models. It is also intended to initiate an Input and Output sensitivity analysis.

Recebido em 04/2008; aceito em 09/2008 após 1 revisão

Received April 2008; accepted September 2008 after one revision

• (1) Angulo-Meza, L.; Biondi Neto, L.; Soares de Mello, J.C.C.B. & Gomes, E.G. (2005). ISYDS Integrated System for Decision Support (SIAD Sistema Integrado de Apoio a Decisão): A Software Package for Data Envelopment Analysis Model. Pesquisa Operacional, 25(3), 493-503.
• (2) Angulo-Meza, L.; Gomes, E.G.; Biondi Neto, L. & Coelho, P.H.G. (2003). Avaliação do ensino nos cursos de pós-graduação em Engenharia: um enfoque quantitativo de avaliação em conjunto. Engevista, 5(9), 41-49.
• (3) Angulo-Meza, L. & Estellita Lins, M.P. (2002). Review of methods for increasing discrimination in data envelopment analysis. Annals of Operations Research, 116(1-4), 225-242.
• (4) Barba-Romero, S. & Pomerol, J.C. (1997). Decisiones Multicriterio: Fundamentos Teóricos e Utilización Práctica Universidad de Alcalá
• (5) Charnes, A.; Cooper, W.W. & Rhodes, E. (1978). Measuring the efficiency of decision-making units. European Journal of Operational Research, 2(6), 429-444.
• (6) Clímaco, J.C.N. & Henggeler Antunes, C. (1989). Implementation of a user-friendly software package a guided tour of TRIMAP. Mathematical and Computer Modelling, 12(10-11), 1299-1309.
• (7) Clímaco, J.C.N. & Antunes, C.H. (2000). Programação Linear Multiobjectivo: Uma perspectiva interactiva. In: Análise Envoltória de Dados e perspectivas de integração no ambiente do Apoio à Decisão [edited by M.P.E. Lins and L. Angulo-Meza], Rio de Janeiro, Editora da COPPE/UFRJ.
• (8) Clímaco, J.C.N. & Antunes, C.H. (1987). TRIMAP an interactive tricriteria linear programming package. Foundations of Control Engineering, 12(3), 101-119.
• (9) Clímaco, J.C.N.; Antunes, C.H. & Alves, M.J.G. (2003). Programação Linear Multiobjectivo. Coimbra, Universidade de Coimbra.
• (10) Clímaco, J.C.N.; Soares de Mello, J.C.C.B. & Angulo-Meza, L. (2008). Performance Measurement From DEA to MOLP. In: Encyclopedia of Decision Making and Decision Support Technologies [edited by F. Adam and P. Humphreys], Hershey, Information Science Reference, 709-715.
• (11) Cooper, W.W.; Ruiz, J.L. & Sirvent, I. (2007). Choosing weights from alternative optimal solutions of dual multiplier models in DEA. European Journal of Operational Research, 180(1), 443-458.
• (12) Doyle, J. & Green, R. (1994). Efficiency and cross-efficiency in DEA: Derivations, meanings and uses. Journal of the Operational Research Society, 45(5), 567-578.
• (13) Golany, B. (1988). An interactive MOLP procedure for the extension of DEA to Effectiveness Analysis. Journal of the Operational Research Society, 39(8), 725-734.
• (14) Li, X.-B. & Reeves, G.R. (1999). A multiple criteria approach to data envelopment analysis. European Journal of Operational Research, 115(3), 507-517.
• (15) Lins, M.P.E.; Angulo-Meza, L. & da Silva, A.C.M. (2004). A multi-objective approach to determine alternative targets in data envelopment analysis. Journal of the Operational Research Society, 55(10), 1090-1101.
• (16) Lins, M.P.E.; Sollero, M.K.V.; Caloba, G.M. & da Silva, A.C.M. (2007). Integrating the regulatory and utility firm perspectives, when measuring the efficiency of electricity distribution. European Journal of Operational Research, 181(3), 1413-1424.
• (17) Quariguasi Frota Neto, J. & Angulo-Meza, L. (2007). Alternative targets for data envelopment analysis through multi-objective linear programming: Rio de Janeiro Odontological Public Health System Case Study. Journal of the Operational Research Society, 58(7), 865-873.
• (18) Rosen, D.; Schaffnit, C. & Paradi, J.C. (1998). Marginal rates and two-dimensional level curves in DEA. Journal of Productivity Analysis, 9(3), 205-232.
• (19) Roy, B. (1987). Meaning and validity of interactive procedures as tools for decision making. European Journal of Operational Research, 31(3), 297-303.
• (20) Senra, L.F.A.D.C.; Nanci, L.C.; Soares de Mello, J.C.C.B. & Angulo-Meza, L. (2007). Estudo sobre métodos de seleção de variáveis em DEA. Pesquisa Operacional, 27(2), 191-207.
• (21) Senra, L.F.A.D.C. & Soares de Mello, J.C.C.B. (2004). Uso de Técnicas de Seleção de Variáveis em DEA para analisar o Setor Elétrico. Relatório de Pesquisa em Engenharia de Produção, 4(4).
• (22) Sexton, T.R.; Silkman, R.H. & Logan, A.J. (1986). Data Envelopment Analysis: Critique and extensions. In: Measuring efficiency: An assessment of data envelopment analysis [edited by H. Silkman], San Francisco, Jossey-Bass Editor.
• (23) Soares de Mello, J.C.C.B.; Angulo-Meza, L. & Branco da Silva, B.P. (2009). A ranking for the Olympic Games with unitary input DEA models. IMA Journal of Management Mathematics, 20(2), 201-211.
• (24) Soares de Mello, J.C.C.B.; Angulo-Meza, L.; Gomes, E.G.; Serapião, B.P. & Estellita Lins, M.P. (2003). Análise de envoltória de dados no estudo da eficiência e dos benchmarks para companhias aéreas brasileiras. Pesquisa Operacional, 23(2), 325-345.
• (25) Soares de Mello, J.C.C.B.; Gomes, E.G.; Angulo-Meza, L. & Biondi Neto, L. (2008a). Cross evaluation using weight restrictions in unitary input DEA models: Theoretical aspects and application to Olympic Games ranking. WSEAS Transactions on Systems, 7(1), 31-39.
• (26) Soares de Mello, J.C.C.B.; Gomes, E.G.; Angulo-Meza, L. & Leta, F.R. (2008b). DEA Advanced Models for Geometric Evaluation of used Lathes. WSEAS Transactions on Systems, 7(5), 500-520.
• (27) Soares de Mello, J.C.C.B.; Gomes, E.G.; Biondi Neto, L. & Lins, M.P.E. (2004). Suavização da Fronteira DEA: O caso BCC tridimensional. Investigação Operacional, 24(1), 89-107.
• (28) Soares de Mello, J.C.C.B.; Estellita Lins, M.P. & Gomes, E.G. (2002). Construction of a smoothed DEA frontier. Pesquisa Operacional, 28(2), 183-201.

*

Corresponding author / autor para quem as correspondências devem ser encaminhadas

Publication Dates

• Publication in this collection
  28 May 2009
• Date of issue
  Apr 2009

History

• Accepted
  Sept 2008
• Received
  Apr 2008