Acessibilidade / Reportar erro

PREDICTION OF EFFICIENCY IN COLOMBIAN HIGHER EDUCATION INSTITUTIONS WITH DATA ENVELOPMENT ANALYSIS AND NEURAL NETWORKS

ABSTRACT

This paper shows the results of a research of the application of data envelopment analysis (DEA) together with artificial neural networks (ANN) of higher education institutions in Colombia during the years 2011-2013, for the purpose of evaluating the technical efficiency of Colombian higher education institutions and subsequently carry out predictions, based on a group of management indicators. Information provided by the Ministry of National Education was used as data source. The results show that this two-stage approach provides the DEA with the predictive potential that it otherwise lacks, enhancing its evaluative qualities; this is also evident in the various research papers consulted. The results also show that 50% of the models built have correct classification rates of 64.58% and 58.33% for training and validation datasets, respectively.

Keywords:
efficiency; classification; higher education

1 INTRODUCTION

Currently it is a priority to take into account principles of rational use of resources and economic efficiency in the administration of public education institutions at their different levels to strengthen their institutional duties (teaching, research and extension). For this reason, the different levels of governance and organization relative to the management of higher education develop strategies and plans to increase efficiency in universities and thus improve their performance. There is a very close relationship between the allocation and efficient use of public resources and for this reason, researchers have spent a lot of time evaluating the efficiency of different education systems in the world.

In this regard, it should be noted that there are quite noticeable differences in terms of the resources available to the different Higher Education Institutions (HEIs) to develop their institutional tasks, which in some way affects the results they obtain. Therefore, it is very possible to find universities presenting a better performance than other universities that have more resources. If we also consider that under the flagship program of the Ministry of National Education, Ser Pilo Paga (Being Smart Pays Off), 98% of the resources are received by private universities (Observatory of the Colombian University, 201624 OBSERVATORY OF THE COLOMBIAN UNIVERSITY. 2016. Available at: Available at: http://www.universidad.edu.co/index.php/noticias/13838-98-4-de-recursos-de-ser-pilo-paga-se-ha-ido-a-las-universidades-privadas . last accessed 10 May 2017.
http://www.universidad.edu.co/index.php/...
) it would be interesting to review the efficiency of Colombian public universities as competitive economic units.

This research uses the CCR model (Charnes et al., 19786 CHARNES A, COOPER WW & RHODES E. 1978. Measuring the efficiency of decision making units. European Journal of Operational Research, 2(6): 429-444.) of the Data Envelopment Analysis (DEA), focused on outputs, to determine whether an HEI is efficient, and subsequently neural networks to predict whether a new observation (HEI) can be classified as efficient. The data used in this study correspond to the management indicators of Colombian public universities during the period 2011-2013 and are taken from the database of the Colombian Ministry of National Education. The efficiency is the ability of an HEI to obtain the maximum outputs indicators from a given set of inputs indicators.

2 THEORETICAL FRAMEWORK

Data Envelopment Analysis (DEA) is an approach to evaluate the performance of a set of homogeneous entities called Decision Units (DMUs) that transform multiple inputs into multiple outputs (Cooper et al., 20119 COOPER WW, SEIFORD LM & ZHU J. 2011. Data envelopment analysis: History, models, and interpretations. In: International Series in Operations Research and Management Science. Springer US.). Charnes, Cooper and Rhodes developed the initial DEA model originally based in an earlier work devised by Farrel in 1957 (Charnes et al., 19786 CHARNES A, COOPER WW & RHODES E. 1978. Measuring the efficiency of decision making units. European Journal of Operational Research, 2(6): 429-444.).

This technique uses linear programming models to compare production units that handle the same amount of resources and they produce the same amount of products. This generates a frontier efficient values and the efficiency indices within the group of production units studied. Uses of DEA have involved not only business firms but also government and non-profit agencies including schools, hospitals among others, the term “Decision Making Unit” (DMU) was therefore introduced to cover, in a flexible manner, any such entity. The performance of a DMU is efficient if and only if it is not possible to improve any input or output without worsening any other input or output (Cooper et al., 20119 COOPER WW, SEIFORD LM & ZHU J. 2011. Data envelopment analysis: History, models, and interpretations. In: International Series in Operations Research and Management Science. Springer US.). In this way, Colombian state universities, can be treated as Decision-Making Units (DMUs).

In the CCR model is determined the relative efficiency of n DMUs to be evaluated, where each unit has m inputs (resources) x1,x2,,xm and s outputs (products) y1,y2,,ys, is required the solution of the following linear program model.

max θ = μ 1 y 10 + + μ s y s 0 subject to ν 1 x 10 + + v m x m 0 = 1 μ 1 y 1 j + + μ s y s j ν 1 x 1 j + + ν m x m j j = 1,2, , n ν 1 , ν 2 , , ν m 0 μ 1 , μ 2 , , μ s 0

The DEA methodology is one of the main techniques to evaluate the performance of productive units both in the public sector and in the private sector. Thus, this technique has been used in the financial sector (Sathye, 200131 SATHYE M. 2001. X-efficiency in Australian banking: An empirical investigation. Journal of Banking and Finance, 25(3): 613-630.; Wang et al., 201438 WANG K, HUANG W, WU J & LIU YN. 2014. Efficiency measures of the Chinese commercial banking system using an additive two-stage DEA. Omega(United Kingdom), 44:5-20.; Kwon & Lee, 201518 KWON HB & LEE J. 2015. Two-stage production modeling of large U.S. banks: A DEAneural network approach. Expert Systems with Applications, 42(19): 6758-6766.; Fukuyama & Matousek, 201714 FUKUYAMA H & MATOUSEK R. 2017. Modelling bank performance: A network DEA approach. European Journal of Operational Research, 259(2): 721-732.), in the energy sector (Sathye, 200131 SATHYE M. 2001. X-efficiency in Australian banking: An empirical investigation. Journal of Banking and Finance, 25(3): 613-630.; Wang et al., 201438 WANG K, HUANG W, WU J & LIU YN. 2014. Efficiency measures of the Chinese commercial banking system using an additive two-stage DEA. Omega(United Kingdom), 44:5-20.; Kwon & Lee, 201518 KWON HB & LEE J. 2015. Two-stage production modeling of large U.S. banks: A DEAneural network approach. Expert Systems with Applications, 42(19): 6758-6766.; Fukuyama & Matousek, 201714 FUKUYAMA H & MATOUSEK R. 2017. Modelling bank performance: A network DEA approach. European Journal of Operational Research, 259(2): 721-732.), in the operational performance of companies (Sathye, 200131 SATHYE M. 2001. X-efficiency in Australian banking: An empirical investigation. Journal of Banking and Finance, 25(3): 613-630.; Wang et al., 201438 WANG K, HUANG W, WU J & LIU YN. 2014. Efficiency measures of the Chinese commercial banking system using an additive two-stage DEA. Omega(United Kingdom), 44:5-20.; Kwon & Lee, 201518 KWON HB & LEE J. 2015. Two-stage production modeling of large U.S. banks: A DEAneural network approach. Expert Systems with Applications, 42(19): 6758-6766.; Fukuyama & Matousek, 201714 FUKUYAMA H & MATOUSEK R. 2017. Modelling bank performance: A network DEA approach. European Journal of Operational Research, 259(2): 721-732.), and in the assessment of research centers Souza201635 SOUZA GS & GOMES EG. 2016. A non-Archimedean DEA model to assess group comparisons. Pesquisa Operacional, 36(3): 533-546., among others.

Artificial Neural Networks (ANNs) simulate the information processing of the human brain, as an interconnected set of simple processing elements, units or nodes, such as the ability to memorize and relate facts and circumstances, and are a powerful tool for modeling non-linear functions through an adaptation or learning process of a set of training patterns (Gurney 200315 GURNEY K. 2003. An introduction to neural networks. CRC press.). These units or nodes receive and transmit signals analogously to biological neural networks (Mehrotra et al., 199721 MEHROTRA K, MOHAN CK & RANKA S. 1997. Elements of artificial neural networks. MIT press.). Neural networks were developed based on the McCulloch and Pitts model, where their object of study is the computation made by neurons, and showed that the neural network system of the human being can be imitated by a mathematical algorithm (Shabanpour et al., 201734 SHABANPOUR H, YOUSEFI S & SAEN RF. 2017. Forecasting efficiency of green suppliers by dynamic data envelopment analysis and rtificial neural networks. Journal of Cleaner Production, 142: 1098-1107.). These computer algorithms consist of many simple elements with highly interconnected that produces different signals based on the weighted sum of the input signals they receive, with ability of modelling complex non-linear relations (Schalkoff, 199732 SCHALKOFF RJ. 1997. Artificial neural networks. McGraw-Hill Higher Education.). The type of network used is defined by the architecture that represents the connection structure between the nodes, by its method of determining the connection weights and by the activation function it uses. The architecture of a network consists of the grouping of the neurons forming layers; there are monolayer and multilayer networks (Fausett, 199411 FAUSETT L. 1994. Fundamentals of neural networks: architectures, algorithms, and applications. Prentice-Hall, Inc.).

2.1 Data Envelopment Analysis and Neural Networks

Neural networks have been used in multiple applications and recently in combination with data envelopment analysis as predictive techniques in a second stage to evaluate the efficiency of productive units. Among these works we can highlight the following: (Çelebi & Bayraktar, 20085 ÇELEBI D & BAYRAKTAR D. 2008. An integrated neural network and data envelopment analysis for supplier evaluation under incomplete information. Expert Systems with Applications, 35(4): 1698-1710.; Emrouznejad & Shale, 200910 EMROUZNEJAD A & SHALE E. 2009. A combined neural network and DEA for measuring efficiency of large scale datasets. Computers and Industrial Engineering, 56(1): 249-254.; Pendharkar, 201127 PENDHARKAR PC. 2011. A hybrid radial basis function and data envelopment analysis neural network for classification. Computers and Operations Research, 38(1): 256-266.; Azadeh et al., 20113 AZADEH A, SHEIKHALISHAHI M & ASADZADEH SM. 2011. A flexible neural networkfuzzy data envelopment analysis approach for location optimization of solar plants with uncertainty and complexity. Renewable Energy, 36(12): 3394-3401.; I-Fei, 2011; Liu, 201219 LIU Z. 2012. An intelligent WPICC evaluation model based on DEA-ANN hybrid algorithm. Journal of Computational Information Systems, 6(13): 4369-4378.; Tosun, 201236 TOSUN O. 2012. Using data envelopment analysis-neural network model to evaluate hospital efficiency. International Journal of Productivity and Quality Management, 9(2): 245-257.; Choobdari et al., 20137 CHOOBDARI N, BARANDARAN B & ALLA KALVANDI H. 2013. Evaluation of Staff Efficiency Using the Combined Model of Neuro/DEA (Case Study: Operational Unit of Gilan Province Gas Company). Life Science Journal, 10: 185-191.; Olanrewaju et al., 201325 OLANREWAJU OA, JIMOH AA & KHOLOPANE PA. 2013. Assessing the energy potential in the South African industry: Acombined IDA-ANN-DEA (Index Decomposition Analysis-Artificial Neural Network-Data Envelopment Analysis) model. Energy, 63: 225-232.; Azadeh et al., 20152 AZADEH A, MIANAEI HS, ASADZADEH SM, SABERI M & SHEIKHALISHAHI M. 2015. A flexible ANN-GA-multivariate algorithm for assessment and optimization of machinery productivity in complex production units. Journal of Manufacturing Systems, 35: 46-75.; Kwon & Lee, 201518 KWON HB & LEE J. 2015. Two-stage production modeling of large U.S. banks: A DEAneural network approach. Expert Systems with Applications, 42(19): 6758-6766.; Park & Kim, 201626 PARK S & KIM J. 2016. Energy efficiency in Korea: analysis using a hybrid DEA model. Geosystem Engineering, 19(3): 143-150.)

2.2 Data Envelopment Analysis and Higher Education

The evaluation of the efficiency of higher education institutions is an area of great interest. The DEA methodology is particularly useful in the evaluation of educational institutions since the economic value of many of the inputs and outputs is difficult to determine, so weighing them is considered appropriate for universities (Colbert et al., 20008 COLBERT A, LEVARY RR & SHANER MC. 2000. Determining the relative efficiency of MBA programs using DEA. European Journal of Operational Research, 125(3): 656-669.). Thanassoulis (2016) has provided insights into the richness of DEA in education. Within the literature, we can highlight recent studies from different parts of the world such as (2013); Nazarko & Šaparauskas (201423 NAZARKO J & ŠAPARAUSKAS J. 2014. Application of DEA method in efficiency evaluation of public higher education institutions. Technological and Economic Development of Economy, 20(1): 25-44.); Ruiz et al. (201529 RUIZ JL, SEGURA JV & SIRVENT I. 2015. Benchmarking and target setting with expert preferences: An application to the evaluation of educational performance of Spanish universities. European Journal of Operational Research, 242(2): 594-605.); Andersson et al. (20171 ANDERSSON C, ANTELIUS J, MÅNSSON J & SUND K. 2017. Technical efficiency and productivity for higher education institutions in Sweden. Scandinavian Journal of Educational Research, 61(2): 205-223.); Fernandez-Palacin et al. (201813 FERNANDEZ-PALACIN F, LOPEZ-SANCHEZ MA & MUOZ-MRQUEZ M. 2018. Stepwise Selection of Variables in DEA Using Contribution Loads. Pesquisa Operacional, 38(1): 31-52.), among others.

3 METHODOLOGY

In this research, we make a quantitative study supported by Data Envelopment Analysis (DEA) of 32 universities within the Colombian Public University System during the years 2011 to 2013. The approach we propose using of a DEA-CCR-O model (CCR model oriented on outputs) with four inputs and six outputs. The inputs of the process are the total number of Academic Staff, Administrative Staff Spending, Financial Resources and Physical Resources. The performance indicators (output) are number of postgraduate and undergraduate degrees enrolled, Results Saber PRO, Indexed Journals, Research publications and Professor Mobility. An output approach is used, since reducing resources for an inefficient university would be unreasonable, because this implies reducing the academic and administrative staff, as well as a reduction of financial resources.

The DEA model is conceived for the purpose of determining the technical efficiency indexes of each Colombian public HEIs and thus identifying the institutions that present the best practices in their performance, later for the efficiency results obtained with the DEA model we using neural networks with the purpose of predicting the classification of a new IES as efficient or not, and the validation of the predictive power of the neural network. The architecture used in this research is a monolayer neuronal network.

3.1 HEIs included in the analysis

In this efficiency assessment of public HEIs, the 32 Colombian public universities belonging to the System of State Universities (SUE, for its Spanish acronym) are considered for three consecutive years (2011-2013). The HEIs in the analysis stage were randomly assigned from HEI-1 to HEI-32 in order to ensure the confidentiality of the results of the evaluation. The institutions that were considered in the study are shown in Table 1.

Table 1:
HEI considered in the study.

3.2 The data and variables

The data of the variables employed in the realization of this study is derived from data which are publicly available on the Colombian Ministry of National Education website (201522 MINISTRY OF NATIONAL EDUCATION OF COLOMBIA. 2015. Available at: Available at: mineducacion.gov.co/sistemasdeinformacion/1735/articles-212353 modelo1.pdf . last accessed 15 July 2017.
mineducacion.gov.co/sistemasdeinformacio...
). The data envelopment analysis requires the identification of input variables, which correspond to the resources used to carry out the institutional duties, and output variables, which are identified as the products or objectives of the HEIs. We selected variables are used both to determine whether or not a given HEI is efficient, and for the subsequent prediction through neural networks, four inputs and six outputs are employed considering previous researches (Tsolas & Charles, 201537 TSOLAS IE & CHARLES V. 2015. Incorporating risk into bank efficiency: A satisficing DEA approach to assess the Greek banking crisis. Expert Systems with Applications, 42(7): 3491-3500.; Selim & Bursalolu, 201533 SELIM S & BURSALOLU SA. 2015. Efficiency of Higher Education in Turkey: A Bootstrapped Two-Stage DEA Approach. International Journal of Statistics and Applications, 5(2): 56-67.; Ramzi & Ayadi, 201628 RAMZI S & AYADI M. 2016. Assessment of Universities Efficiency Using Data Envelopment Analysis: Weights Restrictions and Super-Efficiency Measure. Journal of Applied Management and Investments, 5(1): 40-58.).

Input variables

The first input is the total number of Academic Staff (full-time professors equivalent), includes professors, associate professors, assistant professors, casual professors and lecturers. The second is Administrative Staff Spending, contains the expense of non-academic staff that administers the teaching and research process. The third input is Financial Resources, represents monetary resources provided by the State and the monetary resources produced by the university itself. The fourth input is Physical Resources or area of built physical spaces available for university activities (academic and administrative).

Output variables

The first and second output is the number of postgraduate and undergraduate degrees enrolled, measured in equivalent full-time students. The third output is Results Saber PRO, contains the total quantity of students who are in the upper quintile according to the State exam scores. The fourth output is Indexed Journals indicating the number of academic journals of the HEI classified in the different categories in the Publindex of Colciencias. The fifth result is Research publications that show the number of papers of the researchers in different journals. The sixth output is Professor Mobility or number of professors linked to mobility programme supported by from the HEI to which they belong.

4 RESULTS

4.1 Data Envelopment Analysis (DEA)

The efficiency results of the CCR-O model for the years 2011 to 2013 are shown in Table 2. Therefore, there are 96 observations available, 32 universities for 3 years.

Table 2:
Efficiency of HEIs years 2011 to 2013

4.2 Neural networks

The results in Table 2 indicate that, from the 96 observations, 40 are considered efficient and 56 are inefficient. For the modeling of the neural network, the 96 observations were divided into 72, for training (train), and 24, for validation (test). The validation of the neural network will be carried out through repeated holdout, which consists in taking two sets of random samples for training and validation. This procedure is repeated k times, for which k models are constructed, which are also evaluated k times to improve the estimation of the performance of the classification model.

The modeling of the neural network was conducted through the caret and nnet packages of the R software. The best parameters for the neural network, Size and Decay, are determined through the caret package. The model obtained consists of an artificial neural network (ANN) (10, 3, 1), that is, of 10 inputs, 3 hidden neurons and 1 output, with 37 weights and with the decay parameter of 0.1, as shown in Figure 1. The inputs used for the ANN are the 10 variables of the DEA model mentioned before.

Figure 1:
Estimated neural network model.

Figure 1 shows the representation of the calculated neural network. The black color of the arcs indicates a positive weight, the gray one a negative weight, and the thickness the magnitude of the weight, where we can observe a positive weight of great relative magnitude between the hidden layer H3 and the response (categorical output variable that indicates whether or not the HEI is efficient).

The confusion matrix of the training data shown in Table 3 indicates a true positive (efficient) rate (sensitivity) of 81.25%, a true negative rate (specificity) of 97.50% and an overall correct classification rate of 90.28%.

Table 3:
Confusion matrix training data.

The confusion matrix of the validation data shown in Table 4 indicates a positive (efficient) rate (sensitivity) of 75.00%, a true negative rate (specificity) of 68.75% and an overall correct classification rate of 70.83%.

Table 4:
Confusion matrix validation data.

Figure 2 displays the ROC (Receiver Operating Characteristic) curve for the training data, indicating an AUC (Area Under ROC Curve) of 0.894, which shows that the neural network model has a good classification capacity.

Figure 2:
ROC curve of estimated neural network.

After carrying out the validation by repeated holdout with k = 100 (repeat holdout 100 times), a success rate was found for the training data that varies between a minimum of 34.72% and a maximum of 81.94%, with an average value of 61.76%, and a success rate for the validation data that varies between a minimum of 33.33% and a maximum of 83.33%, with an average value of 58.33%.

Figure 3 shows the Box Plots of the 100 correct classification rates when validating the neural network by means of repeated holdout 100 times, both for the training data and for the validation data, showing that in at least 75% of cases this rate is above 50%.

Figure 3:
Box plots.

The confidence intervals for the correct classification rates constructed by the quintile method shown in Table 5 indicate that 95% of the models built have correct classification rates between 36.77% and 79.17%, and between 33.33% and 79.17% for the training and validation data, respectively. 50% of the models built have correct classification rates of 64.58% and 58.33% for training and validation data, respectively.

Table 5:
Confidence interval.

These differences in the success rates for both the training data and the validation data are due to the fact that every time a model is built, different samples are chosen for training and validation, thus modifying the parameters of the neural network model built, and therefore also its performance.

Also the modeling of the neural network using the 64 observations of the years 2011 and 2012 was also carried out for training and for validation the 32 observations of the year 2013 were used. The model obtained consists of an artificial neural network (ANN) (10, 5, 1), that is, 10 inputs, 5 hidden neurons and 1 output, with 61 weights and the decay parameter of 0.1, as shown in Figure 4. The inputs used for ANN again are the 10 variables of the DEA model.

Figure 4:
Estimated neural network model.

In Figure 4 we can observe a positive weight of great relative magnitude between the hidden layer H3 and the response (categorical output variable that indicates whether the IES is efficient or not).

The confusion matrices of the training data and validation data is show in Table 6 and Table 7, respectively

Table 6:
Confusion matrix training data.

Table 7:
Confusion matrix validation data.

In the Figure 5 displays the ROC (Receiver Operating Characteristic) curve for the training data, indicating an AUC (Area Under ROC Curve) of 0.841, which shows that the neural network model has a good classification capacity.

Figure 5:
ROC curve of estimated neural network.

5 CONCLUSION

The results show that the application of the DEA together with neural networks allows us to predict whether or not a HEI can be classified as efficient, with an adequate correct classification rate. Furthermore, this two-stage approach gives the DEA a predictive potential that it otherwise lacks, thus enhancing its evaluative qualities. Thus, from the point of view of state management, the proposed model can be of great benefit to decision makers because when analyzing the performance of universities by geographic area, they can confirm that the distribution of financial resources from the State should be redesigned to improve the overall efficiency of the system. In other words, this approach can lead to a more realistic distribution of resources based on the results obtained, showing that 50% of the models built have correct classification rates of 64.58% and 58.33% for training and validation data, respectively.

Further research is recommended to identify the most influential variables when predicting whether or not a HEI is efficient, and to include in the neural network variables that have not been considered in determining efficiency by means of the DEA, such as, for example, the transparency index of each of the HEIs, and evaluate the behavior of the neural network. As well as the use of different machine learning techniques instead of neural networks.

It is also recommended to model the neural network considering a predictive model of the number value of the efficiency index and not of the efficient or inefficient classification determined in this study.

REFERENCES

  • 1
    ANDERSSON C, ANTELIUS J, MÅNSSON J & SUND K. 2017. Technical efficiency and productivity for higher education institutions in Sweden. Scandinavian Journal of Educational Research, 61(2): 205-223.
  • 2
    AZADEH A, MIANAEI HS, ASADZADEH SM, SABERI M & SHEIKHALISHAHI M. 2015. A flexible ANN-GA-multivariate algorithm for assessment and optimization of machinery productivity in complex production units. Journal of Manufacturing Systems, 35: 46-75.
  • 3
    AZADEH A, SHEIKHALISHAHI M & ASADZADEH SM. 2011. A flexible neural networkfuzzy data envelopment analysis approach for location optimization of solar plants with uncertainty and complexity. Renewable Energy, 36(12): 3394-3401.
  • 4
    BAYRAKTAR E, TATOGLU E & ZAIM S. 2013. Measuring the relative efficiency of quality management practices in Turkish public and private universities. Journal of the Operational Research Society, 64(12): 1810-1830.
  • 5
    ÇELEBI D & BAYRAKTAR D. 2008. An integrated neural network and data envelopment analysis for supplier evaluation under incomplete information. Expert Systems with Applications, 35(4): 1698-1710.
  • 6
    CHARNES A, COOPER WW & RHODES E. 1978. Measuring the efficiency of decision making units. European Journal of Operational Research, 2(6): 429-444.
  • 7
    CHOOBDARI N, BARANDARAN B & ALLA KALVANDI H. 2013. Evaluation of Staff Efficiency Using the Combined Model of Neuro/DEA (Case Study: Operational Unit of Gilan Province Gas Company). Life Science Journal, 10: 185-191.
  • 8
    COLBERT A, LEVARY RR & SHANER MC. 2000. Determining the relative efficiency of MBA programs using DEA. European Journal of Operational Research, 125(3): 656-669.
  • 9
    COOPER WW, SEIFORD LM & ZHU J. 2011. Data envelopment analysis: History, models, and interpretations. In: International Series in Operations Research and Management Science. Springer US.
  • 10
    EMROUZNEJAD A & SHALE E. 2009. A combined neural network and DEA for measuring efficiency of large scale datasets. Computers and Industrial Engineering, 56(1): 249-254.
  • 11
    FAUSETT L. 1994. Fundamentals of neural networks: architectures, algorithms, and applications. Prentice-Hall, Inc.
  • 12
    FEI R & LIN B. 2016. Energy efficiency and production technology heterogeneity in China’s agricultural sector: A meta-frontier approach. Technological Forecasting and Social Change, 109: 25-34.
  • 13
    FERNANDEZ-PALACIN F, LOPEZ-SANCHEZ MA & MUOZ-MRQUEZ M. 2018. Stepwise Selection of Variables in DEA Using Contribution Loads. Pesquisa Operacional, 38(1): 31-52.
  • 14
    FUKUYAMA H & MATOUSEK R. 2017. Modelling bank performance: A network DEA approach. European Journal of Operational Research, 259(2): 721-732.
  • 15
    GURNEY K. 2003. An introduction to neural networks. CRC press.
  • 16
    HAN Y, GENG Z, ZHU Q & QU Y. 2015. Energy efficiency analysis method based on fuzzy DEA cross-model for ethylene production systems in chemical industry. Energy, 83: 685-695.
  • 17
    I-FEI C. 2011. A two-stage cardholder behavioural scoring model using artificial neural networks and data envelopment analysis. International Journal of Advancements in Computing Technology, 3(2): 87-94.
  • 18
    KWON HB & LEE J. 2015. Two-stage production modeling of large U.S. banks: A DEAneural network approach. Expert Systems with Applications, 42(19): 6758-6766.
  • 19
    LIU Z. 2012. An intelligent WPICC evaluation model based on DEA-ANN hybrid algorithm. Journal of Computational Information Systems, 6(13): 4369-4378.
  • 20
    MALLIKARJUN S, LEWIS HF & SEXTON TR. 2014. Operational performance of U.S. public rail transit and implications for public policy. Socio-Economic Planning Sciences, 48(1): 74-88.
  • 21
    MEHROTRA K, MOHAN CK & RANKA S. 1997. Elements of artificial neural networks. MIT press.
  • 22
    MINISTRY OF NATIONAL EDUCATION OF COLOMBIA. 2015. Available at: Available at: mineducacion.gov.co/sistemasdeinformacion/1735/articles-212353 modelo1.pdf last accessed 15 July 2017.
    » mineducacion.gov.co/sistemasdeinformacion/1735/articles-212353 modelo1.pdf
  • 23
    NAZARKO J & ŠAPARAUSKAS J. 2014. Application of DEA method in efficiency evaluation of public higher education institutions. Technological and Economic Development of Economy, 20(1): 25-44.
  • 24
    OBSERVATORY OF THE COLOMBIAN UNIVERSITY. 2016. Available at: Available at: http://www.universidad.edu.co/index.php/noticias/13838-98-4-de-recursos-de-ser-pilo-paga-se-ha-ido-a-las-universidades-privadas last accessed 10 May 2017.
    » http://www.universidad.edu.co/index.php/noticias/13838-98-4-de-recursos-de-ser-pilo-paga-se-ha-ido-a-las-universidades-privadas
  • 25
    OLANREWAJU OA, JIMOH AA & KHOLOPANE PA. 2013. Assessing the energy potential in the South African industry: Acombined IDA-ANN-DEA (Index Decomposition Analysis-Artificial Neural Network-Data Envelopment Analysis) model. Energy, 63: 225-232.
  • 26
    PARK S & KIM J. 2016. Energy efficiency in Korea: analysis using a hybrid DEA model. Geosystem Engineering, 19(3): 143-150.
  • 27
    PENDHARKAR PC. 2011. A hybrid radial basis function and data envelopment analysis neural network for classification. Computers and Operations Research, 38(1): 256-266.
  • 28
    RAMZI S & AYADI M. 2016. Assessment of Universities Efficiency Using Data Envelopment Analysis: Weights Restrictions and Super-Efficiency Measure. Journal of Applied Management and Investments, 5(1): 40-58.
  • 29
    RUIZ JL, SEGURA JV & SIRVENT I. 2015. Benchmarking and target setting with expert preferences: An application to the evaluation of educational performance of Spanish universities. European Journal of Operational Research, 242(2): 594-605.
  • 30
    SÁNCHEZ-GONZÁLEZ C, SARTO JL & VICENTE L. 2017. The efficiency of mutual fund companies: Evidence from an innovative network SBM approach. Omega(United Kingdom), 71: 114-128.
  • 31
    SATHYE M. 2001. X-efficiency in Australian banking: An empirical investigation. Journal of Banking and Finance, 25(3): 613-630.
  • 32
    SCHALKOFF RJ. 1997. Artificial neural networks. McGraw-Hill Higher Education.
  • 33
    SELIM S & BURSALOLU SA. 2015. Efficiency of Higher Education in Turkey: A Bootstrapped Two-Stage DEA Approach. International Journal of Statistics and Applications, 5(2): 56-67.
  • 34
    SHABANPOUR H, YOUSEFI S & SAEN RF. 2017. Forecasting efficiency of green suppliers by dynamic data envelopment analysis and rtificial neural networks. Journal of Cleaner Production, 142: 1098-1107.
  • 35
    SOUZA GS & GOMES EG. 2016. A non-Archimedean DEA model to assess group comparisons. Pesquisa Operacional, 36(3): 533-546.
  • 36
    TOSUN O. 2012. Using data envelopment analysis-neural network model to evaluate hospital efficiency. International Journal of Productivity and Quality Management, 9(2): 245-257.
  • 37
    TSOLAS IE & CHARLES V. 2015. Incorporating risk into bank efficiency: A satisficing DEA approach to assess the Greek banking crisis. Expert Systems with Applications, 42(7): 3491-3500.
  • 38
    WANG K, HUANG W, WU J & LIU YN. 2014. Efficiency measures of the Chinese commercial banking system using an additive two-stage DEA. Omega(United Kingdom), 44:5-20.
  • 39
    XU B & OUENNICHE J. 2012. A data envelopment analysis-based framework for the relative performance evaluation of competing crude oil prices’ volatility forecasting models. Energy Economics, 34(2): 576-583.

Publication Dates

  • Publication in this collection
    23 Sept 2019
  • Date of issue
    May-Aug 2019

History

  • Received
    01 Jan 2018
  • Accepted
    14 May 2019
Sociedade Brasileira de Pesquisa Operacional Rua Mayrink Veiga, 32 - sala 601 - Centro, 20090-050 , Tel.: +55 21 2263-0499 - Rio de Janeiro - RJ - Brazil
E-mail: sobrapo@sobrapo.org.br