SciELO - Scientific Electronic Library Online

Home Pagealphabetic serial listing  

Services on Demand




Related links


Pesquisa Operacional

Print version ISSN 0101-7438On-line version ISSN 1678-5142

Pesqui. Oper. vol.38 no.3 Rio de Janeiro Sept./Dec. 2018 



Marcela do Carmo Silva1 

Carlos Francisco Simões Gomes1  * 

Castelar Lino Da Costa Junior2 

1Programa de Pós Graduação Engenharia de Produção, R. Passo da Pátria, 156 209 - São Domingos, 24210-240 Niterói, RJ, Brazil. E-mails:;

2Coordenação de Graduação em Engenharia de Produção, R. Passo da Pátria, 156 209 - São Domingos, 24210-240 Niterói, RJ, Brazil. E-mail:


This paper aims to analyse the ordering of Technology Transfer Offices’ (TTO) positions regarding innovation indicators using multicriteria analysis as a tool. The purpose is to identify and analyse the TTOs’ strengths and weaknesses in a specific higher education institution in order to understand their performance via ranking, for obtaining revenues from intellectual property assets possibilities developed by them. The use of a methodology for the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS), using two normalizing methods, was preceded by a treatment of this qualitative data with the Measuring Attractiveness by a Category Based Evaluation Technique (MACBETH). It is based on an applied questionnaire on TTO’s for recognizing the status of the technology transfer condition to developing nations, focusing on an approach to the institutionalization of TTO’s within a national context. The results of these technical applications bring together novel approaches to the processing of data using both methods to support multicultural decisions (using qualitative and quantitative data). These are initially arranged in different ways and present better analytical conditions in respect to the presented technological development. The first combination between TOPSIS and MACBETHS, with two normalized methods, and associated with a Pearson analysis between the criteria.

Keywords: multicriteria decision analysis; decision making/process; MACBETH; TOPSIS; technology transfer offices


Social innovation involves the shedding of companies that turn to developing innovative products and services which generate revenue in and out of its economic sector. With this appears a new kind of company, informally called spin-offs. These companies diffuse innovative products and services to the population in order to solve social unrest, such as assistive technology oriented to support the empowerment of senior citizens. There is a social technology that is largely operated by non-governmental organizations (NGOs) or other companies that coordinate across business sector exchanges.

In this regard, observing innovation and how it is institutionalized becomes believable to the understanding of how technological development is spread so that the entire society benefit. Within the scope of Higher Education Institutions (in Portuguese IES - Instituição de Ensino Superior) and the creation and regulation of Technology Transfer Offices (TTOs).

Brazilian Higher Education Institutions (IES) need scientific basis for obtaining revenues, making laboratories benchmark nucleus for developing innovative technologies (as products or services in any economic sector) profitable for sustaining institutions, without government grant intensively (considering Brazilian educational policies have low government budget).

The decision analysis process usually considers a variety of alternatives, which must be carefully evaluated so that the “best” decision can be chosen. The aim of decision analysis is to provide logical methods to improve the decision-making process of individuals and companies that are focusing on the development of models designed for decision making, under conditions of uncertainty and that could consider multiple decision criteria, for supporting outranking decisions when there is subjectivity (either qualitative perspective) for analysing criteria. As decisions are dynamic, decision-makers (DM) must be convinced that the analysis process is conducted properly and thoroughly, in order to enable the DM to estimate the potential outcome of their decision. This justifies the swift growth in demand for approaches based Multicriteria Decision Aid (MCDA) methods, and explains the rise in adoption of models to support decisions that occurs in situations of uncertainty or even risk (Kadzinski & Tervonen, 2013; Climaco et al., 2009). Some authors prefer the name multiple criteria decision aiding while others use the name multiple criteria decision analysis (Pujadas et al., 2017).

The purpose of this paper is to identify and analyse the strengths and weaknesses of TTOs in a specific higher education institution in order to understand their performance for promoting investments in TTOs with weak profile and also obtaining a better revenue from strong TTOs. This paper intends to address the following research question: “What are the potentialities and weaknesses of these TTOs and how can they improve, in regards to the higher education institution to which they belong?”; “Which TTO’s have strong potential for developing innovative researches to became products and services, considering their infrastructure, human resources qualification and scientifically productivity?”; “Can MCDA combination help to understand better the IES how to support these TTO’s through a questionnaire results, considering subjectivity from those qualitative questions?”

In this way, the aim of this study is to analyze - in a multicriteria method - results obtained from a TTOs’ questionnaire containing both qualitative and quantitative closed questions, and thus, observe the grip on the multicriteria method TOPSIS (Technique for Order Preference by Similarity to Ideal Solution) after the transformation of qualitative responses into quantitative using MACBETH (Measuring Attractiveness by a Category Based Evaluation Technique) method as its base reference.

In this paper, the authors propose a “new TOPSIS” using two normalization methods, in order make sensibly analyse the results (TOPSIS-MACBETH-2N). They are present in equation 9 and 10. This TOPSIS-MACBETH-2N hybrid method permits a sensitive analysis of alternatives; comparing these two TOPSIS’ rankings with MACBETH. Behzadian et al. (2012) review categorized 266 scholarly papers from 103 journals since the year 2000, and there is no hybrid methodology with MACBETH and TOPSIS. The authors did other research, and did not find this combination.

For understanding if there are considerable differences among the MCDA combined methods for supporting the research or not; for understanding TTO’s that can be better substituted for improving revenues for Higher Educational Institutions through new patents, copyrights and trademarks by innovative products and services (Table 4); this paper applied these tools in a different way, observing better TTO’s profiles for investments, using those criteria.

To answer this research question, a survey of 145 TTOs was conducted using the database of an institutional questionnaire applied for this higher education institution. It is based on an applied questionnaire given to TTOs for recognizing the status of the technology transfer condition, focusing on an approach to the institutionalization of TTOs within a regional context.

The remainder of this paper comprises five sections: section 1 introductory, section 2 presents the innovation context being institutionalized in the IES, the section 3 describes the methodology (TOPSIS-MACBETH-2N) applied, section 4 observes the perspectives of the tools used in the qualitative and quantitative context, and the results; and section 5 is devoted to the final considerations of the results.


This section presents

  • Innovation Concepts;

  • Considerations about Technology Transfer Offices in Brazil;

  • Some considerations about Multicriteria;

  • Pearson Correlation;


2.1 Innovation

Institutions are turning to the production of intellectual assets and the copyrights, patents and trademark registrations, in addition to the production of exploratory scientific papers vis-a` -vis the innovations practiced.

The innovation process needs different sources: companies, universities, laboratories, research and development institutes, among others. Technological innovation in companies is a key factor in the maintenance of its activities and for the growth and development of the country (Pereira et al., 2015). O’Kane (2016, paper in press) also understands that although functionally separate, there is a high level of interdependence between the established role identity of scientists and the emerging role identity of TTO executives. This is because scientists require TTO assistance to realize the commercialization ambitions of the research, as obtaining registration of the intellectual property is what increases the IES’ visibility, incomes and revenues as well; for investing in newer researches, infrastructure for laboratories and other institutional demands.

Technological innovation in companies is a key factor in the maintenance of its activities and for the growth and development of the country (Pereira et al., 2015). O’Kane (2016, paper in press) also understands that although functionally separate, there is a high level of interdependence between the established role identity of scientists and the emerging role identity of TTO executives. This is because scientists require TTO assistance to realize the commercialization ambitions of the research, as obtaining registration of the intellectual property is what increases the IES’ visibility, incomes and revenues as well; for investing in newer researches, infrastructure for laboratories and other institutional demands.

Innovation concepts observed extensively until the mid-1970s were the epistemological construction for mass production could increase while keeping sizeable revenue amounts.

However, with a quality management trend on production, discussion arose of bringing the conflicting issues of the production chain to academia, thus keeping high rates of economic development without environmental commitment (Martins et al., 2015). By bringing this vision to academia, business and society began to gain knowledge of the development of research that met the issue of constant social demands with the scientific perception of innovation registered as a common good.

Innovation brings increased competitiveness in the country: disruptions to innovative or provocative factors lead to economic development with added value in the production chain of the various economic sectors.

Innovation can be seen as market experiments seeking broad and extensive changes, which fundamentally structure industries and markets. Thus, this is the fundamental basis of capitalism; hence, this is a production system that needs constant renewal and reinvigoration of its consumer goods and capital. (Pereira et al., 2015).

Because of this new role assigned to universities, issues of intellectual property and technology transfer began to occupy policy maker’s discussion agendas in universities. This was particularly true in public universities, where innovation culture development and a greater research results protecting importance, awareness resulted in the economic benefits sharing with academic researchers (Ritter dos Santos et al., 2013).

2.2 TTOs in Brazilian context

The concept and the identification for something new, aims to analyse and document the adaptations and innovations (even if they are benchmarking), in order that the best practices in innovative production processes raise the national intellectual property (Cornell University et al., 2015).

The enforcement of Innovation Brazilian Law number 10.973 in 2004 became one of the most important milestones in technological innovation promotion in Brazil, because it realigned universities with the knowledge economy upon sharing their knowledge with society by promoting inventions and intellectual property. Apart from the volume of patents originating from universities and research institutes have increased significantly over the last few years, it is important to observe what it is happening inside the IES to understand their performance (Ritter dos Santos et al, 2013).

Considering the TTOs’ objective is to promote innovation and technology transferred from universities to the society in general, this particular IES analysed opportunities to institutionalize TTOs in their campi for opening investments to this institution in its several centres. These TTOs were created to promote registries of intellectual properties regarding the academia innovation, as well the shared innovation with companies and society in general, which searched the institution for supporting their innovative ideas.

Hence, these TTOs produce innovative goods and services can support the launch of the startups inside this IES. Relatively speaking, the increasing number of startups and business incubators are small; due to these circumstances inside this IES, less than 20 startups were registered.

2.3 Multiple criteria decision - concepts

Real-world decision problems are rarely mono-criterion based. They generally incorporate a variety of criteria, often contradictory. In many practical situations, alternatives must be ranked given multiple, conflicting criteria of preference. Such intervention in the decision-making process might occur; and the set of criteria can comprise quantitative as well as qualitative criteria (Gomes et al, 1997). The methods are needed for supporting the evolution of the process.

Decision-making is sometimes a complex and difficult task, especially when it involves finding best alternative, an option, a choice, either an action, e.g., or ranking all alternatives in the presence of multiple, usually conflicting criteria (Kuo, 2017). The vision of providing decision support through the development of knowledge of the context even before the MCDA exists as an established area of knowledge (Ensslin et al., 2010).

Spak (2012) observes multicriteria methods in real problems where a predetermined number of alternatives - in the majority of time with uncertain preferences - with divergences of opinions (inside a group of DM) are utilized.

Multicriteria decision making method plays an important role to select the nondominate one(s) among a number of feasible alternatives evaluated according to multiple criteria in real-life decision-making problems, including uncertainty (Sahin, 2016; Mishra, Chatterjee, 2017).

Research devoted to such problems is most often referred to as multicriteria decision making or multiple criteria decision making (MCDM). MCDM is a usual human activity which helps making decisions mainly in terms of choosing, ranking or sorting alternatives. (Zhang et al, 2017). A number of multi-criteria methodologies have been developed over the elapsing of the time with the aim of providing a systematic framework that considers the multidimensional nature of the real-world problem (Pujadas et al., 2017).

The family of criteria is supposed to satisfy the following consistency conditions (Greco, Mousseau & Słowinski, 2008):

  • (a) Exhaustively - any two alternatives having the same evaluations on all criteria from G should be considered indifferent.

  • (b) Cohesion - imposing correct distinctions of those criteria that are maximizing and thus minimizing (Naciff de Andrade et al., 2016).

  • (c) Monotonicity - when comparing two alternatives, an improvement of one of them on at least one criterion should not deteriorate its comparison to the other alternative.

  • (d) Non-redundancy - excluding criteria whose valued characteristics are just done by other criteria.

Regarding decisors’ profile; the DM ranks alternatives (AI) described by the criteria (Cj), with a weight criterion (W) and grades (classification) that are given for each alternative, in each criterion. Assuming the knowledge of the preferences of the decision-makers and the quality of the evaluation, it can be admitted that one action is as good, better or worse than another, that is, to rank the alternatives (Cardoso et al., 2009).

In the multiple criteria sorting problem, alternatives need to be assigned to one or more classes, based on their evaluations on multiple and potentially conflicting criteria. (Kadzinski et al, 2014). In multiple criteria ranking problems alternatives are compared pairwise, and the results express preferences with the use of comparative notions (Kadzinski et al, 2012).

Multicriteria methods are generally classified in two groups: compensatory methods and noncompensatory methods (when strength in one of the criteria doesn’t compensate weaknesses in others) (Yunna et al., 2016). The first group, assumes the decision-maker is able to explain his preferences rationally; there is a global preference function which aims to synthesize the multiple criteria into a single criterion. Non-compensatory methods adopt different procedures and do not consider a single synthesis criterion (Lopes & Almeida, 2013). Non-compensatory MCDA methods are largely based on the (binary relation) outranking relation. MCDA methods deal with three major types of problem: choice, ranking and sorting (Greco & Słowinski, 2016).

The selection of a multicriteria method depends on the decision-maker’s preferences being aggregated, and not being restricted to a single criterion, namely, reducing everything to just one measure. In many cases, the decision-maker has great difficulty in dealing with the tradeoffs of a problem (Lopes & Almeida, 2013).

Some authors use MCDA for non-compensatory methods (outranking methods) and MCDM to compensatory methods (unique criterion of synthesis methods) (Govindan & Jepsen, 2016). MCDA can also be defined as a set of techniques which are designed to search for a number of alternatives within multiple criteria and conflicting objectives (Gomes et al., 2017).

MCDA is a general term for methods, providing a quantitative approach to support decision making in problems involving several criteria and choices (alternatives or actions) (Botti & Peypoch, 2013; Bailey et al., 2014).

Nowadays, MCDA is facing three important methodological challenges: handling a complex structure of criteria, dealing with interactions between criteria and reducing the cognitive effort of the DMs in interaction with MCDA methods. These challenges are usually handled separately; however, they often concern the same decision problem (Angilella et al., 2016).

MCDM/A is a field of study that originated in Operations Research (OR) and focuses its efforts on solving multiple criteria decision problems (Zaka & Kruszynskib, 2015). These problems are complex decision situations in which several, often contradictory, points of view must be considered. The MCDM/A methodology clearly identifies the major participants of the decision making/aiding process such as: the DM, the analyst and the interveners (stakeholders) and their roles in this process (Kadzinski et al., 2016).

Sometimes the mutual preference independence in a criteria set can be violated. For example, an excellent (comfortable) car with high speed has a price which is associated with comfort and high speed (Corrente et al., 2016). Angilella, Greco and Matarazzo (2010) consider another point, about mutual preference independence: the comprehensive importance of criteria is greater than the sum of the importance of the two criteria or more considered separately. They propose that same decision problem, namely that very often there is positive interaction (synergy) of criteria instead of mutual preference independence problem.

DM commonly uses linguistic variables to evaluate the importance weights of criteria and the ratings of alternatives with respect to various criteria. The concept of linguistic variables is useful in the case of complex or ill-defined situations (Chen, 2015).

As decisions are dynamic, the DM must be convinced that the analysis process is conducted properly and thoroughly, in order to enable the DM to estimate the potential outcome of their decision. This justifies the swift growth in demand for approaches based MCDA methods, and explains the rising in adoption of models to support decision that occurs in situations of uncertainty or even risk (Gomes et al., 2017).

2.4 Pearson’s correlation

Pearson’s correlation coefficient when applied to a population is commonly represented by R(x, y) and may be referred to as the population correlation coefficient or the population Pearson correlation coefficient; verifying statistical independence between variables. The formula 1 (where x and y are variables) represents Pearson’s correlation:

R(x, y)=i=1nxi-Xyi-Yi=1nxi-X2 i=1nyi-Y2, (1)


X=i=1n xin (2)

Y=i=1n yin, (3)

and −1 ≤ R(x, y) ≤ 1.


MACBETH helps to evaluate options by making a qualitative representation in attractiveness preferences terms. The method helps to make a quantitative scale through attractiveness qualitative information. Thus, it is easy to explain the same to the decision-makers, as well as preferences explanation (Roubens et al., 2006). The Preference Ranking Organization METHod for Enrichment of Evaluations (PROMETHEE) was also considered for conducting this research, considering the sensibility of using a set of weightings (Leoneti, 2016). However, as PROMETHEE has a quantitative value for being analysed before the judgments for partial rankings; this particularity was against the data collected from the questionnaire because it was necessary to build a scale for answer options, in those questions where only it was possible to answer the professor or student profiles. Such uniqueness also happened at Gomes et al. (2008) when farmers had a difficult judgment for giving a quantitative opinion among intra and inter criteria.

MACBETH method is a method of semantic judgment. With it, value functions are obtained through semantic judgments made by comparing the attractiveness difference between any two lines of action, always in pairs; after elaborating the judgments on the differences of attractiveness and constructed a cardinal scale of value on the set of alternatives. Then, any inconsistencies are verified and, subsequently, a cardinal value scale is determined that represents the value judgments of the decision maker. The scale obtained and normalized, providing the values of the weights for the alternatives under evaluation, which allows the use of an aggregation model, in general additive.

MACBETH is a support decision method which allows evaluating options considering multiple criteria, using qualitative judgments over attractive differences for weighing up criteria and evaluate options (Bana e Costa et al., 2012).

MACBETH is an interactive approach that uses semantic judgments about the differences in several stimuli attractiveness to help a decision maker quantify the relative attractiveness of each. The proposed method is designed for ranking problem. Thus, what distinguishes MACBETH from the other multicriteria models is that it needs only qualitative judgments about the difference of attractiveness regarding elements at a time, in order to generate numerical scores for the options in each criterion and to weigh the criteria using - according to the method - the base-scale (Dhouib, 2014).

Otherwise, the advantage of an ordinal preference scale transformation to a cardinal scale allows differing the distance among the alternatives: identifying the weak, very weak, strong, very strong etc. The DM compares options for informing the indifference or defining the correlation among the alternative judgments. This is a MACBETH’s great advantage regarding other multiple criteria support decision methods. From a technical point of view, MACBETH enables the construction of a multicriteria additive value model whose benefits of actions are represented by their overall scores computed by adding the products of the scores of the actions to the benefit dimensions by the respective weights (Bana e Costa et al., 2012).

Scales of value are common in several complex problem areas involving multiple criteria identified when presenting conflicting criteria among them which are not understandable or subjective. Conflicting criteria could also be not quantifiable due to not needing the application of a scale of value, that is to say, to attribute value to such criteria as a way of approaching them in the decision process. By using these methods, there are great interactions between the qualitative and quantitative data for making decisions (Spak, 2012).

Per Naciff de Andrade et al. (2016), in order to be possible to ordinally measure the attractiveness of options “x” from a finite set “X” - it is necessary to associate a real number v(x) to each of “x” that satisfies the conditions of: stricter preference (Equation 4) and indifference (Equation 5).

x, yX : xPyv(x)>v(y) (4)

x, yX : xIyv(x)=v(y) (5)

Where: P - preference and I - indifference.

w, x, y, zX with x more attractive than y and w more attractive than z: so, the quotient (v(x) − v(y))/(v(w) − v(z)) measures the attractiveness difference between x and y when the attractiveness difference between w and z is the unit of measure.

Respecting such conditions, a numeric scale of intervals is established (v: XR: xv(x)). The transition of ordinal information at MACBETH to cardinal is done by comparing options in pairs in a qualitative way. The M-MACBETH software deploys the model, as far as the evaluations are inserted at the trials matrix automatically observing the consistency. If there is any inconsistence, possible solutions are shown.


One of the most widely used multi-criteria decision analysis methods is the TOPSIS method, which was proposed by Hwang and Yoon in 1981, and extended by Yoon in 1987, as well as by Lai Hwang, Jou Lai, and Yun Liu in 1993. In the TOPSIS method, the optimal alternative is nearest to the positive ideal solution (PIS) and farthest from the negative ideal solution (NIS), measuring the distance to PIS and to the NIS (Walczak & Rutkowska, 2017; Roszkowska & Wachowicz, 2015). TOPSIS performance attracted attention in terms of accuracy, and TOPSIS had good performance in descriptive analysis (Leoneti, 2016).

The multicriteria support for decision making consists of a group of techniques for supporting a DM to make decisions about a complex problem, evaluating and choosing alternatives for solving it, following different criteria and points of view (Gomes et al., 2017).

This technique allows the decision to be regulated on a weighted criteria basis relevant to the problem to be answered by the DM agents. Here the importance of the criteria is defined by them, in an interactive process with other technical-political actors (Sarrazin & de Smet, 2015).

Regarding the MCDA in Brazil, Longaray et al. (2015) observe within the multicriteria methods most applied from 2004 to 2013 in scientifically studies, bibliometric researches showed the difficulty of identifying in the literature the use of MCDA. Because there are more than ten thousand possibilities of papers to be analysed inside the Brazilian scientifically production and on his (Longaray’s) research; he restricted it to observing only the most cited authors.

Conversely, TOPSIS enhances by being more intuitive and simple for the use of mathematical procedures that contribute to the ease of implementation and application, allowing it to evaluate a non-limited quantity of alternatives and differ from the comparative approaches (Junior & Carpinetti, 2015; Kluczek & Gladysz, 2015).

One of the most widely used multi-criteria decision analysis methods are the TOPSIS method, which was proposed by Hwang and Yoon in 1981, and extended by Yoon in 1987, as well as by Lai Hwang, Jou Lai, and Yun Liu in 1993. (Walczak & Rutkowska, 2017; Roszkowska & Wachowicz, 2015).

The PIS is the one maximizes the most “advantageous” classification each criterion, while the NIS is the one minimizes the classification each criterion. According to Bilbao-Terol et al. (2014), the PIS is made up of all the best possible values based on the benefit criteria, and the NIS consists of all the worst possible values of the cost criteria in order to classify a finite set of alternatives. TOPSIS, therefore, is a method that results in more balance in the evaluation, placing the alternatives in relation to two points of reference: the PIS or ideal point (the method minimizes the distance), and the NIS or the anti-ideal point (method maximizes the distance) (Walczak & Rutkowska, 2017).

According to Kuo (2017), TOPSIS has been widely applied in the past decades, considerable efforts have been made to modify, either extend TOPSIS, and even though to compare, or hybridize it with other MCDM methods. TOPSIS stands out as an intuitive method, characterized by the simplicity of its mathematical procedures, ease of deployment, and because it permits the evaluation of an unlimited quantity of alternatives, unlike methods that use comparative approaches.

According to TOPSIS, the best alternative would be which is next to the positive ideal solution and the most distant to the negative ideal solution (also known as anti-ideal). The positive ideal solution is that which maximizes the criteria of benefits and minimizes the criteria of costs; in other words, the negative ideal solution maximizes the criteria of costs and minimizes criteria of benefits. The positive ideal solution is composed of all the best values achievable from the benefits criteria and the negative ideal solution consists of all the worst values achievable from cost criteria, in order to rank a finite set of alternatives e.g. (Bhutia & Phipon, 2012; Bilbao-Terol et al., 2014).

Since using only MACBETH, the DM is already able to rank the alternatives. However, the use of TOPSIS allows the DM to locate the alternatives between the best and the worst. And the use of two normalizations (formulae present in equation 9 and 10) allowed a sensitivity analysis.

Lima-Junior and Carpinetti (2016) propose the use of fuzzy set; however, for this problem, it was proposed the use of MACBETH to transform a semantic scale into numerical scale. It’s only 4 questions. Where each question has a maximum of five options. Faced with this, this modelling is theoretically simple. It is easier for the decision maker to understand, and more usual for continued applicability at the questionnaire, serving for this paper’s purpose.

The Euclidean calculated distances between A i and A + (benefits) and between A i and A (costs) is realized by the equations (6) and (7):

Di+=j=1n pij-pj+2 (6)

Di-=j=1n pij-pj-2 (7)

while the relative proximity calculations Ci for each alternative Ai in relation to positive ideal solution A+ is done by the equation (8):

Ci=Di-Di+-Di- (8)

where i = 1, ..., m and the index value C i ranges between 0 and 1.

First Normalization method (equation 9):

yij=xiji=1n x2ij, i=1, 2,, m; j=1, 2,,n (9)

Second Normalization Method (equation 10):

yij=xijmax Xi (10)


Different approaches are proposed from various schools of thinking. There are nor better neither worse techniques; only techniques that fit better to a certain situation. It is not easy to say which MCDM approach is more reasonable and reliable for a given decision-making problem, as the selection of MCDM methods itself is a complicated MCDM process. Many DMs apply several MCDM approaches to the same problem, compare their results, and then make the final decision. This approach is difficult to comprehend and complex to implement because it requires extensive technical knowledge in MCDM fields. A combination of different MCDM evaluation techniques to construct a hybrid model may be the correct choice in solving this problem (Wang et al., 2016).

The reasons for integrating two methods together are:

  • MACBETH is used to treat qualitative data (in the questionnaire).

  • TOPSIS uses the criteria to order the alternatives (including the qualitative data previously treated by MACBETH).

  • MACBETH and TOPSIS use different methods to normalize; the aggregation uses the two normalized methods (allowing different arrays associated with each normalization method to rise, thus allowing a sensitivity analysis of the results).

  • TOPSIS e MACBETH are based on additive aggregation of criteria.

3.1 The questionnaire

This research purpose is to understand the innovation evolution as an opportunity for promoting revenue to invest at IES infrastructure; considering TTOs’ profiles by their strength and weakness. Those profiles were shown on the questionnaire (Table 1) by the collected answers; restricting this research to TTOs of science, technology and biomedicine areas. The questions were developed to know the TTOs profiles at the beginning of their creation and for understanding in which they are strong and what kind of investment it is necessary to offer for improving royalties obtained from intellectual properties, agreements and partnerships; when questions observed any sort of weakness for ranking those TTOs (Martins et al., 2015).

Table 1 The questionnaire applied to TTOs. 

Questions Codes
Among the level of degree in the offered courses, in which of them is the TTO attached with a higher degree of certification? q 1
Are there research projects? q 2
What is the TTO highest professors’ coordinators’ degree? q 3
Among TTOs collaborators, what is the highest education degree? q 4
Numbers of praticated activities q 5
Numbers of published articles/papers q 6
Numbers of equipment’s obtained q 7
Numbers of promotions obtained q 8
Number of published books q 9
Number of completed collaborations q 10
Number of elaborated products q 11
Number of completed social projects q 12
Number of completed services q 13
Number of published thesis/essays q 14

This questionnaire was a new one formulated by a strategic department as IES’ scientifically census for understanding which TTO in science, technology and biomedicine areas would have the infrastructure for offering better innovative products and services patents, trademarks and copyrights to IES obtains revenues. TTO’s coordinators answered with numbers the quantity of assets, human resources, scientifically production that their TTO’s have produced since the very beginning of their existence. These questions were answered via an electronic institutional database with confidentiality among them for any TTO knows the answer of the other (only the strategic department knows the answer of each TTO). This scientifically census result was codified in codes for each TTO (e.g. “A17” means TTO “A” number “17”) in order to decide which of them could be offered investment for their productivity and potential productivity (a decision which could be taken by the Rectory and the Provost which those TTO’s responsibilities belong to).

Questionnaire questions were adapted from a government application form made by the Ministry of Science and Technology answered as a scientifically census annually for all higher educational institutions in Brazil.

The methodology for observing this questionnaire data was decision analysis, because they are done in many ways, regardless of the checked methodology or the mental model applied: all the before structuring of the best choice is built based on the same elements. It is possible to share in three principal elements: the DM, alternatives and criteria (Gomes et al., 2017).

The decision makers were Rectory and Provost; the alternatives were TTOs from science, technology and biomedicine areas and criteria adopted were the questionnaire answers transformed into quantitative analyses in MACBETH combined with TOPSIS for ranking TTOs that could be supported in a short term by IES in order to boost revenues for the institution.

After collecting all the answers, the strategic department observed the collected data behaviour in a statistics perspective, for understanding the data quality produced by the TTOs’ answers in the questionnaire; as an analogous thought in observing decision support management analysed by Chaves et al. (2012). The first attempt was to understand the question which was better correlated. In this regard, as Figure 1 shows, using the R Language Program through the command pairs.panels (x) where x is the list with all the answers of the 145 TTOs for each one of the 14 questions: as observed the correlations are small and sometimes negative, visualized also in the plots of dispersion. 0.78 was the best correlation between the question number 14 (q 14) and the question number 6 (q 6); which means articles and papers produced by professors and students who work at these TTOs are good outputs from thesis and dissertations related to the researched areas.

Figure 1 The correlation and dispersion of the questions using the command pairs.panels (x) using the R Language Program. 

Although one of the correlations is 0.78 between the questions (q 14 and q 6) in the questionnaire, the Kaiser-Meyer-Olkin factor adequacy was 0.77 showing a good range. The kurtosis was 4.683885 (leptokurtic: higher than the normal distribution) and the Bartlett’s Test had a p-value < 0 (1.138693e-75).

The skewness shows -2.346562 which means the distribution has a left tail (values below average - positive skewness). After considering the complex behaviour of the questionnaire using Multivariate Normality Tests, it was proposed to observe TTOs in the best position of a rank using MCDA, according to the answers obtained from the questionnaire. Han and Shin (2014) observe that quantitative criteria, methods can be used in a better analysis when there is consistent quantitative data. Thus, the statistic step was concluded from computing MCDA tools results for ranking in a compensatory way the alignment between MACBETH and TOPSIS normalized twice.

It is usually the DM who quantifies and qualifies alternatives and criteria for structuring the decision problem. The DM and the analyst are the same person in most of the cases. Regarding the decision-making analysis; it is recommended to be different individuals, so as to not submit their desires to the decisor and in this way, compromise the results. Chiu et al. (2006) observe the need for supporting DM because the perception, criteria in their process of evolution may sometimes be vague.

Following the trend of sustainable development where TTOs must aim to promote innovation (Elkington, 1999) regarding the demand from social technology transfer, the questionnaire was produced considering the quality of knowledge from the team that works with TTOs and their outputs (e.g.: articles, projects, books, services, social projects, essays, patent registration). With these perspectives in mind, 14 questions were formulated. Four questions were qualitative questions whose answers were transformed into quantitative via the use of MACBETH in the “base” scale: the transformation using the program named M-MACBETH for changing the ordinal scale into cardinal scale (questions: q 1, q 2, q 3 and q 4 - Table 1 and Table 2).

Table 2 The qualitative answers transformed at M-MACBETH base-scale. 

Questions coded Options of answer Codification of the answers at M-MACBETH M-MACBETH “de base” scale
q 1 Doctor D 17
Master M 14
Specialist E 11
Undergraduate G 7
Extension Ext 0
q 2 Yes, with student’s participation S 8
Yes, without student’s participation WS 4
No, there aren’t. N 0
q 3 Doctor D 12
Master M 8
Specialist E 4
Undergraduate G 0
q 4 Doctor D 11
Master M 8
Specialist E 5
Undergraduate G 2
Intern Estag 0

The rest of the 10 questions were quantitative and were simply grouped with these transformed 4 questions computed at M-MACBETH software. Table 1 shows the questionnaire applied to TTOs because these 10 questions were cardinals and quantitative.

Issues are treated as criteria. The answers to questions are scored on the criteria. The projects are the alternatives, formed by the scores in the criteria. For this it is necessary to use the multicriteria to order the projects.

3.2 MACBETH’s application

Considering the weights for each answer option at MACBETH “de base” scale, in this Brazilian IES context, it’s important to observe that human resources who work at TTOs without high degree level education (Doctor e.g. of higher degree), receive a lower grade/weight because it means less opportunity for achieving government subsidies in researches, because some scientifically programs launched by Ministry of Science and Technology demands a coordinator and a Doctor research as part of the TTOs’ team, as well as some international universities only admit agreement cooperation signature if the TTO’s responsible is a Doctor coordinator. Infrastructure is other impact factor because if TTO has enough supportive assets for developing products and services, there will be revenue in short term for university being made by tradeable agreement between companies and the university. Conversely, scientifically production shows TTOs’ productivity profile also showing which TTOs could bring to university revenue in a short term considering intellectual properties tradeable with companies and inside the government as well.

3.3 TOPSIS’ application

Considering that the identification of the most innovative and developed TTOs’ searches Multicriteria Decision Aid depends on the alignment of the qualitative to be transformed into quantitative questions, it was decided - after observing several Multicriteria Decision Aids - using MACBETH and TOPSIS. After deciding the aids, the possibility of testing TOPSIS was observed in a different system from just calculating electronic sheets. It was decided to use the programming language R for this purpose.

Programming language R was organized in 2002 as “The R Foundation for Statistical Computing”, abbreviated as “R Foundation” seated in Vienna, Austria and is active worldwide (R FOUNDATION, 2002). It is considered a free software foundation because everyone can add new packages (new contents of calculations programming) and it is possible for a person without any experience with this software to read the manuals of the packages and the help contents for understanding and calculating any desirable output (if this output has not been available yet, it is possible to add this output offering to the software the package with the input formulas).


Founded values will be in MACBETH’s base-scale that can be normalized and represented in percentiles. Using the M-MACBETH base-scale was displayed at Table 2 where the qualitative answers were transformed into quantitative.

Finally, after getting the transformation needed using M-MACBETH, there are conditions to accept cardinal values from the four questions applied at the questionnaire. This is the reason MACBETH was used: for supporting the TOPSIS method used in the entire questionnaire for ranking TTOs.

R was chosen because of the simple way data is input into the system and how it obtains the rank as output, after inserting action commands in its console from the package named MCDM (Table 3).

Table 3 Applying the MCDM package as basis of calculation the TOPSIS via R

> x < − read.csv (“rstudio2.csv”, header = TRUE, sep = “;”)
> d < − as.matrix(x[,1:14])
> library(topsis)
> w < − c(0.081, 0.090, 0.065, 0.026, 0.048, 0.138, 0.053, 0.081, 0.090, 0.065, 0.026, 0.048, 0.138, 0.053)
> i < − rep(“+”,14)
> output < −topsis(d,w,i)
> output
alt. row score rank alt. row score rank
1 1 0.23323150 37.0 40 40 0.51270187 2.0
2 2 0.14144521 84.0 41 41 0.12878584 96.0
3 3 0.25030013 30.0 42 42 0.11472777 106.0
4 4 0.13031608 93.5 43 43 0.10003992 118.0
5 5 0.24346407 31.0 44 44 0.23383440 36.0
6 6 0.16835934 63.0 45 45 0.35947005 13.0
7 7 0.08327374 131.0 46 46 0.11665346 103.0
8 8 0.44200323 4.0 47 47 0.15526012 74.0
9 9 0.25060285 29.0 48 48 0.16829886 64.0
10 10 0.36619584 11.0 49 49 0.16377551 68.0
11 11 0.26294366 23.0 50 50 0.18183857 54.0
12 12 0.37987110 10.0 51 51 0.16478733 66.0
13 13 0.12878584 96.0 52 52 0.16415691 67.0
14 14 0.18865359 53.0 53 53 0.25895098 25.0
15 15 0.13599546 88.0 54 54 0.16988663 62.0
16 16 0.49778185 3.0 55 55 0.30532252 15.0
17 17 0.38678157 8.0 56 56 0.06907011 139.0
18 18 0.11591935 104.0 57 57 0.29261896 18.0
19 19 0.28973731 19.0 58 58 0.29797531 16.0
20 20 0.21644781 41.0 59 59 0.08657962 123.5
21 21 0.27117633 21.0 60 60 0.22251728 40.0
22 22 0.21120322 44.0 61 61 0.07741437 134.5
23 23 0.12266571 102.0 62 62 0.13767696 85.0
24 24 0.39713673 5.0 63 63 0.08395204 128.0
25 25 0.13042475 92.0 64 64 0.27027553 22.0
26 26 0.12439997 100.0 65 65 0.17432236 59.0
27 27 0.25808288 27.0 66 66 0.08395204 128.0
28 28 0.38884870 7.0 67 67 0.14905041 77.0
29 29 0.13692193 86.0 68 68 0.13654440 87.0
30 30 0.08715050 122.0 69 69 0.07437783 137.0
31 31 0.07684338 136.0 70 70 0.14283694 78.0
32 32 0.31071386 14.0 71 71 0.09776193 119.0
33 33 0.13031608 93.5 72 72 0.39465251 6.0
34 34 0.17285858 61.0 73 73 0.38402681 9.0
35 35 0.12878584 96.0 74 74 0.11335110 108.0
36 36 0.12567601 99.0 75 75 0.14227038 81.0
37 37 0.36155858 12.0 76 76 0.23659404 35.0
38 38 0.15325858 75.0 77 77 0.16026527 71.0
39 39 0.14226222 82.0 78 78 0.18026568 58.0

Using the R Language Program, it can be seen that the TTO number 98 took the first place in the rank, using the MACBETH-TOPSIS method with entropic weights. The next section analyses the MACBETH-TOPSIS-2 using normalization into two different ways of assessment basis.

4.1 MACBETH-TOPSIS-2N: Second normalized method

The first assessment basis used in this normalized method of MACBETH-TOPSIS-2N was the normalization using just the split between the greater number inside the question column, by the number of the row analysed. For example: in the first question, the greater number was 17 split by the first row, answering precisely 17 and giving as a result, the number 1 - this calculation was done for all the answers in all the rows and columns.

The second assessment basis used in normalized method MACBETH-TOPSIS-2N was the normalization using the sum of all the numbers from a question of a column; for example: the first column question has the sum 2,194 that was split by the number of the first row answered in this column-question - 17 - resulting in 0.007748405, where this calculation was also done for all the answers in all the rows and columns. Table 4 shows a comparison among results used in this paper for ranking TTOs.

Table 4 Comparison among MACBETH-TOPSIS-2N and R. 

General TTOs’ rank Rank by splitting the greater number by the others Rank by splitting the sum of the number by the others Rank from R Language - assessment by entropy
1 A98 A9 A98
2 A16 A98 A40
3 A40 A58 A16
4 A24 A40 A8
5 A12 A16 A24
6 A8 A24 A72
7 A28 A73 A28
8 A45 A72 A17
9 A17 A17 A73
10 A53 A12 A12
11 A11 A8 A10
12 A72 A11 A37
13 A19 A28 A45
14 A118 A118 A32
15 A21 A76 A55
16 A22 A130 A58
17 A55 A21 A100
18 A73 A55 A57
19 A32 A45 A19
20 A44 A19 A86
21 A90 A53 A21
22 A57 A60 A64
23 A5 A10 A11
24 A14 A126 A109
25 A113 A37 A53
26 A20 A1 A102
27 A50 A27 A27
28 A109 A36 A113
29 A9 A57 A9
30 A1 A86 A3
31 A34 A100 A5
32 A52 A44 A90
33 A6 A142 A118
34 A119 A97 A108
35 A10 A32 A76
36 A142 A113 A44
37 A48 A82 A1
38 A27 A128 A135
39 A3 A140 A128
40 A130 A64 A60
41 A64 A3 A20
42 A47 A23 A142
43 A60 A90 A138
44 A39 A22 A22
45 A128 A102 A104
46 A67 A108 A126
47 A78 A84 A80
48 A76 A109 A84
49 A58 A133 A133
50 A29 A80 A97
51 A70 A20 A119
52 A108 A5 A140
53 A81 A50 A14
54 A126 A51 A50
55 A104 A34 A81
56 A25 A52 A83
57 A96 A104 A130
58 A4 A127 A78
59 A33 A115 A65
60 A51 A14 A127
61 A80 A81 A34
62 A139 A6 A54
63 A138 A78 A6
64 A133 A87 A48
65 A87 A62 A96
66 A137 A48 A51
67 A135 A96 A52
68 A77 A83 A49
69 A127 A54 A144
70 A115 A119 A106
71 A145 A39 A77
72 A114 A138 A132
73 A102 A143 A111
74 A143 A77 A47
75 A26 A65 A38
76 A74 A135 A87
77 A111 A46 A67
78 A49 A49 A70
79 A54 A120 A122
80 A68 A145 A145
81 A106 A47 A75
82 A121 A122 A39
83 A116 A75 A115
84 A107 A114 A2
85 A65 A107 A62
86 A83 A137 A29
87 A140 A38 A68
88 A38 A18 A15
89 A132 A43 A143
90 A91 A111 A139
91 A86 A67 A137
92 A79 A2 A25
93 A15 A106 A4
94 A129 A29 A33
95 A85 A139 A13
96 A95 A144 A35
97 A71 A70 A41
98 A110 A68 A114
99 A30 A15 A36
100 A2 A129 A26
101 A59 A26 A120
102 A93 A25 A23
103 A13 A4 A46
104 A35 A33 A18
105 A41 A74 A107
106 A37 A132 A42
107 A88 A94 A125
108 A120 A125 A74
109 A97 A13 A88
110 A61 A35 A82
111 A134 A41 A116
112 A46 A116 A91
113 A125 A88 A129
114 A92 A42 A85
115 A62 A79 A95
116 A82 A91 A79
117 A94 A95 A121
118 A112 A85 A43
119 A136 A121 A71
120 A99 A71 A94
121 A23 A89 A110
122 A63 A110 A30
123 A66 A30 A59
124 A101 A59 A93
125 A42 A93 A112
126 A89 A92 A136
127 A144 A61 A63
128 A100 A134 A66
129 A36 A131 A101
130 A43 A112 A89
131 A69 A99 A7
132 A75 A31 A131
133 A122 A136 A124
134 A7 A7 A61
135 A84 A63 A134
136 A141 A66 A31
137 A117 A101 A69
138 A18 A124 A99
139 A131 A56 A56
140 A56 A141 A92
141 A124 A69 A117
142 A123 A117 A123
143 A103 A105 A103
144 A31 A123 A141
145 A105 A103 A105

Different rankings with the same data are noted. It shows that the result is influenced by modeling. Model influences choices suggest possible decisions. In view of this, it was decided to compare them with the purpose of allowing the decision-maker to discern which options would be presented as the highest frequency among the most highly rated.

4.2 TTOs analysis

The domino set of alternatives are {A1; A8; A9; A10; A11; A12; A142; A16; A17; A19; A21; A22; A24; A27; A28; A3; A32; A40; A44; A45; A53; A55; A57}.

Only four criteria need a minimum evaluation:

  • (q 1) - undergraduate;

  • (q 3) - specialist;

  • (q 5) - number 1;

  • (q 7) - number 1.

The Pearson’s correlation obtains (R(x, y)> 0.6) observed at Table 5:

Table 5 Pearson’s analysis. 

Question Question Pearson’s correlation
Numbers of promotions obtained (q 8) Numbers of published articles/papers (q 6) 0.64
Numbers of promotions obtained (q 8) Numbers of equipment obtained (q 7) 0.61
Numbers of promotions obtained (q 8) Number of published thesis/essays (q 14) 0.68

The set of domain alternatives, obtained by the questionnaire results in MACBETH-TOPSIS-2N showing that:

  • (a) When there are coordinators with a doctorate in Brazil, they facilitate the production of books, articles, goods and services without resources. Promoting the desirable products of TTOs for technological innovation to occur through technology transfer.

  • (b) To succeed in calls for funding, the minimum required is: (q 1) undergraduate; (q 3) specialist; (q 5) 1; (q 7) 1. However, this minimum entails a need for a great score in the other criteria. Public calls only with professors with a Doctoral Degree (it is a restrictive item from Brazilian funding agencies).

  • (c) Numbers of promotions obtained is the criteria with more synergy with other criteria.

Some of them had more published papers than others that could be a good output; however, without the infrastructure these TTOs will fail without the support of skilled labour. Since happens this profile, they aren’t a good alternative set for achieving any subsidies from university, because although they have a good scientifically production, the lack of infrastructure and the lack of doctorate coordinators could promote a production constraint for improving university’s revenue.

Another observation is the lesser demand from companies and society in the using of the TTOs from IES as an opportunity of innovation dissemination for promoting partnerships.

This is due mainly to the use of the Innovation Law in Brazil, regarding actions to transfer the knowledge produced to society.

This can also be observed because there is not a routine inside this particular IES in promoting these TTOs as a mechanism of introduction of intellectual property and technology transfer management inside the IES for professors conduct partnerships among their TTOs. For example, medical and chemical TTOs do not have partnerships when producing their products, services or software. By working together, their research would be sped up compared to working separately.

Conversely, the use of different ways of calculating MACBETH-TOPSIS-2N has shown that the position may likely change in two places or more from one method of calculating to another. Such process variability occurs when there are big quantities of data for ranking and the numbers do not have the same behaviour for the survey respondents. For example, some TTOs produced 66 published articles/papers while others did not produce any during a whole production year (considering the type of research spending years for being developed). None of the ranks had the same TTOs at the same place according to the assessment basis, but in the majority of the TTOs, they had the same place in two basic assessment processes.

These three rankings of assessment basis showed how a DM can conduct the decision of qualifying investments at TTOs, considering the quantities or the behavior of these quantities of produced works through the production year. Otherwise, productive TTOs might support financially other TTOs not so productive (because the deployment stage at the research) with their royalties, granting these TTOs requesting more support for their researches; provoking in a middle term better conditions for developing researches at this educational institution.


This paper objective was to identify innovative TTOs from a specific IES in Brazil using MACBETH and TOPSIS MCDAs for supporting the transformation of a qualitative questionnaire’s answers and transforming four questions into quantitative cardinal number questions by being treated with the TOPSIS method in order to obtain TTOs’ rank in three different ways of assessment basis.

Observations showed that TTOs in Brazil have few actions to transfer knowledge to society because there are not partnerships in high volume among universities, society and companies, despite the government instituting an Innovation Law in 2004 for promoting such scientific development.

According to these results, it is suggested that TTOs have to increase the degree of their collaborators for giving better opportunities of investments in the production of articles, projects and intangible assets, with less expected bureaucracy from the norm via opening companies and developing products in Brazil.

By using observed available data from TTOs, this paper has endeavoured to identify how to improve the partnerships between those agents of society which seek the best quality in products and services, in terms of new features and processes, via several economic sectors.

There should also be a likely association in future applications between MACBETH and TOPSIS as a hybrid method for supporting multicriteria decisions. This is because of the assessment basis behaviour that did not have the same position at the three ranks for the same TTOs and this is important to DM to understand how to evaluate the best TTOs production or the TTOs need more investment for transferring social technology.


1 ANGILELLA S, CORRENTE S, GRECO S & SŁOWINSK R. 2016. Robust Ordinal Regression and Stochastic Multiobjective Acceptability Analysis in multiple criteria hierarchy process for the Choquet integral preference model. Omega, 63: 154-169. DOI: 10.1016/ [ Links ]

2 ANGILELLA S, GRECO S & MATARAZZO B. 2010. Non-additive robust ordinal regression: A multiple criteria decision model based on the Choquet integral. European Journal of Operational Research, 201: 277-288. DOI: 10.1016/j.ejor.2009.02.023 [ Links ]

3 BANA E COSTA CA, DE CORTE J-M & VANSNICK J-C. 2012. MACBETH. International Journal of Information Technology and Decision Making, 11(2): 359-387. S0219622012400068 [ Links ]

4 BEHZADIAN M, OTAGHSARA SK, YAZDANI M & IGNATIUS J. 2012. A state-of the-art survey of TOPSIS applications. Expert Systems with Applications, 39: 13051-13069. [ Links ]

5 BHUTIA PW & PHIPON R. 2012. Application of AHP and TOPSIS method for supplier selection problem. IOSR Journal of Engineering, 2(10): 43-50. [ Links ]

6 BILBAO-TEROL A ET AL. 2014. Using TOPSIS for assessing the sustainability of government bond funds. Omega, 49: 1-17. [ Links ]

7 BOLY V ET AL. 2014. Evaluating innovative processes in french firms: Methodological proposition for firm innovation capacity evaluation. Research Policy, 43: 608-622. [ Links ]

8 BOTTI L & PEYPOCH N. 2013. Multicriteria ELECTRE method and destination competitiveness. Tourism Management Perspectives, 6: 108-113. [ Links ]

9 BRASIL. 2004. Lei 10973: Lei de Inovação Tecnológica. Planalto, Casa Civil. [ Links ]

10 CARDOSO RS, XAVIER LC, GOMES CFS & ADISSI PJ. 2009. Uso de SAD no apoio à decisão na destinação de resíduos plásticos e gestão de materiais. Pesqui. Oper., 29(1): 67-95. DOI: http: // ]

11 CHAVES LC, ENSSLIN L & ENSSLIN S. 2012. Mapeamento do tema gestão do apoio à decisão quando analisado sob a ótica de seus resultados. Revista Eletrônica Sistemas & Gestão, 7(3): 336- 348. DOI: 10.7177/sg.2012.v7.n3.a4 [ Links ]

12 CHEN TING-YU. 2015. An interval type-2 fuzzy PROMETHEE method using a likelihood-based outranking comparison approach. Information Fusion, 25: 105-120. [ Links ]

13 CHIU Y-C ET AL. 2006. An evaluation model of new product launch strategy. Technovation, 26: 1244-1252. [ Links ]

14 CLÍMACO JCN, COSTA JA, DIAS LC & MELO P. 2009. Supporting collaborative multicriteria evaluation: the VIP Analysis plug-In for decision deck. International Journal of Decision Support System Technology, 1(4): 1-15. [ Links ]

15 CORNELL UNIVERSITY, INSEAD & WIPO. 2015. The Global Innovation Index 2015: Effective Innovation Policies for Development. Genebra, Suíça. [ Links ]

16 CORRENTE S, GRECO S& ISHIZAKA A. 2016. Combining analytical hierarchy process and Choquet integral within non-additive robust ordinal regression. Omega, 61: 2-18. [ Links ]

17 DHOUIB D. 2014. An extension of MACBETH method for a fuzzy environment to analyze alternatives in reverse logistics for automobile tire wastes. Omega, 42: 25-32. [ Links ]

18 ELKINGTON J. 1999. Cannibals with forks: the triple bottom line of the 21st century business. Capstone, Mankato. [ Links ]

19 ENSSLIN L, GIFFHORN E, ENSSLIN SR, PETRI SM & VIANNA WB. 2010. Avaliação do Desempenho de Empresas Terceirizadas com o Uso da Metodologia Multicritério de Apoio a` Decisão - Construtivista. Pesquisa Operacional, 30(1): 125-152. [ Links ]

20 GOMES CFS, COSTA HG & BARROS AP. 2017. Sensibility analysis of MCDA using prospective in Brazilian energy sector. Journal of Modelling in Management. DOI:10.1108/JM2-01-2016-0005 [ Links ]

21 GOMES LFAM, MURY AR & GOMES CFS. 1997. Multicriteria ranking with ordinal data. System Analysis Modelling Simulation, 27(2): 139-145. [ Links ]

22 GOMES EG, MELLO JCCBS & MANGABEIRA JAC. 2008. Índice multicritério de bem estar social rural em um município da Região Amazônica. Pesquisa Operacional, 28(1): 141-160. DOI: 10.1590/S0101-74382008000100008. [ Links ]

23 GOVINDAN K & JEPSEN MB. 2016. ELECTRE: A comprehensive literature review on methodologies and applications. European Journal of Operational Research, 250: 1-29. [ Links ]

24 GRECO S & SŁOWINSKI R. 2016. Multiple Criteria Hierarchy Process for ELECTRE Tri Methods. European Journal of Operational Research, 252: 191-203. [ Links ]

25 GRECO S, MOUSSEAU V & SŁOWINSKI R. 2008. Ordinal regression revisited: Multiple criteria ranking using a set of additive value functions. European Journal of Operational Research, 191: 416-436. [ Links ]

26 HAN K & SHIN J. 2014. A systematic way of identifying and forecasting technological reverse salients using QFD, bibliometrics, and trend impact analysis: A carbon nanotube biosensor case. Technovation, 34: 559-570. [ Links ]

27 JUNIOR FRL & CARPINETTI LCR. 2015. Comparação entre os métodos Fuzzy TOPSIS e Fuzzy AHP no apoio à tomada de decisão para seleção de fornecedores. Gestão & Produção, 22(1): 17-34. [ Links ]

28 KADZINSKI M & TERVONEN T. 2013. Robust multicriteria ranking with additive value models and holistic pair-wise preference statements. European Journal of Operational Research, 228(1): 169- 180. [ Links ]

29 KADZINSKI M & CIOMEK K. 2016. Integrated framework for preference modelling and robustness analysis for outranking-based multiple criteria sorting with ELECTRE and PROMETHEE. Information Sciences, 352-353: 167-187. [ Links ]

30 KADZINSKI M, GRECO S & SŁOWINSKI R. 2014. Robust Ordinal Regression for Dominance-based Rough Set. Approach to multiple criteria sorting. Information Sciences, 283: 211-228. [ Links ]

31 KADZINSKI M, GRECO S & SŁOWINSKI R. 2012. Extreme ranking analysis in robust ordinal regression Original Research Article. Omega, 40(4): 488-501. [ Links ]

32 KLUCZEK A & GLADYSZ B. 2015. AHP/TOPSIS-based approach to the generation of environmental improvement options for painting process - Results from an industrial case study. Journal of Cleaner Production, 101: 360-367. [ Links ]

33 KUO T. 2017. A modified TOPSIS with a different ranking index. European Journal of Operational Research, 260: 152-160. [ Links ]

34 LEONETI AB. 2016. Considerations regarding the choice of ranking multiple criteria decision making methods. Pesquisa Operacional, 36(2): 259-277. DOI: 10.1590/0101-7438.2016.036.02.0259 [ Links ]

35 LIMA-JUNIOR FR & CARPINETTI LCR. 2016. Combining SCOR@ model and fuzzy TOPSIS for supplier evaluation. Int. J. Production Economics, 174: 128-141. [ Links ]

36 LONGARAY AA ET AL. 2015. Caracterização da Produção Científica Brasileira sobre a Aplicação de Métodos Multicritério de Apoio à Decisão: uma Análise das Publicações entre 2004-2013. XXXV Encontro Nacional de Engenharia de Produção. ENEGEP. Fortaleza, Ceará, Brasil. [ Links ]

37 LOPES YG & DE ALMEIDA AT. 2013. Multicriteria Decision Model For Selecting A Portfolio Of Oil And Gas Exploration Projects. Pesquisa Operacional, 33(3): 417-441. [ Links ]

38 MARTINS EF, LIMA GBA & GOMES HC. 2015. Análise Estratégica Multicritério Sócio-Econômico-Ambiental como Ferramenta de Apoio à Decisão em uma Empresa de Distribuição de Energia. Revista Eletrônica Sistemas & Gestão, 10(1): 96-106. [ Links ]

39 MISHRA M & CHATTERJEE S. 2017. Application of Analytical Hierarchy Process (AHP) algorithm to income insecurity susceptibility mapping e A study in the district of Purulia, India. Socio-Economic Planning Sciences, 1-19. [ Links ]

40 O’KANE C. 2016. Technology transfer executives’ backwards integration: An examination of interactions between university technology transfer executives and principal investigators. Technovation. DOI:10.1016/j.technovation.2016.08.001 [ Links ]

41 PEREIRA FC, VEROCAI HD, CORDEIRO VR, GOMES CFS & COSTA HG. 2015. Bibliometric Analysis of Information Systems Related to Innovation. Procedia Computer Science, 55: 298-307. [ Links ]

42 PUJADAS P, PARDO-BOSCH F, AGUADO-RENTER A & AGUADO A. 2017. MIVES multi-criteria approach for the evaluation, prioritization, and selection of public investment projects. A case study in the city of Barcelona. Land Use Policy, 64: 29-37. DOI: 10.1016/j.landusepol.2017.02.014 [ Links ]

43 R FOUNDATION. 2002. What is R?. Accessed in September the 25th, 2016 from Accessed in September the 25th, 2016 from ]

44 RITTER DOS SANTOS ME & TORKOMIAN ALV. 2013. Technology transfer and innovation: The role of the Brazilian TTOs. International Journal of Technology Management & Sustainable Development, 12(1): 89-111. [ Links ]

45 ROSZKOWSKA E & WACHOWICZ T. 2015. Application of fuzzy TOPSIS to scoring the negotiation offers in ill-structured negotiation problems. European Journal of Operational Research, 242(3): 920-932. [ Links ]

46 ROUBENS M, RUSINOWSKA A & DE SWART H. 2006. Using MACBETH to determine utilities of governments to parties in coalition formation. European Journal of Operational Research, 172(2): 588-603. [ Links ]

47 SARRAZIN R & DE SMET Y. 2015. Applying multicriteria decision analysis to design safe road projects. European Journal of Transport and Infrastructure Research, 15(4): 613-634. [ Links ]

48 SAHIN R. 2016. Fuzzy multicriteria decision making method based on the improved accuracy function for interval-valued intuitionistic fuzzy sets. Soft Computing, 20(7): 2557-2563. DOI 10.1007/s00500-015-1657-x [ Links ]

49 SPAK MDS ET AL. 2012. Produção científica brasileira em métodos multicritérios de tomada de decisão: cienciometria de publicações do ENEGEP entre os anos de 2007 a 2011. XXXII Encontro Nacional de Engenharia de Produção. Rio Grande do Sul. [ Links ]

50 WALCZAK D & RUTKOWSKA A. 2017. Project rankings for participatory budget based on the fuzzy TOPSIS method. European Journal of Operational Research, 260: 706-714. [ Links ]

51 WANG P, ZHU Z & WANG Y. 2016. A novel hybrid MCDM model combining the SAW, TOPSIS and GRA methods based on experimental design. Information Sciences, 345: 27-45. [ Links ]

52 YUNNA W, JINYING Z, JIANPING Y, SHUAI G & HAOBO Z. 2016. Study of decision framework of offshore wind power station site selection based on ELECTRE-III under intuitionistic fuzzy environment - A case of China Energy. Conversion and Management, 113: 66-81. [ Links ]

53 ZAKA J & KRUSZYNSKIB MIROSLAW. 2015. Application of AHP and ELECTRE III/IV methods to multiple level, multiple criteria evaluation of urban transportation projects. Transportation Research Procedia, 10: 820-830. [ Links ]

54 ZHANG W, JU1 Y & LIU X. 2017. Interval-valued intuitionistic fuzzy programming technique for multicriteria group decision making based on Shapley values and incomplete preference information. Soft Computing, 21(19): 5787-5804. DOI 10.1007/s00500-016-2157-3 [ Links ]

Received: September 01, 2017; Accepted: June 29, 2018

*Corresponding author.

Creative Commons License This is an open-access article distributed under the terms of the Creative Commons Attribution License