THE RANK REVERSAL PROBLEM IN MULTI-CRITERIA DECISION MAKING: A LITERATURE REVIEW

ABSTRACT

Despite the importance of multicriteria decision-making (MCDM) techniques for constructing effective decision models, there are many criticisms due to the occurrence of a problem called rank reversal. Nevertheless, there is a lack of a systematic literature review on this important subject which involves different methods. This study reviews the pertinent literature on rank reversal, based on 130 related articles published from 1980 to 2015 in international journals, which were gathered and analyzed according to the following perspectives: multicriteria technique, year and journal in which the papers were published, co-authorship network, rank reversal types, and research goal. Thus our survey provides recommendations for future research, besides useful information and knowledge regarding rank reversal in the MCDM field.

Keywords:
rank reversal; ranking irregularities; multi-criteria decision making; decision theory; literature review

1 INTRODUCTION

Multi-criteria decision-making (MCDM) has been one of the fastest growing areas of Operational Research during the last two decades and it is one of the most used decision methodologies in the sciences, business, government, and engineering for supporting the quality of the decision-making process (^{Wang & Triantaphyllou, 2008}; ^{Maleki & Zahir, 2013}). Despite widespread use of these methodologies, the main methods, such as AHP, TOPSIS, ELECTRE, PROMETHEE and combinations thereof, have been criticized due to the occurrence of a problem called rank reversal (RRP). RR refers to a change in the ordering among alternatives previously defined, after the addition or removal of an alternative from the group previously ordered (^{Lootsma, 1993}; ^{Buede & Maxwell, 1995}; ^{Saaty & Sagir, 2009}; ^{Wang & Luo, 2009}).

The first discussion on rank reversal in MCDM methods dates back to a series of articles published in the 80s by ^{Belton & Gear (1983}, ¹⁹⁸⁵) and ^{Saaty & Vargas (1984a}, ^b) with respect to the Analytic Hierarchy Process (AHP) method (^{Saaty, 1980}). Initially, ^{Belton & Gear (1983}) proposed a new form of normalization called BG (Belton & Gear) in which the priorities of the alternatives should be measured by the maximum value instead of their sum. This seminal paper modified the AHP method, thus preventing the occurrence of reversals and demonstrated RRP for the first time in the literature.

However, ^{Saaty & Vargas (1984a}) responded to Belton & Gear by pointing out that the method that the latter proposed also suffered from ranking reversal, and discussed the main inconsistencies in ^{Saaty & Vargas (1984b}). Nevertheless, ^{Belton & Gear (1985}) argued that there was a misinterpretation of their suggested procedure, stating that their approach would not result in any reversal since the weights of the criteria were amended accordingly.

Since this initial discussion, the studies and papers on RRP have grown considerably, and this expansion is related both to other aspects that generate rank reversal in AHP, as for issues regarding transitivity property (^{Triantaphyllou, 2001}) and a non-discriminating criterion (^{Finan & Hurley, 2002}; ^{Liberatore & Nydick, 2004}; ^{Saaty, 2006}; ^{Lin et al., 2008}; ^{Jung et al., 2009}; ^{Wijnmalen & Wedley, 2009b}; ^{Jan et al., 2011}), even as another multicriteria decision-making methods, such as TOPSIS (Technique for Order of Preferences by Similarity to Ideal Solution) (^{Ren et al., 2007}; ^{Kong, 2011}; ^{García-Cascales & Lamata, 2012}), ELECTRE (Elimination and Choice Expressing Reality) (^{Wang & Triantaphyllou, 2008}; ^{Figueira & Roy, 2009}; ^{Figueira et al., 2013}) and PROMETHEE (Preference Ranking Organization Method of Enrichment Evaluations) (^{De Keyser & Peeters, 1996}; ^{Verly & De Smet, 2013}).

Despite the great interest of researchers in RRP, given its importance for addressing the reliability of MCDM methods, there is still a paucity in the literature regarding published studies on this subject. During the development of our research and analysis, we have found only the work of ^{Maleki & Zahir (2013}) in the literature on rank reversal and MCDM, where the authors evaluated 61 papers on AHP method from 18 international journals. In this study, we extend their paper by analyzing 130 articles on different multicriteria methods from 37 journals, which were examined from five aspects: the multicriteria technique, year and journal in which the papers were published, the co-authorship network, rank reversal situations, and the research goal. Thus, the contributions of our paper are the following: (i) we present a new and broader systematic literature review (SLR) on RRP; (ii) we have carefully analyzed each paper in order to classify the research goal; (iii) we designed a co-autorship network on AHP and rank reversal (RR); and (iv) we summarize the types of rank reversal commonly evaluated in the literature. These findings are important because they can be useful to help prioritize and provide insights for future research on RRP and MCDM.

The remainder of this paper is organized as follows: Section 2 presents the rank reversal problem, related axioms and the assumptions of each MCDM. Section 3 presents the research methodology, i.e. it describes the method for searching for, selecting and analyzing the articles. Section 4 provides an individual analysis of the MCDM techniques regarding the RRP, while Section 5 presents additional results from our SLR. Finally, Section 6 draws some conclusions from this research and makes final remarks.

2 MCDM AND THE RANK REVERSAL PROBLEM

The main focus of the MCDM field is to introduce procedures, methods as well as tools for solving problems and consequently support decision-makers (DM) to make better decisions. In multicriteria decision-making problems, the overall performance of the alternatives are evaluated with respect to several and conflicting criteria, and the objectives are combined based on the DM’s preferences (^{De Almeida et al. 2015}).

The DM’s preferences are elicited by means of preference modeling, considering basic concepts related to preference relations. The main preference relations presented in the literature are: Indifference (I), Strict Preference (P), Weak Preference (Q) and Incomparability (J). See ^{De Almeida et al. (2015}), ^{French (1986}) and ^{Peterson (2009}) for a detailed review. A classical problem in multicriteria decision making is aggregating preference relations on each criterion to obtain a global preference relation on the set of the alternatives considered. Thus, preference relations can be combined, as proposed by ^{De Almeida et al. (2015}), the following being the ones that are the most used: Structure (P,I), Structure (P,Q,I) and Structure (P,Q,I,J).

The Structure (P,I) is complete and transitive and has a symmetric preference relation (I) and an asymmetric preference relation (P). This structure is the basis for the unique criterion of synthesis methods, which are based on a process of an analytical combination of all criteria in order to produce a global evaluation or score for all alternatives. Examples include AHP, SMARTS, MACBETH and TOPSIS. With this structure it is possible to obtain a complete pre-order or a complete order for the alternatives (^{De Almeida et al. 2015}).

According to ^{De Almeida et al. (2015}), outranking methods do not use a unique criterion of synthesis, so many of these methods produce the final recommendation with no scores for alternatives. These methods use the preference structure (P,Q,I,J), including the incomparability relation, and they produce a partial pre-order. The main methods in this group are the ELECTRE and PROMETHEE methods (^{Roy, 1996}; ^{Vincke, 1992}; ^{Belton & Stewart 2002}). This structure is relevant for situations in which the DM is not able to give full preference information.

Despite the great growth and evolution of the MCDM field, these methods are known to present RRP, for which the literature has given different interpretations. Saaty, for example, states “Assume that an individual has expressed preference among a set of alternatives, and that as a result, he or she has developed a ranking for them. Can and should that individual’s preferences and the resulting rankings order of the alternatives be affected if alternatives are added to or deleted from the set?” Researchers such as ^{Saaty & Vargas (1984b}), ^{Saaty (1987a}), ^{Forman (1990}) and ^{Millet & Saaty (2000}) also argued for the legitimacy of this problem. For some authors such as ^{Saaty (1994b}) and ^{Millet & Saaty (2000}) the reversals do not occur often and thus having rank reversals in certain occasions and of certain types may not be indicative of faulty decision making.

On the other hand, ^{Salem & Awasthi (2018}) mentioned that this problem could drive some DMs away from using methods known to have rank reversal and has substantial logical implications for the methodology that is used to make decisions. In fact, ^{Anbaroglu, Heydecker & Cheng (2014}), for example, used the Weighted Product Model (WPM) instead of the classical AHP because it does not suffer from any kind of RR. In addition, ^{Anbaroglu, Heydecker & Cheng (2014}) stated that RRP is a serious limitation of the MCDM field, which could lead researchers to misunderstand the difference between the alternatives examined.

In the literature, rank reversal is the change in ranking of the alternatives as a consequence, for example, of the addition or deletion of an alternative. This means that a DM’s preference ordering between two alternatives has changed when an alternative is added or removed and this clearly contradicts the principle of the independence of irrelevant alternatives. In addition, ^{Finan & Hurley (2002}), ^{Lin et al. (2008}), ^{Jung et al. (2009}) and ^{Jan et al. (2011}), for example, associated RRP with a non-discriminating criterion. In this case, as the DM is indifferent among the alternatives when they are compared on that criterion and the non-discriminating criteria do not differentiate between the alternatives, it is presumably safe to eliminate them from further consideration (^{Wijnmalen & Wedley, 2009b}).

Finally, although there has been a long and unresolved debate on RRP among the practitioners in the MCDM literature, is possible to characterize different typologies of rank reversals from the literature, as described in the following:

1. Type #1: the final rank order of the alternatives changes if an irrelevant alternative is added to (or removed from) the problem. See, for example, ^{Buede & Maxwell (1995}), ^{Zanakis et al. (1998}), ^{Wang & Luo (2009}), ^{García-Cascales & Lamata (2012}), ^{Verly & De Smet (2013}) and ^{Cinelli et al. (2014}).

2. Type #2: the indication of the best alternative changes if a non-optimal alternative is replaced by another worse one. See, for example, ^{Triantaphyllou (2001}) and ^{Wang & Triantaphyllou (2008}).

3. Type #3: the transitivity property is violated if an irrelevant alternative is added to (or removed from) the problem. See, for example, ^{Triantaphyllou (2001}) and ^{Wang & Triantaphyllou (2008}).

4. Type #4: the transitivity property is violated if the initial decision-problem is decomposed into sub-problems, i. e., for the same decision problem and when the same MCDM method is used, the rankings of the smaller problems are in conflict with the overall ranking of the alternatives. See, for example, ^{Triantaphyllou (2001}) and ^{Wang & Triantaphyllou (2008}).

5. Type #5: the final rank order of the alternatives changes if a non-discriminating criterion is removed from the problem. See, for example, ^{Finan & Hurley (2002}), ^{Lin et al. (2008}), ^{Jung et al. (2009}) and ^{Jan et al. (2011}) and ^{Verly & De Smet (2013}).

The MCDM methods analyzed in this paper were evaluated based on different types of RR, where Type #1 was the most widely used in the literature. In addition, many methods have not been tested for all the types, such as the TOPSIS method with respect to Type #3, Type #4, and Type #5.

3 RESEARCH METHODOLOGY

The research methodology of this survey is depicted in Figure 1. The first goal of this research was to gather articles on RRP in the specialized literature of the multicriteria field.

Figure 1
Research methodology.

Thus, we defined the following rules to define our sample of papers:

• The keywords for the initial search were “rank reversal” or “rank irregularities” or “rank preservation” or “rank consistency” and “multi-criteria”.

• The search for articles was made on academic databases including Science Direct, Emerald, Springer-Link Journals, IEEE Xplore, Academic Search Premier, Scielo and World Scientific Net, and the entire number of articles in the literature published was examined.

• To achieve the highest level of relevance, only international journal articles were selected. Thus, conference articles, master and doctoral dissertations, textbooks, unpublished ar-

• ticles, and notes were not included in this review. Similarly, only articles published in English were considered.

Based on these considerations, 130 articles were selected. These articles refer to AHP, TOPSIS, PROMETHEE, ELECTRE, and others multicriteria methods. Each article was carefully reviewed and selected strictly according to the scope of this research as set out in Figure 1.

^Appendix 6 presents a first classification of each paper organized in accordance with the multicriteria method, the year of publication, and the main scientific contribution made by each article. Details about this will be presented in the next section, and so too will the evolution of rank reversal studies for the main multicriteria decision-making methods.

4 INDIVIDUAL APPROACHES

This section presents an analysis of the main papers surveyed, namely the MCDM method Analytic Hierarchy Process (AHP), Technique for Order of Preferences by Similarity to Ideal Solution (TOPSIS), Preference Ranking Organization METHod of Enrichment Evaluations (PROMETHEE), Elimination and Choice Expressing Reality (ELECTRE), and hybrid versions.

4.1 AHP

A total of 99 of 130 articles (76.2 %) analyzed RRP in AHP. After an initial discussion between by ^{Belton & Gear (1983}, ¹⁹⁸⁵) and ^{Saaty & Vargas (1984a}, ^b) as previously presented, ^{Schoner & Wedley (1989}) presented the new version of the AHP method called “Referenced AHP”.

To avoid RR, this approach requires criteria weights to be modified whenever an alternative is added or removed, i.e., the authors considered that criteria weights are dependent upon the alternatives in the set chosen. Thereafter, Foman (1990) argued that the problem was related to non-linearity, and ^{Dyer (1990b}) proposed that AHP judgments should be rescaled in order to ensure “that the criterion weights and the scores of the alternatives on the criteria are normalized with respect to the same range of alternative values”. Harker & Vargas (1990) and ^{Saaty (1990}) replied to Dyer’s paper, and ^{Dyer (1990a}) counter-argued that these authors had misunderstood this process. Finally, ^{Schoner et al. (1992}) proposed that the problem may be easily solved by normalizing local priorities, since the change in the set of alternatives (addition or removal) requires a new normalization of local priorities.

^{Schenkerman (1994}) stated that the AHP method presumes that the DM’s preference structure can be adequately represented by an additive value function. For ^{Vargas (1994}), the RR is an acceptable characteristic of human behavior, and one that cannot be justified simply by saying that some of the basic assumptions of rationality have been violated. This view of the problem was also advocated by other authors, such as ^{Farkas et al. (1999}) and ^{Farkas (2007}).

The AHP has two measurement modes, absolute and relative. In the relative measurement, the local priorities of the alterrnatives are normalized so that they add up to one, whereas in absolute measurement no such normalization is applied to them (^{Saaty, 1994}). In the relative measurement mode, any alternative that is not dominated by some convex combination of other alternatives may become the best one as a result of introducing or deleting other alternatives (^{Salo & Hämäläinen, 1997}). ^{Salo & Hämäläinen (1997}) also pointed out that the fundamental mathematical reason for the occurrence of rank reversal in the relative measurement mode is that the local priorities in the lower level of the hierarchy are normalized so that they add up to 1. When new alternatives are added or removed, the local priorities associated with other alternatives inevitably change and, as a result, the final ranking of alternatives can also be changed (^{Salo & Hämäläinen, 1997}). ^{Belton & Gear (1997}) agree with this explanation.

^{Millet & Saaty (2000}) and ^{Saaty (1994b}) described a structural change in AHP to deal with both situations of measurement modes. These authors proposed two types of synthesis procedures. The first one, which is the one most commonly used, was called the distributive mode. This may cause rank reversal, since there is dependence among the alternatives and the ratings of the alternatives are normalized such that the sum of all of them is 1. In contrast, the second procedure was called the ideal mode. It is able to preserve the ranks, because the score of each alternative is divided only by the score of the best alternative under each criterion.

From these papers, several authors pointed to the classical synthesis procedure as the main cause of RR in AHP (^{Troutt, 1988}; ^{Troutt & Tadisina, 1990}; ^{Barzilai & Golany, 1990}; ^{Zahir, 2009}; ^{Ramanathan & Ramanathan, 2011}). Therefore, multiplicative AHP (MAHP) was designed to solve the RRP (^{Lootsma, 1993}; ^{Barzilai & Lootsma, 1997}; ^{Lootsma & Barzilai, 1997}; ^{Stam & Silva, 2003}). MHAP is a variant of AHP proposed by Saaty with a one-parametric class of geometric scales to quantify human comparative judgement and with a multiplicative structure given by a logarithmic regression to calculate the impact scores of the alternatives at the first evaluation level and a geometric-mean aggregation rule to calculate the final scores at the second level (^{Lootsma, 1993}). According to these authors, MAHP is able to avoid the RRP because the rank order of the impact scores and final scores is scale-independent.

Vargas (1997) was emphatic in rejecting this argument and showed the AHP ideal mode afore-mentioned solves the problems raised by these authors. The same author, explained the different situations where the reversals may occur. See ^{Saaty & Sagir (2009}) for further details.

^{Triantaphyllou (2001}) demonstrated the occurrence of rank reversal in AHP regarding the non-compliance with the property of transitivity and suggested using MAHP to avoid it. ^{Wijnmalen & Wedley (2009a}) explained that the RRP is due to the assumption that the criteria weights should remain unchanged when dealing with sub-problems, i.e, the problem is related to the independence axiom of the AHP method and the commensurability, and not only with regard to the addition/removal of alternatives.

Non-discriminating criterion, also called “Wash criteria” (WC), first pointed out by ^{Finan & Hurley (2002}), became another important issue described in the literature on RRP and AHP, where the DM is indiferent among the alternatives when they are compared with these criteria. ^{Liberatore & Nydick (2004}) recognized the problem which had been pointed to by ^{Finan & Hurley (2002}) and claimed that the problem can be easily solved by reevaluating the relative weights of the criteria after removing WC. This solution also received support from ^{Saaty & Vargas (2006}) and ^{Wijnmalen & Wedley (2009b}). However, ^{Lin et al. (2008}) showed that the proposal of ^{Finan & Hurley (2002}) has shortcomings and also showed how to solve it.

In the latest study on WC, ^{Jan et al. (2011}) showed that the RRP does not occur when a WC is excluded and if a proportional adjustment to revise the upper level relative weights is applied, as suggested by ^{Liberatore & Nydick (2004}) and ^{Saaty & Vargas (2006}). They also reported situations of RR in accordance with ^{Finan & Hurley (2002}) and ^{Lin et al. (2008}) and pointed out that every criterion may also be a WC for the invariant phenomenon proposed by ^{Jung et al. (2009}).

In summary, three versions of AHP were reported in the literature on RRP: AHP, AHP ideal mode and MAHP. There is consensus in our literature review that AHP suffers from RR. As previously described, several authors are in favor of using the AHP ideal mode and MAHP to avoid RR. ^{Stam and Silva (2003}) provided a detailed comparison between MAHP and the AHP, they also discussed several properties of the MAHP, besides showing how the MAHP offers a more flexible preference modeling framework.

Finally, the study of ^{Maleki & Zahir (2013}) reported a literature review on RRP and examined 61 articles related to AHP from 18 international journals. In this paper we have updated the SLR conducted by Maleki by analyzing 130 paper on RR and AHP. Examples of papers we have considered different from Maleki are: ^{Totsenko (2006}), ^{Saaty & Vargas (2009}), ^{Troutt, Tadisina & Pendharkar (2009}), ^{Jan et al. (2011}) and ^{Kulakowski (2015}).

4.2 TOPSIS

Only three of the 130 articles (2.30%) dealt with the RRP in TOPSIS. The first study analyzed was that by ^{Ren et al. (2007}), in which a new method called M-TOPSIS was proposed to avoid the RRP. In short, the M-TOPSIS uses the first four steps of the classical TOPSIS as proposed by ^{Hwang & Yoon (1981}), but a new index called relative proximity [R _i ] is calculated for each alternative. In this step, the authors calculate the distances of each alternative in relation to both the ideal positive (D ⁺ − x axis) and negative (D ⁻ − y axis) solution for calculating min($D_i^{+}$) and max($D_i^{-}$) as the optimized ideal reference point. Finally, the final ranking of the alternatives is given by R _i in ascending order. However if there are two different alternatives a and b, where R _a = R _b , then R _i = $D_i^{+}$ − min($D_i^{+}$) must be calculated for both alternatives. In this case, the best alternative is the one with the lowest value for R _i .

In the article of ^{Kong (2011}), the vector normalization procedure was mentioned as the main reason for the RRP in TOPSIS, since the independence of the alternatives may be affected, since alternatives can be related to each other after the normalization. Furthermore, the optimal solution will also change, since the distances of each alternative to the ideal solutions will always change if there is a change in any ideal solution.

To solve this problem, the author stated that the normalization procedure must comply with the independence of the alternatives while maintaining the ideal solution constant. From this hypothesis, the proposed solution determines both the attribute value that is the most satisfactory and the most unsatisfactory. Therefore, the first one is normalized to be one, while the last is normalized to be zero, thereby guaranteeing that the ideal points will never change when alternatives are added or removed.

Similarly to ^{Kong (2011}), ^{García-Cascales & Lamata (2012}) also indicated that the irregularities in TOPSIS occur due to the procedure of normalization However, in addition, they argued that is also necessary to change the calculation of the positive and negative ideal solutions (PIS and NIS). Thus, the authors proposed a new normalization procedure in absolute terms, not relative, as in the classical TOPSIS method, besides adding two fictitious alternatives with the best and the worst values of each criterion. From this solution, the authors claimed that they had proved that their method avoids the rank reversal.

Finally, ^{García-Cascales & Lamata (2012}) demonstrated that both relative and absolute scales suffer from RR. However, the latter generates lower RR rates because there is better control of the independence between alternatives. On the other hand, ^{Aires & Ferreira (2016}) pointed out that the linear scale transformation (Max) presented the lowest rates of RR when compared to the other normalization processes, such as linear scale transformation (Max-Min), linear scale transformation (Sum) and vector normalization. According to these authors, the “Max method” demonstrated better results because it increased the range of values of the criteria.

4.3 PROMETHEE

Only four of the 130 articles (3.1%) dealt with the RRP in PROMETHEE. Studies of rank reversal in outranking methods are still scarce in the literature. The first authors to address this issue for the PROMETHEE method were ^{De Keyser & Peeters (1996}), but they simply pointed out that this problem existed. The second, which was published 17 years later, was presented by ^{Verly & De Smet (2013}) to empirically test the probability of RR in the PROMETHE I and II methods based on artificial data sets. These tests were conducted with respect both to the addition or removal of alternatives, and to a non-discriminating criterion, as reported by ^{Finan & Hurley (2002}) to AHP and already aforementioned. The results showed that both PROMETHEE suffer from rank reversal.

Additionally, ^{Verly & De Smet (2013}) also highlighted that RRP is directly related to situations where the flow differences are relatively small. Moreover, they demonstrated that RR did not occur in this method when a non-discriminating criterion was tested, and the procedure based on distillation was more stable during the experiments than PROMETHEE I was.

^{Brans & De Smet (2016}) responded to ^{Verly & De Smet (2013}) a view to showing that the PROMETHEE’s rankings are not influenced by the presence or elimination of a non-discriminating criterion, while they may be affected by copies of alternatives. They also pointed out that the ranking between two alternatives a and b in PROMETHEE II remains the same with the removal of a third alternative if | ϕ ⁺(a) - ϕ ⁺(b) |> $\frac{1}{n-1}$ and | ϕ ^-(a) - ϕ ^-(b) |> $\frac{1}{n-1}$. On the other hand, for the PROMETREE I, they demonstrated that the ranking between a and b remains the same if | ϕ ⁺(a) - ϕ ⁺(b) |> $\frac{1}{n-1}$ and | ϕ ^-(a) - ϕ ^-(b) |> $\frac{1}{n-1}$.

4.4 ELECTRE

Only three of the 130 articles (2.3%) specifically discussed the RRP in the ELECTRE family. Initially, ^{Zanakis et al. (1998}) presented a broader analysis regarding different methods, as described in section 3.5. However, the first study made a detailed analysis of ELECTRE II and III methods with respect to RR was published only in 2008 by ^{Wang & Triantaphyllou (2008}) and evaluated them against three criteria: (i) the indication of the best alternative when an alternative “not optimal” is replaced by another worse alternative and should remain the same, while main-taining the relative importance of each criterion; (ii) the transitivity axiom cannot be violated; (iii) the original ranking of the problem and the ranking combination of the decomposed problem should be identical.

Although outranking relations methods may not be transitive, as stated by ^{Figueira & Roy (2009}), the authors analyzed 10,000 simulated decision problems for each method. The results showed that RR occurred for the three criteria examined and the reversal rates generally increased when the number of alternatives was increased. In addition, to strengthen their results, the authors also studied ten real cases randomly selected from the published literature, where 60% of the cases failed in criterion 1. 90% failed in the criterion 2, and the only case submitted to criterion 3 also failed.

Later, ^{Figueira & Roy (2009}) explained and justified the results presented in ^{Wang & Triantaphyllou (2008}). The authors argued that the results obtained with respect to the first criterion do not invalidate the method, they just show its limitations. Moreover, they argued that it well known that methods using outranking relations (not only the ELECTRE methods) do not need to satisfy the transitivity property. According to these authors, “this aspect represents a weakness only if we impose a priori that preferences should be transitive”.

Finally, the last paper analyzed, proposed by ^{Figueira et al. (2013}), presented an overview of ELECTRE methods and their extensions. It highlighted the main features of these methods, recent applications, new methodological developments, axiomatic analysis, software tools and some weaknesses reported in the literature, including the rank reversal analysis. In this paper, as in ^{Figueira & Roy (2009}), the authors relativize the RRP and advocated that RR may occur and, consequently, the property of independence with regard to irrelevant actions can be violated when dealing with outranking relations.

4.5 OTHERS

This section presents 21 (16.2%) articles regarding the RRP when two or more MCDM methods are involved in the analysis and experiments. We also consider in this section MCDM methods which were not individually analyzed in previous sections. Initially, we found five papers (^{Gomes, 1990}; ^{Olson et al. 1995}; ^{Salo & Punkka, 2011}; ^{Maleki & Saadat, 2013}; ^{Soltanifar & Shahghobadi, 2014}) that analyzed the RRP considering individual methods. The first one, proposed by ^{Gomes (1990}), showed that the TODIM (an acronym in Portuguese for Interactive and Multi-Criteria Decision-Making) suffers from RRP, but it also presents a solution for the problem by using the normalization procedure of ^{Belton & Gear (1983}).

The second paper (^{Olson et al., 1995}) was a comparative analysis between the REMBRANDT (Ratio Estimation in Magnitudes or Deci-Bells to Rate Alternatives which are Non-Dominated) and the classical AHP method, where the authors demonstrated that the former is more consistent than AHP in terms of rank reversal. This result is reinforced in the ^{Maleki & Saadat (2013}) article.

The fourth study (^{Soltanifar & Shahghobadi, 2014}) demonstrated for the first time the RRP in DEA (Data Envelopment Analysis), by showing the occurrence or non-occurrence of rank reversal when a copy of a DMU is added to the problem. ^{Salo & Punkka (2011}) is another study that deals with the RRP for the DEA.

Three studies analyzed extensions of AHP and TOPSIS, ^{Lima Junior et al. (2014}), ^{Bulut, Duru & Koçak (2015}) and ^{Iç (2014}). The former work compared the methods Fuzzy-AHP and Fuzzy-TOPSIS in a supplier selection context, in which the authors showed that the Fuzzy-TOPSIS is more consistent than Fuzzy-AHP, since it preserved the original ranking considering the following evaluation situations: (i) the addition of a new alternative, and (ii) the addition of a new criterion. On the other hand, Fuzzy-AHP failed in both evaluation situations.

The goal of ^{Bulut, Duru & Koçak (2015}) was to improve the applicability of the fuzzy-AHP method by using rotational priority investigation (RPI) to eliminate inconsistent results. ^{Iç (2014}) conducted different experiments with a new method called DoE-TOPSIS, which proved the original ranking. ^{Bottero & Ferretti (2011}) analyzed the contribution of the ANP (Analytic Network Process) technique in sustainability assessment of undesirable facilities. The authors suggested the combined use of ANP and BOCR (Benefits, Opportunities, Costs and Risks) model. To demonstrate the improvements of the approach proposed, they conducted a sensitivity analysis to estimate cases of RR. ^{Bairagi et al. (2015}) proposed a new model called “Technique of Precise Order Preference (TPOP)” to overcome the RRP based by using an advanced version of the entropy weighting method.

Unlike previous studies, ^{Buede & Maxwell (1995}) evaluated different methods with a view to examining the frequency and magnitude of rank disagreements using Multi-Attribute Value Theory (MAVT) as the reference. MAVT, according to the authors, is a good reference because it has strong normative foundations and is immune to rank reversal. The first statistic used by the authors to campare the methods was called fraction of ranking disagreements, where the best option given by MAVT was used to count the number of ranking disagreements in 32,000 iterations for each technique. The second statistic computed was the magnitude of the difference between the option chosen by MAVT and the option chosen by each of the other techniques when a ranking disagreement occurred. The methods AHP, TOPSIS, Policy Goal Percentaging Analysis (^{Nagel, 1989}) and the Fuzzy Algorithm (^{Yager, 1978}) were analyzed when new alternatives were added or existing options were deleted from the problem.

The experiments demonstrated of ranking disgreements cases between AHP and MAVT at small percentage levels, and rare cases of significant differences between MAVT and AHP in the fractional value difference. On the other hand, the other three techniques analyzed (Percentaging, TOPSIS and the Fuzzy Algorithm) reached worse results in terms of both the frequency and the magnitude of the value of the ranking disagreements.

In a similar study, ^{Zanakis et al. (1998}) tested the RRP in eight methods: ELECTRE, TOPSIS, MEW (Multiplicative Exponential Weighting), SAW (Simple Additive Weighting), and four different AHP versions. The results concerning the addition of a new alternative in the problem showed that MEW and SAW methods have not produced any rank reversals. This was followed by TOPSIS, the four AHP versions and, finally, ELECTRE. Next, the authors analyzed the effects caused by the number of alternatives, criteria and the distribution of weights. In general, the analysis of the results showed that: (i) more rank reversals occurred in problems with more alternatives; (ii) the number of rank reversals was influenced less by the number of criteria than by the number of alternatives, (iii) more rank reversals were observed under constant weights, and fewer under uniformly distributed weights. In this paper three distributions for weights were simulated: equal weights, uniform distribution, and U-shaped distribution.

However, given the controversy in the MCDM field about comparing rankings from different methods, the results of ^{Buede & Maxwell (1995}), ^{Zanakis et al. (1998}) and others should be interpreted with caution. In more recent papers, ^{Wang & Luo (2009}) and ^{Cinelli et al. (2014}) studied the RRP in different methods to illustrate the problem. ^{Wang & Luo (2009}) explained (and exemplified) that the RRP occurs not only in the AHP but also in many other decision-making methods, such as Borda-Kendall (^{Kendall, 1962}), SAW, TOPSIS, and in DEA. ^{Cinelli et al. (2014}) analyzed the performance of five MCDA methods (Multi-Attribute Utility Theory - MAUT, AHP, PROMETHEE, ELECTRE, and Dominance-based Rough Set Approach - DRSA) with respect to ten criteria in which sustainability assessment tools should be used, such as life cycle perspective, thresholds and uncertainty management, software support and ease of use. Moreover, the authors limited themselves to pointing out the RRP, and the similarities and the differences between the methods.

The last two articles we reviewed for this section were ^{Dede et al. (2015}) and ^{Tomashevskii (2015}). By using the concepts from ^{Saaty & Vargas (1984}), the former analyzed the impact of uncertainty introduced when different experts complete pairwise comparison matrices in the context of fuzzy multicriteria decision making. The RR measure was used in this study as a way of measuring the trustworthiness of the results given that the alternatives are compared pairwise by different DMs.

Finally, the research of ^{Tomashevskii (2015}) was dedicated to analyzing the Eigenvector Method (EM) used in pairwise comparison matrices in decision-making processes. The author infers that the errors that occurred related to rank reversal for EM were caused entirely by the inaccuracy of the measurement scale and the inconsistent judgements of experts. The criteria used to evaluate the reservals were the addition or removal of new alternatives.

5 OTHER CLASSIFICATION RESULTS

This section presents the analysis of the 130 papers according to the following attributes: (i) year of publication; (ii) distribution by journal; (iii) co-authorship network; and (iv) research goal. The results of the co-authorship network refer only to the AHP method, since this is the only method with enough articles published on RRP to conduct this kind of analysis.

5.1 Distribution by year of publication

Table 1 shows the distribution frequency by year for the papers published on RRP in scholarly journals since 1983, from when 23 papers were published to the end of 1991. After that, a total of 34 papers were published between 1992 and 1997. There was a sharp decline to a total of nine papers (6.9%) published between 1998 and 2000, and five (3.8%) between 2001 and 2003. From 2004 to 2015, the number of papers published on RRP has revived, since more than 59 papers (45.4%) of all papers were published within this period.

Table 1
Frequency of papers by year of publication.

Year	N	%
1983-1985	4	3.1
1986-1988	7	5.4
1989-1991	12	9.2
1992-1994	12	9.2
1995-1997	22	16.9
1998-2000	9	6.9
2001-2003	5	3.8
2004-2006	16	12.3
2007-2009	14	10.8
2010-2012	10	7.7
2013-2015	17	14.6
Total	130	100

5.2 Distribution of articles by journal

Table 2 provides the distribution of the articles based on the journal in which they appeared. In this review, there were 37 journals with at least one paper published on the RRP.

Table 2
Distribution of the selected articles by journal.

Journal title	No. of articles	Percentage (%)
European Journal of Operational Research	23	17.7
Journal of Multi-Criteria Decision Analysis	19	14.6
Omega	9	6.9
Management Science	9	6.9
Decision Sciences	7	5.4
Computers & Operations Research	7	5.4
Mathematical and Computer Modelling	6	4.6
Journal of the Operational Research Society	4	3.1
Decision Support Systems	3	2.3
Computers & Industrial Engineering	3	2.3
Expert Systems with Applications	2	1.5
International Journal of Multicriteria Decision Making	2	1.5
International Journal of Operational Research	2	1.5
Linear Algebra and its Applications	2	1.5
Mathematical Modelling	2	1.5
Advanced Materials Research	1	0.8
African Journal of Business Management	1	0.8
Applied Mathematics and Computation	1	0.8
Applied Mathematical Modelling	1	0.8
Applied Mathematics Research eXpress	1	0.8
Applied Soft Computing	1	0.8
Central European Journal of Operations Research	1	0.8
Cybernetics and Systems Analysis	1	0.8
Ecological Indicators	1	0.8
IEEE Transactions on Engineering	1	0.8
European Journal of Marketing	1	0.8
Facilities	1	0.8
Far East Journal of Mathematical Sciences	1	0.8
Fuzzy Sets and Systems	1	0.8
I. J. of Industrial Engineering: Applications and Practice	1	0.8
International Journal of Industrial and Systems Engineering	1	0.8
Interfaces	1	0.8
International Journal Math. Operational Research	1	0.8
International Journal of Management and Decision Making	1	0.8
International Transactions in Operational Research	1	0.8
Journal of Advanced Transportation	1	0.8
Journal of Automation and Information Sciences	1	0.8
Journal of Mathematical Psychology	1	0.8
Knowledge-Based Systems	1	0.8
Operations Research Letters	1	0.8
Quality Engineering	1	0.8
Systems Engineering	1	0.8
TOP	1	0.8
Communications in Statistics - Theory and Methods	1	0.8
Acta Polytechnica Hungarica	1	0.8
Total	130	100

Of these 45 journals, 30 contributed with only one paper, and 15 journals contributed with more than one paper. The European Journal of Operational Research (EJOR) published the largest number of articles, nomely, 23 out of the 130 articles reviewed (17.7%). There was not a significative difference in terms of published papers between the EJOR and the Journal of Multi-Criteria Decision Analysis (19). The Omega and Management Science was in the third position, regarding the number (9) and percentage (6.9%) of the total papers published.

5.3 Co-authorship Network Analysis

This section presents the co-authorship network regarding the RRP in the AHP method, given the number of papers published on this method. Scientific collaboration refers to a broad range of activities, from simple exchanges of opinion to side-by-side work in a laboratory until coauthorship of a paper. Co-authorship is one of the most tangible forms of research collaboration. A coauthorship network is a social network where the interactions can be represented by a dynamically directed network, representing the authors as nodes and the cooperative action of the first author as a link to all other authors (^{Wardil & Hauert, 2014}).

The topology of the co-authorship network of the AHP articles is shown in Figure 2 with 53 articles represented, since the single authors papers were excluded from the analysis. The nodes of the network were represented in accordance with the number of each authored articles, i.e., the higher the number of articles, the more representative and the higher is the author node. The thickness of the lines symbolized the intensity of collaboration among the authors.

Figure 2
Co-authorship network on rank reversal in AHP.

As a general comment regarding the results obtained, there is a large predominance of coauthorship that was established for a single article. In terms of productivity, Vargas, Saaty and Wedley were the authors with most articles published, and the coauthorship ratio involving Vargas & Saaty and Wedley & Schoner was the most common.

Complementarily, we can also use the centrality measure from social network theory (^{Abbasi et al., 2012}; ^{Wardil & Hauert, 2014}) to analyze the results. This measure reports useful information so as to evaluate the authors according to their performances in the social network. Based on the measure of degree centrality, sixteen authors had the highest level of coauthorship. However, when the betweenness centrality measure was analyzed, two of those authors (Saaty and Wedley) were in a good position in the network, this being the shortest path between two other authors. This measure play an important role in controlling the distribution of information in the network. Finally, considering the closeness centrality, Saaty had the minimum distance to other network nodes, representing his effectiveness, centrality and key role in the distribution and circulation of information between other nodes in the network.

5.4 Research goal and causes of RR

In this section we present the final results obtained during our SLR. First, we classify the papers studied into five clusters according to their research goal, as described in the following:

1. Survey: when the authors compiled and analyzed the relevant literature of the RRP from different academic databases.

2. Application: when the authors applied some multicriteria decision-making method and used decision theory and ranking reversal assumptions to improve the reliability of the results produced.

3. Problem Solution: when the authors designed a new procedure to solve the RRP.

4. Simulation: when the authors investigated the performance of different multicriteria procedures or methods by using a set of simulated problems.

5. Problem identification: when the authors studied the nature of the problem, and outlined its causes and conditions.

Table 3 presents the classification of all analyzed papers in this review into each cluster, where the majority of the articles studied the nature of the problem (61.54%), by outlining its causes and conditions. Moreover, disregarding the widely explored AHP method, it appears that the main gap in relation to rank reversal state of the art is the absence of a clear and convincing explanation of its cause, reinforcing that the RRP is still an open problem, despite all the research already carried out and which have been analyzed in this study.

Table 3
Classification of reviewed papers.

Cluster	%	Papers
Survey	0.8	Maleki & Zahir (2013).
Application	9.23	Hämäläinen (1990), Stewart (1992), Triantaphyllou & Mann (1995), Davies (2001), Sundarraj (2004), Wang, Parkan & Luo (2007), Troutt, Tadisina & Pendharkar (2009), Bottero & Ferretti (2011), Ishizaka & Labib (2011), Lima Junior et al. (2014), Cinelli et al. (2014), Bulut, Duru & Koçak (2015).
Problem Solution	14.62	Belton & Gear (1983), Schoner & Wedley (1989), Lootsma (1993), Schoner et al. (1993), Saaty (1994b), Olson et al. (1995a), Leung & Cao (2001), Kujawski(2005), Totsenko (2006), Ramanathan (2006), Ren et al. (2007), Jung et al. (2009), Kong (2011), García-Cascales & Lamata (2012), Maleki & Saadat (2013), Iç (2014), Pankratova & Nedashkovskay (2014), Bairagi et al. (2015) and Salabun (2015).
Simulation	13.85	Saaty & Vargas (1984a), Triantaphyllou & Mann (1989), Saaty & Vargas (1993), Saaty (1994a), Olson et al. (1995b), Paulson & Zahir (1995), Buede & Maxwell (1995), Stam & Silva (1997), Triantaphyllou & Sánchez (1997), Zanakis et al. (1998), Srdjevic (2005), Ishizaka & Lusti (2006), Wang & Triantaphyllou (2008), Wang & Luo (2009), Verly & De Smet (2013), Soltanifar & Shahghobadi (2014), Dede et al. (2015) and Yaraghi et al. (2015).
Problem Identification	61.54	Saaty & Vargas (1984b), Belton & Gear (1985), Saaty (1986), Saaty (1987a), Saaty (1987b), Saaty (1987c), Harker & Vargas (1987), Saaty & Vargas (1987), Troutt (1988), Barzilai & Golany (1990), Dyer (1990a), Dyer (1990b), Saaty (1990), Harker & Vargas (1990), Forman (1990), Troutt & Tadisina (1990), Holder (1990), Gomes (1990), Schoner et al. (1992), Forman (1993), Wedley et al. (1993), Schenkerman (1994), Vargas (1994), Triantaphyllou & Mann (1994), Lai (1995), Pérez (1995), Dodd et al. (1995), Vinod Kumar & Ganesh (1996), De Keyser & Peeters (1996), Barzilai & Lootsma (1997), Salo & Hämäläinen (1997), Belton & Gear (1997), Lootsma & Barzilai (1997), Barzilai (1997), Vargas (1997), Schoner et al. (1997), Stam (1997), Saaty (1997), Yang & Lee (1997), Van Den Honert (1998), Farkas et al. (1999), Sarkis (1999), Zahir (1999), Millet & Saaty (2000), Sinuany-Stern et al. (2000), Aguarón & Moreno-Jiménez (2000), Triantaphyllou (2001), Finan & Hurley (2002), Stam & Silva (2003), Liberatore & Nydick (2004), Farkas et al. (2004), Tavana (2004), Leskinen & Kangas (2005), Raharjo & Endah (2006), Pérez et al. (2006), Saaty (2006), Wang & Elhag (2006), Hochbaum & Levin (2006), Saaty & Vargas (2006), Farkas (2007), Bana e Costa & Vansnick (2008), Lin et al. (2008), Saaty & Sagir (2009), Wijnmalen & Wedley (2009a), Wijnmalen & Wedley (2009b), Zahir (2009), Figueira & Roy (2009), Ramanathan & Ramanathan (2011), Jan et al. (2011), Mirhedayatian & Farzipoor Saen (2011), Salo & Punkka (2011), Hou (2012), Wang & Luo (2012), Vetschera & Almeida (2012), Rodríguez et al. (2013), Figueira et al. (2013), Brans (2015), Morton (2015), Tomashevskii (2015) and Kulakowski (2015).

In the following, we analyzed the papers included in our literature review, to enumerate the causes of RR. In summary, we found that the most common reasons for RR in AHP:

1. Synthesis procedure (see, for example, ^{Troutt (1988}), ^{Barzilai & Golany (1990}), ^{Troutt & Tadisina (1990}), ^{Barzilai & Lootsma (1997}), ^{Stam (1997}), ^{Zahir (2009}) and ^{Ramanathan & Ramanathan (2011})).

2. Normalization procedures (see, for example, ^{Belton & Gear (1983}, ¹⁹⁸⁵), ^{Schoner et al. (1992}), ^{Schenkerman (1994}), ^{Salo & Hämäläinen (1997}), ^{Stam (1997}), ^{Stam & Silva (1997}), ^{Triantaphyllou (2001})).

3. Criteria weights (see, for example, ^{Dyer (1990a}), ^{Dyer (1990b}), ^{Liberatore & Nydick (2004}), ^{Saaty & Vargas (2006}), ^{Lin et al. (2008}), ^{Wijnmalen & Wedley (2009b}) and ^{Jan et al. (2011})).

4. Misuse of the method (see, for example, ^{Harker & Vargas (1987}, 1990) and ^{Saaty (1990})).

5. The uncertainty of the decision-making process (see, for example, ^{Saaty & Vargas (1987}) and ^{Van Den Honert (1998})).

6. Structural dependency between the criteria and alternatives (^{Saaty, 1986}, ^1987a, ^b, ^c).

Lastly, the faults in the normalization procedure and the way to obtain the ideal solutions (see, for example, ^{Ren et al. (2007}), ^{Kong (2011}), and ^{García-Cascales & Lamata (2012})) are the main reasons of RR in TOPSIS, while the distillation Processes (see, for example, ^{Wang & Triantaphyllou (2008})) and flow-score difference are the cited causes of RR in ELECTRE and PROMETHEE, respectively.

6 CONCLUSION

This paper has provided an extensive literature review on MCDM methodologies and rank reversals. We aimed to analyse the collected articles in accordance with the following aspects: the multicriteria technique, the issues discussed, the year and journal in which an article was published, the coauthorship network, and the research goal. A total of 130 journal articles were carefully selected and reviewed in detail. Thus, this paper provides additional valuable knowledge on current studies and makes recommendations for future studies.

Regarding the main results obtained, first of all, the approach which tackled RPP most was AHP and the addition and/or removal of irrelevant alternatives was the most used analysis criterion. In addition, four other situations were also used, as follows: alteration of the indication of the best alternative, decomposition of decision problem, the non-discriminating criterion and the transitivity property.

Second, Vargas, Saaty and Wedley were the authors with the most articles on the RRP, the European Journal of Operational Research was the journal with the most articles published (17.7%), and the period between 1995 and 1997 was the most prolific period of productions on the RRP.

Third, the main gap regarding the RRP seems to be the absence of a convincing explanations of its cause, as well as robust solutions for different methods, such as ELECTRE, PROMETHEE, TOPSIS. Fourth, the revised papers were classified into five clusters: survey, application, problem solution, simulation, and problem identification.

For future studies, the results presented in this study may be increased by analyzing other research papers published in conference proceedings, master’s and doctoral theses. Furthermore, it is suggested that new studies evaluate the occurrence of rank reversal for that more focused studies be conducted to investigate the conditioning factors for the occurrence of rank reversal, in order to really enhance and develop the discussion related to this issue.

ACKNOWLEDGEMENT

We would like to thank the anonymous referees for their valuable comments which greatly improved the quality of the paper. This research work was partially founded by CAPES and CNPq (grant no. 304536/2016-4).

REFERENCES

• ¹
  ABBASI A, HOSSAIN L & LEYDESDORFF L. 2012. Betweenness centrality as a driver of preferential attachment in the evolution of research collaboration networks. Journal of Informetrics, 6(3): 403-412.
• ²
  AGUARÓN J & MORENO-JIMÉNEZ JM. 2000. Local stability intervals in the analytic hierarchy process. European Journal of Operational Research, 125(1): 113-132.
• ³
  AIRES RFF & FERREIRA L. 2016. Reversção de ranking no método TOPSIS: Uma análise comparativa dos procedimentos de normalização. Anais do Simpósio Brasileiro de Pesquisa Operacional - SBPO. Vitória, ES, Brasil, pp. 48.
• ⁴
  ANBAROGLU B, HEYDECKER B & CHENG C. 2014. Spatio-temporal clustering for nonrecurrent traffic congestion detection on urban road networks. Transportation Research C, 48: 47-65.
• ⁵
  BAIRAGI B, DEY B, SARKAR B & SANYAL SK. 2015. A De Novo multi-approaches multicriteria decision making technique with an application in performance evaluation of material handling device. Computers & Industrial Engineering, 87: 267-282.
• ⁶
  BANA E COSTA CA & VANSNICK J-C. 2008. A critical analysis of the eigenvalue method used to derive priorities in AHP. European Journal of Operational Research, 187: 1422-1428.
• ⁷
  BARZILAI J. 1997. Deriving Weights from Pairwise Comparison Matrices. The Journal of the Operational Research Society, 48(12): 1226-1232.
• ⁸
  BARZILAI J & GOLANY B. 1990. Deriving weights from pairwise comparison matrices: The additive case. Operations Research Letters, 9(6): 407-410.
• ⁹
  BARZILAI J & LOOTSMA FA. 1997. Power Relations and Group Aggregation in the Multiplicative AHP and SMART. Journal of Multi-Criteria Decision Analysis, 6(3): 155-165.
• ¹⁰
  BELTON V & GEAR T. 1983. On a short-coming of Saaty’s method of analytic hierarchies. Omega, 11(3): 228-230.
• ¹¹
  BELTON V & GEAR T. 1985. The legitimacy of rank reversal: A comment. Omega, 13(3): 143-144.
• ¹²
  BELTON V & GEAR T. 1997. On the meaning of relative importance. Journal of Multi-Criteria Decision Analysis, 6(6): 335-338.
• ¹³
  BELTON V & STEWART TJ. 2002. Multiple criteria decision analysis: an integrated approach. Boston: Klewer Academic Publishers.
• ¹⁴
  BOTTERO M & FERRETTI V. 2011. An analytic network process-based approach for location problems: the case of a new waste incinerator plant in the Province of Torino (Italy). Journal of Multi-Criteria Decision Analysis, 17(3-4): 63-84.
• ¹⁵
  BRANS J-P. 2015. The PROMETHEE adventure. International Journal of Multicriteria Decision Making, 5(4): 297-308.
• ¹⁶
  BRANS J-P & DE SMET Y. 2016. PROMETHEE Methods. In: GRECO S, EHRGOTT M & FIGUEIRA JR. (Eds.). Multiple Criteria Decision Analysis: State of the Art Surveys (Chap. 6, pp. 187-220). Berlin: Springer.
• ¹⁷
  BUEDE DM & MAXWELL DT. 1995. Rank disagreement: A comparison of multi-criteria methodologies. Journal of Multi-Criteria Decision Analysis, 4(1): 1-21.
• ¹⁸
  BULUT E, DURU O & KOÇAK G. 2015. Rotational priority investigation in fuzzy analytic hierarchy process design: An empirical study on the marine engine selection problem. Applied Mathematical Modelling, 39(2): 913-923.
• ¹⁹
  CINELLI M, COLES SR & KIRWAN K. 2014. Analysis of the potentials of multi criteria decision analysis methods to conduct sustainability assessment. Ecological Indicators, 4: 138-148.
• ²⁰
  DAVIES M. 2001. Adaptive AHP: a review of marketing applications with extensions. European Journal of Marketing, 35(7/8): 872-894.
• ²¹
  DE ALMEIDA AT, CAVALCANTE CAV, ALENCAR MH, FERREIRA RJP, DE ALMEIDA-FILHO AT & GARCEZ TV. 2015. Multicriteria and Multiobjective Models for Risk, Reliability and Maintenance Decision Analysis. (International Series in Operations Research & Management Science. V. 231). New York: Springer.
• ²²
  DE KEYSER W & PEETERS P. 1996. A note on the use of PROMETHEE multicriteria methods. European Journal of Operational Research, 89(3): 457-461.
• ²³
  DEDE G, KAMALAKIS T & SPHICOPOULOS T. 2015. Convergence properties and practical estimation of the probability of rank reversal in pairwise comparisons for multicriteria decision making problems. European Journal of Operational Research, 241(2): 458-468.
• ²⁴
  DODD FJ, DONEGAN HA & MCMASTER TBM. 1995. Inverse inconsistency in analytic hierarchies. European Journal of Operational Research, 80(1): 86-93.
• ²⁵
  DYER JS. 1990a. A Clarification of Remarks on the Analytic Hierarchy Process. Management Science, 36(3): 274-275.
• ²⁶
  DYER JS. 1990b. Remarks on the Analytic Hierarchy Process. Management Science, 36(3): 249-258.
• ²⁷
  FARKAS A. 2007. The analysis of the principal eigenvector of pairwise comparison matrices. Acta Polytechnica Hungarica, 4(2): 99-115.
• ²⁸
  FARKAS A, GYÖRGY A & RÓZSA P. 2004. On the spectrum of pairwise comparison matrices. Linear Algebra and its Applications, 385: 443-462.
• ²⁹
  FARKAS A, RÓZSA P & STUBNYA E. 1999. Transitive matrices and their applications. Linear Algebra and its Applications, 302-303: 423-433.
• ³⁰
  FIGUEIRA JR, GRECO S, ROY B & SLOWINSKI R. 2013. An Overview of ELECTRE Methods and their Recent Extensions. Journal of Multi-Criteria Decision Analysis, 20(1-2): 61-85.
• ³¹
  FIGUEIRA JR & ROY B. 2009. A note on the paper, “Ranking irregularities when evaluating alternatives by using some ELECTRE methods”, by Wang and Triantaphyllou. Omega, 37(3): 731-733.
• ³²
  FINAN JS & HURLEY WJ. 2002. The analytic Hierarchy process: can wash criteria be ignored? Computers & Operations Research, 29(8): 1025-1030.
• ³³
  FORMAN EH. 1990. AHP Is Intended for More Than Expected Value Calculations. Decision Sciences, 21(3): 670-672.
• ³⁴
  FORMAN EH. 1993. Facts and fictions about the analytic hierarchy process. Mathematical and Computer Modelling, 17(4-5): 19-26.
• ³⁵
  FRENCH S. 1986. Decision Theory: An introduction to the Mathematics of Rationality. Chichester: Ellis Horwood.
• ³⁶
  GARCÍA-CASCALES MS & LAMATA MT. 2012. On rank reversal and TOPSIS method. Mathematical and Computer Modelling, 56: 123-132.
• ³⁷
  GOMES LFAM. 1990. Eliminating rank reversal in multicriteria analysis of urban transportation system alternatives. Journal of Advanced Transportation, 24(2): 181-184.
• ³⁸
  GOMES LFAM & GOMES CFS. 2014. Tomada de Decisão Gerencial: Enfoque Multicritério. 5th ed. São Paulo: Atlas.
• ³⁹
  HARKER PT & VARGAS LG. 1987. The Theory of Ratio Scale Estimation: Saaty’s Analytic Hierarchy Process. Management Science, 33(11): 1383-1403.
• ⁴⁰
  HÄMÄLÄINEN RP. 1990. A decision aid in the public debate on nuclear power. European Journal of Operational Research, 48(1): 66-76.
• ⁴¹
  HOCHBAUM DS & LEVIN A. 2006. Methodologies and Algorithms for Group-Rankings Decision. Management Science, 52(9): 1394-1408.
• ⁴²
  HOLDER RD. 1990. Some Comments on the Analytic Hierarchy Process. The Journal of the Operational Research Society, 41(11): 1073-1076.
• ⁴³
  HOU F. 2012. Rank Preserved Aggregation Rules and Application to Reliability Allocation. Communications in Statistics - Theory and Methods, 41(21): 3831-3845.
• ⁴⁴
  HWANG CL & YOON K. 1981. Multiple atribute decision-making methods and applications. Heideberg: Springer.
• ⁴⁵
  IÇ YT. 2014. A TOPSIS based design of experiment approach to assess company ranking. Applied Mathematics and Computation, 227: 630-647.
• ⁴⁶
  ISHIZAKA A & LABIB A. 2011. Review of the main developments in the analytic hierarchy process. Expert Systems with Applications, 38(11): 14336-14345.
• ⁴⁷
  ISHIZAKA A & LUSTI M. 2006. How to derive priorities in AHP: a comparative study. Central European Journal of Operations Research, 14(4): 387-400.
• ⁴⁸
  JAN KH, TUNG C-T & DENG P. 2011. Rank reversal problem related to wash criterion in Analytic hierarchy process (AHP). African Journal of Business Management, 5(20): 8301-8306.
• ⁴⁹
  JUNG S-T, WOU Y-W, LI S-P & JULIAN P. 2009. A revisit to wash criteria in analytic Hierarchy Process. Far East Journal of Mathematical Sciences, 34(1): 31-36.
• ⁵⁰
  KENDALL MM. 1962. Rank Correction Methods. New York: Hafner.
• ⁵¹
  KONG F. 2011. Rank Reversal and Rank Preservation in TOPSIS. Advanced Materials Research, 204-210: 36-41.
• ⁵²
  KULAKOWSKI K. 2015. Notes on order preservation and consistency in AHP. European Journal of Operational Research, 245(1): 333-337.
• ⁵³
  KUJAWSKI E. 2005. A reference-dependent regret model for deterministic tradeoff studies. Systems Engineering, 8(2): 119-137.
• ⁵⁴
  LAI S-K. 1995. A preference-based interpretation of AHP. Omega, 23(4): 453-462.
• ⁵⁵
  LESKINEN P & KANGAS J. 2005. Rank Reversals in Multi-Criteria Decision Analysis with Statistical Modelling of Ratio-Scale Pairwise Comparisons. The Journal of the Operational Research Society, 56(7): 855-861.
• ⁵⁶
  LEUNG LC & CAO D. 2001. On the efficacy of modeling multi-attribute decision problems using AHP and Sinarchy. European Journal of Operational Research, 132(1): 39-49.
• ⁵⁷
  LIBERATORE MJ & NYDICK RL. 2004. Wash criteria and the analytic hierarchy process. Computers & Operations Research, 31(6): 889-892
• ⁵⁸
  LIMA JUNIOR FR, OSIRO L & CARPINETTI LCR. 2014. A comparison between Fuzzy AHP and Fuzzy TOPSIS methods to supplier selection. Applied Soft Computing, 21: 194-209.
• ⁵⁹
  LIN J S-J, CHOU S-Y, CHOUHUANG WT & HSU CP. 2008. Note on “Wash criterion in analytic Hierarchy process”. European Journal of Operational Research, 185(1): 444-447.
• ⁶⁰
  LOOTSMA FA. 1993. Scale sensitivity in the multiplicative AHP and SMART. Journal of Multi-Criteria Decision Analysis, 2(2): 87-110.
• ⁶¹
  LOOTSMA FA & BARZILAI J. 1997. Responseto the Comments by Larichev, Korhonen and Vargason “Power Relations and Group Aggregation in the Multiplicative AHP and SMART”. Journal of Multi-Criteria Decision Analysis , 6(3): 171-174.
• ⁶²
  MALEKI H & SAADAT Y. 2013. Comparison of failure mode and effects analysis by using AHP vs. REMBRANDT system. International Journal of Industrial and Systems Engineering , 14(1): 5-19.
• ⁶³
  MALEKI H & ZAHIR S. 2013. A Comprehensive Literature Review of the Rank Reversal Phenomenon in the Analytic Hierarchy Process. Journal of Multi-Criteria Decision Analysis , 20(3-4): 141-155.
• ⁶⁴
  MILLET I & SAATY TL. 2000. On the relativity of relative measures: accommodating both rank preservation and rank reversals in the AHP. European Journal of Operational Research , 121(1): 205-212.
• ⁶⁵
  MIRHEDAYATIAN SM & FARZIPOOR SAEN R. 2011. A new approach for weight derivation using data envelopment analysis in the analytic hierarchy process. The Journal of the Operational Research Society , 62(8): 1585-1595.
• ⁶⁶
  MORTON A. 2015. Measurement issues in the evaluation of projects in a project portfolio. European Journal of Operational Research , 245(3): 789-796.
• ⁶⁷
  NAGEL SS. 1989. Evaluation Analysis with Microcomputers. Greenwich: JAI Press.
• ⁶⁸
  OLSON DL, FLIEDNER G & CURRIE K. 1995a. Comparison of the REMBRANDT system with Analytic hierarchy process. European Journal of Operational Research , 82(3): 522-539.
• ⁶⁹
  OLSON DL, MOSHKOVICH HM, SCHELLENBERGER R & MECHITOV AI. 1995b. Consistency and Accuracy in Decision Aids: Experiments with Four Multiattribute Systems. Decision Sciences , 26(6): 723-747.
• ⁷⁰
  PAULSON D & ZAHIR S. 1995. Consequences of uncertainty in the analytic hierarchy process: A simulation approach. European Journal of Operational Research , 87(1): 45-56.
• ⁷¹
  PANKRATOVA ND & NEDASHKOVSKAY NI. 2014. Hybrid Method of Multicriteria Evaluation of Decision Alternatives. Cybernetics and Systems Analysis, 50(5): 701-711.
• ⁷²
  PÉREZ J. 1995. Some Comments on Saaty’s AHP. Management Science , 41(6): 1091-1095.
• ⁷³
  PÉREZ J, JIMENO JL & MOKOTOFF E. 2006. Another potential shortcoming of AHP. Top, 14(1): 99-111.
• ⁷⁴
  PETERSON M. 2009. An Introduction to Decision Theory. Cambridge: Cambridge University Press.
• ⁷⁵
  RAHARJO H & ENDAH D. 2006. Evaluating Relationship of Consistency Ratio and Number of Alternatives on Rank Reversal in the AHP. Quality Engineering, 18(1): 39-46.
• ⁷⁶
  RAMANATHAN R. 2006. Data envelopment analysis for weight derivation and aggregation in the Analytic hierarchy process. Computers & Operations Research , 33(5): 1289-1307.
• ⁷⁷
  RAMANATHAN U & RAMANATHAN R. 2011. An investigation into rank reversal properties of the multiplicative AHP. International Journal of Operational Research, 11(1): 54-77.
• ⁷⁸
  REN L, ZHANG Y, WANG Y & SUN Z. 2007. Comparative Analysis of a Novel M-TOPSIS Method and TOPSIS. Applied Mathematics Research eXpress.
• ⁷⁹
  RODRÍGUEZ A, ORTEGA F & CONCEPCIÓN R. 2013. A method for the selection of customized equipment suppliers. Expert Systems with Applications , 40(4): 1170-1176.
• ⁸⁰
  ROY B. 1996. Multicriteria methodology for decision aiding. Boston: Kluwer Academic Publishers.
• ⁸¹
  SAATY TL. 1986. Axiomatic Foundation of the Analytic Hierarchy Process. Management Science, 32(7): 841-855.
• ⁸²
  SAATY TL. 1980. The analytic hierarchy process. New York: McGraw Hill.
• ⁸³
  SAATY TL. 1987a. The analytic hierarchy process: what it is and how it is used. Mathematical Modelling, 9(3-5): 161-176.
• ⁸⁴
  SAATY TL. 1987b. Decision making, new information, ranking and structure. Mathematical Modelling, 8: 125-132.
• ⁸⁵
  SAATY TL. 1987c. Rank Generation, Preservation, and Reversal in the Analytic Hierarchy Decision Process. Decision Sciences, 18(2): 157-177.
• ⁸⁶
  SAATY TL. 1990. An Exposition of the AHP in Reply to the Paper “Remarks on the Analytic Hierarchy Process”. Management Sciences, 36(3): 259-268.
• ⁸⁷
  SAATY TL. 1994a. Highlights and critical points inthe theory and application of the Analytic Hierarchy Process. European Journal of Operational Research , 74(3): 426-447.
• ⁸⁸
  SAATY TL. 1994b. How to Make a Decision: The Analytic Hierarchy Process. Interfaces, 24(6): 19-43.
• ⁸⁹
  SAATY TL. 1997. That is not the analytic Hierarchy process: what the AHP is and what it is not. Journal of Multi-Criteria Decision Analysis , 6(6): 324-335.
• ⁹⁰
  SAATY TL. 2006. Rank from comparisons and from ratings in the Analytic hierarchy/network processes. European Journal of Operational Research , 168(2): 557-570.
• ⁹¹
  SAATY TL & SAGIR M. 2009. An essay on rank preservation and reversal. Mathematical and Computer Modelling, 49: 1230-1243.
• ⁹²
  SAATY TL & VARGAS LG. 1984a. Inconsistency and rank preservation. Journal of Mathematical Psychology, 28(2): 205-214.
• ⁹³
  SAATY TL & VARGAS LG. 1984b. The legitimacy of rank reversal. Omega , 12(5): 513-516.
• ⁹⁴
  SAATY TL & VARGAS LG. 1987. Uncertainty and rank order in the analytic hierarchy process. European Journal of Operational Research , 32(1): 107-117.
• ⁹⁵
  SAATY TL & VARGAS LG. 1993. Experiments on rank preservation and reversal in relative measurement. Mathematical and Computer Modelling, 17(4-5): 13-18.
• ⁹⁶
  SAATY TL & VARGAS LG. 2006. The Analytic Hierarchy Process: wash criteria should not be ignored. International Journal of Management and Decision Making, 7(2): 180-188.
• ⁹⁷
  SALABUN W. 2015. The Characteristic Objects Method: A New Distance-based Approach to Multicriteria Decision-making Problems. Journal of Multi-Criteria Decision Analysis , 22(1-2): 37-50.
• ⁹⁸
  SALEM AAA & AWASTHI A. 2018. Investigating rank reversal in reciprocal fuzzy preference relation based on additive con-sistency: Causes and solutions. Computers & Industrial Engineering , 115: 573-581.
• ⁹⁹
  SALO AA & HÄMÄLÄINEN RP. 1997. On the measurement of preferences in the analytic Hierarchy process. Journal of Multi-Criteria Decision Analysis , 6(6): 309-319.
• ¹⁰⁰
  SALO A & PUNKKA A. 2011. Ranking Intervals and Dominance Relations for Ratio-Based Efficiency Analysis. Management Science, 57(1): 200-214.
• ¹⁰¹
  SARKIS J. 1999. A methodological framework for evaluating environmentally conscious manufacturing programs. Computers & Industrial Engineering , 36(4): 793-810.
• ¹⁰²
  SCHENKERMAN S. 1994. Avoiding rank reversal in AHP decision-support models. European Journal of Operational Research , 74(3): 407-419.
• ¹⁰³
  SCHONER B, CHOO EU & WEDLEY WC. 1997. A Comment on Rank Disagreement: A Comparison of Multi-Criteria Methodologies. Journal of Multi-Criteria Decision Analysis , 6(4): 197-200.
• ¹⁰⁴
  SCHONER B & WEDLEY WC. 1989. Ambiguous Criteria Weights in AHP: Consequences and Solutions. Decision Sciences, 20(3): 462-475.
• ¹⁰⁵
  SCHONER B, WEDLEY WC & CHOO EU. 1992. A Rejoinder to Forman on AHP, with Emphasis on the Requirements of Composite Ratio Scales. Decision Sciences, 23(2): 509-517.
• ¹⁰⁶
  SCHONER B, WEDLEY WC & CHOO EU. 1993. A unified approach to AHP with linking pins. European Journal of Operational Research , 64(3): 384-392.
• ¹⁰⁷
  SINUANY-STERN Z, MEHREZ A & HADAD Y. 2000. An AHP/DEA methodology for ranking decision making units. International Transactions in Operational Research, 7(2): 109-124.
• ¹⁰⁸
  SOLTANIFAR M & SHAHGHOBADI S. 2014. Survey on rank preservation and rank reversal in data envelopment analysis. Knowledge-Based Systems, 60: 10-19.
• ¹⁰⁹
  SRDJEVIC B. 2005. Combining different prioritization methods in the analytic hierarchy process synthesis. Computers & Operations Research, 32(7): 1897-1919.
• ¹¹⁰
  STAM A. 1997. Short Note on “On the Measurement of Preferences in the Analytic Hierarchy Process” by SALO AA & HÄMÄLÄINEN RP. Journal of Multi-Criteria Decision Analysis , 6(6): 338-339.
• ¹¹¹
  STAM A & SILVA APD. 1997. Stochastic Judgments in the AHP: The Measurement of Rank Reversal Probabilities. Decision Sciences, 28(3): 665-688.
• ¹¹²
  STAM A & SILVA APD. 2003. On multiplicative priority rating methods for the AHP. European Journal of Operational Research , 145(1): 92-108.
• ¹¹³
  STEWART T. 1992. A critical survey on the status of multiple criteria decision making theory and practice. Omega , 20(5-6): 569-586.
• ¹¹⁴
  SUNDARRAJ RP. 2004. A Web-based AHP approach to standardize the process of managing service-contracts. Decision Support Systems, 37(3): 343-365.
• ¹¹⁵
  TAVANA M. 2004. A subjective assessment of alternative mission architectures for the human exploration of Mars at NASA using multicriteria decision making. Computers & Operations Research, 31(7): 1147-1164.
• ¹¹⁶
  TOMASHEVSKII IL. 2015. Eigenvector ranking method as a measuring tool: Formulas for errors. European Journal of Operational Research , 240(3): 774-780.
• ¹¹⁷
  TOTSENKO VG. 2006. On Problem of Reversal of Alternatives Ranks while Multicriteria Estimating. Journal of Automation and Information Sciences, 38(6): 1-11.
• ¹¹⁸
  TRIANTAPHYLLOU E. 2001. Two new cases of rank reversals when the AHP and some of its additive variants are used that do not occur with the multiplicative AHP. Journal of Multi-Criteria Decision Analysis , 10(1): 11-25.
• ¹¹⁹
  TRIANTAPHYLLOU E & MANN SH. 1989. An Examination of the Effectiveness of Multi-Dimentional Decision-Making Methods: A Decision-Making Paradox. International Journal of Decision Support Systems, 5: 303-312.
• ¹²⁰
  TRIANTAPHYLLOU E & MANN SH. 1994. A computational evaluation of the original and revised Analytic hierarchy process. Computers & Industrial Engineering , 26(3): 609-618.
• ¹²¹
  TRIANTAPHYLLOU E & MANN SH. 1995. Using the Analytic Hierarchy Process For Decision Making in Engineering Applications: Some Challenges. Journal of Industrial Engineering: Applications and Practice, 2(1): 35-44.
• ¹²²
  TRIANTAPHYLLOU E & SÁNCHEZ A. 1997. A Sensitivity Analysis Approach for Some Deterministic Multi-Criteria Decision-Making Methods. Decision Sciences, 28(1): 151-194.
• ¹²³
  TROUTT M. 1988. Rank reversal and the dependence of priorities on the underlying MAV function. Omega , 16(4): 365-367.
• ¹²⁴
  TROUTT M & TADISINA S. 1990. Corrigendum and further results: Rank reversal and the dependence of priorities on the underlying MAV function. Omega , 18(6): 655-656.
• ¹²⁵
  TROUTT MD, TADISINA SK & PENDHARKAR PC. 2009. Value functions and trade-offs associated with the analytic hierarchy process composition law. International Journal of Mathematics in Operational Research, 1(1-2): 97-120.
• ¹²⁶
  VAN DEN HONERT RC. 1998. Stochastic pairwise comparative judgements and direct ratings of alternatives in the REMBRANDT system. Journal of Multi-Criteria Decision Analysis , 7(2): 87-97.
• ¹²⁷
  VARGAS LG. 1994. Reply to Schenkerman’savoiding rank reversal in AHP decision support models. European Journal of Operational Research , 74(3): 420-425.
• ¹²⁸
  VARGAS LG. 1994. Comments on Barzilai and Lootsma: Why the Multiplicative AHP Is Invalid: A Practical Counterexample. Journal of Multi-Criteria Decision Analysis , 6(3): 169-170.
• ¹²⁹
  VERLY C & DE SMET Y. 2013. Some results about rank reversal instances in the PROMETHEE methods. International Journal of Multicriteria Decision Making , 3(4): 325-345.
• ¹³⁰
  VETSCHERA R & ALMEIDA AT. 2012. A PROMETHEE-based approach to portfolio selection problems. Computers & Operations Research, 39(5): 1010-1020
• ¹³¹
  VINCKE P. 1992. Multicriteria decision-aid. Londres: John Wiley & Sons.
• ¹³²
  VINOD KUMAR N & GANESH LS. 1996. An empirical analysis of the use of the Analytic Hierarchy Process for estimating membership values in a fuzzy set. Fuzzy Sets and Systems, 82(1): 1-16.
• ¹³³
  WANG X & TRIANTAPHYLLOU E. 2008. Ranking irregularities when evaluating alternatives by using some ELECTRE methods. Omega , 36(1): 45-63.
• ¹³⁴
  WANG Y-M & ELHAG TMS. 2006. An approach to avoiding rank reversal in AHP. Decision Support Systems, 42(3): 1474-1480.
• ¹³⁵
  WANG Y-M & LUO Y. 2009. On rank reversal in decision analysis. Mathematical and Computer Modelling, 49(5-6): 1221-1229.
• ¹³⁶
  WANG Y-M & LUO Y. 2012. A note on “A new approach for weight derivation using data envelopment analysis in the analytic hierarchy process”. Mathematical and Computer Modelling, 56(3-4): 49-55.
• ¹³⁷
  WANG Y-M, PARKAN C & LUO Y. 2007. A linear programming method for generating the most favorable weights from a pairwise comparison matrix. Computers & Operations Research, 35(12): 3918-3930.
• ¹³⁸
  WARDIL L & HAUERT C. 2014. Origin and structure of dynamic cooperative networks. Scientific Reports, 4: n/a.
• ¹³⁹
  WEDLEY WC, SCHONER B & CHOO EU. 1993. Clustering, dependence and ratio scales in AHP: Rank reversals and incorrect priorities with a single criterion. Journal of Multi-Criteria Decision Analysis , 2(3): 145-158.
• ¹⁴⁰
  WIJNMALEN DJD & WEDLEY WC. 2009a. Correcting illegitimate rank reversals: proper adjustment of criteria weights prevent alleged AHP intransitivity. Journal of Multi-Criteria Decision Analysis , 15(5-6): 135-141.
• ¹⁴¹
  WIJNMALEN DJD & WEDLEY WC. 2009b. Nondiscriminating criteria in the AHP: removal and rank reversal. Journal of Multi-Criteria Decision Analysis , 15(5-6): 143-149.
• ¹⁴²
  YAGER RR. 1978. Fuzzy decision-making including unequal objectives. Fuzzy Sets and Systems, 1(2): 87-95.
• ¹⁴³
  YANG J & LEE H. 1997. An AHP decision model for facility location selection. Facilities, 15(9-10): 241-254.
• ¹⁴⁴
  YARAGHI N, TABESH P, GUAN P & ZHUANG J. 2015. Comparison of AHP and Monte Carlo AHP Under Different Levels of Uncertainty. IEEE Transactions on Engineering Management, 62(1): 122-132.
• ¹⁴⁵
  ZAHIR S. 1999. Geometry of decision making and the vector space formulation of the analytic hierarchy process. European Journal of Operational Research , 112(2): 373-396.
• ¹⁴⁶
  ZAHIR S. 2009. Normalisation and rank reversals in the additive analytic hierarchy process: a new analysis. International Journal of Operational Research, 4(4): 446-467.
• ¹⁴⁷
  ZANAKIS SH, SOLOMON A, WISHART N & DUBLISH S. 1998. Multi-attribute decision making: A simulation comparison of select methods. European Journal of Operational Research , 107(3): 507-529.

APPENDIX - SUMMARY OF INDIVIDUAL APPROACHES

Table 4
Summary of individual approaches.

Approaches	Authors	Contribution
AHP	Belton & Gear (1983)	BG normalization procedure.
	Saaty & Vargas (1984a)	Evaluation of the deriving ratio estimates regarding inconsistency and preservation ranking.
	Saaty & Vargas (1984b)	Demonstration of ranking inconsistencies in Belton & Gear (1983).
	Belton & Gear (1985)	Explanation on BG normalization procedure.
	Saaty (1986)	Structural dependence of the criteria may cause RRP.
	Saaty (1987a)	Similar to Saaty (1986).
	Saaty (1987b)	RRP may be caused if the importance of the criteria depends both on the number of alternatives and on the strength of their ranking.
	Saaty (1987c)	Similar to Saaty (1986) and Saaty (1987a).
	Harker & Vargas (1987)	RRP is caused by misuse of the method.
	Saaty & Vargas (1987)	RRP is caused by uncertainty of the decision-maker.
	Troutt (1988)	RRP is caused by aggregation rule.
	Triantaphyllou & Mann (1989)	Analysis of four methods regarding rank reversal rate.
	Schoner & Wedley (1989)	RRP is caused by incorrect criteria assessments.
	Barzilai & Golany (1990)	Arithmetic mean is used to solve the RRP.
	Dyer (1990a)	Rankings are arbitrary in the face of inappropriate criteria weights.
	Dyer (1990b)	Similar to Dyer (1990a).
	Saaty (1990)	Similar to Harker & Vargas (1987).
	Harker & Vargas (1990)	Similar to Harker & Vargas (1987) and Saaty (1990).
	Forman (1990)	Referenced AHP does not provide convincing arguments for its use.
	Troutt & Tadisina (1990)	RRP was avoided through the use of Multiattribute Value (MAV).
	Holder (1990)	Reported inconsistencies in the method.
	Hämäläinen (1990)	RRP is mentioned in a specific context of application context.
	Stewart (1992)	RRP is mentioned.
	Schoner et al. (1992)	RRP may be caused by the process of normalizing local priorities.
	Forman (1993)	RRP may be caused by substitution effect.
	Lootsma (1993)	Multiplicative AHP method.
	Schoner et al. (1993)	RRP is mentioned.
	Saaty & Vargas (1993)	Evaluation of inconsistencies and ranking preservation.
	Wedley et al. (1993)	Axiom 3 causes the RRP.
	Saaty (1994a)	Analysis of ranking procedures with implications to RRP.
	Saaty (1994b)	Ideal mode of setting priorities may avoid RRP.
	Schenkerman (1994)	Relative measurement is not a valid manner of measuring.
	Vargas (1994)	RRP is an acceptable phenomenon.
	Triantaphyllou & Mann (1994)	Occurrence of ranking irregularities in AHP or revised AHP.
	Lai (1995)	AHP’- new scaling technique.
	Olson et al. (1995b)	Evaluation of four decision support systems regarding inconsistency and preservation ranking.
	Paulson & Zahir (1995)	RRP and its probability of occurrence over a wide range of uncertainties.
	Pérez (1995)	Relation between the criteria weights and the RRP.
	Dodd et al. (1995)	MAHP method.
	Triantaphyllou & Mann (1995)	RRP is mentioned.
	Vinod Kumar & Ganesh (1996)	Influence of the RRP for estimating membership values in a fuzzy set.
	Barzilai & Lootsma (1997)	Similar to Lootsma (1993).
	Salo & Hämäläinen (1997)	Supermatrix technique does not eliminate the RRP.
	Belton & Gear (1997)	RRP is caused by the relative importance of the average performance on each criterion.
	Lootsma & Barzilai (1997)	Similar to Lootsma (1993) and Barzilai & Lootsma (1997).
	Barzilai (1997)	Similar to Lootsma (1993), Barzilai & Lootsma (1997) and Lootsma & Barzilai (1997).
	Vargas (1997)	Multiplicative AHP is invalid.
	Schoner et al. (1997)	Describe the modified AHP techniques.
	Stam (1997)	Multiplicative AHP method may avoid RRP.
	Stam & Silva (1997)	Evaluation of the RRP in a stochastic context.
	Saaty (1997)	Similar to Saaty (1994b).
	Yang & Lee (1997)	RRP is mentioned.
	Triantaphyllou & Sánchez (1997)	Evaluation of three decision methods regarding to the RRP.
	Van Den Honert (1998)	RRP is caused by uncertainty in the decision-maker’s preference judgments.
	Farkas et al. (1999)	RRP is inherent in the AHP.
	Zahir (1999)	RRP is mentioned.
	Sarkis (1999)	RRP is avoided through the use of ANP method.
	Millet & Saaty (2000)	Similar to Saaty (1994b).
	Sinuany-Stern et al. (2000)	AHP/DEA method.
	Aguarón & Moreno-Jiménez (2000)	Illustration of the RRP regarding to the alteration of the best alternative and transitivity property.
	Davies (2001)	RRP is mentioned.
	Leung & Cao (2001)	RRP is avoided through the use of Sinarchy, a particular form of ANP method.
	Triantaphyllou (2001)	RRP is caused by normalization step and use of additive function.
	Finan & Hurley (2002)	RRP and non-discriminating criterion.
	Stam & Silva (2003)	Similar to Stam (1997).
	Liberatore & Nydick (2004)	Non-discriminating criterion a may not cause RRP.
	Farkas et al. (2004)	Illustration of the RRP.
	Tavana (2004)	RRP is mentioned.
	Sundarraj (2004)	RRP is mentioned.
	Srdjevic (2005)	Multicriteria Preference Synthesis Procedure (MPS).
	Leskinen & Kangas (2005)	RRP is avoided through the use of ratio-scale measurement.
	Kujawski (2005)	Reference-Dependent Regret Model (RDRM).
	Raharjo & Endah (2006)	RRP is caused by consistency ratio and the number of alternatives.
	Pérez et al. (2006)	RRP may be caused by adding indifferent criteria.
	Saaty (2006)	Relative measurement, conditional independence or structural dependence plays an important role in influencing the RRP.
	Wang & Elhag (2006)	RRP is caused by the change of local priorities before and after adding or removing alternatives.
	Totsenko (2006)	Analytic Hierarchy Process Advanced (AHPA).
	Hochbaum & Levin (2006)	RRP is mentioned in a group decision context.
	Saaty & Vargas (2006)	Similar to Liberatore & Nydick (2004).
	Ramanathan (2006)	AHP/DEA method.
	Ishizaka & Lusti (2006)	Evaluation of four AHP ratio scaling methods: the right eigenvalue method, the left eigenvalue method, the geometric mean and the mean of normalized values.
	Farkas (2007)	Similar to Farkas et al. (1999).
	Wang, Parkan & Luo (2007)	DEAHP suffers from RRP.
	Lin et al. (2008)	RRP is not caused by non-discriminating criterion.
	Bana e Costa & Vansnick (2008)	RRP is mentioned.
	Saaty & Sagir (2009)	Preserve rank in all situations may be wrong.
	Wijnmalen & Wedley (2009a)	RRP is caused by the independence axiom.
	Wijnmalen & Wedley (2009b)	Similar to Liberatore & Nydick (2004) and Saaty & Vargas (2006).
	Zahir (2009)	RRP is caused by the aggregation rule.
	Jung et al. (2009)	How avoid RRP from non-discriminating criterion criteria.
	Troutt, Tadisina & Pendharkar (2009)	The number of alternatives being compared in the AHP is the main cause of rank reversal.
	Ramanathan & Ramanathan (2011)	Relation between the Multiplicative AHP and the RRP.
	Ishizaka & Labib (2011)	RRP is mentioned.
	Jan et al. (2011)	Relation between the non-discriminating criterion and the RRP.
	Mirhedayatian & Farzipoor Saen (2011)	Revised DEAHP does not suffer from RRP.
	Wang & Luo (2012)	Revised DEAHP suffers from RRP.
	Maleki & Zahir (2013)	RRP literature review on AHP method.
	Kulakowski (2015)	Order preservation and consistency in AHP method.
	Yaraghi et al. (2015)	RRP is mentioned.
TOPSIS	Ren et al. (2007)	M-TOPSIS method.
	Kong (2011)	New normalization procedure, new PIS/NIS obtaining method.
	García-Cascales & Lamata (2012)	Presentation of a new normalization and a new PIS/NIS calculation procedure for the TOPSIS.
ELECTRE	Wang & Triantaphyllou (2008)	Evaluation of ELECTRE II e ELECTRE III regarding to the RRP.
	Figueira & Roy (2009)	Relation between the RRP and ELECTRE methods.
	Figueira et al. (2013)	Similar to Figueira & Roy (2009).
PROMETHEE	De Keyser & Peeters (1996)	RRP is mentioned.
	Vetschera & Almeida (2012)	RRP is mentioned.
	Verly & De Smet (2013)	Evaluation of PROMETHEE I and PROMETHEE II regarding to the RRP.
	Brans (2015)	RRP is mentioned.
OTHERS	Gomes (1990)	Relation between the RRP and TODIM method.
	Olson et al. (1995a)	RRP is avoided through the use of REMBRANDT method.
	Maleki & Saadat (2013)	Similar to Olson et al. (1995a).
	Salo & Punkka (2011)	RRP is mentioned in DEA.
	Soltanifar & Shahghobadi (2014)	RRP in DEA methods.
	Rodríguez et al. (2013)	RRP in FUZZY-AHP-TOPSIS.
	Lima Junior et al. (2014)	Exploration of the FUZZY-AHP and FUZZY-TOPSIS regarding to the RRP in a specific context.
	Bulut, Duru & Koçak (2015)	RRP is mentioned in FUZZY-AHP.
	Bottero & Ferretti (2011)	Propose the ANP?BOCR method.
	Hou (2012)	FRP-AHP method.
	Pankratova & Nedashkovskay (2014)	Propose the MEDM method.
	Salabun (2015)	Propose the COMET method.
	Bairagi et al. (2015)	Propose the TPOP method.
	Iç (2014)	Propose the DOE-TOPSIS method.
	Morton (2015)	Criticizes of the way of using value functions in the evaluation of portfolio project’s by showing it can lead to a rank reversal.
	Tomashevskii (2015)	RRP is caused by the inaccuracy of the measurement scale and by the inconsistent judgements of experts.
	Dede et al. (2015)	Evaluation of probability ocorrence of the RRP in the context of pairwise comparisons.
	Buede & Maxwell (1995)	Examination of the frequency and magnitude of the RRP in four different methods.
	Zanakis et al. (1998)	Evaluation of eight methods regarding to the RRP.
	Wang & Luo (2009)	Demonstration of the RRP in four different methods.
	Cinelli et al. (2014)	Occurrence of the RRP in five different methods in a context of conduct sustainability assessment.

Publication Dates

• Publication in this collection
  May-Aug 2018

History

• Received
  07 Nov 2017
• Accepted
  23 May 2018