THE RANK REVERSAL PROBLEM IN MULTI-CRITERIA DECISION MAKING : A LITERATURE REVIEW

Despite the importance of multicriteria decision-making (MCDM) techniques for constructing effective decision models, there are many criticisms due to the occurrence of a problem called rank reversal. Nevertheless, there is a lack of a systematic literature review on this important subject which involves different methods. This study reviews the pertinent literature on rank reversal, based on 130 related articles published from 1980 to 2015 in international journals, which were gathered and analyzed according to the following perspectives: multicriteria technique, year and journal in which the papers were published, co-authorship network, rank reversal types, and research goal. Thus our survey provides recommendations for future research, besides useful information and knowledge regarding rank reversal in the MCDM field.


INTRODUCTION
Multi-criteria decision-making (MCDM) has been one of the fastest growing areas of Operational Research during the last two decades and it is one of the most used decision methodologies in the sciences, business, government, and engineering for supporting the quality of the decisionmaking process (Wang & Triantaphyllou, 2008;Maleki & Zahir, 2013).Despite widespread use of these methodologies, the main methods, such as AHP, TOPSIS, ELECTRE, PROMETHEE and combinations thereof, have been criticized due to the occurrence of a problem called rank reversal (RRP).RR refers to a change in the ordering among alternatives previously defined, after the addition or removal of an alternative from the group previously ordered (Lootsma, 1993;Buede & Maxwell, 1995;Saaty & Sagir, 2009;Wang & Luo, 2009).
Despite the great interest of researchers in RRP, given its importance for addressing the reliability of MCDM methods, there is still a paucity in the literature regarding published studies on this subject.During the development of our research and analysis, we have found only the work of Maleki & Zahir (2013) in the literature on rank reversal and MCDM, where the authors evaluated 61 papers on AHP method from 18 international journals.In this study, we extend their paper by analyzing 130 articles on different multicriteria methods from 37 journals, which were examined from five aspects: the multicriteria technique, year and journal in which the papers were published, the co-authorship network, rank reversal situations, and the research goal.Thus, the contributions of our paper are the following: (i) we present a new and broader systematic literature review (SLR) on RRP; (ii) we have carefully analyzed each paper in order to classify the research goal; (iii) we designed a co-autorship network on AHP and rank reversal (RR); and (iv) we summarize the types of rank reversal commonly evaluated in the literature.These findings are important because they can be useful to help prioritize and provide insights for future research on RRP and MCDM.
The remainder of this paper is organized as follows: Section 2 presents the rank reversal problem, related axioms and the assumptions of each MCDM.Section 3 presents the research methodology, i.e. it describes the method for searching for, selecting and analyzing the articles.Section 4 provides an individual analysis of the MCDM techniques regarding the RRP, while Section 5 According to De Almeida et al. (2015), outranking methods do not use a unique criterion of synthesis, so many of these methods produce the final recommendation with no scores for alternatives.These methods use the preference structure (P,Q,I,J), including the incomparability relation, and they produce a partial pre-order.The main methods in this group are the ELECTRE and PROMETHEE methods (Roy, 1996;Vincke, 1992;Belton & Stewart 2002).This structure is relevant for situations in which the DM is not able to give full preference information.
Despite the great growth and evolution of the MCDM field, these methods are known to present RRP, for which the literature has given different interpretations.Saaty, for example, states "Assume that an individual has expressed preference among a set of alternatives, and that as a result, he or she has developed a ranking for them.Can and should that individual's preferences and the resulting rankings order of the alternatives be affected if alternatives are added to or deleted from the set?"Researchers such as Saaty & Vargas (1984b), Saaty (1987a), Forman (1990) and Millet & Saaty (2000) also argued for the legitimacy of this problem.For some authors such as Saaty (1994b) and Millet & Saaty (2000) the reversals do not occur often and thus having rank reversals in certain occasions and of certain types may not be indicative of faulty decision making.
On the other hand, Salem & Awasthi (2018) mentioned that this problem could drive some DMs away from using methods known to have rank reversal and has substantial logical implications for the methodology that is used to make decisions.In fact, Anbaroglu, Heydecker & Cheng (2014), for example, used the Weighted Product Model (WPM) instead of the classical AHP because it does not suffer from any kind of RR.In addition, Anbaroglu, Heydecker & Cheng (2014) stated that RRP is a serious limitation of the MCDM field, which could lead researchers to misunderstand the difference between the alternatives examined.
In the literature, rank reversal is the change in ranking of the alternatives as a consequence, for example, of the addition or deletion of an alternative.This means that a DM's preference ordering between two alternatives has changed when an alternative is added or removed and this clearly contradicts the principle of the independence of irrelevant alternatives.In addition, Finan & Hurley (2002) 2011), for example, associated RRP with a non-discriminating criterion.In this case, as the DM is indifferent among the alternatives when they are compared on that criterion and the non-discriminating criteria do not differentiate between the alternatives, it is presumably safe to eliminate them from further consideration (Wijnmalen & Wedley, 2009b).
Finally, although there has been a long and unresolved debate on RRP among the practitioners in the MCDM literature, is possible to characterize different typologies of rank reversals from the literature, as described in the following: The MCDM methods analyzed in this paper were evaluated based on different types of RR, where Type #1 was the most widely used in the literature.In addition, many methods have not been tested for all the types, such as the TOPSIS method with respect to Type #3, Type #4, and Type #5.

RESEARCH METHODOLOGY
The research methodology of this survey is depicted in Figure 1.The first goal of this research was to gather articles on RRP in the specialized literature of the multicriteria field.
Thus, we defined the following rules to define our sample of papers: • The keywords for the initial search were "rank reversal" or "rank irregularities" or "rank preservation" or "rank consistency" and "multi-criteria".
• The search for articles was made on academic databases including Science Direct, Emerald, Springer-Link Journals, IEEE Xplore, Academic Search Premier, Scielo and World Scientific Net, and the entire number of articles in the literature published was examined.
• To achieve the highest level of relevance, only international journal articles were selected.Thus, conference articles, master and doctoral dissertations, textbooks, unpublished articles, and notes were not included in this review.Similarly, only articles published in English were considered.
Based on these considerations, 130 articles were selected.These articles refer to AHP, TOP-SIS, PROMETHEE, ELECTRE, and others multicriteria methods.Each article was carefully reviewed and selected strictly according to the scope of this research as set out in Figure 1.
Appendix 6 presents a first classification of each paper organized in accordance with the multicriteria method, the year of publication, and the main scientific contribution made by each article.Details about this will be presented in the next section, and so too will the evolution of rank reversal studies for the main multicriteria decision-making methods.

INDIVIDUAL APPROACHES
This section presents an analysis of the main papers surveyed, namely the MCDM method Analytic Hierarchy Process (AHP), Technique for Order of Preferences by Similarity to Ideal Solution (TOPSIS), Preference Ranking Organization METHod of Enrichment Evaluations (PROMETHEE), Elimination and Choice Expressing Reality (ELECTRE), and hybrid versions.

AHP
A total of 99 of 130 articles (76.2 %) analyzed RRP in AHP.After an initial discussion between by Belton & Gear (1983, 1985) and Saaty & Vargas (1984a, b) as previously presented, Schoner & Wedley (1989) presented the new version of the AHP method called "Referenced AHP".To avoid RR, this approach requires criteria weights to be modified whenever an alternative is added or removed, i.e., the authors considered that criteria weights are dependent upon the alternatives in the set chosen.Thereafter, Foman (1990) argued that the problem was related to non-linearity, and Dyer (1990b) proposed that AHP judgments should be rescaled in order to ensure "that the criterion weights and the scores of the alternatives on the criteria are normalized with respect to the same range of alternative values".Harker & Vargas (1990) and Saaty (1990) replied to Dyer's paper, and Dyer (1990a) counter-argued that these authors had misunderstood this process.Finally, Schoner et al. (1992) proposed that the problem may be easily solved by normalizing local priorities, since the change in the set of alternatives (addition or removal) requires a new normalization of local priorities.Schenkerman (1994) stated that the AHP method presumes that the DM's preference structure can be adequately represented by an additive value function.For Vargas (1994), the RR is an acceptable characteristic of human behavior, and one that cannot be justified simply by saying that some of the basic assumptions of rationality have been violated.This view of the problem was also advocated by other authors, such as Farkas et al. (1999) and Farkas (2007).
The AHP has two measurement modes, absolute and relative.In the relative measurement, the local priorities of the alterrnatives are normalized so that they add up to one, whereas in absolute measurement no such normalization is applied to them (Saaty, 1994).In the relative measurement mode, any alternative that is not dominated by some convex combination of other alternatives may become the best one as a result of introducing or deleting other alternatives (Salo & Hämäläinen, 1997).Salo & Hämäläinen (1997) also pointed out that the fundamental mathematical reason for the occurrence of rank reversal in the relative measurement mode is that the local priorities in the lower level of the hierarchy are normalized so that they add up to 1.When new alternatives are added or removed, the local priorities associated with other alternatives inevitably change and, as a result, the final ranking of alternatives can also be changed (Salo & Hämäläinen, 1997).Belton & Gear (1997) agree with this explanation.Millet & Saaty (2000) and Saaty (1994b) described a structural change in AHP to deal with both situations of measurement modes.These authors proposed two types of synthesis procedures.The first one, which is the one most commonly used, was called the distributive mode.This may cause rank reversal, since there is dependence among the alternatives and the ratings of the alternatives are normalized such that the sum of all of them is 1.In contrast, the second procedure was called the ideal mode.It is able to preserve the ranks, because the score of each alternative is divided only by the score of the best alternative under each criterion.
From these papers, several authors pointed to the classical synthesis procedure as the main cause of RR in AHP (Troutt, 1988;Troutt & Tadisina, 1990;Barzilai & Golany, 1990;Zahir, 2009;Ramanathan & Ramanathan, 2011).Therefore, multiplicative AHP (MAHP) was designed to solve the RRP (Lootsma, 1993;Barzilai & Lootsma, 1997;Lootsma & Barzilai, 1997;Stam & Silva, 2003).MHAP is a variant of AHP proposed by Saaty with a one-parametric class of geometric scales to quantify human comparative judgement and with a multiplicative structure given by a logarithmic regression to calculate the impact scores of the alternatives at the first evaluation level and a geometric-mean aggregation rule to calculate the final scores at the second level (Lootsma, 1993).According to these authors, MAHP is able to avoid the RRP because the rank order of the impact scores and final scores is scale-independent.
Vargas (1997) was emphatic in rejecting this argument and showed the AHP ideal mode aforementioned solves the problems raised by these authors.The same author, explained the different situations where the reversals may occur.See Saaty & Sagir (2009) for further details.
Triantaphyllou (2001) demonstrated the occurrence of rank reversal in AHP regarding the noncompliance with the property of transitivity and suggested using MAHP to avoid it.Wijnmalen & Wedley (2009a) explained that the RRP is due to the assumption that the criteria weights should remain unchanged when dealing with sub-problems, i.e, the problem is related to the independence axiom of the AHP method and the commensurability, and not only with regard to the addition/removal of alternatives.
Non-discriminating criterion, also called "Wash criteria" (WC), first pointed out by Finan & Hurley (2002), became another important issue described in the literature on RRP and AHP, where the DM is indiferent among the alternatives when they are compared with these criteria.
Liberatore & Nydick (2004) recognized the problem which had been pointed to by Finan & Hurley (2002) and claimed that the problem can be easily solved by reevaluating the relative weights of the criteria after removing WC.This solution also received support from Saaty & Vargas (2006) and Wijnmalen & Wedley (2009b).However, Lin et al. (2008) showed that the proposal of Finan & Hurley (2002) has shortcomings and also showed how to solve it.
In the latest study on WC, Jan et al. (2011) showed that the RRP does not occur when a WC is excluded and if a proportional adjustment to revise the upper level relative weights is applied, as suggested by Liberatore & Nydick (2004) and Saaty & Vargas (2006).They also reported situations of RR in accordance with Finan & Hurley (2002) and Lin et al. ( 2008) and pointed out that every criterion may also be a WC for the invariant phenomenon proposed by Jung et al.
In summary, three versions of AHP were reported in the literature on RRP: AHP, AHP ideal mode and MAHP.There is consensus in our literature review that AHP suffers from RR.As previously described, several authors are in favor of using the AHP ideal mode and MAHP to avoid RR.Stam and Silva (2003) provided a detailed comparison between MAHP and the AHP, they also discussed several properties of the MAHP, besides showing how the MAHP offers a more flexible preference modeling framework.

TOPSIS
Only three of the 130 articles (2.30%) dealt with the RRP in TOPSIS.The first study analyzed was that by Ren et al. (2007), in which a new method called M-TOPSIS was proposed to avoid the RRP.In short, the M-TOPSIS uses the first four steps of the classical TOPSIS as proposed by Hwang & Yoon (1981), but a new index called relative proximity [R i ] is calculated for each alternative.In this step, the authors calculate the distances of each alternative in relation to both the ideal positive (D + − x axis) and negative (D − − y axis) solution for calculating min(D + i ) and max(D − i ) as the optimized ideal reference point.Finally, the final ranking of the alternatives is given by R i in ascending order.However if there are two different alternatives a and b, where ) must be calculated for both alternatives.In this case, the best alternative is the one with the lowest value for R i .
In the article of Kong (2011), the vector normalization procedure was mentioned as the main reason for the RRP in TOPSIS, since the independence of the alternatives may be affected, since alternatives can be related to each other after the normalization.Furthermore, the optimal solution will also change, since the distances of each alternative to the ideal solutions will always change if there is a change in any ideal solution.
To solve this problem, the author stated that the normalization procedure must comply with the independence of the alternatives while maintaining the ideal solution constant.From this hypothesis, the proposed solution determines both the attribute value that is the most satisfactory and the most unsatisfactory.Therefore, the first one is normalized to be one, while the last is normalized to be zero, thereby guaranteeing that the ideal points will never change when alternatives are added or removed.
Similarly to Kong (2011), García-Cascales & Lamata (2012) also indicated that the irregularities in TOPSIS occur due to the procedure of normalization However, in addition, they argued that is also necessary to change the calculation of the positive and negative ideal solutions (PIS and NIS).Thus, the authors proposed a new normalization procedure in absolute terms, not relative, as in the classical TOPSIS method, besides adding two fictitious alternatives with the best and the worst values of each criterion.From this solution, the authors claimed that they had proved that their method avoids the rank reversal.
Finally, García-Cascales & Lamata (2012) demonstrated that both relative and absolute scales suffer from RR.However, the latter generates lower RR rates because there is better control of the independence between alternatives.On the other hand, Aires & Ferreira (2016) pointed out that the linear scale transformation (Max) presented the lowest rates of RR when compared to the other normalization processes, such as linear scale transformation (Max-Min), linear scale transformation (Sum) and vector normalization.According to these authors, the "Max method" demonstrated better results because it increased the range of values of the criteria.

PROMETHEE
Only four of the 130 articles (3.1%) dealt with the RRP in PROMETHEE.Studies of rank reversal in outranking methods are still scarce in the literature.The first authors to address this issue for the PROMETHEE method were De Keyser & Peeters (1996), but they simply pointed out that this problem existed.The second, which was published 17 years later, was presented by Verly & De Smet (2013) to empirically test the probability of RR in the PROMETHE I and II methods based on artificial data sets.These tests were conducted with respect both to the addition or removal of alternatives, and to a non-discriminating criterion, as reported by Finan & Hurley (2002) to AHP and already aforementioned.The results showed that both PROMETHEE suffer from rank reversal.
Additionally, Verly & De Smet (2013) also highlighted that RRP is directly related to situations where the flow differences are relatively small.Moreover, they demonstrated that RR did not occur in this method when a non-discriminating criterion was tested, and the procedure based on distillation was more stable during the experiments than PROMETHEE I was.

ELECTRE
Only three of the 130 articles (2.3%) specifically discussed the RRP in the ELECTRE family.Initially, Zanakis et al. (1998) presented a broader analysis regarding different methods, as described in section 3.5.However, the first study made a detailed analysis of ELECTRE II and III methods with respect to RR was published only in 2008 by Wang & Triantaphyllou (2008) and evaluated them against three criteria: (i) the indication of the best alternative when an alternative "not optimal" is replaced by another worse alternative and should remain the same, while maintaining the relative importance of each criterion; (ii) the transitivity axiom cannot be violated; (iii) the original ranking of the problem and the ranking combination of the decomposed problem should be identical.
Although outranking relations methods may not be transitive, as stated by Figueira & Roy (2009), the authors analyzed 10,000 simulated decision problems for each method.The results showed that RR occurred for the three criteria examined and the reversal rates generally increased when the number of alternatives was increased.In addition, to strengthen their results, the authors also studied ten real cases randomly selected from the published literature, where 60% of the cases failed in criterion 1. 90% failed in the criterion 2, and the only case submitted to criterion 3 also failed.
Later, Figueira & Roy (2009) explained and justified the results presented in Wang & Triantaphyllou (2008).The authors argued that the results obtained with respect to the first criterion do not invalidate the method, they just show its limitations.Moreover, they argued that it well known that methods using outranking relations (not only the ELECTRE methods) do not need to satisfy the transitivity property.According to these authors, "this aspect represents a weakness only if we impose a priori that preferences should be transitive".
Finally, the last paper analyzed, proposed by Figueira et al. (2013), presented an overview of ELECTRE methods and their extensions.It highlighted the main features of these methods, recent applications, new methodological developments, axiomatic analysis, software tools and some weaknesses reported in the literature, including the rank reversal analysis.In this paper, as in Figueira & Roy (2009), the authors relativize the RRP and advocated that RR may occur and, consequently, the property of independence with regard to irrelevant actions can be violated when dealing with outranking relations.

OTHERS
This section presents 21 (16.2%)articles regarding the RRP when two or more MCDM methods are involved in the analysis and experiments.We also consider in this section MCDM methods which were not individually analyzed in previous sections.Initially, we found five papers (Gomes, 1990 Gomes (1990), showed that the TODIM (an acronym in Portuguese for Interactive and Multi-Criteria Decision-Making) suffers from RRP, but it also presents a solution for the problem by using the normalization procedure of Belton & Gear (1983).
The Unlike previous studies, Buede & Maxwell (1995) evaluated different methods with a view to examining the frequency and magnitude of rank disagreements using Multi-Attribute Value Theory (MAVT) as the reference.MAVT, according to the authors, is a good reference because it has strong normative foundations and is immune to rank reversal.The first statistic used by the authors to campare the methods was called fraction of ranking disagreements, where the best option given by MAVT was used to count the number of ranking disagreements in 32,000 iterations for each technique.The second statistic computed was the magnitude of the difference between the option chosen by MAVT and the option chosen by each of the other techniques when a ranking disagreement occurred.The methods AHP, TOPSIS, Policy Goal Percentaging Analysis (Nagel, 1989) and the Fuzzy Algorithm (Yager, 1978) were analyzed when new alternatives were added or existing options were deleted from the problem.
The experiments demonstrated of ranking disgreements cases between AHP and MAVT at small percentage levels, and rare cases of significant differences between MAVT and AHP in the fractional value difference.On the other hand, the other three techniques analyzed (Percentaging, TOPSIS and the Fuzzy Algorithm) reached worse results in terms of both the frequency and the magnitude of the value of the ranking disagreements.
In a similar study, Zanakis et al. (1998) tested the RRP in eight methods: ELECTRE, TOPSIS, MEW (Multiplicative Exponential Weighting), SAW (Simple Additive Weighting), and four different AHP versions.The results concerning the addition of a new alternative in the problem showed that MEW and SAW methods have not produced any rank reversals.This was followed by TOPSIS, the four AHP versions and, finally, ELECTRE.Next, the authors analyzed the effects caused by the number of alternatives, criteria and the distribution of weights.In general, the analysis of the results showed that: (i) more rank reversals occurred in problems with more alternatives; (ii) the number of rank reversals was influenced less by the number of criteria than by the number of alternatives, (iii) more rank reversals were observed under constant weights, and fewer under uniformly distributed weights.In this paper three distributions for weights were simulated: equal weights, uniform distribution, and U-shaped distribution.
However, given the controversy in the MCDM field about comparing rankings from different methods, the results of Buede & Maxwell (1995) with respect to ten criteria in which sustainability assessment tools should be used, such as life cycle perspective, thresholds and uncertainty management, software support and ease of use.Moreover, the authors limited themselves to pointing out the RRP, and the similarities and the differences between the methods.
The last two articles we reviewed for this section were Dede et al. (2015) and Tomashevskii (2015).By using the concepts from Saaty & Vargas (1984), the former analyzed the impact of uncertainty introduced when different experts complete pairwise comparison matrices in the context of fuzzy multicriteria decision making.The RR measure was used in this study as a way of measuring the trustworthiness of the results given that the alternatives are compared pairwise by different DMs.
Finally, the research of Tomashevskii (2015) was dedicated to analyzing the Eigenvector Method (EM) used in pairwise comparison matrices in decision-making processes.The author infers that the errors that occurred related to rank reversal for EM were caused entirely by the inaccuracy of the measurement scale and the inconsistent judgements of experts.The criteria used to evaluate the reservals were the addition or removal of new alternatives.

OTHER CLASSIFICATION RESULTS
This section presents the analysis of the 130 papers according to the following attributes: (i) year of publication; (ii) distribution by journal; (iii) co-authorship network; and (iv) research goal.
The results of the co-authorship network refer only to the AHP method, since this is the only method with enough articles published on RRP to conduct this kind of analysis.

Distribution by year of publication
Table 1 shows the distribution frequency by year for the papers published on RRP in scholarly journals since 1983, from when 23 papers were published to the end of 1991.After that, a total of 34 papers were published between 1992 and 1997.There was a sharp decline to a total of nine papers (6.9%) published between 1998 and 2000, and five (3.8%) between 2001 and 2003.From 2004 to 2015, the number of papers published on RRP has revived, since more than 59 papers (45.4%) of all papers were published within this period.

Distribution of articles by journal
Table 2 provides the distribution of the articles based on the journal in which they appeared.In this review, there were 37 journals with at least one paper published on the RRP.(19).The Omega and Management Science was in the third position, regarding the number ( 9) and percentage (6.9%) of the total papers published.

Co-authorship Network Analysis
This section presents the co-authorship network regarding the RRP in the AHP method, given the number of papers published on this method.Scientific collaboration refers to a broad range of activities, from simple exchanges of opinion to side-by-side work in a laboratory until coauthorship of a paper.Co-authorship is one of the most tangible forms of research collaboration.A coauthorship network is a social network where the interactions can be represented by a dynamically directed network, representing the authors as nodes and the cooperative action of the first author as a link to all other authors (Wardil & Hauert, 2014).
The topology of the co-authorship network of the AHP articles is shown in Figure 2 with 53 articles represented, since the single authors papers were excluded from the analysis.The nodes of the network were represented in accordance with the number of each authored articles, i.e., the higher the number of articles, the more representative and the higher is the author node.The thickness of the lines symbolized the intensity of collaboration among the authors.
As a general comment regarding the results obtained, there is a large predominance of coauthorship that was established for a single article.In terms of productivity, Vargas, Saaty and Wedley were the authors with most articles published, and the coauthorship ratio involving Vargas & Saaty and Wedley & Schoner was the most common.Complementarily, we can also use the centrality measure from social network theory (Abbasi et al., 2012;Wardil & Hauert, 2014) to analyze the results.This measure reports useful information so as to evaluate the authors according to their performances in the social network.Based on the measure of degree centrality, sixteen authors had the highest level of coauthorship.However, when the betweenness centrality measure was analyzed, two of those authors (Saaty and Wedley) were in a good position in the network, this being the shortest path between two other authors.This measure play an important role in controlling the distribution of information in the network.Finally, considering the closeness centrality, Saaty had the minimum distance to other network nodes, representing his effectiveness, centrality and key role in the distribution and circulation of information between other nodes in the network.

Research goal and causes of RR
In this section we present the final results obtained during our SLR.First, we classify the papers studied into five clusters according to their research goal, as described in the following: 1. Survey: when the authors compiled and analyzed the relevant literature of the RRP from different academic databases.
2. Application: when the authors applied some multicriteria decision-making method and used decision theory and ranking reversal assumptions to improve the reliability of the results produced.
3. Problem Solution: when the authors designed a new procedure to solve the RRP.
Pesquisa Operacional, Vol.38(2), 2018 4. Simulation: when the authors investigated the performance of different multicriteria procedures or methods by using a set of simulated problems.
5. Problem identification: when the authors studied the nature of the problem, and outlined its causes and conditions.
Table 3 presents the classification of all analyzed papers in this review into each cluster, where the majority of the articles studied the nature of the problem (61.54%), by outlining its causes and conditions.Moreover, disregarding the widely explored AHP method, it appears that the main gap in relation to rank reversal state of the art is the absence of a clear and convincing explanation of its cause, reinforcing that the RRP is still an open problem, despite all the research already carried out and which have been analyzed in this study.
Lastly, the faults in the normalization procedure and the way to obtain the ideal solutions (see, for example, Ren et al. (2007), Kong (2011), and García-Cascales & Lamata (2012)) are the main reasons of RR in TOPSIS, while the distillation Processes (see, for example, Wang & Triantaphyllou (2008)) and flow-score difference are the cited causes of RR in ELECTRE and PROMETHEE, respectively.

CONCLUSION
This paper has provided an extensive literature review on MCDM methodologies and rank reversals.We aimed to analyse the collected articles in accordance with the following aspects: the multicriteria technique, the issues discussed, the year and journal in which an article was published, the coauthorship network, and the research goal.A total of 130 journal articles were carefully selected and reviewed in detail.Thus, this paper provides additional valuable knowledge on current studies and makes recommendations for future studies.
Regarding the main results obtained, first of all, the approach which tackled RPP most was AHP and the addition and/or removal of irrelevant alternatives was the most used analysis criterion.
In addition, four other situations were also used, as follows: alteration of the indication of the best alternative, decomposition of decision problem, the non-discriminating criterion and the transitivity property.
Second, Vargas, Saaty and Wedley were the authors with the most articles on the RRP, the European Journal of Operational Research was the journal with the most articles published (17.7%), and the period between 1995 and 1997 was the most prolific period of productions on the RRP.
Third, the main gap regarding the RRP seems to be the absence of a convincing explanations of its cause, as well as robust solutions for different methods, such as ELECTRE, PROMETHEE, TOPSIS.Fourth, the revised papers were classified into five clusters: survey, application, problem solution, simulation, and problem identification.
For future studies, the results presented in this study may be increased by analyzing other research papers published in conference proceedings, master's and doctoral theses.Furthermore, it is suggested that new studies evaluate the occurrence of rank reversal for that more focused studies be conducted to investigate the conditioning factors for the occurrence of rank reversal, in order to really enhance and develop the discussion related to this issue.Saaty & Vargas (1993) Evaluation of inconsistencies and ranking preservation.

Finally, the study
of Maleki & Zahir (2013) reported a literature review on RRP and examined 61 articles related to AHP from 18 international journals.In this paper we have updated the SLR conducted by Maleki by analyzing 130 paper on RR and AHP.Examples of papers we have considered different from Maleki are: Totsenko (2006), Saaty & Vargas (2009), Troutt, Tadisina & Pendharkar (2009), Jan et al. (2011) and Kulakowski (2015).
Brans & De Smet (2016) responded to Verly & De Smet (2013) a view to showing that the PROMETHEE's rankings are not influenced by the presence or elimination of a non-discriminating criterion, while they may be affected by copies of alternatives.They also pointed out that the ranking between two alternatives a and b in PROMETHEE II remains the same with the removal of a third alternative if| φ + (a) − φ + (b) |> 1 n−1 and | φ − (a) − φ − (b) |> 1 n−1 .On the other hand, for the PROMETREE I, they demonstrated that the ranking between a and b remains the same if | φ + (a) − φ + (b) |> 1 n−1 and | φ − (a) − φ − (b) |> 1 n−1 .

Figure 2 -
Figure 2 -Co-authorship network on rank reversal in AHP.
Bairagi et al. (2015)011), 2014)was a comparative analysis between the REMBRANDT (Ratio Estimation in Magnitudes or Deci-Bells to Rate Alternatives which are Non-Dominated) and the classical AHP method, where the authors demonstrated that the former is more consistent than AHP in terms of rank reversal.This result is reinforced in theMaleki & Saadat (2013)article.The fourth study(Soltanifar & Shahghobadi, 2014)demonstrated for the first time the RRP in DEA (Data Envelopment Analysis), by showing the occurrence or non-occurrence of rank reversal when a copy of a DMU is added to the problem.Salo & Punkka (2011) is another study that deals with the RRP for the DEA.The goal of Bulut, Duru & Koc ¸ak (2015) was to improve the applicability of the fuzzy-AHP method by using rotational priority investigation (RPI) to eliminate inconsistent results.Ic ¸(2014) conducted different experiments with a new method called DoE-TOPSIS, which proved the original ranking.Bottero & Ferretti (2011)analyzed the contribution of the ANP (Analytic Network Process) technique in sustainability assessment of undesirable facilities.The authors suggested the combined use of ANP and BOCR (Benefits, Opportunities, Costs and Risks) model.To demonstrate the improvements of the approach proposed, they conducted a sensitivity analysis to estimate cases of RR.Bairagi et al. (2015)proposed a new model called "Technique of Precise Order Preference (TPOP)" to overcome the RRP based by using an advanced version of the entropy weighting method.
lowing evaluation situations: (i) the addition of a new alternative, and (ii) the addition of a new criterion.On the other hand, Fuzzy-AHP failed in both evaluation situations.

Table 1 -
Frequency of papers by year of publication.
Of these 45 journals, 30 contributed with only one paper, and 15 journals contributed with more than one paper.The European Journal of Operational Research (EJOR) published the largest number of articles, nomely, 23 out of the 130 articles reviewed (17.7%).There was not a significative difference in terms of published papers between the EJOR and the Journal of Multi-Criteria Decision Analysis

Table 2 -
Distribution of the selected articles by journal.

Table 3 -
Classification of reviewed papers.
Buede & Maxwell (1995)004)38(2), 2018 RENAN FELINTO DE FARIAS AIRES and LUCIANO FERREIRA 361Saaty & Sagir (2009)Preserve rank in all situations may be wrong.Wijnmalen & Wedley (2009a)RRP is caused by the independence axiom.Wijnmalen & Wedley (2009b)Similar toLiberatore & Nydick (2004)andSaaty & Vargas (2006).Criticizes of the way of using value functions in the evaluation of portfolio project's by showing it can lead to a rank reversal.Tomashevskii (2015)RRP is caused by the inaccuracy of the measurement scale and by the inconsistent judgements of experts.Dede et al. (2015)Evaluation of probability ocorrence of the RRP in the context of pairwise comparisons.Buede & Maxwell (1995)Examination of the frequency and magnitude of the RRP in four different methods.