# Abstract

## Paper Aims

This paper aims to develop an approach to support group decision making combining methods and tools to a holistic MCDA process.

## Originality

Authors have been using Value-Focused Thinking (VFT) for structuring problems with different MCDA methods, but there is a lack of a process that defines a clear transition from VFT to those methods. Here we propose a process to fill this gap.

## Research method

Rich Picture and VFT structure the problem and elicit objectives that become criteria within a decision hierarchy. Analytic Hierarchy Process (AHP) with ratings supports preference elicitation and sensitivity analysis in the judgment weights of decision-makers.

## Main findings

VFT is effective for eliciting the decision structure to AHP; using weight distribution of stakeholders may affect the results, and the multimethodology approach developed here can deal with group decision making.

## Implications for theory and practice

The approach developed is effective in complex environments (complex problems and multiple stakeholders) because it focuses on values and defines a process to bring those values into a multicriteria method. Furthermore, sensitivity analysis with the judgment weights of the different stakeholders may be useful in negotiation.

Keywords
MCDA process; Multimethodology; Ranking problematic; Group preference aggregation

# 1. Introduction

The Brazilian Air Force's mission is: "Maintain the sovereignty of the airspace and integrate the national territory, with a view to the defense of the country boundaries" (Brasil, 2018Brasil. DCA 11-45: concepção estratégica força aérea 100. (2018). Brasília. Retrieved in 2021, May 31, from https://www.fab.mil.br/Download/arquivos/DCA%2011-45_Concepcao_Estrategica_Forca_Aerea_100.pdf.
, p. 20). To achieve this mission, the Brazilian Air Force needs the readiness to employ means that are part of complex systems limited in resources. Thus, it is necessary to use such resources most efficiently.

Air Forces operations are intrinsically complex. This affirmation drives to constant development and improvement of all the readiness structures. To achieve this goal Brazilian Air Force has a tool called Operational Assessment (represented by acronym AVAOP), and Readiness Command is accountable for it. This command has subordinated units, where the operational resources are prepared and trained. One of them, responsible for performing AVAOPs, is the Institute of Operational Applications, based at Sao Jose dos Campos, Sao Paulo state. At that institute, “the staff is trained and skilled in solving complex problems and generating solutions with mathematical models and programming, statistics, and stochastic processes”. (Brasil, 2019Brasil. DCA 11-1: sistemática de planejamento e gestão institucional da aeronáutica. Volume 1 – Planejamento. (2019). Brasília. Retrieved in 2021, May 31, from https://www.fab.mil.br/Download/arquivos/prestacaodecontas/DCA_11_1_2019_SPGIA.pdf
, p. 2).

AVAOPs arise from the identification of Operational Deficiencies by Readiness Command subordinate units, and they are formalized through the Identification Form of Operational Deficiencies. To generate an effective flow of resolutions of the identified deficiencies and resource allocation, a prioritization of AVAOPs is necessary. The model to accomplish this objective, used Value-Focused Thinking (VFT) to structure the problem, and Multicriteria Decision Aiding (MCDA). Another research question is: how to combine methods to decision-making that reflect stakeholders' values and make the transition from VFT to a method for MCDA? Thus, the development of a model to answer that question emerges as the motivation of this work.

The responsible sector is constituted by a group of specialists and the decision-maker, a general officer, who wants the group's opinion and judgment about the alternatives over the attributes. Therefore, this work explores an approach with methods and tools to surface the subjectivity of the decision and make the evaluation replicable, flexible, and robust. It uses Rich Picture and Value-Focused Thinking (VFT) to structure the problem and define a process to elicit a decision hierarchy from a fundamental objective hierarchy from VFT. The decision hierarchy supports the multicriteria decision making and Analytic Hierarchy Process (AHP) is explored with multiple stakeholders, aiding the decision as Aguarón et al. (2019)Aguarón, J., Escobar, M. T., Moreno-Jiménez, J. M., & Turón, A. (2019). AHP-group decision making based on consistency. Mathematics, 7(3), 242. http://dx.doi.org/10.3390/math7030242.
http://dx.doi.org/10.3390/math7030242...
propose.

This study objective was to create an approach for supporting group decision making combining methods and tools to a completely MCDA process, as proposed by Belton & Stewart (2002)Belton, V., & Stewart, T. J. (2002). Multiple criteria decision analysis: an integrated approach. Boston: Springer US. http://dx.doi.org/10.1007/978-1-4615-1495-4.
http://dx.doi.org/10.1007/978-1-4615-149...
. Thus, facilitating the replicability of a multimethodological approach. It consists of five sections: the first one introduces the objectives and motivation; the second section shows a brief literature review of methods and tools, combined here through a multimethodological process, that supports the approach developed and to it be replicable; section 3 shows the application in the case study; the results are shown in section 4, and section 5 brings a discussion and conclusions about the approach.

# 2. Literature review

First, this section presents Problem Structuring Methods (PSMs) and Multicriteria Decision Aiding (MCDA) – the areas in this study – and then, methods and tools that support the intervention.

## 2.1. PSMs

Problem Structuring Methods (PSMs) are a particular area of Soft Operational Research, which considers a situation holistically and ensures sustainability to the proposed recommendations (Ackermann, 2012Ackermann, F. (2012). Problem structuring methods in the dock: arguing the case for soft OR. European Journal of Operational Research, 219(3), 652-658. http://dx.doi.org/10.1016/j.ejor.2011.11.014.
http://dx.doi.org/10.1016/j.ejor.2011.11...
). They were developed between the 1960s and 1980s to tackle complex problems, solving them at least partially (Mingers & Rosenhead, 2004Mingers, J., & Rosenhead, J. (2004). Problem structuring methods in action. European Journal of Operational Research, 152(3), 530-554. http://dx.doi.org/10.1016/S0377-2217(03)00056-0.
http://dx.doi.org/10.1016/S0377-2217(03)...
). To achieve this goal, PSMs shall: 1) enable many alternatives to be brought together into the problem-solution; 2) develop a diagramming that be cognitively accessible to stakeholders with different perspectives and knowledge; 3) operate interactively, allowing adjustments to the representation to the state and stage of discussion among stakeholders and vice-versa; and 4) permit partial or local improvements, rather than global solutions, which marginalize many interests (Mingers & Rosenhead, 2004Mingers, J., & Rosenhead, J. (2004). Problem structuring methods in action. European Journal of Operational Research, 152(3), 530-554. http://dx.doi.org/10.1016/S0377-2217(03)00056-0.
http://dx.doi.org/10.1016/S0377-2217(03)...
).

When providing holistic visualization about the problem, methods that structure the situation, make possible the elicitation of potential solutions. They were already extensively reported in the literature (Rosenhead, 1989Rosenhead, J. (1989). Rational analysis for a problematic world: problem structuring methods for complexity, uncertainty, and conflict. New York: John Wiley., 2006Rosenhead, J. (2006). Past, present, and future of problem structuring methods. The Journal of the Operational Research Society, 57(7), 759-765. http://dx.doi.org/10.1057/palgrave.jors.2602206.
http://dx.doi.org/10.1057/palgrave.jors....
; Mingers & Rosenhead, 2004Mingers, J., & Rosenhead, J. (2004). Problem structuring methods in action. European Journal of Operational Research, 152(3), 530-554. http://dx.doi.org/10.1016/S0377-2217(03)00056-0.
http://dx.doi.org/10.1016/S0377-2217(03)...
; Ackermann & Eden, 2011Ackermann, F., & Eden, C. (2011). Strategic management of stakeholders: theory and practice. Long Range Planning, 44(3), 179-196. http://dx.doi.org/10.1016/j.lrp.2010.08.001.
http://dx.doi.org/10.1016/j.lrp.2010.08....
; Ackermann, 2012Ackermann, F. (2012). Problem structuring methods in the dock: arguing the case for soft OR. European Journal of Operational Research, 219(3), 652-658. http://dx.doi.org/10.1016/j.ejor.2011.11.014.
http://dx.doi.org/10.1016/j.ejor.2011.11...
; Abuabara et al., 2019Abuabara, L., Paucar-Caceres, A., & Burrowes-Cromwell, T. (2019). Consumers values and behavior in the Brazilian coffee-in-capsules market: promoting circular economy. International Journal of Production Research, 57(23), 7269-7288. http://dx.doi.org/10.1080/00207543.2019.1629664.
http://dx.doi.org/10.1080/00207543.2019....
; Dyson et al., 2021Dyson, R. G., O’Brien, F. A., & Shah, D. B. (2021). Soft OR and practice: the contribution of the founders of operations research. INFORMS, 69(3), 727-738. http://dx.doi.org/10.1287/opre.2020.2051.
http://dx.doi.org/10.1287/opre.2020.2051...
; Abuabara & Paucar-Caceres, 2021Abuabara, L., & Paucar-Caceres, A. (2021). Surveying applications of strategic options development and analysis (SODA) from 1989 to 2018. European Journal of Operational Research, 292(3), 1051-1065. http://dx.doi.org/10.1016/j.ejor.2020.11.032.
http://dx.doi.org/10.1016/j.ejor.2020.11...
) and their analytical assistance is characterized by: multiple actors; different perspectives; conflicting interests; important intangibles; and key uncertainties.

Françozo et al. (2021)Françozo, R., Paucar-Caceres, A., & Belderrain, M. C. N. (2021). Combining Value-Focused thinking and soft systems methodology: A systemic framework to structure the planning process at a special educational needs school in Brazil. The Journal of the Operational Research Society. In press. http://dx.doi.org/10.1080/01605682.2021.1880298.
http://dx.doi.org/10.1080/01605682.2021....
have presented VFT in a problem structuring context as done here. It will be discussed forward its use to an MCDA model as proposed by Almeida et al. (2015)Almeida, A. T., Cavalcante, C. A. V., Alencar, M. H., Ferreira, R. J. P., Almeida-Filho, A. T., & Garcez, T. V. (2015). Multicriteria and multiobjective models for risk, reliability and maintenance decision analysis (1. ed.). Recife: Springer. http://dx.doi.org/10.1007/978-3-319-17969-8.
http://dx.doi.org/10.1007/978-3-319-1796...
.

## 2.2. MCDA

With many possible solutions and multiple objectives to be achieved, it is necessary to develop a model for decision aiding, which will consider the conclusions from the PSM application. Thus, a method for MCDA shall be used – support the decision-making with multiple objectives, being some of those conflicting (Almeida et al., 2015Almeida, A. T., Cavalcante, C. A. V., Alencar, M. H., Ferreira, R. J. P., Almeida-Filho, A. T., & Garcez, T. V. (2015). Multicriteria and multiobjective models for risk, reliability and maintenance decision analysis (1. ed.). Recife: Springer. http://dx.doi.org/10.1007/978-3-319-17969-8.
http://dx.doi.org/10.1007/978-3-319-1796...
).

Belton & Stewart (2002)Belton, V., & Stewart, T. J. (2002). Multiple criteria decision analysis: an integrated approach. Boston: Springer US. http://dx.doi.org/10.1007/978-1-4615-1495-4.
http://dx.doi.org/10.1007/978-1-4615-149...
present a way of using PSM to structure the problem before specifying alternatives, defining criteria, and eliciting values, because the problem is not "given" for being tackled, it is a messy situation. Figure 1 illustrates two phases of the designed decision support process. Initially, a divergent phase – data collection during the problem structuring – is used to capture the situation complexity and to capture emerging properties of the system. After this moment, a convergent phase takes place and the model is used to put the complexity more objectively, supporting the alternatives assessment and the action plan (Belton & Stewart, 2002Belton, V., & Stewart, T. J. (2002). Multiple criteria decision analysis: an integrated approach. Boston: Springer US. http://dx.doi.org/10.1007/978-1-4615-1495-4.
http://dx.doi.org/10.1007/978-1-4615-149...
; Montibeller et al., 2008Montibeller, G., Belton, V., Ackermann, F., & Ensslin, L. (2008). Reasoning maps for decision aid: an integrated approach for problem-structuring and multi-criteria evaluation. The Journal of the Operational Research Society, 59(5), 575-589. http://dx.doi.org/10.1057/palgrave.jors.2602347.
http://dx.doi.org/10.1057/palgrave.jors....
).

Figure 1
Two phases of the decision support process. Adapted from Montibeller et al. (2008)Montibeller, G., Belton, V., Ackermann, F., & Ensslin, L. (2008). Reasoning maps for decision aid: an integrated approach for problem-structuring and multi-criteria evaluation. The Journal of the Operational Research Society, 59(5), 575-589. http://dx.doi.org/10.1057/palgrave.jors.2602347.
http://dx.doi.org/10.1057/palgrave.jors....
and Abuabara et al. (2019)Abuabara, L., Paucar-Caceres, A., & Burrowes-Cromwell, T. (2019). Consumers values and behavior in the Brazilian coffee-in-capsules market: promoting circular economy. International Journal of Production Research, 57(23), 7269-7288. http://dx.doi.org/10.1080/00207543.2019.1629664.
http://dx.doi.org/10.1080/00207543.2019....
.

Four types of problematic are defined by Roy (1996)Roy, B. (1996). Multicriteria methodology for decision aiding. Boston: Springer. http://dx.doi.org/10.1007/978-1-4757-2500-1.
http://dx.doi.org/10.1007/978-1-4757-250...
: P.α (choice problematic), refers to the most traditional one, which "the best option" is chosen and is the base for optimization procedures; P.β (sorting problematic) lines up each action to a specific category – the effort is driven to the determination of intrinsic action values, finding your category; P.γ (ranking problematic), help rank actions in order of preferences or build an ordering procedure; e P.δ (description problematic), describe actions and their consequences in a formalized and systematic manner or develop a cognitive procedure.

Belton & Stewart (2002)Belton, V., & Stewart, T. J. (2002). Multiple criteria decision analysis: an integrated approach. Boston: Springer US. http://dx.doi.org/10.1007/978-1-4615-1495-4.
http://dx.doi.org/10.1007/978-1-4615-149...
bring two more types: design problematic, to look for, identify, and create alternatives; and portfolio problematic, to choose the group of alternatives for achieving the objectives subject to some restrictions.

This study searches for a prioritization, order the alternatives for preference, thereby, the ranking problematic, considering that the stakeholders might have different weights in the discussion. Next, the tools and methods utilized in both phases, divergent and convergent, will be explored.

## 2.3. The multimethodological process

As Midgley (2015)Midgley, G. (2015). Systemic intervention: research memorandum 95: Hull University of Business School. Retrieved in 2021, May 31, from https://hull-repository.worktribe.com/output/385209
https://hull-repository.worktribe.com/ou...
discusses, methodological pluralism allows "to breach the castle walls" of methodologies, and if they are seen as dynamics, one can learn from the other on an ongoing basis. Similarly, Mingers & Brocklesby (1997)Mingers, J., & Brocklesby, J. (1997). Multimethodology: towards a framework for mixing methodologies. Omega, 25(5), 489-509. http://dx.doi.org/10.1016/S0305-0483(97)00018-2.
http://dx.doi.org/10.1016/S0305-0483(97)...
say that one multimethodology principle is to use more than one methodology or part of it, from different paradigms, in the same intervention. Figure 2 shows synthetically the process that will guide this work, inspired by the theory proposed by the authors cited.

Figure 2
The multimethodological process.

### 2.3.1. Divergent phase

This phase begins with stakeholder identification, which is an important task for all interventions and the decision-making. Ackermann & Eden (2011)Ackermann, F., & Eden, C. (2011). Strategic management of stakeholders: theory and practice. Long Range Planning, 44(3), 179-196. http://dx.doi.org/10.1016/j.lrp.2010.08.001.
http://dx.doi.org/10.1016/j.lrp.2010.08....
developed a tool for stakeholder analysis that may be taken as a robust, leading example of a typical problem structuring approach grounded in interpretivism (Gregory et al., 2020Gregory, A. J., Atkins, J. P., Midgley, G., & Hodgson, A. M. (2020). Stakeholder identification and engagement in problem structuring interventions. European Journal of Operational Research, 283(1), 321-340. http://dx.doi.org/10.1016/j.ejor.2019.10.044.
http://dx.doi.org/10.1016/j.ejor.2019.10...
). The analysis is based on the power-interest diagram (Figure 3), which the higher on "y-axis", higher is the level of actor's participation (interest), and the farther right on the "x-axis", higher is the influence (power) that affect the decision, being or not interested in the activities.

Figure 3
Power-interest diagram. Adapted from Ackermann & Eden (2011)Ackermann, F., & Eden, C. (2011). Strategic management of stakeholders: theory and practice. Long Range Planning, 44(3), 179-196. http://dx.doi.org/10.1016/j.lrp.2010.08.001.
http://dx.doi.org/10.1016/j.lrp.2010.08....
.

There are four quadrants in the Power-interest diagram: "Players" with a high degree of power e high degree of participation (interest) during the process; "Subjects" with high interest, but a low degree of influence over the decision; "Context Setters" with a high degree of power to perform future changes in the organization, however with low interest; and "Crowd" with a low degree of participation and low degree of influence on decision results. The sorting of quadrants is not statistical, it varies depending on the formal and informal relationships in the organization.

To understand the context, and capture the emergent properties, the Rich Picture tool was developed by Checkland (1981)Checkland, P. (1981). Systems thinking, systems practice. New York: John Wiley. as part of Soft Systems Methodology (SSM). It provides a problematic situation's diagrammatic view, making explicit the system entities, likewise, its structures and stakeholders perspectives over the situation (Georgiou, 2015Georgiou, I. (2015). Unraveling soft systems methodology. International Journal of Economics and Business Research, 9(4), 415-436. http://dx.doi.org/10.1504/IJEBR.2015.069680.
http://dx.doi.org/10.1504/IJEBR.2015.069...
; Checkland & Poulter, 2020Checkland, P., & Poulter, J. (2020). Soft systems methodology. In M. Reynolds & S. Holwell (Eds.), Systems approaches to making change: a practical guide (pp. 201-253). London: Springer. http://dx.doi.org/10.1007/978-1-4471-7472-1_5.
http://dx.doi.org/10.1007/978-1-4471-747...
).

Rich Picture could be part of an intervention, as Abuabara et al. (2019)Abuabara, L., Paucar-Caceres, A., & Burrowes-Cromwell, T. (2019). Consumers values and behavior in the Brazilian coffee-in-capsules market: promoting circular economy. International Journal of Production Research, 57(23), 7269-7288. http://dx.doi.org/10.1080/00207543.2019.1629664.
http://dx.doi.org/10.1080/00207543.2019....
did, to start the discussion over the different group perspectives, into a broader view of the properties that emerge from the relationships of the system parts. Similarly, Monk & Howard (1998)Monk, A., & Howard, S. (1998). Methods & tools: the rich picture: a tool for reasoning about work context. Interaction, 5, 21-30. http://dx.doi.org/10.1145/274430.274434.
http://dx.doi.org/10.1145/274430.274434...
said that Rich Picture depicts stakeholders relationships and interests, and, to be effective, it needs five elements: 1) Structure, referring to the context that will be modified, including geographical boundaries, for example, and people involved in the situation (the most important structure); 2) Process, and the situation dynamic, however without depicting the complexity, focusing on what will be transformed; 3) Concerns from stakeholders and different perspectives; 4) The language of the people depicted in it, and 5) Any drawing technique or textual device that suits their purpose.

This construction helps to understand the problem in a systemic manner (Georgiou, 2015Georgiou, I. (2015). Unraveling soft systems methodology. International Journal of Economics and Business Research, 9(4), 415-436. http://dx.doi.org/10.1504/IJEBR.2015.069680.
http://dx.doi.org/10.1504/IJEBR.2015.069...
), and this is the main reason for its use here. After it, VFT is implemented, shaping the desirable consequences to decision-making.

According to Keeney (1996b)Keeney, R. L. (1996b). Value-focused thinking: Identifying decision opportunities and creating alternatives. European Journal of Operational Research, 92(3), 537-549. http://dx.doi.org/10.1016/0377-2217(96)00004-5.
http://dx.doi.org/10.1016/0377-2217(96)0...
, the thinking focused on values is fundamental to everything that people wish to accomplish and shall be the focus of effort when making decisions. This focus on values is the biggest differential of VFT because it concentrates on two main activities: defining what the decision-maker(s) wants and exploring how to achieve this (Keeney, 1996aKeeney, R. L. (1996a). Value-focused thinking: a path to creative decisionmaking. London: Harvard University Press Cambridge.).

Values are used to assess actual or potential alternatives proposed (Parnell & West, 2008Parnell, G. S., & West, P. D. (2008). Value-focused systems decision making. In Proceedings of the 18th Annual International Symposium of the International Council on Systems Engineering (pp. 1685-1699). Utrecht: INCOSE.). They can be presented by objectives, which shall be achieved and are composed by a verb on infinitive and an object – these objectives are “value bits”, guiding the decision-making process and creating decision opportunities not visualized before (Keeney, 1996a; 2008Keeney, R. L. (1996a). Value-focused thinking: a path to creative decisionmaking. London: Harvard University Press Cambridge.).

Interactions are needed during the accomplishment of a VFT approach. In those interactions, the analyst – or facilitator, as defended by Franco & Montibeller (2010)Franco, L. A., & Montibeller, G. (2010). Facilitated modeling in operational research. European Journal of Operational Research, 205(3), 489-500. http://dx.doi.org/10.1016/j.ejor.2009.09.030.
http://dx.doi.org/10.1016/j.ejor.2009.09...
– elicits the objectives using some devices, such as questions that guide the interview, to stimulate the deepest interests and desires. The output is the hierarchy of fundamental objectives and means-ends network. Fundamental objectives are the translation of the desirable consequences that stakeholders are concerned about and are used to assess the alternatives, whereas the means objectives influence, to some degree, the achievement of such fundamental objectives (Keeney, 2008Keeney, R. L. (2008). Applying value-focused thinking. Military Operations Research, 13(2), 7-17. http://dx.doi.org/10.5711/morj.13.2.7.
http://dx.doi.org/10.5711/morj.13.2.7...
).

An important task traced by Keeney (1996a,; 2007; 2008Keeney, R. L. (1996a). Value-focused thinking: a path to creative decisionmaking. London: Harvard University Press Cambridge.) is the fundamental objectives specification, which shall be done until the alternatives could be assessed concerning them (Keeney, 2007Keeney, R. L. (2007). Developing objectives and attributes. In W. Edwards, R. F. Milles & D. Von Winterfeldt (Eds.), Advances in decision analysis: from foundations to applications (pp. 104-128). Cambridge: Cambridge University Press http://dx.doi.org/10.1017/CBO9780511611308.008.
http://dx.doi.org/10.1017/CBO97805116113...
). They must be specified until an attribute could be assigned to each of the lower-level objectives. If this isn't possible, the alternatives must be assessed through a natural attribute from a mean objective, however, this is not desirable.

Attributes describe the achievement degree of consequences and are measured in each objective (Keeney, 2007Keeney, R. L. (2007). Developing objectives and attributes. In W. Edwards, R. F. Milles & D. Von Winterfeldt (Eds.), Advances in decision analysis: from foundations to applications (pp. 104-128). Cambridge: Cambridge University Press http://dx.doi.org/10.1017/CBO9780511611308.008.
http://dx.doi.org/10.1017/CBO97805116113...
) and consist of three types: natural, construct, or proxy. Natural attributes are a common interpretation: "minimize financial costs" could be the amount of money. On the other hand, construct attributes are developed to measure directly the achievement of an objective. Finally, proxies attribute are used in cases where it is not possible to acquire information about an objective and shall be avoided because they measure the achievement of a mean objective, and this could generate double-counting – one mean objective can be linked in more than one fundamental, for example (Keeney, 1996aKeeney, R. L. (1996a). Value-focused thinking: a path to creative decisionmaking. London: Harvard University Press Cambridge.).

### 2.3.2. Convergent phase

Considering the creation of new alternatives, because the AVAOPs flow – new deficiencies are identified during the months – and for any other kinds of problems, it is assumed that the model shall accept the addition of new options, alternatives. And Saaty (2006)Saaty, T. L. (2006). Rank from comparisons and ratings in the analytic hierarchy/network processes. European Journal of Operational Research, 168(2), 557-570. http://dx.doi.org/10.1016/j.ejor.2004.04.032.
http://dx.doi.org/10.1016/j.ejor.2004.04...
in his argument about AHP with ratings, exposes that, if new alternatives arise, the older ranking remains the same, requiring a smaller number of pairwise comparisons, which would be exhausting otherwise (Mu & Pereyra-Rojas, 2018Mu, E., & Pereyra-Rojas, M. (2018). Practical decision making using super decisions: an introduction to the Analytic Hierarchy Process. Cham: Springer. http://dx.doi.org/10.1007/978-3-319-68369-0.
http://dx.doi.org/10.1007/978-3-319-6836...
). To assess the alternatives, they must be independent of one another, or, at least, their independence must be assumed, preserving the rank (Saaty, 2008Saaty, T. L. (2008). Decision making with the Analytic Hierarchy Process. Scientia Iranica, 9(3), 215-229.).

AHP implementation can be addressed following four steps (Saaty, 2008Saaty, T. L. (2008). Decision making with the Analytic Hierarchy Process. Scientia Iranica, 9(3), 215-229.): 1) define the problem and explore the kind of knowledge sought; 2) structure the decision hierarchy, starting from the top with decision goal, criteria and subcriteria composed by the other intermediate objectives; 3) construct a set of pairwise comparisons; and 4) use priorities from the pairwise comparison to weigh the priorities in the immediately lower level. The implementation can be summarized as shown in Figure 4 and depicted forward.

Figure 4
AHP process with VFT in the first step.

During the first step, criteria and subcriteria are identified exploring the decision-maker's values and concerns (Parnell & West, 2008Parnell, G. S., & West, P. D. (2008). Value-focused systems decision making. In Proceedings of the 18th Annual International Symposium of the International Council on Systems Engineering (pp. 1685-1699). Utrecht: INCOSE.). At this moment, the problem is depicted, surfacing the problem complexity and capturing it to manage the situation and understand how decision-makers shall go forward (Belton & Stewart, 2002Belton, V., & Stewart, T. J. (2002). Multiple criteria decision analysis: an integrated approach. Boston: Springer US. http://dx.doi.org/10.1007/978-1-4615-1495-4.
http://dx.doi.org/10.1007/978-1-4615-149...
). As described before, in this step the problem was structured using a Rich Picture, to give a broader and systemic view over the situation. And VFT, to explore the values, and go further into the transition from this step to the next.

In this study, the transition from 1st step to 2nd step meant the transformation of the hierarchy of fundamental objectives into the decision hierarchy structure and this process can be visualized in Figure 5. For many authors, those criteria and attributes are synonyms (Keeney, 1996aKeeney, R. L. (1996a). Value-focused thinking: a path to creative decisionmaking. London: Harvard University Press Cambridge.), but depicting the definitions from this literature review is plausible to say that they are distinct things. Criteria are something that influences the decision, an objective itself, and attribute measures the achievement of an objective.

Figure 5
Transition from VFT (fundamental objective hierarchy) to AHP (or any other MCDA method that uses a decision hierarchy).

For example, Almeida et al. (2015)Almeida, A. T., Cavalcante, C. A. V., Alencar, M. H., Ferreira, R. J. P., Almeida-Filho, A. T., & Garcez, T. V. (2015). Multicriteria and multiobjective models for risk, reliability and maintenance decision analysis (1. ed.). Recife: Springer. http://dx.doi.org/10.1007/978-3-319-17969-8.
http://dx.doi.org/10.1007/978-3-319-1796...
said that in organizations there are many factors related to the decision process, and these factors, when methodologically formulated, are multiple objectives (multicriteria), being only necessary to measure their achievement. Attributes, according to Keeney (2007)Keeney, R. L. (2007). Developing objectives and attributes. In W. Edwards, R. F. Milles & D. Von Winterfeldt (Eds.), Advances in decision analysis: from foundations to applications (pp. 104-128). Cambridge: Cambridge University Press http://dx.doi.org/10.1017/CBO9780511611308.008.
http://dx.doi.org/10.1017/CBO97805116113...
, measure the achievement of an objective, meaning compliance of the formerly stated. In other words, it can be concluded that criteria are objectives themselves, and a metric linked in it is the attribute – here the ratings of the hierarchy structure are the (constructed) attributes.

In the end, the hierarchy structure defines the global objective, the decision itself, while the criteria and subcriteria are the fundamental objectives from the second level until the lowest, and the ratings reflect the attributes. It is worthy to indicate the main difference from AHP with ratings from traditional AHP: with ratings – intensity levels or categories –, the alternatives are assessed individually concerning each subcriteria or criteria, whereas, when using traditional AHP, it is necessary a pairwise comparison with all alternatives.

Thus, in the 3rd step, when using ratings, there is not a pairwise comparison between alternatives concerning subcriteria or criteria. In this step, these matrices are built with the judgments provided by the decision-makers using Saaty's Fundamental Scale (Saaty, 1977; 1990; 1994; 2001; 2006; 2008Saaty, T. L. (1977). A scaling method for priorities in hierarchical structures. Journal of Mathematical Psychology, 15(3), 234-281. http://dx.doi.org/10.1016/0022-2496(77)90033-5.
http://dx.doi.org/10.1016/0022-2496(77)9...
). Each element is compared concerning the upper level. That is, ratings are compared to subcriteria, the subcriteria are compared to criteria, and criteria are compared to global objective.

Pairwise comparison is illustrated by the matrix A, where A1, A2, and An are being compared: if the element a12 receives the value 3 from Fundamental Scale, meaning that A1 is moderately important if compared to A2, the element a21 receives 1/3.

$A 1 A 2 A n A = A 1 A 2 A n a 1 / a 1 a 1 / a 2 a 1 / a n a 2 / a 1 a 2 / a 2 ⋯ a 2 / a n ⋮ a n / a 1 a n / a 2 a n / a n$ (1)

AHP does not assume transitivity, i.e., if an alternative A is preferred to B, and B is preferred to B, then A is preferred to C. The method admits some inconsistency in the decision-maker judgment process and lower order of magnitude – 10 percent – may be considered tolerable (Saaty, 1994Saaty, T. L. (1994). How to make a decision: the analytic hierarchy process. Interfaces, 24(6), 19-43. http://dx.doi.org/10.1287/inte.24.6.19.
http://dx.doi.org/10.1287/inte.24.6.19...
).

Transitivity is measured through Consistency Index (CI) (Saaty, 1977; 1990; 1994; 2001; 2006; 2008Saaty, T. L. (1977). A scaling method for priorities in hierarchical structures. Journal of Mathematical Psychology, 15(3), 234-281. http://dx.doi.org/10.1016/0022-2496(77)90033-5.
http://dx.doi.org/10.1016/0022-2496(77)9...
), presented by the equation (λmax - n)/(n - 1), where λmax is the largest eigenvalue (mean of elements from product matrix of pairwise comparison matrix and the priority matrix), and n is the number of criteria, subcriteria or ratings and it always has λmax ≥ n. A high index shows that the judgment process is not reliable and there is a high amount of bias and is disturbed, which indicates an unreliable model (Saaty, 1977Saaty, T. L. (1977). A scaling method for priorities in hierarchical structures. Journal of Mathematical Psychology, 15(3), 234-281. http://dx.doi.org/10.1016/0022-2496(77)90033-5.
http://dx.doi.org/10.1016/0022-2496(77)9...
).

After calculating CI, it shall be compared with the Random Index (RI) – generated from aleatory matrices in 1-9 scale, with forced reciprocity, presented by Saaty (1994)Saaty, T. L. (1994). How to make a decision: the analytic hierarchy process. Interfaces, 24(6), 19-43. http://dx.doi.org/10.1287/inte.24.6.19.
http://dx.doi.org/10.1287/inte.24.6.19...
, when calculated to square matrices with n elements (Table 1). The Consistency Ratio (CR) is defined as CR=CI/RI. If CR is less than 0.1, it shows a consistent judgment process.

Table 1
RI for square matrices with n elements.

The priorities are defined by a model synthesis, through the sum of weight vectors (Mu & Pereyra-Rojas, 2018Mu, E., & Pereyra-Rojas, M. (2018). Practical decision making using super decisions: an introduction to the Analytic Hierarchy Process. Cham: Springer. http://dx.doi.org/10.1007/978-3-319-68369-0.
http://dx.doi.org/10.1007/978-3-319-6836...
).

The 4th step, consisting of judgment aggregation, considered the case of multiple stakeholders using the weighted geometric mean, as explored by Aguarón et al. (2019)Aguarón, J., Escobar, M. T., Moreno-Jiménez, J. M., & Turón, A. (2019). AHP-group decision making based on consistency. Mathematics, 7(3), 242. http://dx.doi.org/10.3390/math7030242.
http://dx.doi.org/10.3390/math7030242...
. Let $ki$ be a pairwise judgment of the i-th stakeholder, whose relative weight in the decision process is $wi$. The aggregate judgment $K$ is given by Equation 2. Equation 3 gives the boundary conditions for the relative weight distribution.

$K = ∏ i = 1 n k i w i$ (2)
$∑ i = 1 n w i = 1$ (3)

The sensitivity analysis consists of varying the weight scale range of the two stakeholders involved using a percentage scale that varied from 100:0, in which stakeholder #2 judgments are disregarded, to 0:100, in which stakeholder #1 judgments are disregarded. From the literature associated with sensitivity analysis and multicriteria decision analysis, there were no records of such focus on stakeholder weight distribution sensitivity.

Therefore Figure 2 showed each step of the multimethodology proposed here to deal with the MCDA process, and it was depicted in each subsection. Summarizing the model: 1st) identify stakeholders; 2nd) structure the problem with Rich Picture and VFT; 3rd) make the transition from VFT to AHP as presented in Figure 2; 4th) apply all steps of AHP; and 5th) do the sensitivity analysis.

# 3. Developing the model

In this section, the tasks from the multimethodological process, illustrated in Figure 1, are depicted for the application in ranking AVAOPs and model development.

## 3.1. Divergent phase

The first task in this phase is to identify the key stakeholders. Three key stakeholders were chosen for this intervention. They were two majors and one colonel. The colonel is the chief of the sector that decides which AVAOP would be done, thus, the decision-maker. One major is his assessor. He has more experience than his hierarchical superior – the colonel was only two months in his position and the major was two years. This was relevant because it was a motivation to the colonel to invite the major to the decision-making in itself. The other major is the chief of the sector in the Institute of Operational Applications where the AVAOPs are done.

First, they were interviewed individually to construct the rich picture and elicit the objectives. Figure 6 shows the rich picture which was shaped during these interactions:

Figure 6
Identification of operational deficiencies in Readiness Command (Rich Picture).

Many emergent properties and relevant systems (Georgiou, 2015Georgiou, I. (2015). Unraveling soft systems methodology. International Journal of Economics and Business Research, 9(4), 415-436. http://dx.doi.org/10.1504/IJEBR.2015.069680.
http://dx.doi.org/10.1504/IJEBR.2015.069...
) emerged. Nonetheless, the prioritizing was highlighted (near the right top corner), showing the reason why the model developed here is necessary.

The questionnaire used was based on Keeney (1996a)Keeney, R. L. (1996a). Value-focused thinking: a path to creative decisionmaking. London: Harvard University Press Cambridge.. It also made a literature review of strategic legislation, with a bigger focus on DCA 11-45 (Strategic Conception of Air Force 100), because it is the newer strategic legislation and made significant changes in Brazilian Air Force. The workshops and legislation review yield twenty objectives, after taking out redundancy – redundancy is not a deficiency, it is much easier to recognize the desires and concerns when objectives are enumerated, with no restrictions, and although an objective, arising the most hidden interests (Keeney, 2013Keeney, R. L. (2013). Identifying, prioritizing, and using multiple objectives. EURO Journal on Decision Processes, 1(1-2), 45-67. http://dx.doi.org/10.1007/s40070-013-0002-9.
http://dx.doi.org/10.1007/s40070-013-000...
).

Mean-end network helped to find the fundamental objectives with the WITI Test (Why is it important?). The top-level fundamental objective was “Maximize the Operational Impact”. The term “Operational Impact” has a similar meaning to “operational efficiency” shown in legislation. However, the first one was more explored by stakeholders, highlighting their desires and the consequences that could be achieved with the alternatives.

"Operational Impact" appear also on another strategic norm, the Aeronautics Systematic Planning and Institutional Management, as a criterion to start projects in Air Force: "impact straight in Force operationality, through the inter-relationship with Airspace Power Capacity" (Brasil, 2019Brasil. DCA 11-1: sistemática de planejamento e gestão institucional da aeronáutica. Volume 1 – Planejamento. (2019). Brasília. Retrieved in 2021, May 31, from https://www.fab.mil.br/Download/arquivos/prestacaodecontas/DCA_11_1_2019_SPGIA.pdf
, p. 30). Considering the attributes generation, the fundamental objective cited before was specified. "Maximize the Operational Impact" was specified (what does this objective mean?) in five more objectives, setting the second level of hierarchy. The process proceeded, stopping in the third level (Figure 7).

Figure 7
Hierarchy of Fundamental Objectives.

Finally, the ratings were defined finding the attributes linked in fundamental objectives. Thus, the Hierarchy of Fundamental Objectives is transformed into the decision structure, with the process described in section 2, and AHP starts to get the scale’s constants for the aggregation value function with the pairwise comparison in criteria, subcriteria, and ratings, concluding with the categorization of alternatives.

## 3.2. Convergent phase

Concluded the problem structuring means that 1st step of AHP is done, and transforming the fundamental objectives hierarchy to the decision structure consists of 2nd step – in this work, the second level of fundamental objectives (Figure 7) is the criteria, and the third level is subcriteria – completing the Decision Hierarchy as shown in Figure 8 – and their definitions are synthesized in Tables 2 and 3. The next task is accomplishing pairwise comparisons. Thereby, completing step 3.

Figure 8
Decision Hierarchy.
Table 2
Criterions description.
Table 3
Subcriterions description.

Ratings were built upon attributes paying attention to be as careful as Keeney (1996a) Keeney, R. L. (1996a). Value-focused thinking: a path to creative decisionmaking. London: Harvard University Press Cambridge.argues: "The careful development of a constructed attribute (…) may promote thinking and describe the consequences in a decision situation much better than the "subjective" choice to use a readily available natural attribute" (Keeney, 1996aKeeney, R. L. (1996a). Value-focused thinking: a path to creative decisionmaking. London: Harvard University Press Cambridge., p. 104). Figures 9, 10 and 11 present the decision structure with all elements described.

Figure 9
Complete decision hierarchy.
Figure 10
Zoom in the left side of decision hierarchy.
Figure 11
Zoom in the right side of decision hierarchy.

The next task was collecting judgments for pairwise comparisons. Each decision-maker accomplished one hundred judgments. Eight AVAOPs options were assessed to check the model. Half of them were not solved – AVAOP 1, AVAOP 3, AVAOP 4 and AVAOP 8. The AVAOPs content is hidden, although they are real examples of Operational Deficiencies informed to Readiness Command.

The software Excel supported the workshop used to collect stakeholders' judgments and alternative ratings. To remove most of the bias, the AVAOP's options were uncharacterized, thereby, stakeholders did not know which of them were accomplished. Table 4 exemplifies the alternatives rating, already with scale constants after pairwise comparisons.

Table 4
Result example of the last workshop.

Going on the additive aggregation value function, considering the evenly-distributed weight of both stakeholders, the ranking in Table 5 shows the results and Figure 12 presents them graphically. The consistency ratio of the aggregated judgments varied from 1.4% to 7.1%, within the 10% suggested by Saaty (1990; 1994Saaty, T. L. (1990). How to make a decision: the analytic hierarchy process. European Journal of Operational Research, 48(1), 9-26. http://dx.doi.org/10.1016/0377-2217(90)90057-I.
http://dx.doi.org/10.1016/0377-2217(90)9...
)

Table 5
AVAOP's ranking.
Figure 12
AVAOP's Ranking (graphic).

In a first moment, as possible analysis of the results, the facilitator showed that three of four best AVAOPs were not solved yet, an interesting conclusion presenting that a method is needed for this decision, because it has not been taken completely rationally until the moment of this work.

"To evaluate how the results provided by the model vary with the parameters and whether the assumed simplifications affect the results, a sensitivity analysis is essential" (Almeida et al., 2015Almeida, A. T., Cavalcante, C. A. V., Alencar, M. H., Ferreira, R. J. P., Almeida-Filho, A. T., & Garcez, T. V. (2015). Multicriteria and multiobjective models for risk, reliability and maintenance decision analysis (1. ed.). Recife: Springer. http://dx.doi.org/10.1007/978-3-319-17969-8.
http://dx.doi.org/10.1007/978-3-319-1796...
, p. 224). The sensitivity analysis took place after collecting the stakeholders' judgments. Figure 13 shows how the ranking varies as a function of the weight distribution, as described in section 2.3.2.

Figure 13
AVAOP rankings as a function of the weight distribution.

Considering that, in the case study, the three-best ranked AVAOPs would be implemented by the decision-maker, it is possible to see that the weight system may affect the final list. As implemented, both stakeholders' voice was balanced (50:50), therefore AVAOPs #8, #3, and #6 would be implemented.

## 3.3. Results

The application showed some relevant results about the approach developed:

1. 1

Rich Picture helps with the problem structuring with VFT because the diagrammatic view facilitates the context decision definition. The case studied, helped the stakeholders to see which could be the impact of the decision in the real world, and to surface different decision opportunities.

2. 2

The definition of a clear transition process from VFT fundamental objective hierarchy to AHP with ratings decision hierarchy supported the perception of which values were considered in the decision, helping with the negotiation of alternatives evaluation in the final process of the approach; and

3. 3

Defining weights for decision-makers and showing how the final ranking varies when these weights vary, is a sensitive task. It depends on the group and facilitator's feelings. Thus, it lacks deeper analysis and studies to determine a process that could deal with group heterogeneity and conflicts.

# 4. Discussion

Some works are combining SSM and VFT. Françozo et al. (2021)Françozo, R., Paucar-Caceres, A., & Belderrain, M. C. N. (2021). Combining Value-Focused thinking and soft systems methodology: A systemic framework to structure the planning process at a special educational needs school in Brazil. The Journal of the Operational Research Society. In press. http://dx.doi.org/10.1080/01605682.2021.1880298.
http://dx.doi.org/10.1080/01605682.2021....
, for example, use this combination to structure policies in planning the inclusion of students with special needs in Brazilian Federal High School, however, they do not use an MCDA method to aid the decision. Nevertheless, Abuabara et al. (2019)Abuabara, L., Paucar-Caceres, A., & Burrowes-Cromwell, T. (2019). Consumers values and behavior in the Brazilian coffee-in-capsules market: promoting circular economy. International Journal of Production Research, 57(23), 7269-7288. http://dx.doi.org/10.1080/00207543.2019.1629664.
http://dx.doi.org/10.1080/00207543.2019....
make the exact combination used here; they did not show how they made the transition from VFT to AHP. Almeida et al. (2015)Almeida, A. T., Cavalcante, C. A. V., Alencar, M. H., Ferreira, R. J. P., Almeida-Filho, A. T., & Garcez, T. V. (2015). Multicriteria and multiobjective models for risk, reliability and maintenance decision analysis (1. ed.). Recife: Springer. http://dx.doi.org/10.1007/978-3-319-17969-8.
http://dx.doi.org/10.1007/978-3-319-1796...
clearly define in their procedure for resolving problems and building multicriteria models that "for each objective previously established a criterion or attribute has to be proposed" (Almeida et al., 2015Almeida, A. T., Cavalcante, C. A. V., Alencar, M. H., Ferreira, R. J. P., Almeida-Filho, A. T., & Garcez, T. V. (2015). Multicriteria and multiobjective models for risk, reliability and maintenance decision analysis (1. ed.). Recife: Springer. http://dx.doi.org/10.1007/978-3-319-17969-8.
http://dx.doi.org/10.1007/978-3-319-1796...
, p. 31), although they do not show how people should do it. Here, this gap is filled to make the transition to AHP with ratings, but it is possible to use the same steps with other MCDM/A methods.

The transition process from VFT to AHP to consider the values from stakeholders into the decision hierarchy, and not only for it, but was important to make clear which values were considered. The sensitivity analysis on this ranking problem displays how final results may be affected by the weight distribution of the stakeholders. Firstly, it is important to be as transparent as possible to stakeholders concerning the weight of their judgment. Disclosing clearly how weights are distributed will avoid any dispute that could emerge if the weights were kept secret. Secondly, there are some useful applications of this assessment, such as in private companies.

Shareholders, when having their say in the company's decisions, may use their relative share as means to determine each shareholder's weight. This would allow the application, not only of the AHP but other aggregative methods. A shareholder could evaluate, based on this sensitivity analysis, whether it is attractive to acquire more shares or not to increase their weight in the company's decisions.

Aguarón et al. (2019)Aguarón, J., Escobar, M. T., Moreno-Jiménez, J. M., & Turón, A. (2019). AHP-group decision making based on consistency. Mathematics, 7(3), 242. http://dx.doi.org/10.3390/math7030242.
http://dx.doi.org/10.3390/math7030242...
proposition brought many possibilities for AHP applications, as done here. In this application, the interaction between the DMs has occurred without conflicts because of the hierarchical difference. Nonetheless, Group decision processes are sensitive to the accuracy of the communication process (Almeida et al., 2015Almeida, A. T., Cavalcante, C. A. V., Alencar, M. H., Ferreira, R. J. P., Almeida-Filho, A. T., & Garcez, T. V. (2015). Multicriteria and multiobjective models for risk, reliability and maintenance decision analysis (1. ed.). Recife: Springer. http://dx.doi.org/10.1007/978-3-319-17969-8.
http://dx.doi.org/10.1007/978-3-319-1796...
). Thus, in replications of the model, this issue must have considerable attention.

# 5. Conclusion

The multimethodological process developed here presented a framework to deal with group decision-making holistically. It has involved and engaged stakeholders from the process beginning, with Rich Picture and VFT to structure the problem, eliciting criteria and ratings through fundamental objectives and attributes, exploring workshops to evaluate the alternatives, and with a sensitivity analysis that has brought insights to different groups, building an interactive and adapted method to group decision making, showing a means to deal with in organizations with rigid structure, like military ones and big private companies.

It is important to note that, considering only two stakeholders, there was little concern regarding consistency and coherence - as mentioned by Saaty (1994)Saaty, T. L. (1994). How to make a decision: the analytic hierarchy process. Interfaces, 24(6), 19-43. http://dx.doi.org/10.1287/inte.24.6.19.
http://dx.doi.org/10.1287/inte.24.6.19...
, the method allows inconsistency to some extent. Future work could explore these areas using, for instance, the precise consensus consistency matrix (PCCM) and solve the optimization problem, as proposed by Aguarón et al. (2019)Aguarón, J., Escobar, M. T., Moreno-Jiménez, J. M., & Turón, A. (2019). AHP-group decision making based on consistency. Mathematics, 7(3), 242. http://dx.doi.org/10.3390/math7030242.
http://dx.doi.org/10.3390/math7030242...
, with more than two stakeholders. Thus, a limitation of this study was not defining how negotiation and conflicts could be managed during alternatives evaluation in this approach, opening a vast field to studies, exploring this phase.

Another interesting take that future works could tackle would be personal bias introduced by stakeholders during negotiations or in the judgments phase. A psychological study could find out whether knowing the weight of their judgment beforehand would have an impact or not on the stakeholder's judgments and final results.

• How to cite this article: Logullo, Y., Bigogno-Costa, V., Silva, A., & Belderrain, M. C. (2022). A prioritization approach based on VFT and AHP for group decision making: a case study in the military operations. Production, 32, e20210059. https://doi.org/10.1590/0103-6513.20210059

# References

• Abuabara, L., & Paucar-Caceres, A. (2021). Surveying applications of strategic options development and analysis (SODA) from 1989 to 2018. European Journal of Operational Research, 292(3), 1051-1065. http://dx.doi.org/10.1016/j.ejor.2020.11.032
» http://dx.doi.org/10.1016/j.ejor.2020.11.032
• Abuabara, L., Paucar-Caceres, A., & Burrowes-Cromwell, T. (2019). Consumers values and behavior in the Brazilian coffee-in-capsules market: promoting circular economy. International Journal of Production Research, 57(23), 7269-7288. http://dx.doi.org/10.1080/00207543.2019.1629664
» http://dx.doi.org/10.1080/00207543.2019.1629664
• Ackermann, F. (2012). Problem structuring methods in the dock: arguing the case for soft OR. European Journal of Operational Research, 219(3), 652-658. http://dx.doi.org/10.1016/j.ejor.2011.11.014
» http://dx.doi.org/10.1016/j.ejor.2011.11.014
• Ackermann, F., & Eden, C. (2011). Strategic management of stakeholders: theory and practice. Long Range Planning, 44(3), 179-196. http://dx.doi.org/10.1016/j.lrp.2010.08.001
» http://dx.doi.org/10.1016/j.lrp.2010.08.001
• Aguarón, J., Escobar, M. T., Moreno-Jiménez, J. M., & Turón, A. (2019). AHP-group decision making based on consistency. Mathematics, 7(3), 242. http://dx.doi.org/10.3390/math7030242
» http://dx.doi.org/10.3390/math7030242
• Almeida, A. T., Cavalcante, C. A. V., Alencar, M. H., Ferreira, R. J. P., Almeida-Filho, A. T., & Garcez, T. V. (2015). Multicriteria and multiobjective models for risk, reliability and maintenance decision analysis (1. ed.). Recife: Springer. http://dx.doi.org/10.1007/978-3-319-17969-8
» http://dx.doi.org/10.1007/978-3-319-17969-8
• Belton, V., & Stewart, T. J. (2002). Multiple criteria decision analysis: an integrated approach Boston: Springer US. http://dx.doi.org/10.1007/978-1-4615-1495-4
» http://dx.doi.org/10.1007/978-1-4615-1495-4
• Brasil. DCA 11-1: sistemática de planejamento e gestão institucional da aeronáutica. Volume 1 – Planejamento (2019). Brasília. Retrieved in 2021, May 31, from https://www.fab.mil.br/Download/arquivos/prestacaodecontas/DCA_11_1_2019_SPGIA.pdf
• Brasil. DCA 11-45: concepção estratégica força aérea 100 (2018). Brasília. Retrieved in 2021, May 31, from https://www.fab.mil.br/Download/arquivos/DCA%2011-45_Concepcao_Estrategica_Forca_Aerea_100.pdf
• Checkland, P. (1981). Systems thinking, systems practice New York: John Wiley.
• Checkland, P., & Poulter, J. (2020). Soft systems methodology. In M. Reynolds & S. Holwell (Eds.), Systems approaches to making change: a practical guide (pp. 201-253). London: Springer. http://dx.doi.org/10.1007/978-1-4471-7472-1_5
» http://dx.doi.org/10.1007/978-1-4471-7472-1_5
• Dyson, R. G., O’Brien, F. A., & Shah, D. B. (2021). Soft OR and practice: the contribution of the founders of operations research. INFORMS, 69(3), 727-738. http://dx.doi.org/10.1287/opre.2020.2051
» http://dx.doi.org/10.1287/opre.2020.2051
• Franco, L. A., & Montibeller, G. (2010). Facilitated modeling in operational research. European Journal of Operational Research, 205(3), 489-500. http://dx.doi.org/10.1016/j.ejor.2009.09.030
» http://dx.doi.org/10.1016/j.ejor.2009.09.030
• Françozo, R., Paucar-Caceres, A., & Belderrain, M. C. N. (2021). Combining Value-Focused thinking and soft systems methodology: A systemic framework to structure the planning process at a special educational needs school in Brazil. The Journal of the Operational Research Society In press. http://dx.doi.org/10.1080/01605682.2021.1880298
» http://dx.doi.org/10.1080/01605682.2021.1880298
• Georgiou, I. (2015). Unraveling soft systems methodology. International Journal of Economics and Business Research, 9(4), 415-436. http://dx.doi.org/10.1504/IJEBR.2015.069680
» http://dx.doi.org/10.1504/IJEBR.2015.069680
• Gregory, A. J., Atkins, J. P., Midgley, G., & Hodgson, A. M. (2020). Stakeholder identification and engagement in problem structuring interventions. European Journal of Operational Research, 283(1), 321-340. http://dx.doi.org/10.1016/j.ejor.2019.10.044
» http://dx.doi.org/10.1016/j.ejor.2019.10.044
• Keeney, R. L. (1996a). Value-focused thinking: a path to creative decisionmaking London: Harvard University Press Cambridge.
• Keeney, R. L. (1996b). Value-focused thinking: Identifying decision opportunities and creating alternatives. European Journal of Operational Research, 92(3), 537-549. http://dx.doi.org/10.1016/0377-2217(96)00004-5
» http://dx.doi.org/10.1016/0377-2217(96)00004-5
• Keeney, R. L. (2007). Developing objectives and attributes. In W. Edwards, R. F. Milles & D. Von Winterfeldt (Eds.), Advances in decision analysis: from foundations to applications (pp. 104-128). Cambridge: Cambridge University Press http://dx.doi.org/10.1017/CBO9780511611308.008
» http://dx.doi.org/10.1017/CBO9780511611308.008
• Keeney, R. L. (2008). Applying value-focused thinking. Military Operations Research, 13(2), 7-17. http://dx.doi.org/10.5711/morj.13.2.7
» http://dx.doi.org/10.5711/morj.13.2.7
• Keeney, R. L. (2013). Identifying, prioritizing, and using multiple objectives. EURO Journal on Decision Processes, 1(1-2), 45-67. http://dx.doi.org/10.1007/s40070-013-0002-9
» http://dx.doi.org/10.1007/s40070-013-0002-9
• Midgley, G. (2015). Systemic intervention: research memorandum 95: Hull University of Business School Retrieved in 2021, May 31, from https://hull-repository.worktribe.com/output/385209
» https://hull-repository.worktribe.com/output/385209
• Mingers, J., & Brocklesby, J. (1997). Multimethodology: towards a framework for mixing methodologies. Omega, 25(5), 489-509. http://dx.doi.org/10.1016/S0305-0483(97)00018-2
» http://dx.doi.org/10.1016/S0305-0483(97)00018-2
• Mingers, J., & Rosenhead, J. (2004). Problem structuring methods in action. European Journal of Operational Research, 152(3), 530-554. http://dx.doi.org/10.1016/S0377-2217(03)00056-0
» http://dx.doi.org/10.1016/S0377-2217(03)00056-0
• Monk, A., & Howard, S. (1998). Methods & tools: the rich picture: a tool for reasoning about work context. Interaction, 5, 21-30. http://dx.doi.org/10.1145/274430.274434
» http://dx.doi.org/10.1145/274430.274434
• Montibeller, G., Belton, V., Ackermann, F., & Ensslin, L. (2008). Reasoning maps for decision aid: an integrated approach for problem-structuring and multi-criteria evaluation. The Journal of the Operational Research Society, 59(5), 575-589. http://dx.doi.org/10.1057/palgrave.jors.2602347
» http://dx.doi.org/10.1057/palgrave.jors.2602347
• Mu, E., & Pereyra-Rojas, M. (2018). Practical decision making using super decisions: an introduction to the Analytic Hierarchy Process Cham: Springer. http://dx.doi.org/10.1007/978-3-319-68369-0
» http://dx.doi.org/10.1007/978-3-319-68369-0
• Parnell, G. S., & West, P. D. (2008). Value-focused systems decision making. In Proceedings of the 18th Annual International Symposium of the International Council on Systems Engineering (pp. 1685-1699). Utrecht: INCOSE.
• Rosenhead, J. (1989). Rational analysis for a problematic world: problem structuring methods for complexity, uncertainty, and conflict New York: John Wiley.
• Rosenhead, J. (2006). Past, present, and future of problem structuring methods. The Journal of the Operational Research Society, 57(7), 759-765. http://dx.doi.org/10.1057/palgrave.jors.2602206
» http://dx.doi.org/10.1057/palgrave.jors.2602206
• Roy, B. (1996). Multicriteria methodology for decision aiding Boston: Springer. http://dx.doi.org/10.1007/978-1-4757-2500-1
» http://dx.doi.org/10.1007/978-1-4757-2500-1
• Saaty, T. L. (1977). A scaling method for priorities in hierarchical structures. Journal of Mathematical Psychology, 15(3), 234-281. http://dx.doi.org/10.1016/0022-2496(77)90033-5
» http://dx.doi.org/10.1016/0022-2496(77)90033-5
• Saaty, T. L. (1990). How to make a decision: the analytic hierarchy process. European Journal of Operational Research, 48(1), 9-26. http://dx.doi.org/10.1016/0377-2217(90)90057-I
» http://dx.doi.org/10.1016/0377-2217(90)90057-I
• Saaty, T. L. (1994). How to make a decision: the analytic hierarchy process. Interfaces, 24(6), 19-43. http://dx.doi.org/10.1287/inte.24.6.19
» http://dx.doi.org/10.1287/inte.24.6.19
• Saaty, T. L. (2001, August 2-4). Deriving the AHP 1-9 scale from first principles. In Proceedings of the 6th International Symposium on Analytical Hierarchy Process Berna: ISAHP.
• Saaty, T. L. (2006). Rank from comparisons and ratings in the analytic hierarchy/network processes. European Journal of Operational Research, 168(2), 557-570. http://dx.doi.org/10.1016/j.ejor.2004.04.032
» http://dx.doi.org/10.1016/j.ejor.2004.04.032
• Saaty, T. L. (2008). Decision making with the Analytic Hierarchy Process. Scientia Iranica, 9(3), 215-229.

# Publication Dates

• Publication in this collection
18 Feb 2022
• Date of issue
2022