A MULTICRITERIA DECISION MODEL FOR RISK MANAGEMENT MATURITY EVALUATION

ABSTRACT

This article aims to present a Multicriteria Decision Aiding (MCDA) model for assessing risk management maturity. Therefore, it is proposed to use a Maturity Model (MM) for risk management aligned with the ELECTRE TRI method. The ELECTRE TRI was chosen as the sorting method because it has a strong axiomatic structure based on the relationship of concordance and discordance between the alternative and the profile that delimits each of its classes. To test the proposal, a case study was carried out on a real company in the construction industry. For the development of the risk management maturity assessment model, a questionnaire was applied to collect data related to risk management practices in the organization. After collection, the data were used for modeling in a Decision Support System to apply the ELECTRE TRI, which managed to classify and identify the organization’s risk management maturity at level 3 (managed).

Keywords:
Multicriteria Decision Analysis; risk management; maturity model; ELECTRE TRI

1 INTRODUCTION

Risk management (RM) is a relevant topic for any organization as it offers integrated strategies for evaluating, controlling, and monitoring decisions that involve risks (Hopkin, 2010). To say that an organization reaches maturity in risk management means there is an evolution towards the full development of risk management processes (Hoseini, Hertogh & Bosch-Rekyeldt, 2019). In this regard, Maturity Models (MMs) are instruments that support the measurement of risk maturity in organizations.

MMs offer organizations a simple and effective way to assess their process development. MMs were developed to evaluate the capability and effectiveness of systems in different situations through a coherent, capabilities-based framework (Macgillivray et al., 2007; Sheehan et al., 2021). Despite being widely diffused initially in software engineering, application areas have been expanding rapidly and research has gained increasing importance (Wendler, 2012; Santos-Neto & Costa, 2019). A Risk Management Maturity Model (RMMM) aims to measure the maturity of risk management in projects and/or organizations.

These MMs can assess the current state of the RM and identify where it should prioritize intervention to reach higher levels of maturity (Zou, Chen & Chan, 2010). The foundation of the RM and maturity assessment supports a company in gaining an understanding of its current ERM implementation, as well as the strong and weak aspects of ERM implementation (Zhao, Hwang &Low, 2016).

Due to their usefulness, RMMM can be found in various applications in the literature, such as in civil construction (Hoseini, Hertogh & Bosch-Rekyeldt, 2019; Motaleb, 2017), legal (Unger et al., 2015), corporate (Oliva, 2016), for evaluation of logistic processes (Tubis & Werbińska-Wojciechowska, 2021), and financial technology firm (Alijoyo et al., 2021).

The complexity of the evaluation process proposed by some MMs and the lack of operationalization are considered barriers that hinder the use of MMs as a means of management and organizational diagnosis (Röglinger et al., 2012). Santos-Neto & Costa (2019) identified in their literature review that 24% of the articles found on MM do not clearly present the sorting method used to identify the maturity level.

Thus, this article aims to present a Multicriteria Decision Aiding (MCDA) model for assessing risk management maturity. To this end, the application of an RMMM is aligned with the MCDA ELECTRE TRI method as an alternative for sorting the maturity level. It is believed that the use of MCDA can be a way to standardize the set of procedures necessary for the application of MMHence, MCDA methods can support the Decision Maker (DM) in a problem concerning a sorting of maturity levels by comparing information and characteristics of the maturity model through a set of attributes.

The MCDA approach seeks to support the solution of problems that demand complex decisions, which involve multiple criteria, some conflicting with each other when evaluating the actions (Trojan & Morais, 2012; Gonçalves et al., 2021). Furthermore, MCDA methods admit a systematic view of the problem assessment and are efficient in comparing alternatives via multiple attributes allowing the combination of both subjective and objective attributes (Rodrigues et al., 2022; Lacerda, Santos-Neto & Martins, 2021). Besides that, allow to making a decision by choosing the best one from a set of options in the attendance of multiple and conflict attributes (Santos-Neto & Costa, 2023). Thus, it is expected that the use of MCDA provides scientificity and robustness of the results found, which facilitates the assessment of maturity for a benchmarking approach.

2 BACKGROUND

Organizations look for objective ways to monitor and control their own risk management and identify weaknesses and opportunities for improvement (Wibowo & Taufik, 2017; Hoseini, Hertogh & Bosch-Rekyeldt, 2019). This justifies the establishment of standardized procedures to improve the application of maturity models focused on risk management as an alternative to meet these demands. Furthermore, it covers the gap raised by Santos-Neto & Costa (2019) about the scarcity of evaluation models for the application of MM.

For the RM domain, some different MMs have been developed over the years (Santos-Neto & Costa, 2019). Some MMs in RM are shown in Table 1.

Table 1
Risk Management Maturity Models.

Author	Models	Application Area	Level	Data collection
Yeo and Ren (2009)	RM-CMM (Risk Management Capability Maturity Model)	Complex product systems projects	1. Ad hoc	Questionnaire (1 to 5 scale)
			2. Initial
			3. Defined
			4. Managed
			5. Optimizing
Von Kanel et al. (2010)	ERMMM (Enterprise Risk Management Maturity Model)	Enterprise risk management	1. Ad hoc	Questionnaire (1 to 5 scale)
			2. Basic
			3. Consistent
			4. Strong but
			separate
			5. Integrated
Zou, Chen and Chan (2010)	RM3 (Risk Management Maturity Model)	Construction organizations	1. Initial and ad hoc	Questionnaire (1 to 4 scale)
			2. Repeatable
			3. Managed
			4. Optimized
MacGillivray et al. (2007)	RM-CMM (Risk Management Capability Maturity Model)	Water utility sector	1. Initial	Maturity level matrix (1 to 5 scale)
			2. Repeatable
			3. Defined
			4. Controlled
			5 Optimized
Bhosale, Ravi and Patil (2018)	RMMM (Risk Management Maturity Model)	Road construction projects	1. Initial	Maturity level matrix (1 to 4 scale)
			2. Defined
			3. Managed
			4. Continuous
			5. Improvement

MCDA is a term that describes a collection of approaches that aims to support individuals or groups in the process of making decisions, taking explicit account of multiple criteria (Sapienza et al., 2016). In this process, the decision maker is a key actor, as he/she is one of the main sources of information and is responsible for establishing the constraints, preferences and assessing each alternative. According to Belton & Stewart (2002), the MCDA approach offers the following advantages: it seeks to clarify all the multiple factors involved in a decision, provides a structured analysis for the problem, helps the decision maker by synthesizing and presenting all the information; and even though the process does not provide an “ideal solution”, it allows the decision maker to reach an agreement between his preferences and the possible outcomes.

To test the MCDA method combined with a MM to assess RM, a company in the construction industry was selected to develop a decision model in this study. Therefore, due to the alignment of the MM with the target company’s segment of the test, the MM chosen for testing was the RM3 (Risk Management Maturity Model). According to Zou, Chen & Chan (2010), the model was developed through comparison with other similar MMs, compiling the aspects considered most important for identifying the main characteristics of risk management specifically in construction companies.

Although each author gives a name or establishes different aggregations, in RM3 the dimensions can be translated into five main approaches: management (people and leadership) in relation to risk; organizational risk culture; identifying risks; analyzing risks; and standardized management process of risks.

According to Zou, Chen & Chan (2010), civil construction activity strongly depends on the interaction between employees and leadership. While management must ensure productivity and quality, it is necessary to cultivate an organizational climate that encourages employees. Within RM3, these factors (management and culture) are encompassed by the first two attributes, which define the operational part of the organization. The identification, analysis, and standardization of risks, on the other hand, focus specifically on the resolution of events and at the same time consider the possible risks involved. The combination of these factors leads to a general understanding of risk management for an organization focused on civil construction.

3 RESEARCH METHODOLOGY

For the development of the Multicriteria Decision Model for RM evaluation, we developed a framework for the research model summarized in three phases, as shown in Figure 1.

Figure 1
Framework of the multicriteria decision model for RM evaluation.

In the preliminary phase, we characterized the decision maker and defined the evaluation criteria for the decision problem, and performed data collection. The decision maker is the person responsible for the decision and for establishing relationships and judgment of values that influence the decision process. At this stage, the Project Coordinator of the organization that was the focus of the study was identified as the decision maker who was responsible for evaluating and acting prescriptively on the result of the maturity assessment. The decision maker had experience in the position and extensive contact with the preparation and execution of projects developed by the organization.

For the definition of the set of criteria, the dimensions listed in the RM3 model were selected. A total of five criteria were defined, namely Management perspective (people and leadership) concerning risk, Organizational risk culture, Identifying risks, Analyzing risks, and Standardized risk management process. The criteria are measured to assess maturity at four levels: initial, repeated, managed, and optimized, which are the sorts of the proposed model. They define the organization’s maturity level in RM.

The attributes proposed in RM3 reflected the fundamentals of risk management and were designed to benefit construction companies in measuring and improving their risk management capabilities. The meaning of each criterion is defined as:

1. Management perspective (people and leadership) concerning risk: seeks to assess how much the upper management actively takes part in risk activities, supports and encourages risk management;

2. Organizational risk culture: seeks to measure to what extent team members are taking risk ownerships during project implementation;

3. Identifying risks: assesses whether the organization has a risk identification procedure;

4. Analyzing risks: assesses whether the organization has qualitative and/or quantitative risk analysis tools or procedures;

5. Standardized risk management process: measure if a standardized risk management process is applied to all projects within the organization.

For detailed information regarding the RM3 model, see Zou, Chen & Chan (2010).

A survey questionnaire to assess the organization’s performance regarding the problem criteria was defined for data collection. The questionnaire was a means of allowing the organization’s decision maker and specialists to assess the RM dimensions of each of the processes in a simplified way. The RM3 defined five main dimensions that are translated into five questions/statements each (total of 25), which represented the practices related to RM.

For data collection, five specialists directly in contact with the organization’s processes were selected to answer the questionnaire: coordinator (5 years with the company); resident engineer (8 years with the company); engineering assistant (2 years with the company); production supervisor (7 years with the company); engineering assistant (3 years with the company). All specialists were employees of the company and were included daily in the production routine, at different hierarchical levels.

Each of the experts rated each question/statement of the questionnaire according to a five-point Likert scale, where 1 meant ”The practice is not widely used in the organization” and 5 meant ”The practice is widely used in the organization,” as shown in Table 2. The application questionnaire to verify the level of applicability of risk management practices is found in ^{Appendix A}.

Table 2
Rating Scale.

Scale	Description
1	The practice is not widely used in the organization.
2	There is a strong discussion about using the practice, but no decision.
3	There is the decision and action plan to start using the practice.
4	The practice has been tested and experience can be gained.
5	The practice is widely used in the organization.

In the modeling phase, after data collection, the answers were evaluated using the mode found in each criterion, and the value identified in the criterion was used as the organization’s performance in composing the decision matrix.

To assess the maturity level, the ELECTRE TRI method was selected. The ELECTRE TRI method was chosen as the sorting method by observing the non-compensatory rationality of the decision maker. In other words, it is expected that the substandard performance of one criterion is not compensated by the good performance of another, something that is exploited by ELECTRE TRIFurthermore, ELECTRE TRI has a strong axiomatic structure and uses a Decision Support System (DSS) for application. For this study, the DSS MCDA-ULAVAL was used. MCDA-ULAVAL, a Canadian software developed at Laval University, has free access and an open source (ULAVAL, 2018).

Particularly in the ELECTRE TRI method, the sorting of an alternative, a, into a given class, c _h (h=1, 2, ..., p), is made based on the comparison between the alternative and the profile that defines the limit of each category (Brito, Almeida & Mota, 2010). This factor allows the sorting of only one alternative, if necessary, as the comparison is made against the profile and not with other alternatives.

The sorting criteria g _j (j=1, 2, ..., m) are used to judge the alternative and compare it with the profile, b _h , which represents the upper limit of class c _h and the lower limit of class c _h+1 , as depicted in Figure 2 (Mousseau, Figueira & Naux, 2001).

Figure 2
ELECTRE TRI sorting plan (adapted from Mousseau, Figueira & Naux, 2001).

The ELECTRE methods works with an outranking system, where aSb _h means alternative a is at least as good as profile b _h (Ochoa et al., 2022). To validate this statement, two conditions must be met:

1. Concordance: for it to outrank b _h (aSb _h ), the necessary majority of the criteria must agree;

2. Non-concordance: if the concordance is true, none of the remaining criteria should strongly oppose the aSb _h statement.

Two inter-criteria parameters interfere in the composition of S. The first is called the weight-importance coefficient (w), which indicates how much each criterion contributes to the aSb _h statement. SRF 2.2 software (Figueira & Roy, 2002) was used to define the weight parameter of each attribute to apply the ELECTRE TRI method. Figueira & Roy (2002) proposed a method and a Decision Support System for defining weights of criteria used in some methods of the ELECTRE family. The described method is based on the use of cards and its application is described in Section 4 of this study.

The other parameter is called the veto threshold (v), which represents the smallest difference in performance in a criterion g _j incompatible with aSb _h (Dias et al., 2002). The method proposes a credibility index σ(a, b _h )∈[0, 1], thus the statement aSb _h is considered valid if σ(a, b _h )≥λ, where λ∈[0.5, 1] (Mousseau, Figueira & Naux, 2001). The credibility index (1) is calculated from the partial concordance (2), concordance (3), and discordance (4) indexes.

Credibility Index:

$\sigma \left(a,b_h\right)=c\left(a,b_h\right)\Pi \frac{1-d_j\left(a,b_h\right)}{1-c\left(a,b_h\right)}$ (1)

Partial concordance:

$c_j\left(a,b_h\right)\left\{\begin{array}{r}0se{g}_j\left(b_h\right)-g_j\left(a\right)\ge p_j\left(b_h\right)\\ 1se{g}_j\left(b_h\right)-g_j\left(a\right)\le q_j\left(b_h\right)\\ \frac{p_j\left(b_h\right)+g_j\left(a\right)-g_j\left(b_h\right)}{p_j\left(b_h\right)-q_j\left(b_h\right)}otherwise\end{array}\right.$ (2)

Concordance:

$c\left(a,b_h\right)=\frac{\Sigma w_j{c}_j\left(a,b_h\right)}{\Sigma w_j}$ (3)

Discordance:

$d_j\left(a,b_h\right)\left\{\begin{array}{r}0se{g}_j\left(b_h\right)-g_j\left(a\right)\le p_j\left(b_h\right)\\ 1se{}_g\left(b_h\right)-g_j\left(a\right)>v_j\left(b_h\right)\\ \frac{g_j\left(b_h\right)-g_j\left(a\right)-p_j\left(b_h\right)}{v_j\left(b_h\right)-p_j\left(b_h\right)}otherwise\end{array}\right.$ (4)

When measuring credibility indexes, the result can be evaluated in two ways: Pessimistic procedure and Optimistic procedure. The pessimistic approach is made from successive comparisons between a and b _h , h=1, 2, ..., p, with a being allocated to the first class, c_h+1 , in which aSb_h is verified. The other procedure, optimistic, compares a and b _h , h=p, p-1, ..., 1, with a being allocated to the first class c _h where b _h is preferable to a. More details about the ELECTRE TRI algorithm can be seen in Mousseau, Figueira & Naux (2001).

In the finalization phase, the sensitivity analysis was performed to verify the method’s robustness. For this, the model parameters must be varied to observe possible changes in the model results. For this study, we performed the sensitivity analysis varying the cutting level λ between 0.7 and 1. Finally, to evaluate the results, the scenario identified by the MM result was clarified with the decision maker to determine its consistency with reality.

4 EMPIRICAL APPLICATION

The research model was applied in the form of a case study in a Brazilian company in the construction industry. The company is active in all stages of real estate development, from land acquisition, project design, sales, planning, and construction, including transfers and after-sales.

During the process of data collection, all five specialists from the company were invited to answer the survey questionnaire (^{Appendix A}), rating every statement on a scale from 1 to 5 (Table 2). After that, the decision matrix was constructed using the mode found in the questions related to each of the five assessment criteria for the organization that was the focus of the study. For example, in order to determine the performance of the first dimension, Management Perspective (C1), the mode was applied for the answers to questions 1 to 5. The decision matrix developed for this empirical application can be seen in Table 3.

Table 3
Mode and normalization of the answers.

Criteria/performance	C1 Management perspective	C2 Organizational risk culture	C3 Identifying risks	C4 Analyzing risks	C5 Standardized RM process
Focus organization	4	5	5	5	3

For the application of ELECTRE TRI we defined some parameters along with the decision maker. To define the weight parameter, SRF 2.2 software (Figueira & Roy, 2002) was used to determine the weight of each attribute that would be used to apply the ELECTRE TRI method. The SRF facilitated the process of defining the degree of importance among the criteria for the decision maker since this decision is not always clear. SFR is based on the use of cards and can be applied in three phases.

1. In the first phase, the decision maker is provided with two sets of cards. The first set contains a card for each criterion in the assessed set, and the second has blank cards of the same size. The number of blank cards will depend on the decision maker’s need.

2. In sequence, the decision maker is required to order the set of cards with criteria from least important to most important. If any criterion is of equal importance to another, the card must be placed over the criterion (or criteria) of equal importance.

3. In the third phase, the decision maker is asked to think about the degree of importance between two successive criteria. The determination of weights considers the change in importance between two successive criteria. Then, the decision maker is asked to insert white cards between two successive cards (or subset of successive cards, in case of criteria with equal importance). The greater the difference between the criteria (or the subsets of criteria), the greater the number of white cards.

After these three phases, the DSS provides the normalized weights for each of the criteria. For this application, the decision maker ordered the criteria thus: C3>C4>C5>C1>C2. The weights given were: 5.8 for C2; 15.4 for C4; 23.1 for C5; 26.9 for C4; and 28.8 for C3. More details about the SFR method and the method’s algorithm can be found in Figueira & Roy (2002). Figure 3 illustrates the application of SFR 2.2.

Figure 3
Weights defined with the help of SRF 2.2.

Also, for the application of ELECTRE TRI in the DSS MCDA-ULAVAL according to the ELECTRE TRI algorithm, the following parameters were verified with the decision maker: cutting level (λ) equal to 0.7; veto threshold and both preference (p) and indifference (i) thresholds equals to zero. This means an abrupt transition between the preference range, not considering the weak preference zone or uncertainty (Miranda; De Almeida, 2003). Moreover, the definition of thresholds p, i, and v equal to zero is equivalent to using a true criterion, that is, there is a sudden change between the preference zones for the maturity levels (Rogers & Bruen, 1998).

Other parameters defined for the application of the model were the classes and evaluation profiles. In RM3, the maturity level is defined by the lowest value among the attributes. The attribute with the lowest value is considered the weak point; therefore, improvement actions must prioritize that attribute. For the definition of the level, an evaluation interval [0, 1] is made available for the organization’s evaluation. This interval was used in our modeling to compose the assessment profiles for each class (maturity level), as shown in Table 4.

Table 4
RM3 maturity levels gaps.

	Class	Profile
1	Initial	0,0-0,25
2	Repeatable	0,25-0,50
3	Managed	0,50-0,75
4	Optimized	0,75-1,00

Having defined all the necessary parameters to apply the ELECTRE TRI, we ran the DSS MCDA-ULAVAL from the perspective of evaluating the maturity of RM in the organization that was the focus of the case study, which allowed us to compare the performance in the five criteria with the maturity levels represented by the four classes scaled from 0 to 1 divided between profiles. Figure 4 illustrates the application of the model in MCDA-ULAVAL.

Figure 4
MCDA-ULAVAL Interface.

The execution of the DSS allowed the realization of two different sortings: one pessimistic and the other optimistic. Table 5 shows the two categorizations for the cutting level λ∈[0.7, 1], as defined in the research methodology for the sensitivity analysis.

Table 5
Results and sensitivity analysis.

Cutting Level (λ)	Pessimistic Level	Optimistic Level
0,7	Optimized (4)	Optimized (4)
0,75	Optimized (4)	Optimized (4)
0,8	Managed (3)	Optimized (4)
0,85	Managed (3)	Optimized (4)
0,9	Managed (3)	Optimized (4)
0,95	Managed (3)	Optimized (4)
1	Managed (3)	Optimized (4)

According to the procedure described in Section 3 of this article, when compared to a cutting level of 0.7 to 0.75, the assessed organization is rated at risk management maturity level 4 for both pessimistic and optimistic assessments. However, above 0.75, the pessimistic assessment gives a rating result for level 3 (Managed). The pessimistic procedure is considered in the analysis because it is more rigorous and, therefore, allows a prescriptive analysis of the evaluation toward improvements for the process.

According to Zou, Chen & Chan (2010), the RM3 maturity level 3 is named Managed and represents a scenario in which risk management systems and processes are formalized, implemented, and documented. At this level, the benefits of risk management are understood by all hierarchical levels of the organization. Senior management provides strong support, while employees are empowered to implement risk management processes to take risks. Level 3 maturity is considered sufficient for most organizations where risk management has become an integral part of their daily practices.

By making the result available to the decision maker, it was confirmed that the company’s RM possesses mostly characteristics compatible with the Managed maturity level. It was reported that the organization undertook practices such as the formalization and documentation of risk management, good RM practices are encouraged, and the understanding of its benefits permeate the entire organization.

However, although managers understand risk management as a competitive differential that must be continuously improved, which is a practice associated with level 4 organizations, RM is still not perceived as an integral part of the company’s culture. This disparity can be identified in criterion 2 (Organizational risk culture), had a mode equal to five (maximum value) while the question that assesses the responsibility for RM by all team members (question 8) had a mode equal to 1 (minimum value). Thus, classifying the company’s risk management as managed (level 3) is a result compatible with the real state of maturity, which demonstrates the quality in the use of the ELECTRE TRI multicriteria method as an ally to assess maturity in RM.

5 CONCLUSION

The present paper proposed a Multicriteria Decision Aiding model for assessing risk management maturity. The ELECTRE TRI Method was used as a tool to apply a Maturity Model (MM) for risk management.

As the main contribution of this study, the development of a Multicriteria Decision Model to assess maturity in RM showed us that the MCDA can be used as an important alternative for the application of MMs. Through a case study carried out with the application in a real organization, the proposed model was able to process data to assess the risk management practices proposed by the RM3 model and determine a maturity level in RM aligned with the decision maker’s perspective.

Furthermore, this study acts on the gap evidenced by Röglinger et. al. (2012), Becker et. al. (2009), & Santos-Neto and Costa (2019) when using the ELECTRE TRI algorithm to standardize procedures for the operationalization of the MM application.

Acknowledgements

This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001, the research programme funded by the Brazilian Research Council (CNPq) and the Federal University of Mato Grosso do Sul (UFMS).

References

• ALIJOYO FA, BONITA I & SIRAIT KB. 2021. The Risk Management Maturity Assessment: The Case of Indonesian Fintech Firm. Paper presented at 4th International Conference on Research in Management & Economics, Milan, Italy, May 7-9.
• BECKER J, KNACKSTEDT R & PÖPPELBUSS J. 2009. Developing maturity models for IT management. Business & Information Systems Engineering, 1(3): 213-222.
• BELTON V & STEWART T. 2002. Multiple criteria decision analysis : an integrated approach. Springer New York.
• BHOSALE AS, RAVI K & PATIL SB. 2018. Risk management maturity model for road construction projects: case study. Risk Management, 5(5): 2473-2482.
• BRITO AJ, DE ALMEIDA AT & MOTA CM. 2010. A multicriteria model for risk sorting of natural gas pipelines based on ELECTRE TRI integrating Utility Theory. European Journal of Operational Research, 200(3): 812-821.
• DIAS L, MOUSSEAU V, FIGUEIRA J & CLIMACO J. 2002. An aggregation/disaggregation approach to obtain robust conclusions with ELECTRE TRI. European Journal of Operational Research, 138(2): 332-348.
• FIGUEIRA J & ROY B. 2002. Determining the weights of criteria in the ELECTRE type methods with a revised Simos’ procedure. European Journal of Operational Research, 139(2): 317-326.
• GONÇALVES ATP, ARAÚJO MVPD, MÓL ALR, & ROCHA FAFD. 2021. Application of the Electre Tri method for supplier classification in supply chains. Pesquisa Operacional, 41.
• HOPKIN P. 2010. Fundamentals of risk management: understanding, evaluating and implementing effective risk management. Kogan Page Publishers.
• HOSEINI E, HERTOGH M & BOSCH-REKVELDT M. 2019. Developing a generic risk maturity model (GRMM) for evaluating risk management in construction projects. Journal of Risk Research, 1-20.
• LACERDA NLB, DOS SANTOS-NETO JBS & MARTINS CL. 2021. MCDM Model for Natural Gas Pressure Reducing Station Site Selection. International Journal of Decision Support System Technology (IJDSST), 13(1): 67-84.
• MIRANDA CMGD, ALMEIDA ATD. 2003. Postgraduate evaluation through ELECTRE TRI method: the case of III engineering area of capes. Production, 13: 101-112.
• MACGILLIVRAY BH, SHARP JV, STRUTT JE, HAMILTON PD & POLLARD SJT. 2007. Benchmarking risk management within the international water utility sector. Part II: A survey of eight water utilities. Journal of Risk Research, 10(1): 105-23.
• MOTALEB OH. 2017. A Model of Risk Response Development for Managing Delays in Construction Projects. International Journal of Project Organisation and Management, 9(2): 133-153.
• MOUSSEAU V, FIGUEIRA J & NAUX JP. 2001. Using assignment examples to infer weights for ELECTRE TRI method: Some experimental results. European Journal of Operational Research, 130(2): 263-275.
• OLIVA FL. 2016. A maturity model for enterprise risk management. International Journal of Production Economics 173: 66-79.
• RODRIGUES KT, MARTINS CL, DOS SANTOS NETO JBS, FOGAÇA DR & ENSSLIN SR. 2022. Decision-Making Model to Assess the Organizational Climate in Healthcare Organizations. International Journal of Decision Support System Technology (IJDSST), 14(1): 1-19.
• ROGERS M & BRUEN M. 1998. Choosing realistic values of indifference, preference and veto thresholds for use with environmental criteria within ELECTRE. European Journal of Operational Research, 107(3): 542-551.
• RÖGLINGER M, PÖPPELBUSS J & BECKER J. 2012. Maturity Models in Business Process Management. Business Process Management Journal, 18(2): 328-46.
• SANTOS-NETO JBS & COSTA APCS. 2019. Enterprise maturity models: a systematic literature review. Enterprise Information Systems, 13(5): 719-769.
• SANTOS-NETO JBS & COSTA APCS. 2023. A Multi-Criteria Decision-Making Model for Selecting a Maturity Model. International Journal of Decision Support System Technology (IJDSST), 15(1): 1-15.
• SAPIENZA G, BRESTOVAC G, GRGURINA R & SECELEANU T. 2016. On applying multiple criteria decision analysis in embedded systems design. Design automation for embedded systems, 20: 211-238.
• SHEEHAN B, MURPHY F, KIA AN & KIELY R. 2021. A quantitative bow-tie cyber risk classification and assessment framework. Journal of Risk Research, 1-20.
• OCHOA CAO, INSUA DV, LEYVA LÓPEZ JC & NORIEGA JJS. 2022. Ranking of a set of accounts receivable strategies in a Mexican regional company based on a multicriteria approach. Pesquisa Operacional, 42.
• TROJAN F & MORAIS DC. 2012. Using Electre TRI to support maintenance of water distribution networks. Pesquisa Operacional, 32: 423-442.
• TUBIS AA & WERBIŃSKA-WOJCIECHOWSKA S. 2021. Risk Management Maturity Model for Logistic Processes. Sustainability, 13(2): 659.
• ULAVAL. Available at: Available at: http://cersvr1.fsa.ulaval.ca/mcda/?q=en/node/4 Accessed in December 2020.
  » http://cersvr1.fsa.ulaval.ca/mcda/?q=en/node/4
• UNGER CJ, LECHNER AM, KENWAY J, GLENN V & WALTON A. 2015. A Jurisdictional Maturity Model for Risk Management, Accountability and Continual Improvement of Abandoned Mine Remediation Programs. Resources Policy, 43: 1-10.
• VON KANEL J, COPE EW, DELERIS LA, NAYAK N & TOROK RG. 2010. Three key enablers to successful enterprise risk management. IBM Journal of Research and Development, 54(3).
• WENDLER R. 2012. The Maturity of Maturity Model Research: A Systematic Mapping Study. Information and Software Technology, 54(12): 1317-1339.
• WIBOWO A & TAUFIK J. 2017. Developing a Self-Assessment Model of Risk Management Maturity for Client Organizations of Public Construction Projects: Indonesian Context. Procedia Engineering, 171: 274-81.
• YEO KT & REN YT. 2009. Risk Management Capability Maturity Model for Complex Product Systems (CoPS) Projects. Systems Engineering, 12(4): 275-94.
• ZHAO X, HWANG BG & LOW SP. 2016. An enterprise risk management knowledge-based decision support system for construction firms. Engineering, Construction and Architectural Management, 23(3): 369-384.
• ZOU PXW, CHEN Y & CHAN TY. 2010. Understanding and Improving Your Risk Management Capability: Assessment Model for Construction Organizations. Journal of Construction Engineering and Management - ASCE, 136(8): 854-63.

APPENDIX A - EVALUATION QUESTIONNAIRE

The following statements assess risk management practices in your organization. Mark the most appropriate option with an ’X’:

Dimension: Management perspective (people and leadership) in relation to risk

1. Upper management actively takes part in risk activities, supports and encourages risk management.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

2. Risk management capacity assessments are carried out for each new project in the organization.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

3. Risk management information distributed and communicated to all project participants within the organization.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

4. Risk management tools and techniques (i.e. FMEA, Preliminary Risk Analysis-PRA, Brainstorming, SWOT) are integrated and used in projects.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

5. Resources are dedicated to projects in accordance with the severity of risk events identified.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

Dimension: Organizational risk culture

6. There is a build-up of trust within the organization and project teams in relation to risk management.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

7. Frequently, team members take risk ownership during project implementation.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

8. Responsibilities for managing risks are distributed and carried out by all team members.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

9. Risk events are openly communicated within the organization.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

10. Risk management is widely accepted and practiced in all levels within the organization.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

Dimension: Identifying risks

11. Potential risks are identified each time for new projects.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

12. A systematic identification method (i.e. FMEA, Preliminary Risk Analysis-PRA, Brainstorming, SWOT) is used to ensure major risks are identified.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

13. Information on risks identified is processed, grouped, and communicated to all project participants.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

14. Risks identified are consistently revised and reevaluated throughout the project process.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

15. Actual risks found are compared against initially identified risks.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

Dimension: Analyzing risk

16. All project participants are capable of basic risk analysis skills such as qualitative or quantitative analysis.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

17. The likelihood of occurrence and magnitude of impacts of a risk is thoroughly assessed upon identification.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

18. Qualitative and/or quantitative risk analysis tools and applications are used to assess identified risks.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

19. After analyzing the analytical results of risks identified, it is used to aid in decision-making for risk responses.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

20. The results of risk analysis are used as a basis for resource allocation and distribution to projects.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

Dimension: Standardized risk management process

21. Risks are consistently identified, analyzed, responded to, and continuously monitored throughout the project life cycle.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

22. The flow of risk management information is passed on and communicated throughout the entire project life cycle.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

23. Risk management processes are woven into the daily business processes of the organization.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

24. A standardized risk management process is applied to all projects within the organization.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

25. The risk management process is reviewed frequently to ensure the process is effective.

	The practice is not widely used in the organization.
	There is a strong discussion about using the practice, but no decision.
	There is the decision and action plan to start using the practice.
	The practice has been tested and experience can be gained.
	The practice is widely used in the organization.

Publication Dates

• Publication in this collection
  09 Oct 2023
• Date of issue
  2023

History

• Received
  06 Dec 2022
• Accepted
  11 Apr 2023