Scenario analysis in the BNDES experience: integrating operational risk management with the measurement of capital

ABSTRACT

Internal operational risk models have not yet been established as a methodology for calculating regulatory capital. These models, which must be integrated with operational risk management, have been criticized for the subjectivity of some of their fundamental elements. The purpose of this paper is to demonstrate the use of the "scenario analysis" element in the Loss Distribution Approach (LDA) methodology for calculating regulatory capital relative to operational risk, based on the experience of the Brazilian Development Bank (BNDES) in integrating operational risk management with the measurement of capital. The proposed methodology, which applied the Delphi technique through questionnaires, enabled: (i) the measurement of regulatory capital considering feasible scenarios; (ii) the identification of tail and body scenarios for the aggregate distribution of losses, which are not reflected in the internal loss database; (iii) the identification and comprehensive measurement of BNDES’s operational risks; (iv) the obtainment of information that can guide risk management with regard to identifying risks that must be given prioritized treatment; (v) the development of a risk culture, with a view to involving specialists from different units; (vi) the use of a methodology that can be understood by all business experts, who are the ones that are aware of the risks of their activities.

Keywords:
scenario analysis; operational risk; internal models; BNDES; Linear discriminant analysis (LDA)

RESUMO

Os modelos internos de risco operacional ainda não se estabeleceram como metodologia para cálculo de capital regulamentar. Esses modelos, que devem estar integrados à gestão do risco operacional, têm sido criticados pela subjetividade de alguns de seus elementos fundamentais. Este artigo tem como objetivo demonstrar a utilização do elemento “análise de cenários” na metodologia loss distribution approach (LDA) para cálculo do capital regulamentar referente ao risco operacional, tendo como base a experiência do Banco Nacional de Desenvolvimento Econômico e Social (BNDES) na integração da gestão do risco operacional com a mensuração do capital. A metodologia proposta, que aplicou a técnica Delphi por meio de questionários, possibilitou: (i) a mensuração do capital regulamentar considerando cenários factíveis; (ii) a identificação de cenários de cauda e de corpo da distribuição agregada de perdas, que não estão refletidos na base de dados interna de perdas; (iii) a identificação e a mensuração dos riscos operacionais do BNDES de forma abrangente; (iv) a obtenção de informações que podem direcionar a gestão do risco no que se refere à identificação de riscos que devem ter o tratamento priorizado; (v) o desenvolvimento de uma cultura de riscos, tendo em vista o envolvimento de especialistas de diversas unidades; (vi) a utilização de uma metodologia compreensível a todos os especialistas de negócios, que são os que conhecem os riscos de suas atividades.

Palavras-chave:
análise de cenários; risco operacional; modelos internos; BNDES; metodologia LDA

1. INTRODUCTION

The specificities of the financial system characterize it as one of the sectors that is most associated with risks. The intermediation of resources between investors and borrowers means that the commitment to the financial equilibrium of these institutions affects thousands of individuals and companies, as well as the local financial system and even the international one when the operations of an institution go beyond the local financial market. This propagator effect is associated with systemic risk, defined by the Bank for International Settlements (BCBS, 1994) as the risk of one participant’s non-fulfillment of its contractual obligations affecting others in the fulfillment of theirs. This risk can create a chain reaction of greater financial difficulties.

Given the importance of the financial sector and the risks that are involved in it, in 1988 the BIS, an international financial organization in which 60 supervisors from countries around the world participate, published the Capital Accord (Basel I) (Basel Committee on Banking Supervision - BCBS, 1988), which proposed guidelines for capital requirements in banks in order to address possible credit risk losses. In 1993, the risk market was incorporated into Basel I. Finally, in 2004, the Second Capital Accord (Basel II) was established, incorporating operational risk.

Given the concern about these risks, minimum capital requirements were defined for financial institutions (reference equity). In the case of Brazilian financial institutions, in December 2017 the reference equity should represent at least 9.25% of risk-weighted assets, which are composed of amounts related to credit, market, and operational risks.

Operational risk is defined as the possibility of occurrence of losses resulting from failure, deficiency, or inadequacy of internal processes, people, and systems, or of external events (Banco Central do Brasil - BC, 2006) and is present in any of an institution’s activities, whether financial or not, given that failures can occur in any process. Such scope makes managing and measuring this risk challenging.

Despite Basel II proposing guidelines for internal operational risk models in 2004, these models have not yet been established as a methodology for calculating capital. In Brazil, no financial institution uses internal operational risk models to calculate regulatory capital.

The non-consolidation of internal operational risk models can be explained by the following factors: (i) the search for totally objective models that are consistent with the subjectivity inherent to operational risk; (ii) the existing gap between the theoretical methodologies and the reality of managing this risk in financial institutions; and (iii) the challenge of forming a database of comprehensive losses.

According to Wahlstrom (2009), in order to achieve the aims of management and operational control on a day-to-day basis, the assumptions that support risk measuring models need to describe the reality, otherwise they fail because they do not guide decision-making. The author also suggests that studies that satisfy purely academic demands should focus on practice and start discussions together with users and players to find solutions to real problems.

The gap between the theoretical methodologies and the reality of financial institutions, in turn, can be explained by the inexistence of research that applies these methodologies to real situations, given that the information about losses from operations risk is treated as confidential information due to the possibility of it compromising the image of these institutions.

The methodology that has been used most for calculating capital related to operational risk is the loss distribution approach (LDA). As required by Basel II, this methodology considers four elements: internal data on losses, external data on losses, scenario analysis, and indicators related to the business environment and to internal controls (Business Environment and Internal Control Factors - BEICFs).

The aim of this study is to demonstrate the use of the scenario analysis element in the LDA methodology to calculate regulatory capital related to operational risk as an integrated element of risk management and capital measurement, using the experience of the Brazilian Development Bank (BNDES) in managing operational risk.

2. LITERATURE REVIEW

The minimum regulatory capital amount related to operational risk should be constituted to absorb severe losses resulting from operational risk events that can compromise the continuity of financial institutions.

Via Circular Letter n. 3,640/2013 (BC, 2013a), the BC defined standardized approaches (RWA_OPAD) for calculating this amount. The advanced measurement approach - AMA, based on internal models at the institution, was defined via Circular Letter n. 3,647/2013 (BC, 2013b), which despite having established the minimum requirements for using the AMA approach for calculating the amount related to operational risk (RWA_OAMA), did not specify the methodology that should be used.

The Basel Committee (BCBS, 2004) defines the four minimum elements of the AMA approach:

• The “data on internal losses” element refers to the losses occurring at the institution and is an essential pre-requisite for the development and functioning of a reliable system for measuring operational risk;

• The “data on external losses” element refers to the operational losses occurring at other financial institutions and should be mainly used when there are reasons to believe that the institution is exposed to infrequent, but potentially severe losses;

• The “scenario analysis” element, derived from the opinion of experts, should be used to evaluate the institution’s exposure to high severity events. This element adds a future vision that is not given by the data on internal and external losses elements; and

• The “key factors of the business environment and internal controls” element should reflect the change in operational risk profile. These factors will give the risk assessments a forward-looking view, recognizing both improvements and deteriorations in the risk profile in a more immediate way.

Rodriguez, Domínguez, and Marín (2009) state that the LDA methodology is the most sensible for use in internal models for measuring operational risk. Arbeláez and Ceballos (2010) claim that the LDA methodology is the most widely used within the scope of the AMA approach.

In this section, the general guidelines for the LDA methodology and the scenario analysis element will be presented, given the aim of this study: to propose a methodology that integrates the scenario analysis element with the LDA methodology for calculating the minimum capital required to cover operational risk.

2.1 LDA Methodology

According to Cruz, Peters, and Shevchenko (2015), the LDA methodology is based on the annual modeling of the frequency N and severity X1, X2, …, X_n of the operational risk events of each cell in the matrix of risk of losses classified by business line [defined by the BC in Article 4 of Circular Letter 3,640/2013 (BC, 2013a) and by the Basel Committee in Annex 8 of Basel II] and by type of operational risk event [defined by the National Monetary Council in Article 32 of Resolution 4,557/2017 (BC, 2017)]. So, the annual loss for each risk cell is calculated by aggregating the severities for a one-year time horizon (Z(j)) and the total loss in a particular year is obtained by the sum of all the cells in the risk matrix (Z).

$$Z\left(j\right)=X_1\left(j\right)+X_2\left(j\right)+\mathrm{...}+X_N\left(j\right)$$ (1)

$$Z=\Sigma Z\left(j\right)$$ (2)

where N is the number of loss events, X is the value of the loss per event, and j is each cell in the loss matrix classified by business line and type of operational risk event.

As basic steps of the LDA methodology, Fontnouvelle, Dejesus-Rueff, Jordan, and Rosengren (2006), and Frachot, Moudoulaud, and Roncalli (2003) indicate: (i) estimation of severity; (ii) estimation of frequency; (iii) computation of capital via the aggregate distribution of losses obtained based on the frequency and severity distributions using the Monte Carlo simulation or another equivalent technique.

The severity estimation covers selecting the theoretical distribution that best adjusts to the loss values (empirical distribution). According to Cruz et al. (2015), it is possible to use one distribution to model the body and another to model the tail. This occurs because the data on losses from operational risk present fat tails. Thus, one single distribution is not always enough to represent the empirical data.

The frequency estimation covers selecting the discrete theoretical distribution that best adjusts to the annual number of events.

After estimating the parameters of the theoretical distributions, it is necessary to select the distribution that will be considered in the model. Cruz et al. (2015) indicate that the selection can occur via: (i) qualitative diagnostic, such as Q-Q plots and P-P plots; (ii) tail diagnostic, used to evaluate more adequate distributions for fat severity distribution tails, such as mean excess (ME) plots and Hill plots; (iii) Akaike information criteria; and (iv) goodness-of-fit (GOF) tests, such as Kolmogorov-Smirnov (KS), chi-square, Cramer-von-Mises (CVM), and Anderson-Darling (AD). The aim of these tests is to indicate which theoretical distributions best adjust to the empirical distribution.

After calibrating the frequency and severity distributions, the capital computation is carried out after the convolution of the theoretical distributions selected. The most widely used methods for convolution of distributions in internal operational risk models are: simulation, such as the Monte Carlo simulation, and methods that are considered exact, such as the Panjer resource (2006) and fast Fourier transform (FFT).

The regulatory capital is defined as the value at risk (VaR) related to the 99.9 percentile of the aggregate distribution, which is the percentile of the annual distribution of losses Z for the next year (OpVar).

2.2 Scenario Analysis

The Basel Committee and the BC do not establish the way to use the scenario analysis element and there is also no consensus in the literature regarding the best way to use this element, which is considered very subjective in various studies presented on this topic.

Ergashev (2012) defines scenarios as hypothetical operational risk events at an institution or as is widely known in the financial literature, inherent risks. They can thus be useful for prospective adjustments in the frequency and severity of historical events for anticipating changes in the institution’s risk profile.

In order to elaborate this study, a review was carried out of how previous studies have addressed the scenario analysis element within the scope of the LDA methodology. The methodologies used in these studies are summarized below.

For Frachot et al. (2003), scenario analysis expresses the perception of experts and experienced managers regarding the risk of their activities. This perception would not be reflected in the historical data on losses and should be connected with the frequency and severity estimates. As a methodology for using scenario analysis in the LDA methodology, the authors indicated restricting parameters of the frequency and severity distributions. The authors did not present any methodology for defining the scenarios.

McConnell and Davies (2006) proposed the use of the Bow-Tie diagram, a technique used to analyze aviation company safety and generate operational risk scenarios, which aims to answer two fundamental questions regarding the frequency and severity of the scenarios. According to the authors, the Bow-Tie diagram makes assumptions, carries out analyses, explains conclusions related to risk management, and requires an additional step, which is the estimation of the numerical outputs for use in the scenarios in the AMA models. However, this additional step was not shown in the study.

Alderweireld, Garcia, and Léonard (2006) defined a scenario analysis quantification - SAQ technique with the following main objectives: (i) constructing a questionnaire that was understood by all operational risk managers; and (ii) transforming the questionnaire into an accurate capital element. The questionnaires had four questions related to the frequency of losses and to the average value. The distribution of frequency considered was the Poission one and the severity distribution was defined for distributions with two parameters (first two moments) defined in accordance with the average. According to the authors, the methodology presented enabled the construction of the distribution of losses perceived by the experts, even by those that had no knowledge of statistics. However, the authors did not suggest any way of combining these scenarios with the other elements of the LDA methodology.

In addition, Steinhoff and Baule (2006) argued that if the experts are directly asked what type of distribution is appropriate for a particular category of risk, their answer is not normally useful since when faced with this question they are being pressured to think about an unfamiliar statistical dimension. The argument used by Steinhoff and Baule (2006) is that this way of determining scenarios, besides not being popular in the banking industry, is normally incomprehensible for the experts. In this context, the authors proposed a methodology for defining scenarios using questionnaires with questions related to the severity and frequency of events. For the authors, the scenarios obtained could be directly transformed into parametric distributions or used as expert estimates to form a “virtual” database. In the didactic example presented by the authors, as the scenarios and data on internal losses were adjusted by the same severity distribution, it was possible to reflect the two elements in the distribution parameters.

Aue and Kalkbrener (2007) presented the quantitative aspects of the LDA model of Deutsche Bank. The fundamental structure of the model was conceived to meet the requirements of Basel II. The scenario analysis element was used to supplement the data on internal losses with the aim of modeling the severity distribution (direct input).

Folpmers (2008) implemented the LDA methodology using the opinion of experts. The Poisson distribution parameters were used to model the frequency and the triangular distribution parameters were used to model the severity. For the author, these two distributions were chosen because they use parameters that can be easily interpreted: the annual average of the number of events, in the Poisson case, and maximum, average, and minimum loss, in the triangular case, which would enable an effective process for extracting expert knowledge.

Pontes (2009) cites some methods for someway attaching the opinion of experts to the model: (i) fuzzy logic; (ii) Bayensian inference; (iii) dynamic systems; and (iv) parameter estimation by experts. After analyzing these techniques, which make the use of historical data and expert judgments feasible, the author applied the LDA methodology to a non-financial company, using a method in which the experts infer frequency and severity distribution parameters.

Guegan and Hassani (2011) proposed a methodology for experts to define scenarios to create a synthetic database. The experts defined generalized extreme value (GEV) distribution parameters for each cell in the Basel matrix and for different levels of granularity. The authors compared the financial institution’s capital value using the proposed methodology and the LDA methodology (data on internal losses). With this, they showed that the capital value is higher for the scenario definition methodology than for the LDA methodology.

Also with regards to applying the scenario definition method, Rippel and Teply (2011) presented a methodology for combining internal losses with the scenario analysis that used the worst case of loss defined for one scenario in particular and the average loss given by a probability of loss distribution defined for a scenario. In both approaches, the values of the scenarios were added to the records on historical losses to determine the frequency and severity distribution parameters.

Momen, Kimiagari, and Noorbakhsh (2012) applied the LDA method in a commercial bank. In this case, the scenario analysis element was used as a direct input; that is, the scenario values were used together with the internal data for frequency and severity modeling. For the authors, this type of methodology for defining scenarios is much better understood by managers and adds the benefit of expert opinion to the capital measurement.

However, Ergashev (2012) quantified scenarios with the frequency estimate that represented the worst case in a time interval and with the severity estimate that represented the lowest value of the severity interval defined by the experts. The author compared the severity distribution adjusted to the data on internal losses with the scenarios and adjusted the corresponding quantiles of the distribution when the scenarios were not compatible with the distribution.

Shevchenko and Peters (2013) indicated that scenarios can be used both in managing operational risk and in the LDA methodology. According to them, the elicitation of experts is a challenge, given that many managers and employees may not have knowledge regarding statistics and probability theory, which can lead to a misleading result and poor understanding. The study did not present any methodology for defining scenarios.

Hassani and Renaudin (2013) proposed a methodology for combining the internal and external data elements and scenarios obtained in workshops, using the Bayensian approach. However, the study did not present any methodology for defining scenarios nor was there any application in a real case.

Karam (2014) applied the LDA methodology by combining internal data on losses with the scenarios using Bayensian networks. The study also did not present any methodology for defining scenarios. The author draws attention to the fact that the capital requirement is highly influenced by these estimates. This study is similar to the one from Lambrigger, Shevc Henko, and Wuthrich (2007).

Despite not having presented any proposed methodology for defining scenarios, Ames, Schuermann, and Scott (2015) argued that the scenarios should be more important for the internal operational risk models, given that they are a formal mechanism for incorporating the opinion of experts into the models. The authors indicated that the use of this element aims to increase and not reduce the capital, which is highly sensitive to this element. Finally, the authors indicated that the subjectivity of the scenarios can be used to obtain a reward in exchange for a lower regulatory capital.

Dutta and Babbel (2014) proposed the change of measure (COM) approach, which combines scenario analysis with data on internal losses and enables the impact of each scenario on the capital value to be evaluated. Via the proposed model, the scenarios are defined in workshops by experts, who define estimates for the severity and for the frequency of the scenarios in a period equivalent to that of the internal losses database. The frequency and severity distribution parameters of the data on internal losses are adjusted to reflect the probability of the scenarios. The authors applied the methodology using 16 scenarios in a financial institution whose name was not revealed.

It is observed that these studies present some characteristics that do not favor the application of these methodologies in a real case because:

• they are totally theoretical (Alderweireld et al., 2006; Ergashev, 2012; Ergashev, Blei & Abdymomunov, 2015; Frachot et al., 2003; Hassani & Renaudin, 2013; Karam, 2014; Lambrigger et al., 2007; Mc Connell & Davies, 2006; Shevchenko & Peters, 2013; Steinhoff & Baule, 2006). The studies do not show real difficulties that can be faced by financial institutions. The confidentiality with which the information on operational risk is treated may explain why applications of real cases are rare in theoretical studies.

• the proposed methodologies require statistical knowledge that may not be common to all the experts that define the scenarios (Frachot et al., 2003; Guegan & Hassani, 2011; Hassani & Renaudin, 2013; Karam, 2014; Lambrigger et al., 2007; Pontes, 2009; Schevchenko & Peters, 2013), for example: adjustments in parameters of the frequency and severity distributions.

• the studies were applied one-off, such as: Dutta and Babbel (2014), Folpmers (2008), and Rippel and Teply (2011).

This study contributes to the topic by proposing a methodology that is easily understood by business unit experts in order to obtain scenarios. To propose the methodology, some assumptions were extracted from the studies chosen, such as: (i) the use of distributions with easily understood parameters, such as the Poisson distribution and triangular distribution (Folpmers, 2008); (ii) the use of information such as direct input in the LDA methodology (Aue & Kalkbrener, 2007; Momen et al., 2012).

3. METHODOLOGY

Based on the literature review and on the observations derived from the experiences of BNDES in operational risk management, the model shown in this study was structured as indicated in Figure 1.

Figure 1
Structure of the model implemented

The model aims to define the scenarios by measuring the operational risk events identified by the units. The risk measurement is made by the experts and compared until there is a consensus in relation to the measurement.

After defining the scenarios, as in the stages indicated in Table 1, a synthetic database was formed, which together with the internal data on losses (direct input), was used to measure the capital within the scope of the LDA methodology.

Table 1
Stages of the methodology

Stages of the methodology		Aim
1st	Zero round survey of occurred and potential events in the areas (first round of the questionnaire)	To survey the operational risks of all the areas
2nd	Survey of opinion to define the measurement methodology	To identify the methodology that the potential experts judge to be the most adequate for measuring the events
3rd	Measurement of risks by the experts (second round of the questionnaire)	To measure the events reported by the areas in the first round of the questionnaire
4th	Comparison of the measurements from the second round (third round of the questionnaire)	To compare the measurements from the second round of the questionnaire

• Source: Elaborated by the authors.

In the first stage of the methodology, the units should report operational risk events that in their view have already occurred or may occur in their processes. The report should occur in a non-structured or zero round way with open questions to guarantee that the events are indicated freely (extraction of scenarios using a bottom-up approach). To facilitate this reflection, the units could have access to different information, such as data on external losses, indicators related to the business environment etc.

In the second step of the methodology, a survey of opinion was carried out regarding how comfortable the potential experts feel in relation to the most widely used methodologies for measuring the scenarios, given their knowledge of statistics and probability theory. The survey was motivated by some studies, such as the one from Schevchenko et al. (2013), who indicated that the elicitation of experts is a challenge given that many managers and employees may not have knowledge of statistics and probability theory, which could lead the research to a misleading and poorly understood result.

In the third and fourth steps of the methodology, the events reported in the first were measured by the experts. The measurements were compared so that the experts could indicate whether they would alter measurements after having knowledge regarding the measurements carried out by the other experts.

After the measurement, the scenarios formed a synthetic database of losses that was used together with the data on internal losses to calculate the capital, in accordance with the steps indicated by Cruz et al. (2015).

3.1 Application of the Methodology

BNDES is a federal public company controlled entirely by the Union and that was created in 1952. Its main objective is to provide long term financing, to address the deficiency of private sources of this type of financing, and to invest in different segments of the Brazilian economy.

The application of this study at BNDES occurred as a result of the institution’s interest in developing an internal model for operational risk, and the bank gave us permission to use its information as long as confidential company information was properly protected.

3.1.3 First stage - Applying the methodology for identifying the scenarios - First round of the questionnaire.

After presentations were carried out to each one of the 24 areas of BNDES with regards to filling in the questionnaire and about the concepts of operational risk, the questionnaires were applied. Each department filled in the questionnaire, indicating the occurred and potential events, as well as their effects and the mitigation measures.

The areas of BNDES reported 321 occurred and potential loss events. Of these, 98 were excluded from the model for the following reasons: (i) events that are already captured for recording in the database on losses and reflected in the internal losses; (ii) events that can interrupt the activities of BNDES because they have a specific methodology for measuring financial impact, which includes other impacts besides possible losses, but not frequency. Thus, 223 were sent for measurement in the second round of the questionnaire.

Given the secrecy with which this information is treated by BNDES, the events indicated cannot be detailed in this study.

3.1.2 Survey of opinion to define the measurement methodology.

The survey, carried out in September 2015, was sent to 524 employees with the position of trust of head of department and manager, which are the positions of the potential experts who would measure the events. Of these, 170 employees answered the survey.

The results analysis indicated that most of the respondents (73%) claimed not to have advanced knowledge of statistics and probability. In relation to the methodologies presented, 68% considered that they were able to estimate how many times an event had occurred or could occur in a particular time period, 53% did not consider that they were able to estimate the minimum, average, and maximum loss values, and 60% did not consider that they were able to estimate probability distribution parameters. Finally, between the two methodologies presented, 79% considered that they were able to estimate the number of times that an event could occur with the minimum, average, and maximum loss values that the probability distribution parameters could estimate.

In light of the results of the survey and with the aim of mitigating the possibility of misleading results, the methodology selected for application in this study was measurement of the number of times that an event can occur in a particular time period and the minimum, average, and maximum loss values.

3.1.3 Application of the methodology for identifying the scenarios - Second round.

The questionnaires from this round were sent in November 2015 to each expert indicated by the areas via email. Two hundred thirty-nine experts took part in this round.

The following time interval options were given to the experts to estimate the frequency of the events: five, ten, or fifteen years, so that it was possible to adapt the frequency of the events to the five-year internal losses database used.

With the aim of facilitating the measurement of the severity of the events, the areas could use some internal references. For example, if the risk associated with the event could mean the resources released in a credit operation were not recovered, the areas could use their credit operations as a reference. The highest value credit operation would be equivalent to the maximum loss that the event could cause, the average value credit operation would be the average loss, and the lowest value credit operation would be the minimum loss. These internal references for defining the severity of the scenarios were called internal business indicators (IBIs).

Of the 223 events measured in this round:

• 77 were excluded for these main reasons: (i) they may cause a judicial process that would require a specific methodology to consider jurisprudence, districts, and other information; (ii) they might not cause a loss after a more detailed evaluation by the experts; and (iii) they should have been analyzed by a different area from the one that indicated the risk (the severity measurement could not be carried out by the same area that indicated the risk, for example, when the loss related to the inavailability of systems can only be measured by the areas that use the systems and not by an area such as Information Technology.

• 70 were measured in non-compliance with the proposed methodology, despite the instructions given, and with this they were excluded. These cases contemplated: (i) measurements with indications that they were carried out together, despite the instructions given; (ii) risks measured by only one expert due to the others being on vacation or a business trip; and (iii) measurements considered complementary in the cases in which the scope of measurement was different, such as a flaw in the credit operation documentation, in which the experts are assigned to different management units, and consequently with different operations portfolios.

After the exclusions, 76 events remained to compare the measurements in the third round.

3.1.4 Applying the methodology to identify the scenarios - 3 ^rd round.

After the experts measured the scenarios, all the measurements carried out for the same scenario were sent to them, without identifying the others, for them to indicate whether they agreed or whether they would change the measurements from the second round.

Of the 76 events whose measurements were compared in the third round:

• 23 were excluded in the third round for the same reasons as the events excluded in the second round; and

• seven were measured in non-compliance with the proposed methodology, despite the instructions given, and with this they were excluded.

After the exclusions, 46 events remained to use as scenarios within the scope of the LDA methodology.

In the cases in which the experts did not reach a consensus (divergent measurements), the measurements whose fundamentals were presented were considered. Some experts presented arguments contesting other measurements.

Finally, of the 46 events that remained to compare the measurements in the third round, 10 were measured via IBIs and 36 via the indication of maximum, average, and minimum losses.

3.2 Limitations of the Method

Since a case study is concerned, the generalizations of this study should be made with caution, given that the study was carried out at an institution that, like others, has its own characteristics and structure.

In the application of the questionnaires, it is possible that, intentionally or not, the areas did not report events or even underestimated losses (Ames et al., 2015). It is recommended that in other applications of this methodology comparisons are carried out between the database of losses and other questionnaires, or even with the internal controls and auditing report, in order to reduce the chances of some events not being considered.

Finally, there is no way to guarantee that the experts really carried out the measurements individually with the desired independence.

4. PRESENTATION AND ANALYSIS OF THE RESULTS

In this section the results of applying the proposed methodology are presented along with the analysis of these results.

From this section onwards, the measurements of the events used to form the synthetic database of losses will be considered to calculate the operational risk capital. These scenarios consider risks existing in the institution that are not reflected in the internal losses database.

4.1 Constitution of the Synthetic Database of Losses

After defining the number of measurements to be considered, a database of synthetic losses was created with the scenarios measured in conformity with the proposed methodology, in accordance with the following criteria: (i) being constituted for a period of five years (the same period as the database of internal losses used in this study); (ii) considering the frequency of each measurement based on the number of records created, for example: if the scenario can occur once for a five-year period, only one record related to the scenario was created; (iii) also considering only one record for an event occurring in a period greater than five years (this procedure is considered to be conservative, given that the scenario could materialize only after five years); and (iv) considering the averages of the triangular distributions to define the values of the losses. The use of random values for these distributions would lead to instability in the databases with each simulation carried out.

Table 2 indicates some descriptive statistics from the database of synthetic losses.

Table 2
Descriptive statistics from the database of synthetic losses

	Database of synthetic losses for the scenarios measured in conformity with the methodology
Total number of records	277
Total value of records (in R$)	4,378,733,593.00
Maximum value (in R$)	538,335,837.00
Average value (in R$)	15,807,703.00
Minimum value (in R$)	1,350.00
Maximum annual frequency	12
Average annual frequency	4
Minimum annual frequency	1

• Source: Elaborated by the authors.

4.2 Application of the LDA Model - Internal Losses

Taking the first four steps defined by Frachot et al. (2003) for application of the LDA model as a reference, in this section the minimum capital required values (OpVaR) using the LDA methodology will be presented, considering only the internal losses occurring in the last five years (Model 1) and the internal losses plus the scenarios measured in accordance with the proposed methodology (Model 2). The OpVision was the system used to calculate the capital via the LDA methodology.

4.2.1 Severity estimation.

The tail and the body of the severity distribution were modeled separately. To model them, the OpVision calculated the parameters of the following continuous distributions: burr, empirical, generalized extreme value (GEV), gamma, inverse Gaussian, G and H, linear, lognormal, heavy tailed lognormal, gamma mixture, lognormal-gamma mixture, lognormal mixture, and generalized Pareto and Weibull. After calculating the parameters, the system orders the distributions into descending order of the minimum p-value between the KS and AD tests, indicating: (i) a green sign when the lowest p-value from the KS and AD tests is higher than 0.01; (ii) a yellow sign when the lowest p-value from the KS and AD tests is between 0.001 and 0.01; (iii) a red sign when the lowest p-value from the KS and AD tests is less than 0.001.

As none of the distributions presented a p-value higher than 0.01, the empirical distribution was selected to model the body of the distributions of the two models.

To model the tail of the distributions of the two models, nine parameters were calculated. In the case of Model 1, the lognormal mixture was the only distribution that presented p-values higher than 0.01 in all of the tests; for this reason it was selected. With regards to Model 2, despite the lognormal mixture distribution having presented a minimum p-value higher than 0.01 for the tail of the distribution, the empirical distribution was used to model the whole distribution, as will also be indicated in section 4.2.3.

4.2.2 Frequency estimation.

To estimate the distribution that best fits the frequency of the distributions of the two models, the OpVision calculated the parameters for the discrete distributions: empirical, negative binomial, Poisson, and Cox process. All the theoretical distributions presented p-values lower than 0.001. For this reason, the empirical distribution was used in the two models.

4.2.3 Computing the capital and confidence interval.

The value of the OpVar of Model 1, obtained via the Monte Carlo simulation with the determination of a relative error of 1% at most, was R$ 219,688,403.00. This value represents the 99.9% percentile of the aggregate distribution of the losses. The relative error of 1% indicates that, in the capital computation, the percentile of the distribution of losses was correctly calculated with a margin of error of 1% at most.

The value of the OpVar of Model 2, obtained via the Monte Carlo simulation with the determination of a relative error of 1% at most, was R$ 3,084,486,612.00, which represents the 99.9% percentile of the aggregate distribution of the losses.

The tail and the body of the severity distribution of Model 2 were not modeled separately, despite the lognormal mixture distribution having presented a minimum p-value higher than 0.01 for the tail of the distribution, because it did not present a realistic value.

4.3 Backtesting

The aim of the backtesting was to compare the results of the models with the data on internal losses with the aim of verifying the performance of the OpVar calculation.

Given that a five-year database was considered and the minimum period of the database for calculating the capital defined by the BC is three years (BC, 2013a), it was possible to carry out the backtesting considering three and four year periods.

Table 3 summarizes the results found.

Table 3
Backtesting

	Severity			Frequency
Backtesting	Distribution of the body	Distribution of the tail	Distribution	OpVar (R$)
3-year database (2011-2013)	Empirical	-	Empirical	4,014,730,096.00
4-year database (2011-2014)	Empirical	-	Empirical	3,461,236,776.00

• Source: OpVision.

The scenarios were determined for a period of five years, with year 1 being the most recent and having the highest number of records in value and quantity, and year 5 being the oldest. To carry out the three-year backtesting (2011 to 2013), years 4 and 5 were eliminated from the scenarios. To carry out the four-year backtesting (2011 to 2014), year 5 was eliminated from the scenarios. Thus, the years with the lowest number of records were eliminated in order to keep the highest number of scenarios. For this reason (keeping the highest value scenarios and reducing the time window), the OpVar of the three-year backtesting is higher than that of the four-year one.

In both cases, the OpVar values were much higher than the net losses occurring in 2014 and 2015, which cannot be disclosed for confidentiality reasons. Thus, the OpVar values calculated using the proposed methodology would be sufficient to face the occurred losses.

4.4 Comparison of the OpVar with the Minimum Capital Required by the Basic Indicator Approach (BIA)

The BIA was defined by the BC via Circular Letter 3,640/2013 (BC, 2013a) as one of the standardized approaches for calculating the minimum capital related to the operational risk of Brazilian financial institutions. This is the approach that is currently used by BNDES for calculating capital.

To sum up, with the BIA approach the capital value is equal to 15% of the gross income from the financial intermediation, ignoring provisions. The higher the gross income from the financial intermediation, the higher the capital value. This is the main reason for criticizing the standardized approaches.

Figure 2 compares the minimum capital values required for BNDES’s operational risk via the BIA approach (base date December/2015) and via the LDA methodology, using only the data on internal losses (Model 1) and using the data on internal losses and the scenarios measured in accordance with the methodology proposed in this study (Model 2).

Figure 2
Comparison of the minimum capital required of the Brazilian Development Bank (BNDES) LDA = loss distribution approach.

The capital value derived from Model 1 is much lower than the capital value derived from using the BIA approach; however, it is sufficient to cover BNDES’s historical net operational losses, which cannot be disclosed due to the secrecy with which the institution classifies this information.

The capital value derived from Model 2 reflects operational risk events that have not occurred but that could occur in the view of the areas of BNDES and their experts. The capital should be enough to cover these risks, even though they have not materialized.

The fact that Figure 2 indicates that the capital value derived from the proposed methodology is close to the capital value derived from the BIA methodology is a mere coincidence, since the capital value derived via the BIA approach is simply a function of the gross income from BNDES’s financial intermediation in recent years.

5. CONCLUSION

The internal models for calculating the regulatory capital requirement related to operational risk were not developed with the credit and market risk models. This can be attributed to the characteristics of operational risk, which do not depend on one contract or on one investment to materialize. For this reason, besides severity, frequency (occurring or not) is a variable to be considered in these models.

The absence of extensive internal databases that contemplate all the risks to which a financial institution is exposed created the need to use other elements, such as data on external losses and scenarios, to overcome this deficiency. However, these elements are criticized because of their subjectivity.

In addition, the regulators demand that these models are integrated into risk management. Obtaining an extensive database and knowledge of the risks to which an institution is exposed require effective risk management.

Thus, the aim of this study was to show how the scenario analysis element can be used in order to integrate operational risk management with the measurement of capital.

The proposed methodology enabled the following results: (i) the use of a methodology that is understood by the experts from the different areas, who are the ones that know the risks of their activities, and which did not require knowledge of statistics and probability theory, which impedes defining and measuring the scenarios; (ii) the measurement of regulatory capital considering feasible scenarios to which BNDES is exposed (adjusted to its reality), attending to the regulators regarding the use of the scenario analysis element; (iii) the identification of scenarios not only from the tail, but also from the body of the aggregate distribution of losses, which are not reflected in the database of losses; and (vi) the obtainment of information that can guide risk management with regards to identifying risks that should receive prioritized treatment.

In terms of advantages in relation to previous studies, this study: (i) used a methodology for the comprehensive analysis of scenarios in the institution; (ii) presented, in a practical and direct way, how to define and measure the scenarios, as well as using them to calculate capital; (iii) used real data to demonstrate the methodology, which is rare in studies related to operational risk because of the company secrecy involving such information. Many studies address the problem without, however, presenting any practical application of a methodology.

Finally, defining and evaluating scenarios provides an important flow of information on risk management (Scenario-based AMA Working Group, 2003) among the units, which develops the institution’s risk culture. Thus, the proposed methodology has contributed to the development of a risk culture.

Regarding the external validity, the proposed method can be applied in any institution, given that it requires basic knowledge about statistics and probability theory from the experts. However, it is possible that the fact that BNDES is a public company with a low employee turnover means that the experts carry out the activities for enough time for them to have broad knowledge of the risks involved in their execution.

Despite all the benefits of implementing the proposed method, it requires a considerable amount of time from the experts and especially from the operational risk management unit, which consolidates all the information and supports all those involved. This represents a relevant cost in relation to the standardized approaches.

In addition, the use of internal models will probably not lead to capital savings since these models contemplate robust estimates of expected and unexpected losses, which are not considered in the standardized approaches.

It is important to highlight that successive applications of this methodology will result in the experts being more mature in relation to operational risk and will consequently increase the quality and precision of the measurements carried out.

Some improvements can be implemented in future applications, such as a single measurement of risks that can occur in various units. As a suggestion for future studies, the proposed methodology could be used to carry out stress tests, if a percentile of the distribution is defined to establish the severity of the scenario instead of the average of the triangular distribution, for example.

Finally, the Basel Committee reported the replacement of all current standardized approaches and of the internal models for measuring operational risk, from 2022, by a single standardized model that uses internal losses for measuring capital: standardized measurement approach (SMA) (Basel Committe on Banking Supervision - BCBS, 2017). Despite the SMA reducing some weaknesses of the current standardized approaches and considering the database of internal losses, this methodology determines the capital value with reference to the past (occurred losses). It does not reflect potential risks (scenarios) and the risk profile of institutions. Maintaining the internal models at the same time as the SMA approach would be a way of enabling the comparison between the institutions’ capitals and continuing to incentivize the search for a robust internal model for measuring operational risk.

REFERENCES

• Alderweireld, T., Garcia, J., & Léonard, L. (2006). A pratical operational risk scenario analysis quantification. Risk Magazine, 19(2), 93-95.
• Ames, M., Schuermann, T., & Scott, H. S. (2015). Bank capital for operational risk: a tale of fragility and instability. Journal of Risk Management in Financial Instituitions, 8(3), 227-243.
• Arbeláez, L. C. F., & Ceballos, E. V. (2010). Alternativas fundamentales para cuantificar el riesgo operacional. Ecos de Economia, 14(30), 1-43.
• Aue, F., & Kalkbrener, M. (2007). LDA at work: Deutshe bank’s approach to quantifying operational risk. Journal of Operational Risk, 1(4), 49-93.
• Banco Central do Brasil. (2006). Resolução n. 3.380 de 29 de junho de 2006. (2006, 29 de junho). Dispõe sobre a implementação de estrutura de gerenciamento do risco operacional. Retrieved from https://www3.bcb.gov.br/normativo/detalharNormativo.do?N=106196825&method=detalharNormativo
  » https://www3.bcb.gov.br/normativo/detalharNormativo.do?N=106196825&method=detalharNormativo
• Banco Central do Brasil. (2013a). Circular n. 3.640 de 4 de março de 2013. (2013, 4 de março). Estabelece os procedimentos para o cálculo da parcela dos ativos ponderados pelo risco (RWA), relativa ao cálculo do capital requerido para o risco operacional mediante abordagem padronizada (RWA_OPAD), de que trata a Resolução nº 4.193, de 1º de março de 2013. Retrieved from http://www.bcb.gov.br/pre/normativos/circ/2013/pdf/circ_3640_v1_O.pdf
  » http://www.bcb.gov.br/pre/normativos/circ/2013/pdf/circ_3640_v1_O.pdf
• Banco Central Do Brasil. (2013b). Circular n. 3.647 de 4 de março de 2013. (2013, 1 de outubro). Estabelece os requisitos mínimos para a utilização de abordagem avançada, baseada em modelo interno, no cálculo da parcela relativa ao risco operacional (RWA_OAMA), dos ativos ponderados pelo risco (RWA), de que trata a Resolução nº 4.193, de 1º de março de 2013. Retrieved from http://www.bcb.gov.br/htms/Normativ/CIRCULAR3647.pdf
  » http://www.bcb.gov.br/htms/Normativ/CIRCULAR3647.pdf
• Banco Central do Brasil. (2017). Resolução n. 4.557 de 23 de fevereiro de 2017. (2017, 1 de março). Dispõe sobre a estrutura de gerenciamento de riscos e a estrutura de gerenciamento de capital. Retrieved from http://www.bcb.gov.br/pre/normativos/busca/downloadNormativo.asp?arquivo=/Lists/Normativos/Attachments/50344/Res_4557_v1_O.pdf
  » http://www.bcb.gov.br/pre/normativos/busca/downloadNormativo.asp?arquivo=/Lists/Normativos/Attachments/50344/Res_4557_v1_O.pdf
• Basel Committe on Banking Supervision. (1988). International Convergence of Capital Measurement and Capital Standards Retrieved from http://www.bis.org/publ/bcbs04a.pdf
  » http://www.bis.org/publ/bcbs04a.pdf
• Basel Committe on Banking Supervision. (1994). 64th Annual Report Retrieved from https://www.bis.org/publ/arpdf/archive/ar1994_en.pdf
  » https://www.bis.org/publ/arpdf/archive/ar1994_en.pdf
• Basel Committe on Banking Supervision. (2004). International Convergence of Capital Measurement and Capital Standards . A revised framework Retrieved from http://www.bis.org/publ/bcbs107.pdf
  » http://www.bis.org/publ/bcbs107.pdf
• Basel Committe on Banking Supervision. (2017). Finalising Basel III Recuperado de http://www.bis.org/bcbs/publ/d424.pdf.
  » http://www.bis.org/bcbs/publ/d424.pdf.
• Cruz, M. G., Peters, G. W., & Shevchenko, P. V. (2015). Fundamental aspects of operational risk and insurance analytics - A handbook of operational risk (8th ed.). Hoboken, NJ: Wiley.
• Dutta, K. K., & Babbel, D. F. (2014). Scenario analysis in the measurement of operational risk capital: a change of measure approach. Journal of Risk and Insurance, 81(2), 303-334.
• Ergashev, B. A. A. (2012). A theoretical framework for incorporating scenarios into operational risk modeling. Journal of Finance Services Research, 41(3), 145-161.
• Ergashev, B. A., Blei, S., Abdymomunov, A. (2015). Integrating stress scenarios into risk quantification models. Journal of Finance Services Research, 47(3), 57-79.
• Folpmers, M. A. (2008). Practical guide to measuring operational risk using subjective data through copulas and scenarios analysis. Journal of Operational Risk , 3(3), 63-74.
• Fontnouvelle, P., Dejesus-Rueff, V., Jordan, J. S., & Rosengren, E. S. (2006). Capital and risk: new evidence on implications of large operational losses. Journal of Money, Credit and Banking, 38(7), 1819-1846.
• Frachot, A., Moudoulaud, O., & Roncalli, T. (2003). Loss distribution approach in practice. Social science research network Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1032592
  » http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1032592
• Guegan, D., Hassani, B. (2011). A mathematical resurgence of risk management: na extreme modeling of expert opinions. Documents de travail du Centre d’Economie de la Sorbonne. Retrieved from https://halshs.archives-ouvertes.fr/halshs-00639666
  » https://halshs.archives-ouvertes.fr/halshs-00639666
• Hassani, B., & Renaudin, A. (2013). The cascade bayesian approach for a controlled integration of internal data, external data end scenarios Retrieved from https://halshs.archives-ouvertes.fr/halshs-00795046/document
  » https://halshs.archives-ouvertes.fr/halshs-00795046/document
• Karam, E. (2014). Measuring and managing operational risk in the insurance and banking sectors. Business administration Lyon, France: Université Claude Bernard.
• Lambrigger, D., Shevc Henko, P. V., & Wuthrich, M. V. (2007). The quantification of operational risk using internal data, relevant external data and expert opinion. Journal of Operational Risk , 2(3), 3-27.
• McConnell, P., & Davies, M. (2006). Safety first - Scenario analysis under Basel II Retrieved from http://www.continuitycentral.com/feature0338.htm
  » http://www.continuitycentral.com/feature0338.htm
• Momen, O., Kimiagari, A., & Noorbakhsh, E. (2012). Modelling the operational risk in Iranian commercial banks: case study of a private bank. Journal of Industrial Engineering International, 8(15), 1819-1846.
• Panjer, H. H. (2006). Operational risk modeling analytics. Wiley Series in probability and statistics Hoboken, NJ: Wiley .
• Pontes, A. R. (2009). Risco operacional: uma aplicação do modelo de distribuição de perdas agregadas para risco de produção em uma empresa não financeira (Master’s Thesis, Escola de Pós-Graduação em Economia, Fundação Getúlio Vargas, Rio de Janeiro). Retrieved from http://bibliotecadigital.fgv.br/dspace/handle/10438/7860
  » http://bibliotecadigital.fgv.br/dspace/handle/10438/7860
• Rippel, M., & Teply, P. (2011). Operational risk - Scenario analysis. Prague Economic Papers, 20(1), 23-39.
• Rodriguez, E. J. J., Domínguez, J. M. F., & Marín, J. L. M. (2009). Comparative analysis of operational risk approaches within Basel regulatory frame-work: case study of Spanish saving bank. Journal of Financial Management and Analysis, 22(1), 1-15.
• Scenario based AMA Working Group. Scenario-based AMA, May 2003, Final version 1.0. Retrieved from https://www.newyorkfed.org/medialibrary/media/newsevents/events/banking/2003/con0529d.pdf
  » https://www.newyorkfed.org/medialibrary/media/newsevents/events/banking/2003/con0529d.pdf
• Shevchenko, P. V., & Peters, G. W. (2013). Loss distribution approach for operational risk capital modelling under Basel II: combining different data sources for risk estimation. Journal of Governance and Regulation, 2(3), 33-57.
• Steinhoff, C., & Baule, R. (2006). How to validate operational risk distributions. Op Risk & Compliance, 1(8), 36-39.
• Wahlstrom, G. (2009). Risk management versus operational action: Basel II in a Swedish context. Management Accounting Research, 20(1), 53-68.

Publication Dates

• Publication in this collection
  May 2018

History

• Received
  01 Dec 2016
• Accepted
  14 Dec 2017