Assessment of safety indicators in high-risk industries in the context of Resilience Engineering: a systematic literature review

Avaliação de indicadores de segurança em indústrias de altor risco no contexto da Engenharia de Resiliência: uma revisão sistemática da literatura

Abstracts

Abstract

Measuring organizational safety performance is critical in managing production systems. This practice allows decisions to be made objectively and based on data analysis. In the study of resilience engineering applied to the safety of production systems, there is interest on the part of the scientific community and companies in identifying, in the different processes of organizations, indicators to measure resilient performance in safety management. This study aims to discuss and analyze qualitative and quantitative methods to identify the potential for resilience in safety management and support new research on resilience engineering applied to safety performance management in industries with high technological risks. The methodology used included a systematic review of national and international literature from the last ten years. The results obtained provide a critical analysis of the methods used to define safety indicators applied through resilience engineering to occupational safety management in organizations that deal with high technological risks. Qualitative methods for generating safety indicators from the perspective of Resilience Engineering have proven to be a better way, as they apply the concepts of Safety I and Safety II and early warning indicators.

Keywords:
Resilience Engineering; Safety indicators; High technological risk

Resumo

Medir o desempenho de segurança nas organizações é um elemento crítico na gestão de sistemas de produção. Essa prática permite que as decisões sejam tomadas de forma objetiva e com base na análise de dados. No estudo da engenharia de resiliência aplicada à segurança dos sistemas produtivos, há interesse por parte da comunidade científica e das empresas em identificar, nos diferentes processos das organizações, indicadores para medir o desempenho resiliente na gestão da segurança. Este estudo tem como objetivo discutir e analisar métodos qualitativos e quantitativos para identificar o potencial de resiliência na gestão de segurança e subsidiar novas pesquisas sobre engenharia de resiliência aplicada à gestão de desempenho de segurança em indústrias com altos riscos tecnológicos. A metodologia utilizada incluiu uma revisão sistemática da literatura nacional e internacional dos últimos dez anos. Os resultados obtidos fornecem uma análise crítica dos métodos utilizados para definir indicadores de segurança aplicados por meio da engenharia de resiliência à gestão da segurança do trabalho em organizações que lidam com altos riscos tecnológicos. Métodos qualitativos para geração de indicadores de segurança sob a ótica da Engenharia de Resiliência têm se mostrado um caminho melhor, pois aplicam os conceitos de Segurança I e Segurança II e indicadores de alerta precoce.

Palavras-chave:
Engenharia de Resiliência; Indicadores de segurança; Alto risco tecnológico

1 Introduction

In the management of production processes, measuring performance in organizations is a crucial element, as it allows decisions to be taken objectively through data analysis. The measurement parameters used are, in the vast majority, performance indicators, which is quantitative information or relevant fact that expresses the performance of a product or organizational process in terms of efficiency, effectiveness or level of satisfaction, and that, in general, allows you to monitor its evolution over time. It is the product of a chain that begins with the need to observe an object, an event, or a phenomenon (Fundação Nacional da Qualidade, 2015).

Indicators can be divided into groups called lagging and leading. The first reflects performance information relating to the past in safety management, enabling the system to learn from what has happened, while the second provides information so that control actions can be taken in time to avoid undesirable outcomes or an unacceptable change in one or more of the main results of the process; or even that management systems can anticipate and mitigate adverse changes based on the analyzed data (Hollnagel et al., 2017).

Florin & Linkov (2016) state that safety indicators created through the perspective of Resilience Engineering (RE) can support decision-making about risks, enabling governance systems in companies to develop resilience-based strategies to help deal with unexpected and sudden events, such as major accidents.

Of course, organizations that attempt to measure safety performance, both in terms of occupational safety (which affects the worker) and in terms of process safety (which affects neighboring communities and the environment). Both safety areas safety require management processes based on performance data (Wreathall, 2006). There are two tools used in a complementary way to measure performance. The first one are indicators associated with front-line performance (sharp end), near-miss reports, detours, etc. In most of these applications, eight to twelve job or task factors are identified that encapsulate the dominant contributions to performance problems. The second tool is based on organizational effectiveness models that focus on the core processes by which an organization achieves its mission (Wreathall, 2006).

Some attributes of organizational indicators used to improve performance were identified by the literature review carried out by Reason (1997), containing sixty-five different models that describe the relationship of processes to achieve successful and safe results. This revision resulted in the proposition of seven main guidelines/attributes, namely: senior management commitment; awareness; preparation; flexibility; culture of reporting events; learning culture; opacity, or consciousness. Later, such attributes were also used by Wreathall (2006) in studies on highly resilient organizations.

According to Hollnagel et al. (2021), resilience is not a property or quality of a system, therefore, it is not something that can be measured or managed on its own. In his first book co-authored with Woods & Leveson – Resilience Engineering: Concepts and Precepts –, Hollnagel et al. (2006), argued that resilience was something a system did and not something it had. Thus, “a system (organization) cannot be resilient, but a system can have the potential for resilient performance” (Hollnagel, 2015, p. 1). The author identified four characteristics of the system that allows assessment of resilience potential: anticipation; response; monitoring; and learning.

In this way, the processes of change and evolution serves to expand the concept of resilient performance, as it is no longer a matter of assessing only the ability of organizations to recover from threats and stresses suffered, but about the way in which they anticipate, as needed and under a variety of conditions, seeking to respond appropriately to disruptions and opportunities. Resilience is then about the performance of systems in general, not just about how secure they remain (Hollnagel et al., 2006).

Regarding the definitions of Resilience Engineering, one of the difficulties of the scientific community is related to both the concept of resilience and the concept of this subject for its application. For the purposes of this research, the following definition described by the international organization Resilience Engineering Association (REA, 2023) was adopted:

Resilience Engineering adopts a transdisciplinary perspective that focuses on developing theories and practices necessary to enable complex socio-technical systems and organizations to continue operations and provide essential services in the face of expected, tense or unexpected situations. Resilience Engineering deals with complexity, non-linearity, interdependencies, emergence, formal and informal social structures, threats and opportunities that originate from past, current or potential failures and successes, with the aim of understanding, designing and implementing resilient systems and operations.

In this regard, Woods & Wreathall (2003) presented an overview and future perspectives on Resilience Engineering, pointing out the need to progress in research with the following focuses: i) development of a system of measures that indicates resilience in organizations ; ii) development of support for decisions on productions and safety trade- offs; iii) development of new forms of feedback on the side effects of organizational changes and decisions on risk.

Peñaloza (2020) conducted a systematic review with the aim of assessing safety performance measurement systems, seeking to identify whether Resilience Engineering offers a new perspective for them and whether the concepts of this subject have been put into practice. The results of this review showed, albeit prematurely, that the premises of Resilience Engineering have been used as a lens for defining safety performance indicator systems.

In another systematic review, Ranasinghe et al. (2020) explored which Resilience Engineering indicators are considered important for safety management and the development and assessment of a resilient work environment in high technological risk industries, mainly those related to renovation and civil construction. They identified that there are four commonly used indicators: senior management commitment, awareness, learning and flexibility, that is, all of those have a strong relationship with Resilience Engineering. However, no systematic review study on the analysis of safety indicators in high technological risk industries from the perspective of RE was identified. On the other hand, it is worth highlighting that, although many authors consider themselves to be assessing whether an organization is resilient or not, Hollnagel (2015) states that, in fact, what they assess is its resilience potential.

Therefore, as a way of contributing to filling this gap and advancing knowledge, this work sought to answer the following question: how to identify and analyze the resilience potential in organizations that operates and deals with high technological risk through Safety indicators created based on Resilience Engineering? To this end, a literature review was carried out on the quantitative and qualitative methods that have been used to generate such indicators.

This review article presents the methodology adopted, the eligibility criteria, followed by results, discussion, conclusions, references, and appendices.

2 Methodology

2.2 Instrument used to organize the flow of research information.

The instrument used to organize the flow of information collected in this research was based on the recommendations of the PRISMA platform. It is an evidence-based minimum set of guidelines for systematic reviews and meta-analyses that focuses mainly on reporting such reviews in different fields of knowledge, in addition to being also useful for the critical assessment of published systematic reviews (Moher et al., 2015).

In general, PRISMA proposes a checklist containing a total of 27 items as presented in ^{Appendix 1}, that must be met and a diagram with the flow of the research process consisting of three phases: identification of studies, screening, and inclusion, as shown in Figure 1 (Page et al., 2021).

Figure 1
. Flow of the selection process prepared by the author in 2022, based on the PRISMA methodology. Source: Page et al. (2021).

According to Page et al. (2021), we had some advances since the first time the PRISMA document was published in 2009, as the methodological guidelines have undergone several updates and their latest version dates back to 2020. Among the innovations, several processes were proposed to synthesize and present results when meta-analysis is not possible or appropriate.

2.2 Research steps

This research was developed with the following steps:

1. definition of the research question;

2. search for the topic in gray literature (books, theses, dissertations, simple and expanded summaries, etc.) with the aim of finding studies published in congresses and/or symposiums.

3. definition of descriptors for carrying out horizontal scanning in electronic databases;

4. study selection process through complete reading, summarization, categorization and application of eligibility criteria;

5. identification and separation of qualitative and quantitative methods of the studies assessed;

6. extraction of data and variables from each study using forms;

7. analysis of the risk of bias among the included studies and assessment of methodological quality using the Joanna Briggs Institute checklist (2022);

8. synthesis of data found in the literature.

This article had two major sources as databases. The first one was the scientific databases for horizontal scanning through the CAPES Periódicos portal, while the second were the websites and databases for accessing gray literature. International gray literature was searched on repository websites such as Perlego and Google Scholar. It was also possible to use the website Research Gate to screen the references cited in the selected articles, which is a social network of researchers linked to various national and international institutions. Thus, it was possible to consult the authors of the selected studies to clarify doubts about the studies analyzed by this review. Systematic searches were conducted in three databases: PUBMED, Web of Science and Google Scholar. The temporal delimitation of the study ranged from the beginning of 2012 to January 2022

• The PUBMED database was also consulted to identify and extract studies that had the same objective as this research. In this way, consultation was conducted with the following descriptors: (“Resilience Engineering”[All Fields] AND “measure”[All Fields]) AND “indicators”[All Fields]) AND “High-Risk”[All Fields].

• Systematic searches were conducted using the following bases and respective descriptors: in the Web Of Science database the descriptors “Resilience Engineering” (title) or “ Indicators of Resilience Engineering ” (all fields) and “ Safety ” (all fields) and “High-Risk industry ” (all fields) and “ Safety Management ” (all fields) and “ indicators ”.

• Google Scholar was consulted with the aim of identifying related studies. An advanced search was conducted in this database using the following descriptors: “Resilience engineering indicators” or “metrics” and “ safety ”. Then, articles that had the same scope as the present study were extracted.

For more details about the search strategy with the filters used in each database, it is recommended to consult the PROSPERO protocol (CRD42022347710) presented in ^{Appendix 2} (Ilhanez et al., 2023). The bases were consulted between the months of September and October 2022.

Bibliometric data were extracted using software Vosvewier and Mendeley Desktop and Citation, which extracted reports made it possible to collect the main bibliometric data analyzed. With all this data, it was possible to create a spreadsheet with references where only the selected studies were included. It is important to highlight those five reviewers worked independently analyzing all the material collected during the literature search. To extract and analyze the data, a spreadsheet with a bibliometric survey was used, which aimed to facilitate the process of analyzing data relating to the studies. A research protocol was developed to define the inclusion criteria and systematic search strategy, objectives, and review question.

2.3 Eligibility criteria

The following exclusion criteria were adopted for the selected studies, as shown in Figure 1 below: publication within ten years and the cut-off date defined as the year 2012. Articles that intended to evaluate the resilience potential in safety processes were separated, including only studies that encompassed industrial production systems with high technological risk, with the exception of technological risks of natural origin. Systematic review studies with approximate themes or on Resilience Engineering and indicators were also considered. Articles in English or Portuguese were selected, although it was noted that even national publications were published in English.

2.4 Selection of studies for analysis

For the process of identifying and selecting relevant articles, it was selected two independent judges and, in case of disagreement between them, a consensus meeting was held with the participation of a third judge. During this stage, differences were evident in the characteristics of the selected studies, in the types of intervention and results, identifying high methodological heterogeneity and in the design of the studies. Therefore, it was not possible to compare the data quantitatively and interpret them based on the meta-analysis.

3 Results

3.1 Characteristics of the studies

The 23 articles found were distributed in nine (9) different journals and six (6) events, among them, three (3) published in conferences and another three published in symposiums. Safety Health at Work was the magazine with the highest number of publications, with 17%, followed by Safety Science, with 13%; in third place were Jornal magazines loss Prevention in the Process Industry and Human Factor and Ergonomics in Manufacturing & Service Industry, with 9% each. The studies were geographically distributed as follows, regarding their reporting country:(7) Iran; (5) Brazil; (2) USA; (3) Italy; (2) Norway; (1) Australia; (1) Japan; (1) China; (1) Mexico. the chronology of studies over the last 10 years is shown in Figure 2:

Figure 2
Distribution of studies over ten years. Source: Prepared by the author (2022).

Quantitative and qualitative studies were evaluated based on checklists developed by the Joanna Briggs Institute (2022), existing in the document Critical Appraisal Tools, being applied to qualitative research, systematic reviews, and cross-sectional analytical studies. Some studies that presented methods to assess resilience through indicators were excluded for the following reasons: publication time greater than 10 years, being related to medium or low-risk industry, or considering technological risks of natural origin, these studies are presented in ^{Appendix 3}. In the graph above, Figure 2, it is observed that there is a variation in the number of publications on the topic of this research. Between 2017 and 2020 was the period with the greatest number of scientific productions on indicators and assessment of resilience in organizations.

Of the studies analyzed it was observed that sixteen have the impact factors of their respective journals, according to the descending list below: Saurin & Werle (2017) = 7.247 / Azadeh et al. (2014) = 6,392 / Peñaloza et al. (2020) = 4,877 / Patriarca et al. (2019) = 4,877 / Shirali et al. (2018) = 4,045 / Ranasinghe et al. (2020) = 4,045 / Azadeh et al. (2017a) = 4,045 / Patriarca et al. (2018) = 4,045 / Jain et al. (2018) = 3,916 / Nelson et al. (2016) = 3.916 / Chuang et al. (2020) = 3.752 / Shirali et al. (2016) = 2.818 / Rabbani et al. (2019) = 1.721 / Azadeh et al. (2017b) = 1.699 / Zarrin & Azadeh (2019) = 1.699 / Pflanz & Levis (2012) = 0.833.

3.1.1 Characterization of studies considering different industrial sectors and their complexity

The classification of institutions according to their typologies and different complexities has been a concern of organizational theory over time. Many decades have passed since Charles Perrow (1984) addressed this topic in chapter three (Complexity, Coupling and Catastrophe) from his work: “Normal Accidents Living with High-Risk Technologies”.

The author classified organizational tasks into two dimensions: the first is related to the nature of interactions within the organization's functioning (linear or complex) and the second, to the nature of the coupling of existing events and processes (loose or tight). Perrow (1984) also indicated some examples of industries with linear interactions, namely: production line factories and civil construction, construction and assembly industries. Highly complex industries have complex interactions and rigid couplings, little room for maneuver and limitations regarding the replacement of labor or material, as well as multiple control parameters with potential interactions; Furthermore, such systems (organizations) are more vulnerable to disasters (major accidents).

A more current study describes the main characteristics of complex high-risk sociotechnical systems, including i) many elements that interact dynamically; ii) great diversity of elements, iii) unexpected variability; iv) resilience, (Righi & Saurin, 2015).

The industry segments found in this research were considered complex socio-technical systems of high technological risk, with characteristics similar to those cited by Perrow (1984), which encompass the following sectors: Process Chemical Industry; Oil Industry; Nuclear Industry; Airline Industry; Naval Military Industry; and Health Industry.

It is also noteworthy that the concept of high-risk complex sociotechnical systems has been updated by several authors since the publication of “Normal Accidents” in 1984. For Le Coze (2023), the matrix of industries with characteristics of high-risk complex systems risk should be updated, considering decades of technology and industry evolution; In this context, he cites Covid-19 as an example, which demonstrated how emergency health systems are also considered complex systems considered critical.

3.2 Individual results of quantitative studies

The studies were separated and classified by quantitative and qualitative methods, and those that used quantitative methods to evaluate Resilience Engineering factors are presented below.

The study by Shirali et al. (2018) aimed to assess the validity of a construct as a measuring instrument developed to assess resilience in socio-technical systems. As a result, the instrument proved to be reliable for this purpose, although it did not analyze the risk of bias between participants and did not have the capacity to measure all dimensions of resilience.

Four studies published by researcher Ali Azadeh, from the University of Tehrân, presented the following objectives and results:

1. Azadeh et al. (2014) – They intended to evaluate the resilience factors in a petrochemical plant. The results showed that preparation, awareness and flexibility were the most important factors among the nine RE factors analyzed. Sixty questionnaires were distributed, half to specialists and the other half to workers. The authors did not define a clear sampling strategy.

2. Azadeh et al. (2017b) – The aim was to investigate the reciprocal impacts of managerial and organizational factors on RE. The results demonstrated that organizational factors had a greater impact on the organization's resilience than managerial factors. It was also observed that RE factors related to learning and flexibility had a greater influence on managerial and organizational factors,

3. Azadeh et al. (2017a) – This research aimed to develop a verified tool for assessing RE factors in maintenance departments in a large oil industry. The study confirmed a close relationship between RE factors and performance modeling factors called performance shaping factor – According to CCPS (Center for Chemical Process Safety, 2005), this factor is defined by any inherent characteristic of an individual, such as personality, level of fatigue, skill and knowledge; and the work situation, such as task demands, factory policies, interface design, training, and ergonomics.

4. Zarrin & Azadeh (2019) – They proposed to map the influences of RE on the health, safety, environmental and ergonomics management system. The results showed that RE principles have a high impact on the company's Safety, Health and Environment (HSE) management. It also revealed that senior management commitment has the greatest impact on the environment, learning has the greatest impact on health, preparation has the greatest impact on ergonomics and awareness has the greatest impact on safety.

The research presented by Shirali et al. (2016) aimed to represent a new vision for and assessing RE factors in a process industry using a wide range of indicators. Results showed that three indicators – “cross-scale interaction”; “margins” and “anticipation” – were the best assessed resilience factors, however, the remaining indicators were below the optimal value. It also revealed that the maintenance, hydrogen and control III units were in poor resilience conditions.

Rabbani et al. (2019) proposed evaluating the organization's resilient safety culture. The results demonstrated that among the ten departments assessed in a petrochemical plant, the “process and production” and “quality control and polymer chemical operations” departments demonstrated a more resilient safety culture, while the “inspection, laboratory and maintenance” had a lower rate and needed attention.

Nelson et al. (2016) described the analysis of operational data of the information contained in the corrective action program at a nuclear plant, using Fuzzy software Loagic Toolbox. The methodology considered human error and organizational factors, due to their large contribution to consequential events, as well as unsafe conditions and behaviors. The data used to feed the indicators were nuclear plant equipment failures and reports of operational errors, in addition to corrective actions. This data generated an indicator calculated by the software based on the severity of the failures.

Grecco et al. (2013) adopted a fuzzy approach to establish a method for assessing resilience in organizations based on proactive safety performance indicators, defined in accordance with RE principles. The result showed that the radiopharmaceutical sector of a company, based on the limit value of (0.6) as the minimum acceptable, presented a deficiency in attribute 1, defined as commitment from senior management, which received a value below. One of the weaknesses of this study was that it still needs to be applied to a company as a whole.

The study by Patriarca et al. (2018) assessed the four main resilience skills through a questionnaire based on Hollnagel's (2011)Resilience Assessment Grid (RAG) method and then submitted the results to the Analytic matrix Hierarchy Process (AHP), by Saaty (2004), in an anesthesiology department of a hospital in central Europe. The results showed that the resilience analysis grid, that is, a weighted set of probing questions, can be used in different domains, with a Safety II approach (Hollnagel, 2014).

Another indicator developed based on the RE concept is in the quantitative study by Saurin & Werle (2017). The objective was to learn how to analyze slack resources as resilience phenomena of different natures in complex socio-technical systems, offering insights into the design of the work system. The Units of Slack (UoS) indicator is that each pool of similar slack resources that share the same purpose corresponds to a UoS., e.g.: redundant equipment in a surgical ward corresponds to one UoS.

Given this, the author proposes to evaluate the extent to which each UoS mitigates the sources of unexpected results (variabilities) in terms of resources, and this can be done through a scale. These UoS. are nothing more than a redundancy system of resources for a specific critical need, such as instrumentation in a surgical room. The UoS. identified over time and correlated with accidents or near misses become important safety indicators.

Although this study presents a series of contributions in terms of adequate allocation to provide minimum resources relative to demand, the structure created to analyze slack does not account for the costs of maintaining the UoS. versus the costs of absorbing waste resulting from variability. Furthermore, the current version of this method does not capture the dynamic nature of the sources of variability and UoS, which can change in terms of intensity and coupling over time.

3.3 Summary of quantitative study results

Due to the heterogeneity of the studies in relation to their designs, interventions, outcome measures and contexts, it was not possible to conduct a meta-analysis of the results, which are presented through a descriptive approach to the selected studies, according to Page et al. (2021).

In general, quantitative studies did not present a well-defined sampling plan. Another common point was the main attributes of the resilience measurement variables, such as: senior management commitment, awareness, learning and flexibility, all with a strong connection to RE. Apart from the study by Patriarca et al. (2018), none jointly proposed an analysis of the four main systemic potentials of RE, which are: monitor, respond, learn, anticipate. All presented an ascending assessment of resilience factors through a standard questionnaire applied to the workforce.

A common limiting factor among the studies was that none of them had the ability to calculate all dimensions of resilience, as this would be unfeasible due to the large volume of variables generated; therefore, some authors cited this limitation in their studies. The following quantitative methods were the most used to generate resilience factor scores based on the scales of the questionnaires applied: Fuzzy logic, (Zadeh, 1965); Analytical Hierarchical Process (PHA) (Saaty, 2004); graph theory (Euler, 1736); Data Envelopment Analysis (AED) (Charnes et al., 1978), Principal Component Analysis (PCA) (Pearson, 1901). It was observed that Fuzzy logic is the most used among studies. Ninety percent of the studies performed a reliability test using Cronbach's Alpha coefficient (Cronbach, 1951). The studies are predominantly cross-sectional or cross-sectional analytical, in epidemiological terms, and all conduct a diagnosis of the organization at a specific moment. The main objectives of the studies, methods and respective findings are set out in ^{Appendix 4}. Bias risk analyses are presented in ^{Appendix 5}. The objectives, location and context of the studies included in the research, as well as the main attributes of the indicators found in the quantitative studies are in Table 1, below:

Table 1
Summary of studies included in the research and attributes of quantitative indicators.

Studies included in the research
Author(s)	Shirali et al. (2018)	Azadeh et al. (2017b)	Azadeh et al. (2017a)	Shirali et al. (2016)	Azadeh et al. (2014)	Rabbani et al. (2019)	Zarrin & Azadeh (2019)	Grecco et al. (2013)	Saurin & Werle (2017)	Nelson et al. (2016)	Patriarca et al. (2018)
goal	Assess content validity, construct validity (by exploratory and confirmatory factor analyses) and reliability (by Cronbach 's alpha and test-retest) to measure the resilient culture in safety of socio-technical systems.	Investigate the reciprocal impacts of managerial and organizational factors and Resilience Engineering on safety through mathematical programming.	Develop a verified tool for the assessment of Resilience Engineering in the safety of maintenance organizations in the oil and gas industry.	Assess Resilience Engineering factors in the safety of a process industry using a wide range of indicators.	Assess resilience factors in the safety of a petrochemical plant, which can be expanded to other industries.	Introduce a new theory-based performance optimization algorithm of graphs, in the matrix approach and in statistical methods for assessment of the Resilience Engineering culture.	Map the influences of Resilience Engineering on the health, safety, environment, and ergonomics management system using the fuzzy cognitive map and the number Z.	Fuzzy Set Theory (FST) approach to assess resilience in organizations based on proactive safety performance indicators, defined in accordance with the principles of Resilience Engineering.	Know how to analyze slack resources of different natures in complex sociotechnical systems to face variability (including incidental variability).	Analyze operational data from the Corrective Action Program with 120,000 corrective actions over 10 years	Resilience Assessment Grid resilience analysis grid and the AHP analytical hierarchy process
Local	Location: Iran	Location: Iran	Location: Iran	Location: Iran	Location: Iran	Location: Iran	Location: Iran	Location: Brazil	Location: Brazil	Location: USA	Location: Rome
	Respondents (R.): 312	Answer: 41	A: 99	A: 32	A.: 60	R.: 10	A: 71	Answer: 12	R.: 45	R.: 120,000	Answer: 12
Context	Oil refinery located in Southwest Iran.	Refinery in southern Iran.	The assessment was conducted in 11 departments in the maintenance sector of a state-owned oil company in Tehran province.	Process industry in Iran.	The respondents were managers, engineers, experienced workers who work at the plant, experts in this production system.	An old, worn-out plant, as well as old and inadequate safety management practices, according to the authors.	Implemented at a large petrochemical plant located in Iran.	The nuclear installation where the radiopharmaceutical packaging shipping process is conducted.	The study was conducted in a maternity ward of a large hospital, where there were safety risks for patients and professionals.	Ten years of corrective actions, from 2005 to 2014, at a nuclear plant in Idaho, USA.	The research was conducted at the Department of Anesthesiology, Critical Care and Pain Medicine, at the Sapienza University of Rome, Italy.
Indicator attributes
	Fair culture	Senior management commitment	Mutual Trust and Transparency	Ability to cushion	Senior management commitment	Senior management commitment	Senior management commitment	Senior management commitment	Irregular patient arrival rate	Job Tasks	Anticipation
	Change management	Learning	Improvement in adaptive skills	Flexibility	Reporting culture	Reporting culture	2 – Health and learning culture	Learning	Ward occupancy rate	Corrective action plan	Response
	Learning Culture	Conscience	Improvement in smart skills	Margin	Learning culture	Learning culture	Ergonomics and preparation	Flexibility	Lack of anesthesiologist on duty		Monitoring
	Risk management and analysis	Flexibility	Promoting cooperation	Tolerance	Conscience	Knowledge	safety and awareness	Conscience	Neonatal ICU occupancy rate		Learning
	Preparation	Self-organization	Clarification of safe work procedures	Cross-scale interaction	Preparation	Preparation		Fair culture	Accessibility of the environment built
	Flexibility	Redundancy	Employee empowerment	Learning	Flexibility	Flexibility		Preparation	Bureaucratic procedures
	Reporting culture			Attention	Team work	Self-organization			Psychological support for the patient
	Top management commitment			Response	Redundancy	Fault tolerance			Lack of blood reserve
	Conscience			Anticipation	Fault tolerance	Team work			Patient's clinical condition
	safety management system					Redundancy
	Accident investigation; staff involvement; competence

• Source: Prepared by the author (2022).

3.3.1 Individual results of qualitative studies

The studies by Chuang et al. (2020) and Sakuda & Kitamura (2020) used the RAG method to measure the organization's potential resilient performance in relation to safety in different industries. The first assessed the potential for resilient performance in a hospital emergency department with the aim of identifying the status of the four capabilities – to monitor, to respond, to anticipate and to learn. This study by Chuang et al. (2020) demonstrated that monitoring was the most deficient function of the emergency department and concluded that the RAG method questions must be well prepared by a team of people who are experts in the production process of the assessed department.

The second study used the method to prevent degradation of resilience potential at the Fukushima Daiichi nuclear power plant. Due to the post-traumatic scenario of the 2011 accident, the authors faced several resistances, however, they managed to find a way to accept the research using the adapted RAG based on the weaknesses of the nuclear plant. Therefore, the study revealed that this adaptation of the method can be a useful tool to facilitate awareness of the plant's possible weaknesses, in addition to pointing out a way to improve monitoring and response potential through the identification of weaknesses. However, due to the difficulties, the research did not demonstrate the scores for the four main attributes of resilience.

Some similar characteristics were presented in the qualitative studies by Øien & Nielsen (2012), Patriarca et al. (2019), and Jain et al. (2018). They proposed tools for detection and early warning through proactive indicators that would make it possible to detect major accidents in advance, based on literature reviews to analyze the statistics of these occurrences and the main causes and failures in the defenses that have triggered these types of accidents in the past. Hollnagel 's study et al. (2021) propose something similar, however, based on weak signals. Weak signals are sources of information not covered by traditional Safety Management Systems, covering what they are not prepared for, what they do not expect and what they otherwise do not notice. Weak signals comprise the many small events that are below the notification or severity threshold, as well as the generally unrecognized patterns of performance the common habits, routines, and common compensations – that most often lead to the expected results but, from time to time, originate unexpected and unwanted results (Hollnagel et al., 2021).

In the study by Øien & Nielsen (2012) called Resilience method based Early Warning Indicators (REWI), developed a set of self-assessment measures that provide information to senior managers and safety professionals within an organization about the fundamental attributes of organizational resilience. The objectives of the method are to provide early warnings to prevent serious accidents and improve long-term safety and organizational performance. It can be seen as an effort to improve the organization's anticipation potential.

The proposal by Patriarca et al. (2019) adapted the REWI method by Øien & Nielsen (2012) to propose a game in a chemical factory, more specifically in an ammonia production unit. The proposed approach aimed to encourage workers' engagement in workplace safety and, more generally, to overcome psychological barriers to the participation of factory employees in the REWI method. These three aforementioned early detection methods will always depend on an assertive response from the organization in the face of an indicator that signals latent flaws and the fragility of organizational barriers. However, if efforts to manage these types of indicators are not rewarded by an assertive response from the implementation team, the method will be doomed to failure.

Hollnagel's study et al. (2021), the authors demonstrated an evolution from the RAG method to the Systemic Potential Management (SPM), developed based on a weak signals design. The European Organization for the Safety of Air Navigation developed this project with the aim of detecting weak early warning signals that can prevent the occurrence of accidents or events that pose a threat to the air traffic management and control system, in order to demonstrate a concern common to several organizations and their governance systems at risk and strengthen the potential for anticipation to avoid threats. An evolution observed between the RAG and SPM methods is that the old four skills (monitoring, responding, learning and anticipating) are now called the four systemic potentials.

SPM is a tool for managing an organization's performance and how changes are implemented. SPM uses four sets of questions – called foreground questions – to assess changes as a means of managing them. In contrast to background questions, foreground questions should be used repeatedly throughout the project. Systemic potentials effectively describe or define what an organization should do – the performance criteria – and answer the questions (the performance potential profile) – consequently representing how well these criteria have been achieved (Hollnagel et al., 2021).

A critical element for the RAG and SPM methods is knowing how to prepare the questions well in order to have a correct diagnosis of what is intended to be assessed. It is necessary to have an implementation team with experience within the company's processes where they are being applied. Another important factor is that respondents must undergo a preparatory interview, otherwise, the diagnosis runs the risk of having meaningless generic questions and answers. Another limitation is that the method will always depend on employees' perception of the four main systemic potentials of resilience: monitoring, response, learning and anticipation.

A technological tool such as software was proposed in the studies by Huber et al. (2012) and de Souza et al. (2021) to generate resilience indicators and guarantee system safety. The first study proposed a framework to indicate where the organization is located within the capabilities needed to deal with disruptions that may affect it, including accidents, in an air taxi company. Through questionnaires filled out in the system, indicators were generated with the following attributes: commitment, awareness, adaptability, and efficiency. Questions were prepared following these four principles and placed on a rating scale.

The study by Souza et al. (2021), applied to the nuclear industry, intended to use RE concepts in understanding the organization through the analysis of tasks and cognitive activities to develop indicators by the human agents themselves working at the operational level. Questionnaires were filled out in the system to detect organizational difficulties, resources, knowledge and others that could arise when conducting the tasks. The study focused on identifying sources of task variability, following the premises of cognitive analysis for data collection, which aims to understanding the work performed from the perspective of frontline workers, gaining their tacit knowledge and informal strategies for dealing with complexity.

The operator points out in the questionnaires which of these elements’ present problems or difficulties for their daily performance. This analysis is conducted in real-time by the system and allows managers to know what the operators' deviations or difficulties are. As a disadvantage, the method will always depend on the workers' interpretation of the questions, which can cause bias in the participants' responses. Furthermore, it is not known to what extent workers will be willing to reveal their difficulties.

As previously stated, two secondary studies of important contribution to this research were identified. The first was by Peñaloza et al. (2020), which aimed to assess safety performance measurement systems with a focus on identifying whether RE offers a new perspective on safety for these systems and whether the concepts of this discipline have been put into practice. The results of this review demonstrated that, although still quite premature, the RE premises have been used as a lens for defining safety performance indicator systems.

The most relevant contribution of the study by Peñaloza et al. (2020) for this review, more than just knowing whether RE assumptions are used for safety performance measurement systems, it was to identify, through gray literature, which are the main attributes that performance measurement systems should have. Namely: I) they must support the monitoring of daily variability; II) they must provide real-time feedback to those directly involved in the execution and supervision of production activities; III) they should facilitate learning about what goes well, as well as what goes wrong; IV) they must provide insights into managing trade-offs between Safety and other business dimensions; V) they must evolve due to the changing nature of complex socio-technical systems.

The second systematic review was by Ranasinghe et al. (2020), whose main objective was to explore RE indicators that were identified as important in the development and assessment of a resilient work environment in high-risk industries, particularly in the renovation and construction industry. The results showed that the four commonly used indicators were: senior management commitment, awareness, learning, and flexibility, all of which have a strong relationship with RE. However, this review did not consider qualitatively defined indicators, only quantitative methods. As already mentioned, indicators are not just quantitative data, they are also relevant facts that can signal a phenomenon (Fundação Nacional da Qualidade, 2015).

Pflanz & Levis (2012) presented guidelines for measuring organizational resilience based on proxy measures, such as error tolerance, ability to respond to unexpected events and level of connectivity between elements of the system or organization. The study adopts the Petri Net software, which designs a network architecture using linear algebra elements and demonstrates three main phases of resilience: a) prevention, b) survival, and c) recovery. The concepts presented in this study also apply to organizational resilience, however, this research only focused on the survival phase of resilience – the phase in which the systems' capacity is affected or deformed.

Finally, Herrera et al. (2014) proposes another application of resilience indicators in the aviation industry, more specifically in an air traffic control department at an airport. The authors developed a scale framework to identify a group of resilience indicators. This study applied the concept of Safety I and Safety IIHollnagel et al. (2017) and was developed from observations on air traffic controller offices (ATCO). The scale structure detects not only incidents, accidents, and deviations, but the presence of human-machine system capacity at times when operations went beyond expectations, or were threatened, at times when a suitable landing was not possible, and the pilots had to ram the aircraft.

During go-around maneuvers – (considered operational incidents) it was possible to identify instances of problem solutions and maneuvers proposed by air traffic controllers, which avoided collisions or aircraft crashes (major accidents), thanks to the skills of these professionals in solving such problems, most of which were not included in procedures. Regarding to RE, the scaling structure proposed by the study supports the identification and extraction of resilience indicators for mapping into patterns (Herrera et al., 2014), thus facilitating learning about what goes well, in addition of what goes wrong (Peñaloza, 2020).

3.3.2 summary of qualitative study results

As previously stated, the results pointed to a tendency in qualitative studies to seek proactive indicators that can anticipate accidents or major events. Some studies called resilience-based early warning indicators; others called them weak signals for accident detection. Another common characteristic of qualitative studies was a more longitudinal approach, recognizing through the methods the need to revisit processes and collect data again due to the constant changes that complex socio-technical systems undergo. The studies showed greater interest in the attributes of the indicators than in their metrics. Finally, two studies did not demonstrate congruence between the methodology and the proposed objectives with the representation and analysis of the results. The research by Jain et al. (2018), which although presents a process resilience analysis framework and considers technical and social factors, the latter were not assessed and included in the results. The study by Sakuda & Kitamura (2020) applied the RAG methodology, however it demonstrated the scoring of the four main resilience skills, as recommended by the method. The purposes, locations, context and main attributes of the Safety indicators, with a Resilience Engineering perspective, found in qualitative studies, are shown in Table 2 below. The analysis of the methodological quality of qualitative studies is found in ^{Appendix 5}.

Table 2
Summary of studies included in the research and attributes of qualitative indicators.

Studies included in the research
Author(s)	Hollnagel et al. (2021)	Øien & Nielsen (2012)	Herrera et al. (2014)	Peñaloza et al. (2020)	Souza et al. (2021)	Patriarca et al. (2019)	Chuang et al. (2020)	Sakuda & Kitamura (2020)	Jain et al. (2018)	Huber et al. (2012)	Pflanz & Levis (2012)
goal	Assess the management of the systemic resilience potential of organizational systems and detect weak signals that could be warnings of major accidents, promoting the safe management of air traffic.	Establish proactive safety indicators for monitoring oil spill risk in the Goliat field, through a resilience model based on early warnings.	Describe the scale structure and its application to a concrete case through the combination of Enterprise Architecture with Resilience engineering, to measure the resilience potential of the ATM system – Air Traffic Management	Identify whether Resilience Engineering offers a new perspective on Safety performance measurement systems and understand how the resilience engineer has put this into practice in Safety performance measurement systems.	Apply Resilience Engineering concepts to understanding the organization by analyzing tasks and cognitive activities to develop indicators.	Promote and improve your resilience to unwanted events with indicators used for early warning in gamification.	The study aims to redesign a graph using the RAG (Resilience Analysis Grid), to generate indicators for evaluating Safety resilience.	Use the RAG method (Resilience Analysis Grid) to prevent degradation of the safety resilience potential of the Fukushima nuclear plant.	Present a new framework, incorporating technical and social factors in an integrated approach to assess resilience in the process industry through indicators.	Propose a framework to indicate where the organization is located, within the capabilities needed to deal with disruptions that may affect it.	Describes approach as based on error tolerance, ability to respond to unexpected events in the moment of survival when the system (organization) is threatened or interrupted and recovers. It also demonstrates that the level of collaboration between the human elements of the organization helps with adaptive capacity and the survival phase.
Local	Location: Sweden, EUROCONTROL	Location: Norway	Location: Sweden	Location: Brazil	Location: Rio de Janeiro, Brazil	Location: Rome, Italy	Location: Taiwan, China	Location: Japan	Location: USA	Location: Rio de Janeiro, Brazil	Location: Naval Industry, USA,
Context	The study derives from a project called “Weak Signals”, an initiative by the EUROCONTROL organization in conjunction with European universities.	Goliat Field, in the Barent Sea. More specifically, the Goliat Field is located in the Norwegian part of the Barents Sea.	Arlanda Airport, Sweden. Applied in an air traffic operations tower.	Federal University of Rio Grande do Sul (UFRGS), for complex high-risk systems.	Federal University of Rio de Janeiro (UFRJ), Rio de Janeiro. Developed for the nuclear industry.	Rome Italy. In the ammonia manufacturing sector of a chemical process industry.	Emergency unit of a hospital in Taiwan.	Fukushima Nuclear Power Plant, Japan.	The study was developed at the University of Texas.	Institute of Nuclear Engineering, Cidade Universitária, Rio de Janeiro, RJ, Brazil.	Systems Architecture Laboratory, George Mason University, Fairfax
Attribute of Qualitative Indicators
	Anticipation	1.1.1. Number of years of knowledge of the production system	(+) Potential conflict in the go-around maneuver administered by two Air Traffic Control Officers (ACTO)	a) system of indicators must support the monitoring of daily variability	Training and experience	Number of hours of risky courses in the last 12 months	1 – Anticipation	1 – Anticipation	(ED) Error Tolerant Design	Conscience	1 - Tolerance
	Response	1.1.2. Information about risks through courses, documents, HAZOP, AQRs	(+) ATCOs (Air Traffic Control Officers) – prevent a potential collision between aircraft	b) indicator systems must provide real-time feedback to those directly involved in the execution and supervision of production activities	Communication	No. of barrier failures, e.g.: failures of PSVs (safety valve)	2 – Response	2 – Response	(R) Recoverability	Efficiency	2 - Flexibility
	Monitoring	1.1.3. Reporting of near misses and incidents	(-) Unexpected condition (not the go-around itself, but the fact that the aircraft did not follow the controller's instructions)	c) indicator systems should facilitate learning about what is going well, as well as what is going wrong	Facilities and equipment	Information on the quality of barriers (technical safety)	3 – Monitoring	3 – Monitoring	(P) Plasticity	Adaptability	3 - Capacity
	Learning	1.1.4. Information on the quality of barriers (technical safety)		d) indicator systems should offer insights into managing trade- offs between Safety and other dimensions of the business	Condition of materials	Number of cases in which a response decision was delayed in the last three months	4 – Learning	4 – Learning	(P) Social Resilience	Commitment	4 - Collaboration
		1.1.5. Information on the quality of barrier support functions (operational safety)		e) indicator systems must evolve due to the changing nature of complex socio-technical systems		Planning and scheduling	Number of hours of risky courses in the last 12 months	1 – Anticipation	1 – Anticipation

• Source: Prepared by the author (2022). Analysis of the main attributes of qualitative indicators.

4 Discussion

This systematic review demonstrated a tendency for quantitative studies to analyze organizational resilience based on indicators with attributes published by Reason (1997). The author conducted a research into the main elements of a safety culture. At the time, the author discovered the following subcultures as attributes of a safety culture: 1) leadership commitment; 2) awareness; 3) preparation; 4) flexibility; 5) reporting culture; 6) learning culture; and 7) fair culture. Later, these attributes were used by Wreathall (2006), in chapter seventeen of the work Resilience Engineering; however, not all complex socio-technical systems with a robust safety culture have proven to be synonymous with resilient safety performance, as per event statistics demonstrated in the study by Jain et al. (2018).

According to Hopkins (2002), it must be emphasized, however, that culture as a mentality tends to ignore the latent conditions that are behind all workplace accidents, highlighting workers' attitudes as the cause of accidents. Another factor is that the elements of a safety culture proposed by Reason (1997) were conceived within a safety perspective excluding bad results - Safety I (Hollnagel, 2014). The attributes of the indicators are less aligned with the concept of Safety II (Hollnagel, 2014), coming from Resilience Engineering.

The indicators proposed by quantitative studies, for the most part, proposed assessing the management of organizations to find out whether they were resilient or not, a reactive approach. The quantitative variables proposed in the studies were obtained through the application of questionnaires to the workforce or survey of operational errors and failures, as well as failures of critical equipment; then, the various calculation methods were applied. As a limitation, these studies will always depend on the perception of the workforce in relation to resilience factors and safety management. Only half of the quantitative studies conducted interviews with the workforce, a recommended practice to avoid interpretive errors in questionnaires.

Unlike quantitative studies, which presented methods and characteristics of cross-sectional or analytical cross-sectional studies, qualitative studies recognized the limitation of using only a momentary diagnosis of the organization, therefore, they developed methods and proposed longitudinal approaches. Unlike the quantitative study by Nelson et al. (2016), the qualitative study by Herrera et al. (2014) used the concept of indicator as a phenomenon or relevant fact (Fundação Nacional da Qualidade, 2015) and developed a scale to extract indicators based on the new concepts of Safety I and Safety II (Hollnagel, 2014), as shown in Table 1.

Finally, qualitative studies showed greater proximity to attributes revealed in the secondary study by Peñaloza (2020) for measuring safety performance based on RE in complex socio-technical systems. These attributes are: II) providing real-time feedback to those directly involved in the execution and supervision of the production activity, which was demonstrated in the study by Souza et al. (2021) and Huber et al. (2012); III) facilitating learning about what goes well, in addition to what goes wrong, as presented in the study by Herrera et al. (2014); and IV) offer insights into managing trade-offs between Safety and other business dimensions. The study by Øien & Nielsen (2012) recommends the implementation of a computerized system to support decision-making during conflicts between production and safety.

An important contribution regarding quantitative studies was observed in the study by Saurin & Werle (2017), A framework for the analysis of slack in socio-technical systems. This method could be important to mitigate impacts if it were adopted by hospital emergency systems before adverse events occur. The slack applied to the lack of spaces and resources to care for the affected population, if adopted by hospital emergency systems, could mitigate external variability, such as the pandemic, and save lives in emergency hospital cells, considered critical systems.

This method would be very useful to improve the use of resources and exert a capacity to buffer or slack adverse effects. Resource buffer is also a concept that can be designed by project management systems, for example, the concept of buffer capacity. What is nothing more than buffer designed as a redundancy of resources to ensure the functioning of critical systems in projects.

5 Conclusion

This study aimed to identify and analyze the potential for resilience in organizations, as well as the qualitative and quantitative methods proposed by the authors of the bibliographic review considered for the development of Safety indicators used within the perspective of resilience engineering, in organizations that deal with high technological risk and that work through complex socio-technical systems. Its contribution is to support future research that aims to assess resilience to support Safety management processes in high technological risk organizations. The different indicator approaches from a Resilience Engineering perspective can support operational or managerial decision-making, especially governance systems at risk. As a limitation, the loss of some information by the excluded studies is a common characteristic of this systematic literature review process. Another common difficulty to all studies is the actual definitions of the phenomenon of resilience, as well as Resilience Engineering, as there is a vast literature with different definitions and concepts, without a single definition that can be followed.

The results of the applied analysis demonstrated that quantitative methods for developing and defining safety indicators, from the perspective of Resilience Engineering, should prioritize recurring periodic measurements, that is, longitudinal studies in future research, taking into account subtle and continuous changes which are characteristic of the changing nature of complex socio-technical systems, as well as the emergence of latent conditions that originate both at the managerial level and at the front line of operations. For this reason, indicators of the resilience potential of Safety processes should not collect data in a short period of time, but rather through monitoring conducted repeatedly. Furthermore, few quantitative studies have demonstrated metrics to calculate the organization's systemic potentials, called anticipation, response, monitoring and learning.

Based on the studies analyzed, one aspect that requires more research is the use of qualitative methods to generate indicators when operators are forced to deal with unexpected incidents or variability. Even if metrics are not used for this type of scenario, these are important markers, as regulations or adaptations occur outside of the prescribed work, which are conducted to overcome such situations and avoid major accidents. Such cases are important and should be considered as an ability to deal with the unexpected and maintain operations even in the face of adversity.

Qualitative methods to generate Safety indicators from the perspective of Resilience Engineering demonstrated a better path, as linear calculation models have the common disadvantage of not being able to evaluate all dimensions of resilience, since this is a multidimensional phenomenon.

Appendix 1 Prisma compliance checklist.

PRISMA 2020 Checklist

Section and Topic	Item #	Item Verification	Location where the item is Lines in Lauda
TITLE
Title	1	Identifies the publication as a systematic review.	1:3
			4:6
SUMMARY
Summary	2	See PRISMA 2020 Checklist for Abstracts.	AT
INTRODUCTION
Rationale	3	It bases the review in the context of existing knowledge.	45:141
Goals	4	Explicitly presents the objective(s) or question(s) regarding the review.	134:139
METHODS
Eligibility Criteria	5	Specifies the inclusion and exclusion criteria for the review and how the studies were grouped for the syntheses.	214:236
			305:307
Information sources	6	Specifies all databases, registries, websites, organizations, reference lists and other sources searched or consulted to identify studies. Specifies the last date each source was searched or consulted.	177:212
Search strategy	7	Displays complete search strategies for all databases, records and websites, including all filters and limits used.	177:212
Selection process	8	Specifies the methods used to decide whether a study meets the review inclusion criteria, including how many reviewers screened each record and selected publication, whether they worked independently, and, if applicable, details of automation tools used in the process .	214:223
			204:212
			225:232
Data collection process	9	Specifies the methods used to collect data from publications, including how many reviewers collected information from each publication, whether they worked independently, all processes for obtaining or confirming data by study investigators, and, if applicable, details of automation tools used.	204:212
			225:232
			145:159
Item data	10a	Lists and defines all results for which data was searched. Specifies whether all results compatible with each domain in each study were searched (e.g. for all measures, moments, analyses) and, if not, specifies the methods used to decide which results to collect.	141:201
			239:592
			Frames 1 and 2
			APPENDICES
	10b	List and define all other variables for which data were researched (e.g., participant and intervention characteristics, funding sources). Describes the assumptions used about missing or unclear information.	Frames 1 and 2
Section and Topic	Item #	Item Verification	Location where the item is Lines in Lauda
Assessment of the risk of bias in studies	11	Specifies the methods used to assess the risk of bias of the included studies, including details about the instrument(s) used, how many reviewers evaluated each study and whether they worked independently, and, if applicable, details of tools of automation used in the process.	174:175
			204:212
			225:232
Effect measures	12	Specifies for each outcome the measure(s) of effect (e.g. relative risk and mean difference) used in summarizing or presenting the results.	397:400
Synthesis method	13a	write the processes used to decide which studies are eligible for each synthesis (e.g. present the characteristics of the intervention presented in the study and compare with the groups planned for each synthesis (item #5)).	225:232
	13b	Describes all necessary methods of preparing data for presentation or synthesis, such as dealing with missing data in summary statistics, or data conversions.	AT
	13c	Describes all methods used to present or display individual results of studies and syntheses.	239:423
			425:592
	13d	Describes all methods used to summarize the results and provides a justification for the choice(s). If a meta-analysis was performed, describe the model(s) and method(s) for identifying the presence and extent of statistical heterogeneity, and softwareused.	Not applicable.
			225:232
			397:400

Section and Topic	Item #	Item Verification	Location where the item is Not applicable see
	13e	Describes all methods used to explore possible causes of heterogeneity among study results (e.g., subgroup analysis, meta-regression).
			397:400
			225: 232
	13f	Describes all sensitivity analyzes performed to assess the robustness of the synthesis of results.	239:423
			425:592
			APPENDIX 5
Assessment of reported bias	14	Describes all methods used to assess the risk of bias due to missing results in a synthesis (due to information bias).	181:187
			180: 182
Assessment of the degree of confidence	15	Describes all methods used to assess the certainty (or confidence) in the body of evidence for a result.	APPENDIX 5
RESULTS
Study selection	16a	Describes the results of the search and selection process, from the number of records identified in the search to the number of studies included in the review, ideally using a flowchart.	234: 236
			239:268
	16b	Cites studies that appear to meet the inclusion criteria, but were excluded, and explains the reasons for exclusion.	APPENDIX 3
Characteristics of the studies	17	Cites each study included and presents its characteristics.	Frames 1 and 2
			304:423
			425:592
Section and Topic	Item #	Item Verification	Location where the item is Not applicable see
Risk of bias in studies	18	Presents the risk of bias assessment for each included study.
			APPENDIX 5
Individual study results	19	For all results from each study, present: (a) summary statistics for each group (where appropriate) and (b) an estimate of the effect and its precision (e.g. confidence/credibility interval), ideally using tables or structured graphics.	Not applicable see
			397:400
Synthesis results	20a	For each synthesis, summary of the characteristics and risk of bias among the selected studies.	APPENDIX 5
			Frames 1 and 2
	20b	Presents the results of all statistical syntheses carried out. If a meta-analysis was performed, it presents for each result the summary of the estimate and its precision (e.g. confidence/credibility interval) and measures of statistical heterogeneity. If groups are compared, describe the direction of the effect.	Not applicable see
			397:400
	20c	Presents the results of all investigations of possible causes of heterogeneity among study results.	Frames 1 and 2
			304:423
			425:592
	20d	Presents results of all sensitivity analyzes performed to assess the robustness of the synthesized results.	Not applicable see
			397:400
Reported biases	21	It presents the assessment of the risk of bias due to the lack of results (resulting from information bias) for each synthesis evaluated.	174: 175
			APPENDIX 5
Significance level	22	Presents the assessment of certainty (or confidence) in the body of evidence for each evaluated result.	Not applicable see
			397:400
DISCUSSION
Discussion	23a	Provides a general interpretation of results in the context of other evidence.	862: 878
			896:818
	23b	Discusses all limitations of the evidence included in the review.	239:423
			425:592
			APPENDIX 5
	23c	Discusses all limitations of the review processes used.	928: 929
	23d	Discusses the implications of the results for practice, policy and future research.	919: 955
OTHER INFORMATION
Registration of	24a	Provide information about the registration of the review, including the name and registration number, or state that the review is not registered.	ID: CRD42022347710 PROSPERO

Section and Topic	Item #	Item Verification	Location where the item is
protocol	24b	Indicates where the review protocol can be accessed, or states that the protocol has not been prepared.	PROSPERO Platform ID: CRD42022347710
	24c	Describes and explains any changes to the information provided in the registration or protocol.	Some changes were made to the title until the final stage, as well as to the authors.
Support	25	Describes the sources of funding or unfunded support that support the review, and the role of the funders or sponsors of the review.	Not applicable
Conflict of interests	26	Declare all conflicts of interest of review authors.	Not applicable
Availability of data, codes and other materials	27	Reports which of the following materials are publicly accessible and where they can be found: model data collection forms extracted from included studies, data used for analysis; analytical code, any other material used in the review.	PROSPERO platform ID: CRD42022347710

Translated by: Verónica Abreu*, Sónia Gonçalves-Lopes*, José Luís Sousa* and Verónica Oliveira / *ESS Jean Piaget - Vila Nova de Gaia – Portugal. From: Page et al. (2021).

Appendix 2 Research registration protocol.

Appendix 3 Summary of excluded studies.

Summary of excluded articles
Reasons for Exclusion	References
Medium risk industry	Rubio-Romero et al. (2018)
It is not a high-risk technological industry	Chen et al. (2018)
Duplicate study with study above	Chen et al. (2017)
Study duplicated with that of Shirali et al. (2016)	Azadeh et al. (2016)
Duplicate study with that of Zarrin & Azadeh (2019)	Asadzadeh et al. (2013)
Duplicate study with the study Shirali et al. (2016)	Shirali et al. (2013)
Publication time	Saurin & Carim Junior (2011)
Publication time	Costella et al. (2009)
Publication time	Herrera et al. (2011)

Source: Prepared by the author (2022). List of excluded studies that seem to meet the criteria.

Appendix 4 Objectives, methods of the study and their main findings.

Quantitative Study	Local	Main goals	Measuring instruments	Main Results (Findings)
Shirali et al. (2018)	Southwest, Iran, Oil Refinery	To assess the content validity, construct validity (by exploratory and confirmatory factor analyses) and reliability (by Cronbach 's alpha and test-retest) of an instrument developed to measure the resilient culture in Safety of socio-technical systems.	Likert Scale (CVI - Content Validity Index, T-test, Lawshe, Crombach 's alpha)	The results of the tests conducted indicate that the instrument is valid and reliable.
Azadeh et al. (2017b)	Southern Iran, Oil Refinery	This study aims to investigate the reciprocal impacts of managerial and organizational factors and resilience engineering through a unique mathematical programming approach.	DEA (Data Involvement Analysis)	It was found that organizational factors have a greater impact on resilience than managerial factors.
				It was also found that resilience factors related to learning and flexibility have a greater influence on managerial and organizational factors.
Azadeh et al. (2017a)	Terhan Provincial Oil Company	The objective of the study was to develop a verified tool for the assessment of Resilience Engineering in maintenance organizations in the oil and gas industry.	(AHP) Analytical Hierarchical Process/ algorithm K- means /DEA (Data Involvement Analysis)	The study also confirmed the close relationship between Resilience Engineering and human-related performance modeling factors, suggesting that services to promote resilience engineering will lead to such factors being in good condition so that human error can be eliminated. reduced and safety can be improved.
Shirali et al. (2016)	Process Industry in Iran	This study aims to represent a new vision for assessing Resilience Engineering factors in a process industry using a wide range of indicators. This article aimed at a quantitative assessment of RE based on nine indicators of resilience, namely buffer capacity, margin, tolerance, cross-scale interactions, learning culture, flexibility, anticipation, attention and response using PCA and numerical taxonomy (NT) in a process plant.	Principal Component Analysis/Numerical Taxonomy/Delphi/ Superman – reliability of results	The results of the analysis showed that three indicators of interactions between scales, margin and anticipation were at a level below best practice, respectively. Furthermore, the results of the unit analysis showed that the maintenance, hydrogen and control unit 3 were in the worst condition in relation to the resilience factors raised.
Azadeh et al. (2014)	Petrochemical Industry in Iran	The main objective of this study is to assess the resilience factors of a petrochemical plant, which can be expanded to other industries. It is achieved through a Fuzzy Cognitive Map (FCMs) method, which considers interactions between factors due to their calculated final weights.	Graph theory and fuzzy logic (Fuzzy Cognitive Map)	The results showed that preparation, awareness and flexibility are the most important factors among all nine RE factors. Furthermore, redundancy and teamwork play a small role among RE factors.
Rabbani et al. (2019)	Petrochemical Industry in Iran	To this end, the aim of this study is to present a new performance optimization algorithm based on graph theory, matrix approach and statistical methods for evaluating the resilience engineering culture of a given organization.	Graph theory and matrix approach	Among the 10 departments assessed in terms of resilience factors, “production process”, “quality control” and “polymer chemical operation” were considered the most effective in the resilience culture of the petrochemical plant among the departments considered by the study. The results also indicated that the “inspection”, “laboratory” and “maintenance” departments had the lowest resilience index and require the most attention. Low resilience index means weak system capacity in case of unexpected accidents or crises.
Zarrin & Azadeh (2019)	Petrochemical Plant in Northwest Iran	The aim of the study was to map the influences of resilience engineering on the health, safety, environment and ergonomics management system, using the Z number cognitive map.	The concept of Z- numbers with Fuzzy Cognitive Map (FCM) approach is integrated and a new approach called Z- number Cognitive Map is proposed.	The results showed that the principles of Resilience Engineering have a high impact on HSE management.
				The results also indicated that senior management commitment has the greatest impact (0.827) on the environment, learning has the greatest impact (0.792) on health, preparation has the greatest impact (0.786) on ergonomics, and awareness has the greatest impact (0.776) on Safety.
Grecco et al. (2013)	Brazil, Radiopharmaceutical Industry	The aim of this research is to adopt a fuzzy approach Set Theory (FST) to establish a method for assessing resilience in organizations based on proactive Safety performance indicators, defined according to the principles of resilience engineering.	Fuzzy Logic using the Fuzzy Set Theory method to measure resilience factors	We consider an attendance level greater than or equal to 0.6 to be satisfactory. The result of the average assessment showed that the radiopharmaceutical packaging shipping sector presented a satisfactory learning culture, awareness of flexibility, fair culture, and preparation. However, this sector presented problems related to senior management commitment.
Patriarca et al. (2018)	Central European Region.	Based on the four categories (monitoring, response, anticipation and learning), this article aims to define a semi-quantitative analysis to measure organizational resilience in complex socio-technical systems, combining the RAG (Resilience Assessment Grid) resilience analysis grid and the process of AHP analytical hierarchy.	RAG (Resilience Assessment Grid) and AHP - Saaty Matrix	The result of the resilience analysis grid, i.e. a weighted set of probing questions, can be used in different domains as a supporting tool in a broader managerial action oriented towards Safety -II, to bring management safety at the core organization's business.
Nelson et al. (2016)	Idaho National Laboratory, USA	This work describes the analysis of operational data of the information contained in the Corrective Action Program. The methodology considers human error and organizational factors because of their large contribution to consequential events. The results include a tool developed from the data to be used to identify, predict and reduce the likelihood of significant consequential events.	MATLAB Fuzzy Logic Toolbox (version 2.1.1; The Math Works, Inc., Natick, MA, USA)	The results include a tool developed from the data to be used to identify, predict and reduce the likelihood of significant consequential events. This tool is based on the resilience curve that was constructed from the plant's operational data.
Saurin & Werle (2017)	Maternity, Rio Grande do Sul, Brazil	The aim of the study was to learn how to analyze slack resources of different natures in complex socio-technical systems, offering insights into work system design to cope with the unexpected variabilities of complex systems.	Descriptive statistics, such as means, standard deviations and coefficients of variation, were used to analyze the questionnaire data, which produced information that was directly used.	Benefits of the framework for assessing slack: (i) it allows the joint analysis of slack resources of different natures, based on a shared construct (UoS), metric (protection score) and classification scheme; (ii) allows prioritization of sources of variability, based on how effectively they are covered by slack resources ; (iii) sheds light on sources of variability and idle resources that arise from self-organization and informal work practices; this reflects a focus on the work done [16] while leaving room for imagined work, emphasizing the regulations' time off requirements; (iv) UoS identified over time and correlate this with accidents and near misses: the measured clearance is expected to be an important safety indicator.
			Slack Units (UoS): each pool of similar slack resources that share the same purpose, corresponds to a UoS, for example, redundant equipment in a surgical ward, regardless of the number of extra equipment.
			Next, the extent to which each UoS mitigates each source of variability must be assessed. This assessment is based on a 7-point scale.
Hollnagel et al. (2021)	Sweden, Eurocontrol	Build a basis for the resilient performance of complex socio-technical systems that manage air traffic through a qualitative method of assessing resilient performance in an organization that controls air traffic. Proposing a questionnaire and Likert scale with scores for potentials: monitor, respond, learn and anticipate.	Web-shaped chart with Likert scale scores to measure: monitoring, response, learning and anticipation.	Qualitative method based on a psychometric questionnaire that assesses the resilience potential of organizational systems that aim to promote air traffic management, using the potential for response, monitoring, anticipation and learning.
Øien & Nielsen (2012)	Goliat Field in the Barent Sea	Develop a resilience-based approach, mainly due to the ability to provide warnings for unknown (atypical) scenarios, not covered by a risk assessment. Improving the capacity for anticipation, through indicators, to avoid a major accident.	Awareness of risks	It shows that the method can be applied in any oil and gas industry. It also shows that the proposed general issue indicators can provide early warning for major accidents.
			1.1.1. Number of years of knowledge of the production system.
			1.1.2. Information about risks through courses, documents, HAZOP, AQRs, in the last 3 months.
			1.1.3. Quality of reports of near misses and incidents.
			1.1.4. Information on the quality of barriers (technical safety).
			1.1.5. Information on the quality of barrier support functions (operational safety).
			1.1.6. Discussion of HSE issues/status at regular meetings.
			1.1.7. Risk/resilience communication at all levels of the organization.
			Anticipation
			1.2.1. Risk/hazard identification number (HAZID etc.).
			1.2.2. Learn from your own experiences and accidents.
			1.2.3. Learn from other experiences and accidents.
Herrera et al. (2014)	Arlanda Airport in Sweden	It proposed a scaling framework with the aim of identifying sets of resilience indicators. Applied the Safety II concept of safety as the presence of capability.	Resilience indicators are nothing more than instantiations of problem solutions that procedures do not cover. Problems presented during the go-around maneuver at a central air traffic management tower.	With regard to Resilience Engineering, the SCALE framework supports the identification and extraction of resilience indicators and their mapping into patterns.
			Examples:
			(+) Potential conflict in the go-around maneuver administered by two Air Traffic Control Officers (ACTO)
			(+) ATCOs (Air Traffic Control Officers) - Prevent a potential collision between aircraft.
Peñaloza et al. (2020)	Brazil, UFRG	A systematic review of the literature aimed to identify whether RE offers a new perspective on safety performance measurement systems, and to understand how RE has been put into practice in safety performance measurement systems.	a) system of indicators must support the monitoring of daily variability;	However, there have been several studies that are moderately or strongly aligned with these guidelines, suggesting that RE has been implicitly adopted to some extent in performance measurement systems.
			b) indicator systems must provide real-time feedback to those directly involved
			in the execution and supervision of production activities;
			c) system of indicators should facilitate learning about what is going well, as well as what is going wrong;
			d) indicator system should offer insights into trade- offs management between Safety and other dimensions of the business;
			e) indicator system must evolve due to the changing nature of complex socio-technical systems.
Souza et al. (2021)	UFRJ, Rio de Janeiro	It proposed applying resilience engineering concepts to understanding the organization through the analysis of tasks and cognitive activities to develop indicators by the human agents themselves working at the operational level, in addition to the initial specification of a computational tool to support this understanding.	Radial (spider) graphs with scores for the following factors to perform operational tasks, e.g .:	Provides a technological opportunity to detect deviations of work towards its safety boundaries. It allows managers to photographically analyze how activities were or are being conducted throughout the day. Radial graphs are obtained with indicators that affect the team's difficulties in achieving the task's objectives.
			training and experience;
			communication;
			facilities and equipment;
			group activities and work interfaces.
Patriarca et al. (2019)	Italy, Rome	The proposed approach aims to encourage workers' engagement in workplace safety and, more generally, to overcome psychological barriers to their participation.	Number of years of experience with NH3 production.	The method is still viewed with great skepticism, as it involves a structure for putting together games. Perhaps with cost and benefit analysis it could be implemented electronically, since assembling the structure and design of the game can be costly.
		The approach was explored in a case study within the chemical industry. In particular, the safety-critical sector of ammonia production was addressed, with the aim of promoting and improving its resilience to unwanted events.	Number of hours of risky courses in the last 12 months.
			Information on the quality of barriers (technical safety).
			Number of barrier failures, e.g. PRV failures.
			Number of exceptions handled in the last month
			Number of cases of unsuccessful communication between operators during operational maneuvers.
			Number of cases in which a response decision was delayed in the last three months.
Chuang et al. (2020)	China, Medical Emergency Department of a Hospital in Taiwan	Hollnagel Resilience Assessment Grid (RAG) into a customized RAG (ED-RAG) to support resilience management in a Hospital Emergency Department in Taiwan. ED-RAG (Emergency Department - RAG)	Scores from 0 to 4 for specific questions prepared by a team of researchers and employees who knew the department. With this, it was possible to define grades from 0 to 100% for the four resilience skills: potential to learn (86.11%); response potential (61.56); anticipate (56.25%); monitor (33.93).	The ED-RAG represents a snapshot of the resilience of EDs under specific conditions. It can be performed multiple times by a single hospital to monitor the directions and content of improvement that can complement conventional safety management toward resilience. Some considerations are necessary to be successful when hospitals use it. Future studies to overcome potential methodological shortcomings of the ED-RAG are needed. Among the four skills evaluated, the only one that was below was monitoring.
Sakuda & Kitamura (2020)	Fukushima Nuclear Power Plant, Japan	Used RAG to prevent degradation of resilience potential in a nuclear power plant.	Spider chart with Likert scale scores to measure monitoring, response, learning and anticipation.	It is found that fragility-based RAG can be a useful tool to facilitate awareness about possible weaknesses of nuclear power plant.
Jain et al. (2018)	University of Texas, USA	Presented a generic framework for measuring the probability of resilience of a process system based on metrics in which a Bayesian Network (BN) can be developed using an equation.	Resilience Analysis Framework (PRAF), the metrics were developed considering the performance indicators established by API RP 754 (API, 2010) and based on the resilience aspects of the Early Detections (ED) process, Error Tolerant Design (ETD), ability to recovery (recoverability) (R) and (plasticity) plasticity (P). These were categorized into technical (ED, ETD, R) and social resilience (P).	The Process Resilience Analysis Framework is
			Note: uses the calculation of these factors in a Bayesian Network, however, it did not demonstrate values.	presented in the context of improving risk and safety management.
				As illustrated, the key aspects of process system resilience are detection (ED), error-tolerant design (ETD), recoverability (R), and plasticity (P). In application, the use of these aspects serves to drive predictability.
Huber et al. (2012)	Institute of Nuclear Engineering, Cidade Universitária, Rio de Janeiro, RJ, Brazil	The aim of this study was to propose a framework to indicate where the organization is located, within the capabilities necessary to deal with the disturbances that may affect it (the organization's level of resilience).	Conscience;	In the experts' assessment of the air taxi company's resilience application, the method and its division of activities into three levels resulted in activities compatible with the capabilities, interests and availability of those involved, and was therefore well accepted. The combined concept of practical development (a representative, participatory, iterative and interactive method) of resilience indicators has also been well accepted and should be tested in more areas of work.
			efficiency;
			adaptability;
			commitment.
Pflanz & Levis (2012)	Maritime Industry, US Navy Command and Control Center in Georgia, USA	This article describes a quantitative approach to assess the expected resilience of a command-and-control system. It presents guidelines for measuring organizational resilience, based on proxy measures, such as tolerance to errors, responsiveness to unexpected events and level of connectivity between elements of the system or organization.	Petri Net software presents linear algebra network modeling that demonstrates how to calculate resilience.	It presented guidelines for measuring organizational resilience based on proxy measures such as error tolerance, responsiveness to unexpected events, and level of connectivity between system elements.
Ranasinghe et al. (2020)	Newcastle University	The aim of this research is to explore RE indicators that have been identified as important in developing and evaluating resilient work environments in high-risk industries, particularly construction renovation.	Commitment from senior management;	The results show that the four commonly used indicators were: senior management commitment, awareness, learning and flexibility, all of which have a strong relationship with RE. The findings of this study are useful for interested parties in making decisions about the most important RE indicators in the context of their research or practice, as this avoids ambiguity and disparity in identifying RE indicators.
			awareness
			apprenticeship
			flexibility

Source: Prepared by the author (2022).

Appendix 5 Risk of bias analysis.

Joanna Briggs Checklist (Quantitative Studies)	Azadeh et al. (2014)	Azadeh et al. (2017a)	Azadeh et al. (2017b)	Shirali et al. (2018)	Shirali et al. (2016)	Rabbani et al. (2019)	Zarrin & Azadeh (2019)	Grecco et al. (2013)	Nelson et al. (2016)	Saurin & Werle (2017)	Patriarca et al. (2018)
Were the criteria for sample inclusion clearly defined?	High risk	High risk	High risk	Low risk	High risk	Low risk	High risk	Low risk	Low risk	It is not clear	It is not clear
Have the study topic and setting been written in detail?	It is not clear	High risk	Low risk	Low risk	Low risk	Low risk	Low risk	Low risk	Low risk	Low risk	Low risk
Was exposure measured validly and reliably?	Low risk	Low risk	Low risk	Low risk	Low risk	Low risk	Low risk	Low risk	Low risk	Low risk	It is not clear
Were standard objectives and criteria used to measure the condition?	Low risk	Low risk	Low risk	Low risk	Low risk	Low risk	Low risk	Low risk	Low risk	Low risk	Low risk
Have confounding factors been identified?	High risk	High risk	High risk	Low risk	High risk	Low risk	High risk	Low risk	Low risk	Low risk	Low risk
Were there strategies to address the stated confounding factors?	High risk	High risk	High risk	High risk	High risk	Low risk	High risk	Low risk	Low risk	Low risk	It is not clear
Were the results measured validly and reliably?	It is not clear	Low risk	It is not clear	Low risk	Low risk	Low risk	Low risk	Low risk	Low risk	Low risk	Low risk
Was appropriate statistical analysis used?	Low risk	High risk	High risk	Low risk	Low risk	It is not clear	Low risk	Low risk	Low risk	Low risk	Low risk

Note: The common limitation of quantitative studies is that none were able to present an instrument to evaluate or measure all dimensions of the resilience phenomenon in high-risk industries. Source:Joanna Briggs Institute (2022). Checklist applied to analyze the risk of bias in quantitative studies.

Checklist Joanna Briggs Institute (Qualitative Studies)	Hollnagel et al. (2021)	Herrera et al. (2014)	Øien & Nielsen (2012)	Patriarca et al. (2019)	Jain et al. (2018)	Huber et al. (2012)	Souza et al. (2021)	Chuang et al. (2020)	Sakuda & Kitamura (2020)	Pflanz & Levis (2012)
Is there congruence between the stated philosophical perspective and the research methodology?	Low risk	Low risk	Low risk	High risk	Low risk	Low risk	Low risk	Low risk	Low risk	Low risk
Is there congruence between the research methodology and the research question or objectives?	Low risk	Low risk	Low risk	It is not clear	Low risk	Low risk	Low risk	Low risk	High risk	Low risk
Is there congruence between the research methodology and the methods used to collect data?	Low risk	Low risk	Low risk	It is not clear	It is not clear	Low risk	Low risk	Low risk	It is not clear	Low risk
Is there congruence between the research methodology and data representation and analysis?	Low risk	Low risk	Low risk	Low risk	High risk	Low risk	Low risk	Low risk	High risk	Low risk
Is there congruence between the research methodology and the interpretation of the results?	Low risk	Low risk	Low risk	Low risk	High risk	Low risk	Low risk	Low risk	High risk	Low risk
Is there a statement locating the researcher culturally or theoretically?	Not applicable	Not applicable	Not applicable	Not applicable	Not applicable	Not applicable	Not applicable	Not applicable	Not applicable	Not applicable
Is there influence of the researcher on the research, and vice versa, addressed?	Not applicable	Not applicable	Not applicable	Not applicable	Not applicable	Not applicable	Not applicable	Not applicable	Not applicable	Not applicable
Are participants and their voices adequately represented?	It is not clear	It is not clear	It is not clear	High risk	Not applicable	Low risk	Low risk	Low risk	Low risk	Low risk
Do the conclusions drawn in the research report flow from the analysis, or interpretation, of the data?	Low risk	Low risk	Low risk	It is not clear	Low risk	Low risk	Low risk	Low risk	Low risk	Low risk

Source:Joanna Briggs Institute (2022). Checklist applied to analyze the risk of bias in qualitative studies.

Joanna Briggs Institute Checklist (Systematic Review Reports)	Peñaloza et al. (2020)	Ranasinghe et al. (2020)
Is the review question clearly and explicitly stated?	Low risk	High risk
Were the inclusion criteria appropriate for the review question?	It is not clear	Low risk
Was the search strategy adequate?	Low risk	Low risk
Were the sources and resources used to search for studies adequate?	Low risk	Low risk
Were the study assessment criteria adequate?	Low risk	Low risk
Was the critical assessment conducted by two or more reviewers independently?	It is not clear	Low risk
Were there methods to minimize errors in data extraction?	It is not clear	Low risk
Were the methods used to combine the studies adequate?	Low risk	Low risk
Has the likelihood of publication bias been assessed?	Low risk	It is not clear
Were the recommendations for policy and/or practice supported by the reported data?	Low risk	Low risk
Were specific guidelines for new research appropriate?	Low risk	Low risk

Source:Joanna Briggs Institute (2022). Checklist applied to analyze the risk of bias of secondary studies.

• Financial support: None.
• How to cite: Ilhanez, M. P. L., Silva, E. R., Mattos, U. A. O., Frankenfeld, K. P., & Silva, L. C. (2023). Assessment of safety indicators in high-risk industries in the context of Resilience Engineering: a systematic literature review. Gestão & Produção, 30, e3423. https://doi.org/10.1590/1806-9649-2023v30e3423

References

• American Petroleum Institute – API. (2010). Process Safety Performance Indicators for the Refining and Petrochemical Industries Washington, DC: API.
• Asadzadeh, S. M., Azadeh, A., Negahban, A., & Sotoudeh, A. (2013). Assessment and improvement of integrated HSE and macroergonomics factors by fuzzy cognitive maps: the case of a large gas refinery. Journal of Loss Prevention in the Process Industries, 26(6), 1015-1026. http://dx.doi.org/10.1016/j.jlp.2013.03.007
  » http://dx.doi.org/10.1016/j.jlp.2013.03.007
• Azadeh, A., Asadzadeh, S. M., & Tanhaeean, M. (2017a). A consensus based AHP improved for assessment of resilience engineering in maintenance organizations. Journal of Loss Prevention in the Process Industries, 47, 151-160. http://dx.doi.org/10.1016/j.jlp.2017.02.028
  » http://dx.doi.org/10.1016/j.jlp.2017.02.028
• Azadeh, A., Salehi, V., Arvan, M., & Dolatkhah, M. (2014). Assessment of resilience engineering factors in high-risk environments by fuzzy cognitive maps: a petrochemical plant. Safety Science, 68, 99-107. http://dx.doi.org/10.1016/j.ssci.2014.03.004
  » http://dx.doi.org/10.1016/j.ssci.2014.03.004
• Azadeh, A., Salehi, V., Mirzayi, M., & Roudi, E. (2017b). Combinatorial optimization of resilience engineering and organizational factors in a gas refinery by a unique mathematical programming approach. Human Factors and Ergonomics in Manufacturing, 27(1), 53-65. http://dx.doi.org/10.1002/hfm.20690
  » http://dx.doi.org/10.1002/hfm.20690
• Azadeh, A., Zarrin, M., & Hamid, M. (2016). A novel framework for improvement of road accidents considering decision-making styles of drivers in a large metropolitan area. Accident; Analysis and Prevention, 87, 17-33. http://dx.doi.org/10.1016/j.aap.2015.11.007 PMid:26651129.
  » http://dx.doi.org/10.1016/j.aap.2015.11.007
• Center for Chemical Process Safety. (2005). Combined Glossary of Terms Retrieved in 2022, December 15, from https://paradoxintellectual.com/uploads/3/0/9/9/3099442/ccps-combined-glossary_of_terms.pdf
  » https://paradoxintellectual.com/uploads/3/0/9/9/3099442/ccps-combined-glossary_of_terms.pdf
• Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efficiency of decision making units. European Journal of Operational Research, 2(6), 429-444. http://dx.doi.org/10.1016/0377-2217(78)90138-8
  » http://dx.doi.org/10.1016/0377-2217(78)90138-8
• Chen, Y., McCabe, B., & Hyatt, D. (2017). A belief network model to predict safety performance of construction workers – from the perspective of organizational resilience. In The Canadian Society for Civil Engineering’s 6th International, 11th Construction Specialty Conference, Vancouver, Canada.
• Chen, Y., McCabe, B., & Hyatt, D. (2018). A resilience safety climate model predicting construction safety performance. Safety Science, 109, 434-445. http://dx.doi.org/10.1016/j.ssci.2018.07.003
  » http://dx.doi.org/10.1016/j.ssci.2018.07.003
• Chuang, S., Ou, J., Hollnagel, E., & Hou, S. (2020). Measurement of resilience potential - development of a resilience assessment grid for emergency departments. PLoS One, 15(9), e0239472. http://dx.doi.org/10.1371/journal.pone.0239472 PMid:32956391.
  » http://dx.doi.org/10.1371/journal.pone.0239472
• Costella, M. F., Saurin, T. A., & Guimarães, L. B. M. (2009). A method for assessing health and safety management systems from the resilience engineering perspective. Safety Science, 47(8), 1056-1067. http://dx.doi.org/10.1016/j.ssci.2008.11.006
  » http://dx.doi.org/10.1016/j.ssci.2008.11.006
• Cronbach, L. J. (1951). Coefficient alpha and the internal structure of test. Psychometrika, 16(3), 297-334. http://dx.doi.org/10.1007/BF02310555
  » http://dx.doi.org/10.1007/BF02310555
• Euler, L. (1736). Graph Theory: Eulerian graphs. Retrieved in 2022, November 7, from http://www.inf.ufsc.br/grafos/temas/euleriano/euleriano.htm
  » http://www.inf.ufsc.br/grafos/temas/euleriano/euleriano.htm
• Florin, M., & Linkov, I. (2016). Resilience: an edited collection of authored pieces comparing, contrasting, and integrating risk and resilience with an emphasis on ways to measure resilience. Lausanne: EPFL International Risk Governance Center (IRGC). http://doi.org/10.5075/epfl-irgc-228206
  » http://doi.org/10.5075/epfl-irgc-228206
• Fundação Nacional da Qualidade. (2015). Transformando o Sistema de Indicadores Retrieved in 2022, November 7, from https://fnq.org.br/comunidade/product/livro-transformando-o-sistema-de-indicadores/
  » https://fnq.org.br/comunidade/product/livro-transformando-o-sistema-de-indicadores/
• Grecco, C., Vidal, M., Cosenza, C., Santos, I., & Carvalho, P. (2013). A Fuzzy Model to Assess Resilience for Safety Management. In Proceedings of the 5th Symposium on Resilience Engineering, Managing trade-offs (pp. 259-264). Soesterberg: The Ohio State University.
• Herrera, I. A., Hollnagel, E., & Håbrekke, S. (2011). Proposing safety performance indicators for helicopter offshore on the Norwegian Continental Shelf. In PSAM 10 - Tenth Conference on Probabilistic Safety Assessment and Management Seattle, Wa, United States.
• Herrera, I. A., Pasquini, A., Ragosta, M., & Vennesland, A. (2014). The SCALES framework for identifying and extracting resilience related indicators: Preliminary findings of a go-around case study. In Proceedings of the SESAR Innovation Days (pp. 1-8). Brussels: SESAR Joint Undertaking.
• Hollnagel, E. (2011). RAG – Resilience Analysis Grid. In E. Hollnagel, J. Pariès & J. Wreathall (Eds.), Resilience engineering in practice: a guidebook (pp. 275-295). London: CRC Press. https://doi.org/10.1201/9781317065265
  » https://doi.org/10.1201/9781317065265
• Hollnagel, E. (2014). Safety-I and Safety-II: the past and future of safety management (1st ed.). London: CRC Press.
• Hollnagel, E. (2015). Introduction to the Resilience Analysis Grid (RAG) Retrieved in 2022, November 5, from https://www.erikhollnagel.com/onewebmedia/RAG%20Outline%20V2.pdf
  » https://www.erikhollnagel.com/onewebmedia/RAG%20Outline%20V2.pdf
• Hollnagel, E., Leonhardt, J., & Licu, T. (2021). The Systemic Potentials Management: building a basis for resilient performance Retrieved in 2022, November 5, from https://skybrary.aero/sites/default/files/bookshelf/32380.pdf
  » https://skybrary.aero/sites/default/files/bookshelf/32380.pdf
• Hollnagel, E., Pariès, J., & Wreathall, J. (2017). Resilience Engineering in Practice London: CRC Press. Retrieved in 2022, November 5, from https://www.taylorfrancis.com/books/edit/10.1201/9781317065265/resilience-engineering-practice-erik-hollnagel-jean-pari%C3%A8s-john-wreathall
  » https://www.taylorfrancis.com/books/edit/10.1201/9781317065265/resilience-engineering-practice-erik-hollnagel-jean-pari%C3%A8s-john-wreathall
• Hollnagel, E., Woods, D., & Leveson, N. (2006). Resilience Engineering: Concepts and Precepts Hampshire: Ashgate. Retrieved in 2022, November 5, from https://books.google.com.br/books?hl=pt-BR&lr=&id=rygf6axAH7UC&oi=fnd&pg=PP1&dq=Hollnagel,+E.,+Woods,+D., +%26+Leveson,+N.+(Eds.).+(2012).+Engineering:+Concepts+and+Precepts.+&ots=iq9GSWb_7g&sig=2BVo2fV361GJGK6d1tVmdrQ5kF8#v=onepage&q&f=false.
  » https://books.google.com.br/books?hl=pt-BR&lr=&id=rygf6axAH7UC&oi=fnd&pg=PP1&dq=Hollnagel,+E.,+Woods,+D.,
• Hopkins, A. (2002). Safety culture, mindfulness and safe behavior: converging ideas? Retrieved in 2022, December 5, from http://hdl.handle.net/1885/41764
  » http://hdl.handle.net/1885/41764
• Huber, G. J., Gomes, J. O., & de Carvalho, P. V. R. (2012). A program to support the construction and evaluation of resilience indicators. Work (Reading, Mass.), 41(Suppl.1), 2810-2816. http://dx.doi.org/10.3233/WOR-2012-0528-2810 PMid:22317145.
  » http://dx.doi.org/10.3233/WOR-2012-0528-2810
• Ilhanez, M. P. L. I., Silva, E. R., Mattos, U., Frankefeld, K. & Silva, L. (2023). The resilience engineering as perspective for safety indicators in high technological risk industries: a systematic literature review Retrieved in 2023, July 24, from https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42022347710
  » https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42022347710
• Jain, P., Pasman, H. J., Waldram, S., Pistikopoulos, E. N., & Mannan, M. S. (2018). Process Resilience Analysis Framework (PRAF): a systems approach for improved risk and safety management. Journal of Loss Prevention in the Process Industries, 53, 61-73. http://dx.doi.org/10.1016/j.jlp.2017.08.006
  » http://dx.doi.org/10.1016/j.jlp.2017.08.006
• Joanna Briggs Institute. (2022). Critical Appraisal Tools Retrieved in 2022, August 8, from https://jbi.global/critical-appraisal-tools
  » https://jbi.global/critical-appraisal-tools
• Le Coze, J. (2023). Coupling and complexity at the global scale: flows, networks, interconnectedness and synchronicity (eg Covid-19). Safety Science, 165, 106193. http://dx.doi.org/10.1016/j.ssci.2023.106193
  » http://dx.doi.org/10.1016/j.ssci.2023.106193
• Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews, 4(1), 1. http://dx.doi.org/10.1186/2046-4053-4-1 PMid:25554246.
  » http://dx.doi.org/10.1186/2046-4053-4-1
• Nelson, P. F., Martín-Del-Campo, C., Hallbert, B., & Mosleh, A. (2016). Development of a leading performance indicator from operational experience and resilience in a nuclear power plant. Nuclear Engineering and Technology, 48(1), 114-128. http://dx.doi.org/10.1016/j.net.2015.10.010
  » http://dx.doi.org/10.1016/j.net.2015.10.010
• Øien, K., & Nielsen, L. (2012). Proactive resilience-based indicators: The case of the deepwater horizon accident. In Proceedings of the International Conference on Health, Safety and Environment in Oil and Gas Exploration and Production (pp. 972-983). Perth: OnePetro.
• Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., McGuinness, L. A., Stewart, L. A., Thomas, J., Tricco, A. C., Welch, V. A., Whiting, P., & Moher, D. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ (Clinical Research Ed.), 372(71), n71. http://dx.doi.org/10.1136/bmj.n71 PMid:33782057.
  » http://dx.doi.org/10.1136/bmj.n71
• Patriarca, R., di Gravio, G., Costantino, F., Falegnami, A., & Bilotta, F. (2018). An analytic framework to assess organizational resilience. Safety and Health at Work, 9(3), 265-276. http://dx.doi.org/10.1016/j.shaw.2017.10.005 PMid:30370158.
  » http://dx.doi.org/10.1016/j.shaw.2017.10.005
• Patriarca, R., Falegnami, A., de Nicola, A., Villani, M. L., & Paltrinieri, N. (2019). Serious games for industrial safety: an approach for developing resilience early warning indicators. Safety Science, 118, 316-331. http://dx.doi.org/10.1016/j.ssci.2019.05.031
  » http://dx.doi.org/10.1016/j.ssci.2019.05.031
• Pearson, K. (1901). LIII. On lines and planes of closest fit to systems of points in space. The London, Edinburgh and Dublin Philosophical Magazine and Journal of Science, 2(11), 559-572. http://dx.doi.org/10.1080/14786440109462720
  » http://dx.doi.org/10.1080/14786440109462720
• Peñaloza, G. A. (2020). A framework to assess Safety Performance Measurement Systems for construction projects based on the Resilience Engineering perspective (Doctoral degree). Federal University of Rio Grande do Sul, Porto Alegre.
• Peñaloza, G. A., Saurin, T. A., Formoso, C. T., & Herrera, I. A. (2020). An engineering resilience perspective of safety performance measurement systems: A systematic literature review. Safety Science, 130, 2020. http://dx.doi.org/10.1016/j.ssci.2020.104864
  » http://dx.doi.org/10.1016/j.ssci.2020.104864
• Perrow, C. (1984). Normal accidents: living with high-risk technologies. USA: BasicBooks. Retrieved in 2022, December 5, from https://www.theisrm.org/public-library/Charles%20Perrow%20-%20Normal%20Accidents.pdf
  » https://www.theisrm.org/public-library/Charles%20Perrow%20-%20Normal%20Accidents.pdf
• Pflanz, M., & Levis, A. (2012). An approach to evaluating resilience in command and control architectures. Procedia Computer Science, 8, 141-146. http://dx.doi.org/10.1016/j.procs.2012.01.030
  » http://dx.doi.org/10.1016/j.procs.2012.01.030
• Rabbani, M., Yazdanparast, R., & Mobini, M. (2019). An algorithm for performance evaluation of resilience engineering culture based on graph theory and matrix approach. International Journal of Systems Assurance Engineering and Management, 10(2), 228-241. http://dx.doi.org/10.1007/s13198-019-00774-0
  » http://dx.doi.org/10.1007/s13198-019-00774-0
• Ranasinghe, U., Jefferies, M., Davis, P., & Pillay, M. (2020). Resilience engineering indicators and safety management: a systematic review. Safety and Health at Work, 11(2), 127-135. http://dx.doi.org/10.1016/j.shaw.2020.03.009 PMid:32596006.
  » http://dx.doi.org/10.1016/j.shaw.2020.03.009
• Reason, J. (1997). Managing the risks of organizational accidents. London: Routledge.
• Resilience Engineering Association. (2023). Welcome to Resilience Engineering Association Retrieved in 2022, December 5, from https://www.resilience-engineering-association.org/
  » https://www.resilience-engineering-association.org/
• Righi, A., & Saurin, T. (2015). Complex socio-technical systems: characterization and management guidelines. Applied Ergonomics, 50, 19-30. http://dx.doi.org/10.1016/j.apergo.2015.02.003 PMid:25959314.
  » http://dx.doi.org/10.1016/j.apergo.2015.02.003
• Rubio-Romero, J. C., Pardo-Ferreira, M. C., Varga-Salto, J., & Galindo-Reyes, F. (2018). Composite leading indicator to assess the resilience engineering in occupational health & safety in municipal solid waste management companies. Safety Science, 108, 161-172. http://dx.doi.org/10.1016/j.ssci.2018.04.014
  » http://dx.doi.org/10.1016/j.ssci.2018.04.014
• Saaty, T. L. (2004). Decision making – the Analytic Hierarchy and Network Processes (AHP/ANP). Journal of Systems Science and Systems Engineering, 13(1), 1-35. http://dx.doi.org/10.1007/s11518-006-0151-5
  » http://dx.doi.org/10.1007/s11518-006-0151-5
• Sakuda, H., & Kitamura, M. (2020). Resilience Assessment Grid (RAG) for facilitating safety awareness of nuclear power plant personnel. In Proceedings of the 8th REA Symposium on Resilience Engineering: Scaling up and Speeding up (pp. 1-9). Kalmar: Linnaeus University. https://doi.org/10.15626/rea8.05
  » https://doi.org/10.15626/rea8.05
• Saurin, T. A., & Carim Júnior, G. C. (2011). Evaluation and improvement of a method for assessing HSMS from the resilience engineering perspective: a case study of an electricity distributor. Safety Science, 49(2), 355-368. http://dx.doi.org/10.1016/j.ssci.2010.09.017
  » http://dx.doi.org/10.1016/j.ssci.2010.09.017
• Saurin, T. A., & Werle, N. B. (2017). A framework for the analysis of slack in socio-technical systems. Reliability Engineering & System Safety, 167, 439-451. http://dx.doi.org/10.1016/j.ress.2017.06.023
  » http://dx.doi.org/10.1016/j.ress.2017.06.023
• Shirali, G., Mohammadfam, I., & Ebrahimipour, V. (2013). A new method for quantitative assessment of resilience engineering by PCA and NT approach: a case study in a process industry. Reliability Engineering & System Safety, 119, 88-94. http://dx.doi.org/10.1016/j.ress.2013.05.003
  » http://dx.doi.org/10.1016/j.ress.2013.05.003
• Shirali, G., Motamedzade, M., Mohammadfam, I., Ebrahimipour, V., & Moghimbeigi, A. (2016). Assessment of resilience engineering factors based on system properties in a process industry. Cognition Technology and Work, 18(1), 19-31. http://dx.doi.org/10.1007/s10111-015-0343-1
  » http://dx.doi.org/10.1007/s10111-015-0343-1
• Shirali, G., Shekari, M., & Angali, K. A. (2018). Assessing reliability and validity of an instrument for measuring resilience safety culture in sociotechnical systems. Safety and Health at Work, 9(3), 296-307. http://dx.doi.org/10.1016/j.shaw.2017.07.010 PMid:30370161.
  » http://dx.doi.org/10.1016/j.shaw.2017.07.010
• Souza, A. P., Gomes, J. O., & Carvalho, P. V. R. (2021). Uma abordagem para o monitoramento de indicadores de resiliência em organizações. Revista Ação Ergonômica, 6(2), 1-11. Retrieved in 2022, December 5, from https://www.revistaacaoergonomica.org/article/627d539ca9539504123c8ba3/pdf/abergo-6-2-1.pdf
  » https://www.revistaacaoergonomica.org/article/627d539ca9539504123c8ba3/pdf/abergo-6-2-1.pdf
• Woods, D. D., & Wreathall, J. (2003). Managing risk proactively: the emergence of resilience engineering Columbus: Ohio University. Retrieved in 2022, December 5, from https://www.researchgate.net/publication/228711828_Managing_Risk_Proactively_The_Emergence_of_Resilience_Engineering
  » https://www.researchgate.net/publication/228711828_Managing_Risk_Proactively_The_Emergence_of_Resilience_Engineering
• Wreathall, J. (2006). Challenges for a practice of resilience engineering: properties of resilient organizations: an initial view. In E. Hollnagel, D. Woods & N. Leveson (Eds.). Resilience engineering: concepts and precepts (pp. 275-285). Hampshire: Ashgate. Retrieved in 2022, November 8, from https://books.google.com.br/books?hl=pt-BR&lr=&id=rygf6axAH7UC&oi=fnd&pg=PP1&dq=Hollnagel,+E.,+Woods,+D., +%26+Leveson,+N.+(Eds.).+(2012).+Engineering:+Concepts+and+Precepts.+&ots=iq9GSWb_7g&sig=2BVo2fV361GJGK6d1tVmdrQ5kF8#v=onepage&q&f=false.
  » https://books.google.com.br/books?hl=pt-BR&lr=&id=rygf6axAH7UC&oi=fnd&pg=PP1&dq=Hollnagel,+E.,+Woods,+D.,
• Zadeh, L. A. (1965). Fuzzy sets. Information and Control, 8(3), 338-353. http://dx.doi.org/10.1016/S0019-9958(65)90241-X
  » http://dx.doi.org/10.1016/S0019-9958(65)90241-X
• Zarrin, M., & Azadeh, A. (2019). Mapping the influences of resilience engineering on health, safety, and environment and ergonomics management system by using Z-number cognitive map. Human Factors and Ergonomics in Manufacturing, 29(2), 141-153. http://dx.doi.org/10.1002/hfm.20766
  » http://dx.doi.org/10.1002/hfm.20766

Publication Dates

• Publication in this collection
  27 Nov 2023
• Date of issue
  2023

History

• Received
  27 July 2023
• Accepted
  28 Sept 2023