Assessing resilience in a project-oriented organization: An exploratory case study using the Resilience Analysis Grid

Avaliação da resiliência em uma organização orientada a projetos: Um estudo de caso exploratório usando o Resilience Analysis Grid

Abstract

Purpose:  The study proposes a framework for assessing organizational resilience using the Resilience Analysis Grid (RAG), a questionnaire-based tool, in a project-oriented organization. The proposed method is assessed through a case study, aimed at identifying systemic and individual factors that pose risk to the potential of resilient performance in a public technology development organization.

Originality/value:  The study approaches resilience from a new perspective, borrowed from the safety management field, and applies it in project management concepts. In addition, it presents the first instantiation of the RAG in a project management context, proposing the exaptation of the framework.

Design/methodology/approach:  The transposition of the RAG framework is supported by the design science research paradigm. In the case study, the questionnaire is built cooperatively with the stakeholders and administered to all projects in the organizational portfolio. The results are aggregated through the Analytical Hierarchy Process (AHP) and processed in order to highlight singular and systemic aspects of the projects’ performance in the organization. Finally, strategic actions are proposed to mitigate the risks identified.

Findings:  The methodological design proved to be fit to the purpose, from the perspective of the stakeholders and project managers involved in the study. Through its instantiation in the case study, it allowed identifying sources of risk such as organizational struggle with learning, and prioritizing task and resource allocation, among others, subsidizing the proposition of strategic actions for systemic intervention.

Keywords:
resilience; resilience engineering; project management; Resilience Analysis Grid; RAG

Resumo

Objetivo:  O estudo propõe um framework para avaliar a resiliência organizacional utilizando o Resilience Analysis Grid (RAG), uma ferramenta baseada em um questionário, em uma organização orientada a projetos. O método proposto é avaliado em um estudo de caso, a fim de identificar fatores de risco sistêmicos e individuais para o potencial de performance resiliente em uma organização pública de desenvolvimento de tecnologia.

Originalidade/valor:  O estudo apoia-se em uma nova perspectiva de resiliência, emprestada do campo de gestão da segurança, aplicada em um contexto de gestão de projetos. Apresenta, ainda, a primeira aplicação do RAG no contexto de pesquisa em gestão, propondo a exaptação da ferramenta.

Design/metodologia/abordagem:  A transposição do RAG é apoiada no paradigma do design science research. No estudo de caso, o questionário é construído de maneira participativa pelos stakeholders e é aplicado a todos os projetos do portfólio organizacional. Os resultados são agregados através de uma estrutura hierárquica analítica (AHP) e processados a fim de evidenciar aspectos singulares e sistêmicos referentes aos projetos na organização. Por fim, são propostas ações para mitigar riscos identificados.

Resultados:  O design metodológico mostrou-se adequado, conforme perspectiva dos stakeholders e dos gerentes de projeto participantes do estudo. Através da instanciação no estudo de caso, foram a identificação de fatores de risco como dificuldade organizacional em lidar com o aprendizado e a priorização de tarefas e alocação de recursos, entre outros, subsidiando a proposição de ações estratégicas de intervenção no sistema.

Palavras-chave:
resiliência; engenharia de resiliência; gestão de projetos; Resilience Analysis Grid ; RAG

INTRODUCTION

Complex technology and product development projects are subject to high degrees of uncertainty and risks, by the very nature of projects themselves: unique, temporary, resource restricted and multidisciplinary, and by the novel nature of the technologies under development (Floricel & Miller, 2001; Oehmen et al, 2015). Managing such projects poses challenges to project managers (PM’s) and management teams in the integration of the scope, schedule, and cost to meet the agreed business objectives. On the other hand, to cope with the increasing complexity of the work environment, resilience has emerged as a trending topic. The input (resilience AND (work OR project)), over the last five ye+ars as for August 2024, returned over 20,000 results when searched on the SCOPUS engine, 35,000 at Periódicos and 800,000 at Google Scholar. Resilience has been a trend in risk analysis and risk management (Aven, 2016) and it is now featured as one of the principles guiding the project management industry by the Project Management Body of Knowledge, the PMBoK Guide (Project Management Institute, 2021). This paper aims at proposing a participative framework for assessing resilience in a project-oriented organization, based on well-established methods and paradigms from various scientific domains, mainly supported on recent principles and perspectives borrowed from risk and safety science, and applying this proposed framework in a case study.

The literature on organizational resilience typically defines resilience as an ability to recover from disturbing events and mishaps, or to bounce back (Sheffi, 2007), while Vogus and Sutcliffe (2007) consider that it manifests in “the maintenance of positive adjustment under challenging conditions such that the organization emerges from those conditions strengthened and more resourceful” (p. 3418). This becomes more evident when in times of crisis, or amidst a disaster, as discussed by Oliveira et al. (2023), most of the efforts to measure resilience depend on the occurrence of such unwanted events (Hosseini et al., 2016). This definition approaches resilience by its absence, rather than by its presence, aligned with the best-established definition of safety: the absence of unacceptable risks (Aven, 2014). It is derived from linear thinking and the belief that an unwanted outcome is the result of an unwanted action and vice-versa, also known as the causality credo (Hollnagel, 2014). Complex systems, on the other hand, are characterized by passing the tipping point “when causality breaks down” (Editorial, 2009, p. 1), and display unpredictable, emerging properties (Foote, 2007), leading to the common statement that a system is more than the sum of its constituting parts. Hollnagel (2012a) defines such systems as intractable: their functioning is not entirely known, they show high levels of interdependence with other systems and within their constituting parts, and, by the time a description of the system is completed, it has already changed its nature.

Resilience engineering (RE) is, then, developed within the field of safety science and safety management as an effort to understand complex sociotechnical systems and tackle issues that traditional risk assessments have been unable to cope with (Hollnagel, 2012b; Patriarca, et al., 2018a). In this new paradigm, resilience is defined as “the intrinsic ability of a system or an organization to adjust its functionality prior to, during, or following events and thereby sustain required operations both under expected and unexpected conditions [emphasis added]” (Hollnagel, 2010, p. 1). Among the many frameworks that emerged in the area, the Resilience Analysis Grid (RAG) was proposed by Erik Hollnagel in 2010 to assess organizational resilience in safety management systems using a questionnaire, which should be tailored to the specifics of each system under scrutiny. It assesses resilient performance through four potentials: responding, monitoring, anticipating, and learning, as in Figure 1.

Figure 1
The four potentials of resilience

The RAG questionnaire is primarily qualitative, and there are several probing questions from which researchers and practitioners can derive new, focused items for their needs (such as Hollnagel, 2015). There have been several applications of the RAG questionnaire across different safety management contexts, such as healthcare (Patriarca et al., 2018b; Chuang et al., The RAG questionnaire is primarily qualitative, and there are several probing questions from which researchers and practitioners can derive new, focused items for their needs (such as in Hollnagel, 2015). There have been several applications of the RAG questionnaire across different safety management contexts, such as healthcare (Patriarca et al., 2018b; Chuang et al., 2020), aviation (Silva & Cardoso Junior, 2021), manufacturing (Trajkova et al., 2021), construction sites (Pardo-Ferreira et al., 2018), and even in correctional services (Steen et al., 2021). However, there is no record of studies that apply the RAG framework to assess resilience in non-safety-management contexts, therefore disclosing a gap in the literature.

Projects share many aspects with the systems addressed by resilience engineering. It requires management of efficiency-thoroughness trade-offs, albeit not necessarily safety trade-offs (as in Hollnagel, 2009a), and competing goals. They have to cope with the inherent variability of this complex work environment (Oehmen et al., 2015), and make decisions of which aftermath may become apparent after a long timeframe (Strafaci, 2008). Although each project is unique, they compose the organizational portfolio and they are connected through the governance structure, meaning that the context is also posed by the organization. Understanding individual and systemic aspects, highlighting which challenges faced by them are within their sphere of influence, and which lie at the organizational level, will provide insights to decision makers at the organization and support the proposition of actions to bridge the gaps. By approaching projects as (temporary) organizations, it is possible to apply concepts and frameworks of organizational resilience. The RAG, as proposed by Hollnagel, integrated with of the Analytical Hierarchy Process (AHP) as done by Patriarca et al. (2018b), is a strong candidate to assess each project individually, and aggregate the results at the host organizational level using the hierarchical structure of the portfolio.

A post-RAG analysis phase is proposed to support decision making and to propose strategic actions. This extension of the RAG is supported by the design science research (DSR), a scientific paradigm aimed at the development of artifacts to solve practical problems (Thuan et al. 2019). As to demonstrate the proposed framework, it will be applied in a case study to assess the potential for resilient performance in a public technology-development facility in Brazil, with a diverse portfolio, and which has been facing challenges to deliver its projects.

Summing up, there are two key motivations to this work: a gap in the organizational resilience literature, and a problem-solving one, emerging from the context of the organization where the case study is conducted, posing a research opportunity. Based both on the context and the motivations, this paper aims to respond to two research questions: how can we evaluate the potential for resilient performance in a project-oriented organization (RQ1) and, in the case study, what are the systemic and what are the particular aspects that may affect the potential for resilient performance among the projects in the organization (RQ2).

THEORETICAL FOUNDATION

Organizational resilience in perspective

Following the definition of resilience posed in the management literature, forecasting unwanted scenarios is a key aspect of risk management and business continuity management (Jain et al., 2020). It can, however, be tricky to be performed in complex management systems, especially when signals are weak (Silva, 2015) or when the outcome of decisions will only become apparent after a long period of time (Strafaci, 2008). This definition of resilience, therefore, does not allow for an assessment in times when operational conditions are as expected.

In project management, particularly in engineering systems, the literature on resilience follows a similar definition. Wied et al. (2019) discuss definitions of resilience in engineering systems which are tightly coupled to unexpected events or extraordinary operational conditions, with either positive or negative outcomes. From the three perspectives on project resilience, Piperca and Floricel (2023) approach projects as complex systems, which can be designed to be reliable and able to rebound after a disturbance in expected operational scenarios. In their conceptual work, resilience is supported by other organizing processes, which allow for anticipating future scenarios and realizing the actual context. This approach goes in line with that adopted in resilience engineering.

As stated in the introduction, the definition of resilience adopted in this study is borrowed from safety science. It was developed in a context where traditional risk and safety analysis were not able to cope with the emerging complexity of the work environment, which has two different components: the sharp end, in which the work is effectively carried out, and where people perform the required work while interacting with potentially hazardous processes, and the blunt end, composed by organizational layers that may affect or interfere with the work conditions in the sharp end, but which do not participate effectively in it (Hollnagel, 2014). From the sharp-end perspective, work can only be carried out by adjusting their performance in accordance to the situation. This performance variability is key to the concept of work-as-done (WAD), which is the description of the work performed in the sharp-end, opposed to the concept of work-as-imagined (WAI), which is a blunt-end perspective on how the work should be performed (Hollnagel, 2014). This difference is key to the emerging definition of resilience as the ability to manage variability. Resilience engineering rises as a new safety management paradigm, which allowed the development of “theories, methods and tools to deliberately manage the adaptive ability of organizations in order to function effectively and safely” (Nemeth & Herrera, 2015, p. 1). Hollnagel would introduce the idea that resilience is “something a system does rather than [...] something [...] a system has” (2011b, p. 275). By doing so, resilience could be managed and engineered as a process performed by an organization, composed of the four potentials, described by Hollnagel (2009b, 2018) as follows:

• Responding: the ability to deal with expected and unexpected situations, either using planned actions, by small adjustments, or by implementing a new set of actions;

• Monitoring: the ability to gather information which could potentially affect the system’s performance, within the time frame of its operation, both from the environment and from the system’s own performance;

• Anticipating: the ability to predict or forecast future scenarios, risks and opportunities, disruptions, which may rely on a model of the system and its operational context; and

• Learning: the ability to acquire knowledge from previous events and update the understanding about the system itself and its operational context, either by learning from one’s own experience, or from third parties, either from formal or tacit knowledge (Patriarca et al., 2021).

This perspective in organization and project studies has been indirectly presented by Weick and Sutcliffe (2015), who recognize that being sensitive to operational conditions and committed to resilience are fundamental to the ability of managing unexpected scenarios. In addition, Naderpajouh et al. (2020) recognize resilience as an interdisciplinary concept, which should be reflected in resilience-driven studies, and call for a further exploration of resilience paradigms, expanding from traditional uncertainty- and risk-driven approaches. Recent management studies already apply RE concepts, such as Steen et al. (2023), who propose a framework for integrating business continuity and resilience management, and Bigogno-Costa et al. (2021), who performed a resilience assessment of a subset of the engineering management process in a product-development project. The RAG, however, allows for an assessment from a bird’s-eye view of the organization, which will later subsidize further strategic actions or investigations. Project studies can be conducted from micro (individual) level to macro (societal) level (Geraldi & Söderlund, 2018). To that extent, the RAG questionnaire has also been applied across different levels of abstraction as defined by Bergström and Dekker (2014), such as individuals from different groups in a hospital staff (Patriarca et al. 2018b), departments within an organization (Steen et al., 2021) different organizations in a given sector (Djunaidi et al., 2021) or sectors in different countries (Falegnami et al., 2018). This documented ability of the RAG application studies drives this study, which aims to assess resilience further from the typical analyses, which focus mainly on effect on performance (Hosseini et al., 2016).

Designing for intervention

In order to achieve the aim of the study, which is to perform a resilience assessment using the RAG framework as an instrument, it is necessary to tailor the questionnaire to the organizational context. To advance towards this goal, this study is supported on two methodological paradigms: action research and design science research (DSR).

Action research is an empirical-based research mode, aimed at solving a collective problem, in which researchers and stakeholders cooperate in order to develop a solution (Thiollent, 2018). In that same paradigm, soft operational research and problem structuring methods provide means for structuring complex, wicked problems, in order to intervene in real-life problems (Midgley, 2008), such as facilitated modeling (Franco & Montibeller, 2010) and systemic intervention (Midgley, 2015).

Design science research (DSR), on the other hand, is a scientific paradigm proposed by Herbert A. Simon (1996), who argues about the need of a new scientific corpus to deal with creating artifacts, consisting of a set of properties, as a means to solving problems, or meeting a set of objectives. This new paradigm opposes formal and explanatory sciences to a new exploratory paradigm, aiming at developing “knowledge for the design and realization of artifacts, i.e., to solve construction problems, or to be used in the improvement of the performance of existing entities, i.e., to solve improvement problems” (Van Aken, 2004, p. 224). The DSR paradigm also differs from the formal and explanatory paradigms by taking a pragmatic approach to problem, focusing on applying knowledge to taking action (Romme, 2003; Van Aken, 2004), and by shifting the ontological perspective, in which the problem is created by the researcher, who design an instrument to address it - therefore its artificial nature, instead of just being out there (Holmström et al., 2009).

Both action research and DSR require collective action from stakeholders to meet their objectives. However, DSR takes a more prescriptive direction, as opposed to that of action research, which seeks an understanding of how things behave (Lacerda et al., 2013) - which does not pose a methodological conflict, rather being complementary. They also differ on the role of the researcher, who plays a role of a facilitator in the action research, whereas, in the DSR, they work as a designer of the artifact which will meet the needs of the end-users (Lacerda et al., 2013). In this research project, the researcher plays both the facilitator and the artifact designer roles.

It is noteworthy that Hollnagel (2010) expected that the scope of RAG would be expanded from its initial applications, and Righi et al. (2015) discussed how DSR can aid advancing the field. From artifacts development perspective, this exploratory study is the first step of the introduction of the RAG in a different context (Offermann et al., 2009). From the definitions of Gregor and Hevner (2013), it can be positioned both as an exaptation, i.e., the use of an established solution to a problem in another field, and an improvement, in this study, bringing additions to the original framework to enhance its capability.

With this theoretical and methodological background in mind, our hypotheses to respectively address the research questions are that the Resilience Analysis Grid, with the adequate tailoring, is capable of evaluating the potential for resilient performance in a project-oriented organization (H1) and that the results of the RAG questionnaire, given an adequate processing, will aid the identification of the systemic and particular aspects among the projects (H2).

METHODOLOGICAL PROCEDURES

Supported on the action research and DSR paradigms, the developments of the RAG framework (Hollnagel, 2011b) and by its extension with the analytical hierarchy process (AHP), as introduced by Patriarca et al. (2018b), Figure 2 shows the proposed research design for this study.

Figure 2
Research design for the assessment of potential for resilient performance using the RAG

Preliminary research

This step, which may be called “step 0”, aims at understanding the current status of the organization and what successful performance means to it. This is pivotal to establish what the aim of the system under assessment is to address its ongoing issues, and it will drive the following steps. Data is collected through document analysis, looking for recurring issues and struggles in progress reports and project closing reports, and semi-structured interviews with key actors in the governance to understand how the organization is structured and how it actually functions. It is advisable to allocate an organizational focal point, who has a broad knowledge of the organization and its projects – typically the portfolio manager. They shall follow the research project, establishing the aim of the assessment and later managing the benefits of its findings.

RAG 1: Defining and describing the organization

Provides a description of the organization in terms of structure, people and resources, as well as timeframe and, by so, determining the scope of the analysis (Hollnagel, 2011a). The information gathered in the previous step may be complemented with a further document analysis, interviews and direct observation, if possible.

RAG 2: Designing the questionnaire

The engagement of stakeholders is key for developing a questionnaire that addresses the problem from the perspective of those affected by the issue identified in the early stages of the study. This step is aided by the participation of subject-matter experts (SME’s), who have a broad understanding of the organizational aspects and of the ongoing projects. In order to get a systemic perspective (Franco & Montibeller, 2010), the SME’s came from two distinct groups: from the sharp-end of the work, meaning they deal on a daily basis with managing projects in the organization, and from the blunt-end, representing the actors in the governance structure, which has a significant impact on the operational conditions (Turner, 2020). Criteria for participating involved experience in project management in the organization and relevant background in projects, such as training and certifications.

Another framework which supports the development of the questionnaire is the Analytic Hierarchy Process (AHP). The AHP is a compensatory decision-making method from the operational research (OR) field (Saaty, 2004). The AHP simplifies the process of making decisions, with respect to a given objective, by breaking down decisions into a hierarchical structure in order to establish relative priorities among a set of alternatives. Pairwise judgements would set reference guides for stakeholders and decision-makers by comparing two similar alternatives, enabling a more consistent assessment than absolute judgements. The number of pairwise judgements, however, shows a quadratic growth behavior as a function to the size of the set of alternatives. Not only can it be too tiring for raters to perform pairwise judgements in large sets of alternatives, Miller (1956) sets a psychological boundary of 7 ± 2 elements so that consistency is reached at the end of the process. As an alternative, Saaty (2006) proposes the use of absolute judgments, supported by the building of a relative scale: the AHP with ratings. Now, each alternative (lowest level) will be assessed through a set of criteria, weighted with respect to each other through pairwise judgements, and each criterion has a set of possible intensities, which will be ranked based on a pairwise judgment with each other.

Originally, step 2 of the RAG framework is not broken down in substeps (Hollnagel, 2011a). For this study, however, it has been tailored to accommodate the development cycles for questionnaires as a scientific instrument, such as in Gil (2021), Sarantakos (2012) and Tsang et al. (2017), splitting the step into four substeps: setting questionnaire requirements (2A), eliciting items (2B), assessing items (2C), and releasing the questionnaire (2D).

Setting requirements (RAG 2A): The requirements drive the design of the questionnaire, which shall meet the research objectives, and the needs and expectations of its stakeholders, who have an understanding of the end user in the organization. Requirements range from defining parameters, such as item structure (open- or close-ended) and expected time to completion, to how the items should be distributed across and within the different potentials. Requirements can be sourced both from the aforementioned literature on questionnaire design, and from specific demands of the subject, which were collected through a survey. New requirements may arise concurrently with the phases 2B and 2C.

Eliciting items (RAG 2B): Items can be elicited through a participative process, as performed similarly in other RAG studies, using structured interviews (Nilsson, 2015), focus groups (De Linhares et al., 2021) or Delphi (Pardo-Ferreira et al., 2018). In this case, the SME’s provide contributing factors (CF) within the four resilience potentials through a semi-structured interview, based on the probe questions as suggested by Hollnagel (2015). They also respond to why a CF contributes to that resilient potential, and how it contributes - which can be in terms of actions, tasks or expected behavior. Questions are sourced from “how” enunciations. Before the interview, a handout is provided to the SME’s, explaining the aim of the interview and leveling up the vocabulary. After the interviews, similar contributions are aggregated through thematic analysis (Clarke & Braun, 2017), forming a single and extensive database of items.

Assessing items (RAG 2C): The SME’s assess the items elicited in the previous phase in criteria as used by Pardo-Ferreira et al. (2018) and Chuang et al. (2020): coherence and relevance to the resilient potential, and clarity of the writing. Each SME established their own importance to the criteria, and the judgements were aggregated using the AHP with ratings to provide each item’s score. Through their feedback, items had their phrasing improved. The aim is to create a solid base from which the best-ranked items are selected. In this case study, one design decision was to structure the responses to the items based on a 5-point Likert scale. However, instead of following the standard phrasing, they were tailored to reflect the work-asdone at the host organization, based on expectation of ideal performance and on the governance cycle. The scoring was also tailored, based on the perception of the group of SME’s and determined using the multi-stakeholder AHP aggregation process, as in Logullo et al. (2022).

Releasing the questionnaire (RAG 2D): With the final items’ score, and based on the requirements established in RAG 2A, a subset of items is proposed to the group of SME’s. Upon their agreement, a version is released for administration.

RAG 3: Administering the questionnaire

The questionnaire is administered in two moments: the pre-test, when a subset of the population interacts with the newly developed instrument as a means to verify whether it meets its requirements and the primary research objectives. During the pre-test administration, the researcher may observe and interact with the respondents, monitoring time to respond and checking if there are any issues with the research instrument. Participants will, then, respond to a post-pre-test survey to verify performance requirements which rely on their sentiment, such as mental burden, ease of use, content coverage, and language adequacy, and to state if the results of the questionnaire are representative of their perception of their work. This is a form of face validation, which determines whether the instrument is “understandable and relevant to the target population” (Tsang et al., 2017, p. S87). Despite being a weaker form of questionnaire validation, it verifies that the actions taken respond to the research question RQ1. The survey also works as a form of member checking, where participants “examine and validate the accuracy of the findings” (Ahmed, 2024, p. 2). In addition, a report with the summary of results is presented to the SME’s so they can provide their feedback as a complementary form of member checking of the aggregate results. If the pre-test is successful, the second moment is the final administration, when the entire population participates.

RAG 4: Aggregating the data

The hierarchical structure established using the AHP enables a bottom-up aggregation framework using the weights of each item, based on the final rating in step 2C, then through the contributing factors up to the potential level. The aggregation is performed in three main structures: in a project level, in the organizational level, and within the themes. Figure 3 illustrates both aggregation cases. Final results aggregations up to the resilient potentials are displayed in a radar chart, which is typical for RAG applications (e.g. Hollnagel, 2011b), and in other resilience assessment studies, as in Santos and Spers (2023). For lower-tier levels, the choice, due to the amount of information, is a mixed representation with a bar chart (as done by Pęciłło, 2020) for individual projects, and solid dots for the organizational aggregate score.

Figure 3
Example of how scores are aggregated within each project to contributing factor and resilient potential levels (left) and up to the organizational level (right)

Post-RAG analyses

This novel step is proposed after the results are aggregated, in which two analyses will provide insights for the proposal of strategic actions. The first is the contributing factors for organizational potential for resilient performance assessment. To aid decision-making, the organizational focal point establishes reference points for prioritizing contributing factors, categorizing them in critical, attention, and acceptable. For simplicity, and considering this was the first administration of the RAG in the organization, the focal point decided that the first quartile would be critical, and the second quartile would be the attention category. However, this is at the discretion of the decision maker, advised by the researcher or practitioner.

The second is systemic-particular analysis, which aims at identifying contributing factors in which projects perform similarly, which could mean an organizational trait, as well as contributing factors in which the performance differs significantly across projects. This latter assessment will indicate projects from which good practices can be shared across the organization in order to raise the potential for resilient performance. This analysis is performed by establishing the organizational score, plus and minus a pre-agreed margin, as a baseline. A novel, table-based approach is proposed, using a non-compensatory model (Almeida, 2013), through which points are given for each time a given project score deviates from the reference margin: +1 is awarded when the result is above the reference, and -1 is given when the result is below the reference. From a project’s perspective, counting positive or negative scores, and comparing these indicators, may clarify which projects are over- or underperforming compared with the organizational score. This will indicate if a project, or a set of projects, has an overall result above, or below that of the organization, seeking these particular aspects of the projects’ performance. From the contributing factor’s perspective, counting the scores, both positive and negative, and its sum will allow assessing which are:

• Systemic: no project scores outside the baseline;

• One stands out: one single project scores outside the baseline; and

• Non-systemic: six or more projects score outside the baseline.

The post-RAG analysis, by design, aims at validating the hypothesis H2, associated with the research question RQ2. It is supported by the DSR objective to solve problems by the design of artifacts to acquire and process data from contexts, such as management research (Van Aken, 2004). From the results of both analyses, a set of strategic actions is proposed.

To summarize the actors’ involvement across the research steps, Table 1 provides a comprehensive guide of the participants and in which stages they are engaged.

Table 1
Participants involved in the research and stages in which they are engaged

Participants		Stages of involvement
Title	Role	Step 0	RAG phase							Post-RAG
			1	2A	2B	2C	2D	3	4
Organizational focal point	Establishes the aim of the assessment, considering the current status of the organization and its strategic goals, and manages the benefits of the research outcomes	x	x	i	i	i	i	i	i	i
Subject-matter experts (SME’s)	Aid the development of the research instrument (questionnaire) and should be sourced both from the governance structure (blunt-end) and from the ongoing project management teams (sharp-end)	o		x	x	x	x		*
Project managers or project management team members	Key subject of the study, provides the information of the ongoing situation of the projects in the organization by responding to the questionnaire. In the end, they are informed of the results of the assessment.							x		i

• Note: x = effective engagement in the stage and contributes towards its objective; o = optional engagement; i = informed about the ongoing situation of the research and its outcomes; * = in the particular case of the pre-test cycle, SME’s are informed about the results of the questionnaire administration and shall determine that it is ready for the final administration.

Case study description

To assess the framework, a real-world case study was conducted in a public technology-development facility in Brazil, whose aim is to develop high-end products and technologies in the aerospace industry through projects. As a public organization, some of its challenges are facing fund cuts and struggling with acquiring and retaining personnel to work in its projects.

From an organizational perspective, success is to deliver these products and capabilities while maintaining organizational governance structure, and abiding to the laws and regulations of the public service - that is, by an efficient use of resources, both human and financial, and by delivering these outcomes within the agreed timeframe. Documents from the organizational project management office (PMO) show that the projects have been struggling to meet these objectives and, therefore, they would benefit from a resilience assessment. The projects compete internally for resources, mostly engineers and test facilities, and with support activities. There are interfaces with private organizations, which can be either suppliers or subcontractors, but the main client is the Brazilian government. The organizational value delivery system is presented in Figure 4, in which it is possible to see that the portfolio consists of seven projects, divided in two categories:

Figure 4
Case study organizational value delivery system

• Two priority projects which develop full-scale systems, involving several disciplines across the organization; and

• Five internal projects, developing technologies, subsystems or systems which will be integrated in existing, or future full-scale systems, and typically involve less disciplines.

For this study, projects are considered organizations within themselves which, when integrated in their categories and in the portfolio, contribute to the overall organizational performance. This allows each project to be approached as a system on its own, whereas the organization is considered a system-of-systems (as defined in Ackoff, 1971). Support services integrate the latter; however, their impact is assessed individually to each project organization, hereon simply referred to as “project”.

The organizational portfolio manager has been invited to participate as the organizational focal point. To contribute to the questionnaire design, five SME’s were invited to participate in the development of the questionnaire. Invitations were based on their positions in the blunt-end and sharp-end of the project management governance structure, as well as their experience in projects in the organization:

• Blunt-end: Two representatives from the governance structure: the organizational portfolio manager and one member of the Project Management Office. Both have over 15 years of experience in the organization, over 10 years of experience working in projects, and basic training in project management.

• Sharp-end: Three participants, who are involved in project management teams. Their experience ranges from 5 to 10 years working on projects in the organization. They are trained on project management on a specialization level; they have training in specific project management knowledge areas, and in product life cycle. They have at least one certification in project management.

All participants, including project managers and project team members who responded to the questionnaire, signed informed consent forms, and both participant and organizational confidentiality were ensured throughout the process.

RESULTS

This section displays the outcome of the case study across each phase of the RAG framework, with the addition of the post-RAG phase. The case study took place from September 2021 to May 2022.

RAG 1: Scope of the analysis

The organizational focal point agreed that all seven ongoing projects will be subject to the study, and that the aim is to take a snapshot of their current status and assess the gaps to the desired condition of successfully delivering the projects. Each project system is bounded by the PM’s sphere of influence and the host organization, and the resources are those made available by the organization.

RAG 2: Designing the questionnaire

The questionnaire requirements, during phase 2A, were enunciated and validated with the aid of the group of SME’s, who established parameters for size (50 ± 5 items) and time to respond (less than 60 min). To reflect the aim of performing a broad assessment of the projects, the questionnaire should be closed-ended, which allowed for a semi-quantitative approach, meaning that the numbers do not effectively reflect reality, but allow for a more objective, straightforward comparison between two qualitative judgements. They also determined that items should be distributed based on the number of contributing factors for each resilient potential, and that items should cover as many topics as possible

During the items elicitation phase (RAG 2B), 37 contributing factors and 131 items were elicited through the semi-structured interviews with the SME’s. Later, items were assessed in three criteria: coherence, relevance, and clarity during the assessing phase (RAG 2C). In total, after two feedback cycles, 60 items were revisited, and 1 was rejected, due to lack of coherence to the corresponding resilient potential. As emphasized in the methodological procedures, the aim is to establish a comprehensive database of items that are clear, well-written and representative of the work-as-done.

Finally, based on the requirements established on phase 2A and on the scores from phase 2C, a subset of items is selected and released in the RAG 2D step. For the pre-test version, 52 items, distributed across 34 contributing factors, were selected. The group of SME’s agreed upon the set of items, which are described in Table 2.

Table 2
Contributing factors and rationale of the selected items for the first RAG questionnaire release

CF ID	Contributing factor	Item ID	Item rationale
Responding (15/42)
R.1	Standards and procedures for project management	R.1.1	Roles of PM and others involved in the project governance structure
		R.1.7	Documents and artifacts to be produced by the project as a function of the project lifecycle
R.2	Management of key project aspects	R.2.1	Scope implementation
		R.2.2	Schedule and time allocation
		R.2.3	Cost and budget allocation
R.3	Communication and stakeholder management	R.3.4	Responding to information demands
R.4	Decision-making in the project environment	R.4.2	Dependency on decisions beyond the PM’s scope, and time for this decision-making
R.5	PM and project team maturity, experience and capabilities	R.5.5	Previous experience in project management or project environments
R.6	Change management	R.6.1	Having a clear process for categorizing, deciding upon, and implementing changes in the project
R.7	Risk management	R.7.2	Risk response: triggers and contingency plans
R.8	Resources management	R.8.1	Allocation of human resources
		R.8.3	Allocation of physical resources (materials, equipment, infrastructure) external to the organization
R.9	R.9.4	Organizational assets	Tool for quick internal communications
R.10	Organizational aspects	R.10.1	Adequation and burden of the governance structure
		R.10.3	Prioritizing project-related activities over non-project-related activities
Monitoring (14/32)
M.1	Project data and indicators	M.1.1	Ability to collect representative data from the project
M.2	Standards and procedures for project management	M.2.1	Comprehensive activities list for PM’s on how to monitor and control a project
M.3	Communication and stakeholder management	M.3.1	Ability to communicate the most up-to-date information on both expected and unexpected demands
M.4	Management of key project aspects	M.4.1	Monitoring the scope implementation
		M.4.2	Monitoring the project schedule
		M.4.3	Monitoring the project costs
		M.4.4	Tracking the project objectives
M.5	PM and project team maturity, experience and capabilities	M.5.3	Keeping track of the status of the team
M.7	Risk management	M.7.1	Risk monitoring: monitoring of identified risks
		M.7.2	Risk monitoring: monitoring of risk triggers
M.8	Resources management	M.8.3	Monitoring availability of physical resources in the organization
M.9	Organizational assets	M.9.1	Integrated enterprise governance system for keeping project data and information
M.10	Organizational aspects	M.10.1	Organizational follow-up of project status
		M.10.4	Organizational follow-up of its own status (stocks, knowledge resources, etc).
Anticipating (15/36)
A.1	Management of key project aspects (Planning project lifecycle)	A.1.4	Periodically revisiting the project objectives
A.2	Standards and procedures for project management	A.2.3	Tools for planning and performing status forecast
A.3	Lessons notes and reports	A.3.1	Reviewing other projects documentation and learned lessons
A.4	Project data and indicators	A.4.2	Forecasting project scenarios based on performance indicators
A.5	PM and project team maturity, experience and capabilities	A.5.2	Tracking team expectations and future plans
A.6	Communication and stakeholder management	A.6.1	Stakeholder identification and analysis
A.7	Risk management	A.7.1.1	Risk identification: broadness of risk identification throughout the project scope
		A.7.2.2	Risk assessment: providing a clear risk prioritization
		A.7.3.1	Risk response planning: planning proactive responses to identified and prioritized risks
A.8	Resources management	A.8.1	Projecting future project human resources demands in time
		A.8.2	Projecting future project physical resources demands in time
A.10	Organizational assets	A.10.2	Organizational strategic plan
A.11	Organizational aspects	A.11.1	Revisiting organizational strategic objectives
		A.11.2	Incentives and support to risk management culture
		A.11.5	Organizational efforts to train and prepare current and future personnel in project management
Learning (8/20)
L.1	Standards and procedures for project management	L.1.2	Standardizing documentation for registering, keeping, and sharing lessons learned
L.2	Management of project lessons learned	L.2.2	Time allocated for collecting learned lessons
L.3	Sharing and disseminating knowledge across the projects and the organization	L.3.3	Time between learning a lesson and its effective implementation
L.5	Organizational assets	L.5.1	Organizational database of learned lessons in current and previous projects
L.6	Organizational aspects	L.6.1	Organizational structure for supporting learned lessons activities and book-keeping
		L.6.2	Activities and culture related to lessons learned
		L.6.5	Efforts for sharing and disseminating knowledge and learned lessons across different projects
		L.6.6	Openness to error

RAG 3 and 4: Questionnaire administration and results aggregation

Pre-test administration. The pre-test administration selected 4 of the 7 ongoing projects in the organization. After the interaction with the research instrument, participants provided feedback through the post-pre-test survey. All requirements were verified, and only minor fixes, such as phrasing, were needed. Through this survey, they also agreed that the questionnaire was showed representative results, covering broadly the aspects of the project they manage and that the results reflect the reality of their project, as seen in Figure 5. The results meet the face validity criteria, therefore confirming the hypothesis associated with RQ1.

Figure 5
Results of the post-pre-test survey, collecting feedback from participants, indicating that the questionnaire showed representative results when assessing the projects they manage

Final administration. All seven projects took part in this phase. The responses were aggregated using the AHP with ratings structure and the weighting system based on the scores of each item, as exemplified in Figure 3. For the aggregati,on at the organizational level using the AHP structure, each portfolio was allocated an equal weight - 0.5 - and, within each portfolio, projects were equally weighted. Figure 6 shows the overall score for the resilient potentials in each project and in the host organization, whereas Figure 7 shows the results within each potential.

Figure 6
Results of the aggregation for the potential levels (R: responding, M: monitoring, A: anticipating, L: learning) for each project, and the overall organizational score (ORG), including the reference values for the first (Q1) and second quartile (Q2)

Figure 7
Results for the contributing factors in responding, monitoring, anticipating, and learning for each project, and the overall organizational score (ORG), including the reference values for the first (Q1) and second quartile (Q2)

The immediate conclusion of the results shown in Figure 6 is that all projects show a similar pattern for potential of resilient performance: monitoring is highly emphasized, which is expected, considering how robust the governance structure is required for public organizations. Responding and anticipating are balanced, and the worst scoring potential for resilient performance is learning, marginally making it outside the critical threshold. Project P1, however, outscores all projects in a potential-level, and pulls the organizational score upwards. For the contributing factors in each potential, the graphs in Figure 7 may provide some insights, although they are visually messy, so the post-RAG analysis is performed.

Post-RAG analyses

Contributing factors for organizational resilient performance. The first assessment addressed the contributing factors at an organizational level. Some of the contributing factors in the critical category (below Q1 on Figure 7) raise awareness across the organization, for instance, R.5 “PM and team maturity, experience, and capability”. Another poor scoring CF from responding is R.10 “Organizational aspects”. From the themes of the items in R.10, there are two emerging issues: burden of the governance structure and how non-project-related activities are prioritized over project-related ones, meaning that the projects are spending more time on activities which do not aggregate value to their projects, therefore hindering their ability to respond effectively to the projects’ needs.

Systemic-particular analysis. Based on whether projects perform above or below the organizational values, with a marginal offset established by the focal point by ± 0.15, Table 3 shows how scores have been allocated.

Table 3
Comparing performance between the projects and the organization, with a marginal offset of ± 0.15

	P.1	P.2	I.1	I.2	I.3	I.4	I.5	No. +1	No. -1	Sum
R.1	+1	0	-1	-1	-1	+1	-1	2	4	6
R.2	+1	-1	0	-1	-1	-1	-1	1	5	6
R.3	+1	-1	-1	+1	+1	+1	-1	4	3	7
R.4	0	-1	0	-1	+1	-1	0	1	3	4
R.5	0	0	0	0	0	0	0	0	0	0
R.6	0	+1	-1	-1	-1	0	0	1	3	4
R.7	+1	+1	-1	-1	-1	+1	-1	3	4	7
R.8	0	0	-1	-1	-1	+1	0	1	3	4
R.9	+1	-1	-1	-1	+1	+1	-1	3	4	7
R.10	+1	0	0	0	0	0	0	1	0	1
M.1	+1	0	0	-1	-1	0	+1	2	2	4
M.2	0	0	+1	-1	+1	0	0	2	1	3
M.3	+1	0	-1	0	0	0	0	1	1	2
M.4	0	0	0	0	-1	0	0	0	1	1
M.5	0	0	-1	0	-1	0	0	0	2	2
M.7	0	0	0	-1	0	-1	+1	1	2	3
M.8	0	0	-1	0	-1	0	0	0	2	2
M.9	0	0	0	0	0	0	0	0	0	0
M.10	0	0	0	0	-1	+1	0	1	1	2
A.1	+1	-1	-1	-1	0	0	0	1	3	4
A.2	0	0	0	0	0	0	0	0	0	0
A.3	+1	0	0	0	-1	0	+1	2	1	3
A.4	0	0	0	0	0	0	0	0	0	0
A.5	+1	+1	-1	-1	-1	+1	+1	4	3	7
A.6	+1	+1	0	-1	-1	-1	+1	3	3	6
A.7	0	+1	-1	-1	0	0	0	1	2	3
A.8	+1	-1	0	+1	-1	-1	-1	2	4	6
A.10	0	0	0	-1	0	-1	-1	0	3	3
A.11	+1	0	0	0	+1	-1	0	2	1	3
L.1	0	0	0	0	0	+1	0	1	0	1
L.2	0	+1	-1	-1	-1	-1	+1	2	4	6
L.3	+1	+1	-1	-1	-1	-1	-1	2	5	7
L.5	+1	0	0	0	-1	-1	-1	1	3	4
L.6	0	0	0	-1	0	0	0	0	1	1
	P1	P2	I1	I2	I3	I4	I5		Sum color code:
No. +1	16	7	1	2	5	8	6		Systemic
No. -1	0	6	14	18	17	10	9		One stands out
Differential	16	1	-13	-16	-12	-2	-3		Non-systemic

From the project perspective, project P1 stands out with 16 CF scoring above organizational reference points, whereas projects I1, I2, and I3 have a final balance of -13, -16, and -12 of CF outside the organizational thresholds. As a conclusion, project P1 is the best adjusted project in the organizational structure, and for which it best meets their needs. P1 stands out compared to the other priority project, P2, which has a balance of only 1 contributing factor above the baseline. Although both projects differ significantly from one another in its key aspects, the results show that they also differ when considering how the organization meets its needs. On the other hand, project I1, I2, and I3 show significant results below the reference values, in comparison with the remainder, and it raises the question whether they fit in the organizational strategic objectives and, if so, why they are struggling so much, compared with the other projects.

From the contributing factors perspective, only four are systemic: R.5 “PM and team maturity, experience, and capability”, M.9 “Organizational assets”, A.2 “Standards and procedures”, and A.4 “Project data and indicators”. The potential for resilient performance in these CF is consistent across all seven projects, pointing towards issues on an organizational level. R.5 highlights a weakness in the PM career and professional development within the organization, which resonates in the practitioners’ capabilities across all potentials. M.9 shows that, despite how prone the organization is to monitoring, the processes and organizational tools do not fit the needs of the projects and of the governance structure. A.2 points towards a gap in organizational sources to guide planning activities across the projects. Finally, A.4 indicates that the quality of the data used to manage the projects and make predictions, or planning upon them, is very poor.

Strategic Actions Proposal. Supported on the data, there are five strategic actions proposed to address gaps and to enhance the potential for resilient performance.

• SA #01: Investigates organizational aspects that are hindering learning capabilities, since this is the worst scoring potential in the organization.

• SA #02: Investigates why there is waste of time and effort of the project management teams in activities which do not contribute to the project objectives.

• SA #03: Addresses systemic contributing factors which are scoring in the “critical” range (R.5, M.9, A.2, and A.4).

• SA #04: Addresses high scorers for good practices and low scorers, to understand why performance differs significantly in these contributing factors;

• SA #05: Investigates the impact of not having project P1 - comparing the organization with and without this project - and address the gap between P1 and the remaining projects, especially the other priority project P2; and

• SA #06: Investigates the deficiencies of projects I1, I2, and I3, and assess whether they fit or not in the organizational strategic objectives. If so, the organization should revisit its approach to these projects.

Strategic actions #01 to #03 address systemic, organizational traits, which affect all projects similarly, whereas SA #04 to #06 tackle different projects individually. This instantiation of the post-RAG, including the instrument developed and applied for the systemic-particular analysis in Table 3, provides evidence that it is possible to identify issues on an organizational level, and on each constituent of the system, therefore meeting the hypothesis H2 established to respond to the research question RQ2.

DISCUSSIONS

Aimed at assessing safety management in complex socio-technical systems, the RAG framework identifies sources of risk, from an operational perspective, which should be addressed as to improve potential for resilient performance. In a project management context, risk management aims at improving odds of project success (Project Management Institute, 2021). Therefore, the RAG instantiation of a project management application should tackle the issues and sources of risk that may affect overall project and organizational success.

The frame portrayed by the RAG results suggest that the organization is driving its efforts primarily to meet its governance demands. The intensive focus on monitoring, a fundamental function of project management, still faces drawbacks from lack of integration of data (M.9) and it is not adequately supported by the other potentials. These issues may also hinder the Anticipating potential by providing lagging indicators, negatively resonating on the potential for Responding. On the other hand, the demand for yearly planning cycles works positively towards the Anticipating potential. The governance structure, in this case, is as a two-sided coin, working simultaneously in favor and against the value delivery system, and cannot be eliminated. SA #02 is suggested, from a RE perspective, to bring in some more variability to aid coping with balancing this trade-off.

The discrepancy between projects within the portfolios is also expressed through the RAG results. In the public sector, political decisions in prioritizing projects in portfolio management is a recurrent concern (Alves et al., 2022), therefore SA #05 and #06 may shed light on strategic issues. Considering the aim of the study, it would not be meaningful to compare projects in different portfolios or their aggregated score – which could be performed, if desired, using the AHP structure. The experience of RAG to compare different groups, such as Patriarca et al. (2018b) and Steen et al. (2021) successfully highlighted the gaps between them.

Developing the questionnaire is the most time-demanding activity in implementing the RAG, following the experience of Chuang et al. (2020) and Patriarca et al. (2018b). The RAG 2 stage lasted approximately five months, including the pre-test phase. It also requires an intensive engagement of the SME’s, which is facilitated through the “less formal and more qualitative” approach of the RAG (Woltjer et al., 2013, p. 7). Once the questionnaire is ready to be administered, the data collection phase takes a few days’ time and may be done remotely.

In order to perform an effective assessment, the aim of the study must be clear from start (Hollnagel, 2011a). Therefore, criteria for organizational success should be identified in “step 0” and further validated in the organizational description in RAG 1. These objectives may be aligned with the organizational mission, vision and objectives; however it can be something different. If there can be a gap between the work-as-imagined, or work-asprescribed by standards and procedures, and the actual work as performed in the sharp end, referred as work-as-done (Hollnagel, 2014), there may also be a gap between the organizational strategy as formally established and what actually is going on a daily basis, such as a “strategy-as-done”. In this case, it is possible to administer the RAG either aiming towards the current organizational objectives, supporting the goal of this “strategy-as-done”, or towards a point in the future, a desired state of the business. In either case, this early decision will affect remainder of the assessment.

The post-RAG improved identifying the main organizational issues, presenting straightforward indicators to be addressed. The bar chart, for a large number of contributing factors, is an improvement of the radar chart, a typical RAG representation instrument, and where the area below the curve may change depending on the order of the variables (Chuang at al., 2020). The RAG structure gives flexibility to convey its results (Rodriguez et al., 2020), so the choice should meet the end of performing an effective communication to the stakeholders.

When applied in a safety management context, the RAG should be integrated with other sources of information (Wahl et al., 2020) and the same is applicable to non-safety-critical environments. As an example, Bigogno-Costa et al. (2022) investigate a project team using social network and the functional resonance analysis method to describe interactions between the staff and actors outside the project and identify sources of risk from a complex socio-technical system perspective. The benefit of performing the RAG instantiation is to identify complexity-induced risks and to collect information to support decision making, aiding the allocation of efforts to address the issues more effectively. It also provides a sound business case for investing in resilience, as advocated by Lee et al. (2013).

As an instrument, the RAG provides a proxy measurement of resilience (Hollnagel, 2015), meaning that it is still relevant to assess how performance is affected when unexpected events occur, and compare the two different states of the organization. Moreover, even with the proposed post-RAG step, the framework does not perform an assessment of the entire risk governance structure, currently encompassing only generation of knowledge and decision making, leaving out implementation of actions (The International Risk Governance Council, 2018). Further studies should address the closed loop, which incorporates the results of the implemented actions. By then, it will be possible to position the research as action design research (as defined by Sein et al, 2011).

It is also unfeasible to validate the RAG questionnaire in a single administration cycle, considering how small the population is (Tsang et al., 2017), and randomization is not achievable. To tackle these issues, content and construct contributes to instrument validity (Tsang et al., 2017), whereas member checking, as performed during the pre-test phase, strengthens confirmability (Ahmed, 2024) and concur to validity (Hendren et al., 2022). Triangulation adds credibility in the results (Hendren et al., 2022) – in this work, project status reports gathered during step 0 and RAG 1 phases indicated similar issues which were identified in the RAG 4 and post-RAG phases. Furthermore, transferability can be met through thick descriptions – a comprehensive description of “research, context, participants and methods” that provide significant information so researchers and practitioners may reflect whether the findings are relevant for them (Ahmed, 2024, p. 2). Finally, from the DSR perspective, the instantiation is a key step in the evaluation of the solution designed and, therefore, meets the scientific objectives of developing the RAG as a class of problems (Gregor & Hevner, 2013; Sein et al., 2011; Offermann et al., 2009).

Resilience studies address issues in complex systems. The current work environment in project management point towards an increasing integration with technology, constituting complex socio-technical systems, which are not alien to the scope of resilience engineering. The RAG provides a good introduction to the RE for those unfamiliar with its principles (Nemeth et al., 2017), and management studies can benefit from different approaches. By bearing in mind the similarities and differences between safety and non-safety critical applications, RE frameworks may shed light where that traditional risk and resilience assessments have been unable to – meeting the very aim of this field.

FINAL CONSIDERATIONS

Resilience is a key concept in current operational and management studies. To cope with the increasing complexity in the work environment, particularly in projects in engineering systems, this paper borrowed the definition of resilience from resilience engineering, which tackles problems in complex socio-technical systems. The research design met the expected results, in which the Resilience Analysis Grid framework, in the case study, demonstrated its capability to generate knowledge about the potential for resilient performance in all the projects in the organization, enabling a complexitydriven risk assessment of the system. The questionnaire, with 52 items distributed across 34 contributing factors in the four resilient potentials: responding, monitoring, anticipating, and learning, provides a bird’s-eye perspective that a typical risk assessment may not provide, such as providing evidence that there is a systemic issue with organizational learning, and that all projects struggle with how the organization prioritizes its activities. It also highlighted imbalances between the projects in the organization, which might be expected, but can point towards an issue to meet the overall organizational objectives.

A limitation of the study concerns subjectiveness. Although the semiquantitative approach provides some objectiveness, it depends on the judgement of the personnel involved. Thick descriptions, diversity of subject-matter experts, member checking and triangulation aid coping with biases and subjectiveness in a complex, people-driven context. Subsequent studies may explore the difference between instruments – not just RAG – designed from each end of the work process and one which, as reported here, has been designed with the collaboration between blunt- and sharp-end workers. Future investigations should also address means to bridge the gap to more objective assessments and how they can be performed.

Since this is a single case study, it is unfeasible to generalize the results to a broader conclusion, being the first instantiation of the instrument in a new context. The main contribution of this article, therefore, is a framework for resilience assessment. As part of the DSR scientific research cycle, further instantiations in similar and other management systems, especially closed-loop ones, should address the RAG effectiveness on driving resilient performance in management systems and its ability to enhance day-to-day operations, considering both expected and unexpected scenarios.

The RAG is a versatile instrument, which should be tailored to meet the specifics of the system of interest and may aid researchers and practitioners in coping with complexity beyond the scope of safety-critical systems. On the other hand, there is an inherent burden in its implementation which may be investigated in future studies which apply the closed-loop multiple times to a single system. Some concerns raised are whether the same questionnaire can be applied multiple times or if it should be redesigned, and how frequently, and whether the RAG may result in a pitfall where indicators become more important than the actual aim to perform resiliently.

• RAM does not have information about open data regarding this manuscript.
• RAM does not have permission from the authors or evaluators to publish this article’s review.

REFERENCES

• Ackoff, R. L. (1971). Towards a system of systems concepts. Management Science, 17(11), 661–671. https://doi.org/10.1287/mnsc.17.11.661
  » https://doi.org/10.1287/mnsc.17.11.661
• Ahmed, S. K. (2024). The pillars of trustworthiness in qualitative research. Journal of Medicine, Surgery, and Public Health, 2, Article 100051. https://doi.org/10.1016/j.glmedi.2024.100051
  » https://doi.org/10.1016/j.glmedi.2024.100051
• Almeida, A. T. d. (2013). Processo de decisão nas organizações: Construindo modelos de decisão multicritério Editora Atlas.
• Alves, A. M., Ceolin, A. C., & Moraes, I. C. (2022). Gerenciamento de portfólio de projetos no setor público. Id on line. Revista de Psicologia 16(59), 250–269. https://doi.org/10.14295/idonline.v16i59.3386
  » https://doi.org/10.14295/idonline.v16i59.3386
• Aven, T. (2014). What is safety science? Safety Science, 67, 15–20. https://doi.org/10.1016/j.ssci.2013.07.026
  » https://doi.org/10.1016/j.ssci.2013.07.026
• Aven, T. (2016). Risk assessment and risk management: Review of recent advances on their foundation. European Journal of Operational Research, 253(1), 1–13. https://doi.org/10.1016/j.ejor.2015.12.023
  » https://doi.org/10.1016/j.ejor.2015.12.023
• Bergström, J., & Dekker, S. (2014). Bridging the macro and the micro by considering the meso: Reflections on the fractal nature of resilience. Ecology and Society, 19(4), Article 22. https://doi.org/10.5751/ES-06956-190422
  » https://doi.org/10.5751/ES-06956-190422
• Bigogno-Costa, V., Cardoso Junior, M. M., Saurin, T. A., & Grøtan, T. O. (2022) Using the functional resonance analysis method for modeling social interactions in socio-technical systems: An exploratory study. In Leva M. C., Patelli, E., Podofillini, L., & Wilson, S. Proceedings of the 32nd European Safety and Reliability Conference (ESREL 2022), (pp. 391–398). Research Publishing. https://doi.org/10.3850/978-981-18-5183-4_R1008-441-cd
  » https://doi.org/10.3850/978-981-18-5183-4_R1008-441-cd
• Bigogno-Costa, V., Cardoso Junior, M. M., & Urbina, L. M. S. (2021). Project management process resilience: Assessing and improving the project review process using FRAM. In Rossit, D. A., Tohmé, F., & Mejía Delgadillo, G. Production research: 10th International Conference of Production Research – Americas, ICPR-Americas 2020, Bahía Blanca, Argentina, December 9-11, 2020, Revised Selected Papers, Part II (pp. 141–155). Springer International Publishing. https://doi.org/10.1007/978-3-030-76310-7_11
  » https://doi.org/10.1007/978-3-030-76310-7_11
• Chuang, S., Ou, J. C., Hollnagel, E., & Hou, S. K. (2020). Measurement of resilience potential - development of a resilience assessment grid for emergency departments. PLoS ONE, 15, Article e0239472. https://doi.org/10.1371/journal.pone.0239472
  » https://doi.org/10.1371/journal.pone.0239472
• Clarke, V., & Braun, V. (2017). Thematic analysis. The Journal of Positive Psychology, 12(3), 297–298. https://doi.org/10.1080/17439760.2016.1262613
  » https://doi.org/10.1080/17439760.2016.1262613
• De Linhares, T. Q., Maia, Y. L. & Ferreira Frutuoso e Melo, P. F. (2021). The phased application of STAMP, FRAM and RAG as a strategy to improve complex sociotechnical system safety. Progress in Nuclear Energy, 131, Article 103571. https://doi.org/10.1016/j.pnucene.2020.103571
  » https://doi.org/10.1016/j.pnucene.2020.103571
• Djunaidi, Z., Tantia, A. A., & Wirawan, M. (2021). Analysis of the safety resilience implementation in the maritime industry at public and private companies (A case study in indonesia). Safety, 7(3), Article 56. https://doi.org/10.3390/safety7030056
  » https://doi.org/10.3390/safety7030056
• Editorial: No man is an island. (2009). Nature Physics, 5, 1. https://doi.org/10.1038/nphys1162
  » https://doi.org/10.1038/nphys1162
• Falegnami, A., Bilotta, F., Pugliese, F., Costantino, F., Di Gravio, G., Tronci, M., & Patriarca, R. (2018). A multicountry comparative survey about organizational resilience in anaesthesia. Journal of Evaluation in Clinical Practice, 24(6), 1347–1357. https://doi.org/10.1111/jep.13054
  » https://doi.org/10.1111/jep.13054
• Foote, R. (2007). Mathematics and complex systems. Science, 318, 410–412. https://doi.org/10.1126/science.1141754
  » https://doi.org/10.1126/science.1141754
• Floricel, S.; Miller, R. (2001). Strategizing for anticipated risk and turbulence in large-scale engineering projects. International Journal of Project Management, 19(8), 445–455. https://doi.org/10.1016/S0263-7863(01)00047-3
  » https://doi.org/10.1016/S0263-7863(01)00047-3
• Franco, L. A., & Montibeller, G. (2010). Facilitated modeling in operational research. European Journal of Operational Research, 205(3), 489–500. https://doi.org/10.1016/j.ejor.2009.09.030
  » https://doi.org/10.1016/j.ejor.2009.09.030
• Geraldi, J., & Söderlund, J. (2018). Project studies: What it is, and where it is going. International Journal of Project Management, 36(1), 55–70. https://doi.org/10.1016/j.ijproman.2017.06.004
  » https://doi.org/10.1016/j.ijproman.2017.06.004
• Gil, A. C. (2021). Métodos e técnicas de pesquisa social (7th ed.). Editora Atlas.
• Gregor, S., & Hevner, A. R. (2013). Positioning and presenting design science research for maximum impact. MIS Quarterly, 37(2), 337–55. https://doi.org/10.25300/MISQ/2013/37.2.01
  » https://doi.org/10.25300/MISQ/2013/37.2.01
• Hendren, K., Newcomer, K., Pandey, S. K., Smith, M., & Sumner, N. (2022). How qualitative research methods can be leveraged to strengthen mixed methods research in public policy and public administration? Public Administration Review, 83(3), 468-485. https://doi.org/10.1111/puar.13528
  » https://doi.org/10.1111/puar.13528
• Hollnagel, E. (2009a). The ETTO principle: Efficiency-thoroughness trade-off. Ashgate.
• Hollnagel, E. (2009b). The four cornerstones of resilience engineering. In Nemeth, C. P., Hollnagel, E., & Dekker, S. (Eds.), Resilience engineering perspectives, volume 2 (pp. 117-133). CRC Press.
• Hollnagel, E. (2010). How resilient is your organisation? An introduction to the resilience analysis grid (RAG). In Sustainable transformation: Building a resilient organization, May 2010, Toronto, Canada. Retrieved March 01, 2022, from https://hal-mines-paristech.archives-ouvertes.fr/hal-00613986.
  » https://hal-mines-paristech.archives-ouvertes.fr/hal-00613986
• Hollnagel, E. (2011a). Epilogue: RAG - the resilience analysis grid. In Hollnagel, E., Pariès, J., Woods, D. D., & Wreathall, J. Resilience engineering in practice (pp. 275-296). Ashgate.
• Hollnagel, E. (2011b). Prologue: The scope of resilience engineering. In Hollnagel, E., Pariès, J., Woods, D. D., & Wreathall, J. Resilience engineering in practice (pp. xxix-xxxix). Ashgate.
• Hollnagel, E. (2012a). Coping with complexity: Past, present and future. Cognition, Technology and Work, 14(3), 199–205. https://doi.org/10.1007/s10111-011-0202-7
  » https://doi.org/10.1007/s10111-011-0202-7
• Hollnagel, E. (2012b). FRAM: The functional resonance analysis method Ashgate.
• Hollnagel, E. (2014). Safety-I and safety-II: The past and future of safety management. Ashgate.
• Hollnagel, E. (2015). RAG – Resilience analysis grid Retrieved April 19, 2022, from https://erikhollnagel.com/onewebmedia/RAG%20Outline%20V2.pdf.
  » https://erikhollnagel.com/onewebmedia/RAG%20Outline%20V2.pdf
• Hollnagel, E. (2018). Safety-II in practice: Developing the resilience potentials Routledge.
• Holmström, J., Ketoviki, M., & Hameri, A. P. (2009). Bridging practice and theory: A design science approach. Decision Sciences, 40(1), 65–87. https://doi.org/10.1111/j.1540-5915.2008.00221.x
  » https://doi.org/10.1111/j.1540-5915.2008.00221.x
• Hosseini, S., Barker, K., & Ramirez-Marquez, J. E. (2016). A review of definitions and measures of system resilience. Reliability Engineering and System Safety, 145, 47–61. https://doi.org/10.1016/j.ress.2015.08.006
  » https://doi.org/10.1016/j.ress.2015.08.006
• Jain, P., Pasman, H. J., & Mannan, M. S. (2020). Process system resilience: From risk management to business continuity and sustainability. International Journal of Business Continuity and Risk Management, 10(1), 47–66. https://doi.org/10.1504/IJBCRM.2020.105615.
  » https://doi.org/10.1504/IJBCRM.2020.105615
• Lacerda, D. P., Dresch, A., Proença, A., & Antunes Júnior, J. A. V. (2013). Design science research: Método de pesquisa para a engenharia de produção. Gestão & Produção, 20(4), 741–761. https://doi.org/10.1590/S0104530X2013005000014
  » https://doi.org/10.1590/S0104530X2013005000014
• Lee, A. V., Vargo, J., & Seville, E. (2013). Developing a tool to measure and compare organizations’ resilience. Natural Hazards Review, 14(1), 29–41. https://doi.org/10.1061/(asce)nh.1527-6996.0000075
  » https://doi.org/10.1061/(asce)nh.1527-6996.0000075
• Logullo, Y., Bigogno-Costa, V., Silva, A., & Belderrain, M. C. (2022). A prioritization approach based on VFT and AHP for group decision making: A case study in the military operations. Production, 32, Article e20210059. https://doi.org/10.1590/0103-6513.20210059
  » https://doi.org/10.1590/0103-6513.20210059
• Midgley, G. (2008). Systems thinking, complexity and the philosophy of science. Emergence: Complexity and Organization, 10(4), 55–73. https://journal.emergentpublications.com/Article/351bda0e-01e1-45e5-b199-4f67903 c5ba8/jats
  » https://journal.emergentpublications.com/Article/351bda0e-01e1-45e5-b199-4f67903 c5ba8/jats
• Midgley, G. (2015). Systemic intervention. In Bradbury, H. (ed.). The SAGE handbook of action research (3rd ed.), (pp. 157–166). SAGE Publishing.
• Miller, G. A. (1956). The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychological Review, 63(2), 81–97. https://doi.org/10.1037/h0043158
  » https://doi.org/10.1037/h0043158
• Naderpajouh, N., Matinheikki, J., Keeys, L. A., Aldrich, D. P., & Linkov, I. (2020). Resilience and projects: An interdisciplinary crossroad. Project Leadership and Society, 1, Article 100001. https://doi.org/10.1016/j.plas.2020.100001
  » https://doi.org/10.1016/j.plas. 2020.100001
• Nemeth, C., Lay, B., Blume, J., Stephenson, J., & Holbrook, J. (2017, June 26-29). Using the RAG to assess International Space Station organizational resilience [Paper presentation]. The 7th Symposium on Resilience Engineering, Liège, Belgium. https://bit.ly/3SRh5NX.
  » https://bit.ly/3SRh5NX
• Nemeth, C. P., & Herrera, I. (2015). Building change: Resilience engineering after ten years. Reliability Engineering & System Safety, 141, 1–4. https://doi.org/10.1016/j.ress.2015.04.006
  » https://doi.org/10.1016/j.ress.2015.04.006
• Nilsson, J. (2015). Self-assessed resilience within municipal governments in Sweden [Master’s thesis, Linköpings Universitet]. Digitala Vetenskapliga Arkivet.
• Oehmen, J., Thuesen, C., Ruiz, P. P., & Geraldi, J. (2015). Complexity management for projects, programmes, and portfolios: An engineering systems perspective. PMI White Paper Retrieved December 21, 2023, from https://www.pmi.org/learning/library/complexity-management-projectsprogrammes-portfolios-11141.
  » https://www.pmi.org/learning/library/complexity-management-projectsprogrammes-portfolios-11141.
• Offermann, P., Levina, O., Schönherr, M., & Bub, U. (2009). Outline of a design science research process. In DESRIST ‘09: Proceedings of the 4th international conference on design science research in information systems and technology, (pp. 1–11), Association for Computing Machinery. https://doi.org/10.1145/1555619.1555629
  » https://doi.org/10.1145/1555619.1555629
• Oliveira, C. R., Alcadipani, R., Madariaga, P. I., Coraiola, D. M., & Teixeira, M. L. M. (2023). Theorizing and researching contemporary organizations in contexts of crisis and extreme events. Revista de Administração Mackenzie, 24(6), 1–5. https://doi.org/10.1590/1678-6971/eRAMD230400.en.
  » https://doi.org/10.1590/1678-6971/eRAMD230400.en
• Pardo-Ferreira, M. C., Rubio-Romero, J. C., & Martínez-Rojas, M. (2018). Applying resilience engineering to improve safety management in a construction site: Design and validation of a questionnaire. International Journal of Industrial and Manufacturing Engineering, 12(9), 1236–1241. https://doi.org/10.5281/zenodo.1474691
  » https://doi.org/10.5281/zenodo.1474691
• Patriarca, R., Bergström, J., Di Gravio, G., & Costantino, F. (2018a). Resilience engineering: Current status of the research and future challenges. Safety Science, 102, 79–100. https://doi.org/10.1016/j.ssci.2017.10.005
  » https://doi.org/10.1016/j.ssci.2017.10.005
• Patriarca, R., Di Gravio, G., Costantino, F., Falegnami, A., & Bilotta, F. (2018b). An Analytic Framework to Assess Organizational Resilience. Safety and Health at Work, 9(3), 265–276. https://doi.org/10.1016/j.shaw.2017.10.005
  » https://doi.org/10.1016/j.shaw. 2017.10.005
• Patriarca, R., Falegnami, A., Costantino, F., Di Gravio, G., De Nicola, A, & Villani, M. L. (2021). WAx: An integrated conceptual framework for the analysis of cyber-socio-technical systems. Safety Science, 136, Article 105142. https://doi.org/10.1016/j.ssci.2020.105142
  » https://doi.org/10.1016/j.ssci.2020.105142
• Pe˛ciłło, M. (2020). Identification of gaps in safety management systems from the resilience engineering perspective in upper and lower-tier enterprises. Safety Science, 130, Article 104851. https://doi.org/10.1016/j.ssci.2020.104851
  » https://doi.org/10.1016/j.ssci.2020.104851
• Piperca, S., & Floricel, S. (2023). Understanding project resilience: Designed, cultivated, or emergent? International Journal of Project Management, 41, Article 102453. https://doi.org/10.1016/j.ijproman.2023.102453.
  » https://doi.org/10.1016/j.ijproman.2023.102453
• Project Management Institute. (2021). A guide to the project management body of knowledge: The PMBoK guide (7th ed.). Project Management Institute.
• Righi, A. W., Saurin, T. A., & Wachs, P. (2015). A systematic literature review of resilience engineering: Research areas and a research agenda proposal. Reliability Engineering and System Safety, 141, 142–152. https://doi.org/10.1016/j.ress.2015.03.007
  » https://doi.org/10.1016/j.ress.2015.03.007
• Rodriguez, M., Lawson, E., & Butler, D. (2020). A study of the resilience analysis grid method and its applicability to the water sector in England and Wales. Water and Environment Journal, 34(4), 623–633. https://doi.org/10.1111/wej.12539
  » https://doi.org/10.1111/wej.12539
• Romme, A. G. L. (2003). Making a difference: Organization as design. Organization Science, 14(5), 558–573. https://doi.org/10.1287/orsc.14.5.558.16769
  » https://doi.org/10.1287/orsc.14.5.558.16769
• Saaty, T. L. (2004). Decision making — the Analytic Hierarchy and Network Processes (AHP/ANP). Journal of Systems Science and Systems Engineering, 13, 1–35. https://doi.org/10.1007/s11518-006-0151-5
  » https://doi.org/10.1007/s11518-006-0151-5
• Saaty, T. L. (2006). Rank from comparisons from ratings in the analytic hierarchy/network processes. European Journal of Operational Research, 168(2), 557–570. https://doi.org/10.1016/j.ejor.2004.04.032
  » https://doi.org/10.1016/j.ejor.2004.04.032
• Santos, A. Y. P., & Spers, R. G. (2023). How to measure organizational resilience? Validating a simplified model. Revista de Administração Mackenzie, 24(6), 1–32. https://doi.org/10.1590/1678-6971/eRAMD230007.en
  » https://doi.org/10.1590/1678-6971/eRAMD230007.en
• Sarantakos, S. (2012). Social research (4th ed.). Palgrave Macmillan.
• Sein, M. K., Henfridsson, O., Purao, S., Rossi, M., & Lindgren, R. (2011). Action design research. MIS Quarterly, 35(1), 37–56. https://doi.org/10.2307/23043488
  » https://doi.org/10.2307/23043488
• Sheffi, Y. (2007). The resilient enterprise: Overcoming vulnerability for competitive advantage MIT Press.
• Silva, M. (2015). A systematic review of foresight in project management literature. Procedia Computer Science, 64, 792–799. https://doi.org/10.1016/j.procs.2015.08.630
  » https://doi.org/10.1016/j. procs.2015.08.630
• Silva, M. H. d. O. C., & Cardoso Junior, M. M. (2021, July 20-21). Paradigm shift in safety management in the Brazilian army aviation: Proposal and validation of a RAG questionnaire [Paper presentation]. 38th International Symposium on Military Operational Research, London, UK. http://bit.ly/3NZ5kTP.
  » http://bit.ly/3NZ5kTP
• Simon, H. A. (1996). The sciences of the artificial (3rd ed.). MIT Press.
• Steen, R., Haug, O. J., & Patriarca, R. (2023). Business continuity and resilience management: A conceptual framework. Journal of Contingencies and Crisis Management, 32¸ Article e12501 https://doi.org/10.1111/1468-5973.12501
  » https://doi.org/10.1111/1468-5973.12501
• Steen, R., Ingvaldsen, G., & Patriarca, R. (2021). Engineering resilience in a prison’s performance management system. Safety Science, 142, Article 105367. https://doi.org/10.1016/j.ssci.2021.105367
  » https://doi.org/10.1016/j.ssci.2021.105367
• Strafaci, A. (2008). What does BIM mean for civil engineers? CE News, Transportation Retrieved March 24, 2023, from http://images.autodesk.com/adsk/files/what_does_bim_mean_for_civil_engineers_ce_news_1008.pdf.
  » http://images.autodesk.com/adsk/files/what_does_bim_mean_for_civil_engineers_ce_news_1008.pdf
• The International Risk Governance Council. (2018). Risk governance: Towards an integrative approach. White Paper No. 1. International Risk Governance Council (IRGC). https://irgc.org/wp-content/uploads/2018/09/IRGC_WP_No_1_Risk_Governance__reprinted_version_3.pdf.
  » https://irgc.org/wp-content/uploads/2018/09/IRGC_WP_No_1_Risk_Governance__reprinted_version_3.pdf
• Thiollent, M. (2018). Metodologia da pesquisa-ação (18th ed.). Cortez Editora.
• Thuan, N. H., Drechsler, A., & Antunes, P. (2019). Construction of design science research questions. Communications of the Association for Information Systems, 44, 332–363. https://doi.org/10.17705/1CAIS.04420
  » https://doi.org/10.17705/1CAIS.04420
• Trajkova, K., Sterjovski, N., & Andonov, M. (2021). Organizational resilience – paradigm for corporate sustainability during COVID-19 in the private sector of North Macedonia. In V. Bevanda (Ed.), 5th EMAN Conference Proceedings (pp. 253–262). Association of Economists and Managers of the Balkans. https://doi.org/10.31410/eman.2021.253
  » https://doi.org/10.31410/eman.2021.253
• Tsang, S., Royse, C. F., & Terkawi, A. S. (2017). Guidelines for developing, translating, and validating a questionnaire in perioperative and pain medicine. Saudi Journal of Anaesthesia, 11(Suppl 1), S80–S89. https://doi.org/10.4103/sja.SJA_203_17
  » https://doi.org/10.4103/sja.SJA_203_17
• Turner, R. (2020). Investigating how governmentality and governance influence decision making on projects. Project Leadership and Society, 1, Article100003. https://doi.org/10.1016/j.plas.2020.100003
  » https://doi.org/10.1016/j.plas.2020.100003
• Van Aken, J. E. (2004). Management research based on the paradigm of the design sciences: The quest for field-tested and grounded technological rules. Journal of Management Studies, 41(2), 219–361. https://doi.org/10.1111/j.1467-6486.2004.00430.x
  » https://doi.org/10.1111/j.1467-6486.2004.00430.x
• Vogus, T. J., & Sutcliffe, K. M. (2007). Organizational resilience: Towards a theory and research agenda. 2007 IEEE International Conference on Systems, Man and Cybernetics (pp. 3418–3422). https://doi.org/10.1109/ICSMC.2007.4414160
  » https://doi.org/10.1109/ICSMC.2007.4414160
• Wahl, A., Kongsvik, T., & Antonsen, S. (2020). Balancing safety I and safety II: Learning to manage performance variability at sea using simulator-based training. Reliability Engineering & System Safety, 195, Article 106698. https://doi.org/10.1016/j.ress.2019.106698
  » https://doi.org/10.1016/j.ress.2019.106698
• Weick, K., & Sutcliffe, K. (2015). Managing the unexpected: Sustained performance in a complex world (3rd ed.). Wiley.
• Wied, M., Oehmen, J., & Welo, T. (2019). Conceptualizing resilience in engineering systems: An analysis of the literature. Systems Engineering, 23, 3–13. https://doi.org/10.1002/sys.21491
  » https://doi.org/10.1002/sys.21491
• Woltjer, R., Pinska-Chauvin, E., Laursen, T., & Joseffsson, B. (2013, November 26-28). Resilience engineering in air traffic management: Increasing resilience through safety assessment in SESAR [Paper presentation]. Third SESAR Innovation Days, Stockholm, Sweden. https://bit.ly/3yWd6bZ
  » https://bit.ly/3yWd6bZ

Edited by

• Editor-in-chief
  Almir Martins Vieira

Edited by

• Publishing coordination
  Andreia Cominetti

Data availability

RAM does not have information about open data regarding this manuscript.

Publication Dates

• Publication in this collection
  27 Oct 2025
• Date of issue
  2025

History

• Received
  12 Jan 2024
• Accepted
  05 May 2025