Unknown Unknowns in Innovative Projects: Early Signs Sensemaking

Abstract

This study aims to understand early signs’ sensemaking relevance to identifying unknown unknowns on innovative projects. When managing these projects, early signs of changes in the environment, combined with a sensemaking process, can help identify them prior to their occurrence and mitigate unwanted effects of these uncertainties. To this end, we conducted a field research to assess 16 projects, totaling 35 events representative of this kind of uncertainty. Based on in-depth interviews, we performed an initial qualitative analysis, and then applied non-parametric statistical tests. Early signs perception in cases where external factors may cause the occurrence of unknown unknowns show that this perception and the search for information are relevant to identifying unk unks. Furthermore, some biases, such as overconfidence and unrealistic optimism, can hinder it. There was a high prevalence of project managers detecting early signs of change in their projects at the beginning of the project. However, the majority of perceived early signs can be considered to be the first symptoms of a problem. Thus, as a contribution, we suggest: learning processes, which can provide sensemaking for early signs in unknown unknowns; project managers stimulating their team to be alert to environmental changes that may affect the project; and discussion of early signs perceived by the team during stakeholder management.

Key words:
unknown unknowns; project management; early signs; sensemaking; project uncertainty

Introduction

Business management, as it is known, must undergo deep and dramatic changes: first, to increase performance in answering market demands; and second to achieve new ways of working, such as through how collaboration and creativity (Hamel, 2007) has been stimulated by the Internet, and counting on the values and attitudes of the new workforce. This high level of novelty brings unpredictable uncertainty to innovative projects (Loch, DeMeyer, & Pich, 2006; Loch, Solt, & Bailey, 2008). As the uncertainty in a project increases, the project team tends to diminish knowledge exchange and combination (Mehta, Hall, & Byrd, 2014). Dealing with this kind of uncertainty is a challenge to Project Management [PM] (Ramasesh & Browning, 2014).

Unpredictable uncertainty is also called unknown unknowns [unk unks] (Wideman, 1992), because there is a complete unawareness of what will occur in the future. De Meyer, Loch and Pich (2002) divided this uncertainty into two other kinds: (a) unforeseen uncertainty, when the event is unlikely to occur or when there is unawareness about its existence, and (b) chaos, when the project starts without stable goals and assumptions. However, Snowden and Boone (2007) and Thamhain (2013) affirm that chaos does not have clear cause-and-effect relationships. Unforeseen uncertainty, in turn, is complex (Snowden & Boone, 2007) and requires emergent actions when it occurs. It was called an accident by Thamhain, (2013, p. 23), because it “can be identified in principle, but the occurrence probability and its specific scope and impact on project performance are difficult, if not impossible, to predict”. In this article, we will be use unk unks when referring to this type of uncertainty.

One way to identify unk unks is through the perception of an early sign. The real world is flooded with information, which is often ambiguous, inaccurate and incomplete; much information may be early signs of current cycle interruption, a break, which may be beneficial or harmful for business (Ansoff, 1975). Identification of an early sign can greatly contribute to making better decisions in the early stages of a project (Haji-Kazemi, Andersen, & Krane, 2013). Usually, however, project managers do not act upon them (Williams, Klakegg, Walker, Andersen, & Magnussen, 2012). In order to recognize such a sign, besides realizing that something has changed, its meaning must be understood. Sensemaking is the ideal tool for understanding meaning (Weick, 1995).

These distinct subjects (early signs, sensemaking, and uncertainty) have previously been studied in PM. Several studies (Haji-Kazemi, Andersen, & Klakegg, 2015; Kuosa, 2010; Williams et al., 2012) have assessed early signs management in projects. Stigliani and Ravasi (2012) studied cognitive processes in collective sensemaking in design projects, as Fellows and Liu (2016) assessed cultural influence in sensemaking in projects and innovations. Ramasesh and Browning (2014) have proposed a framework to transform unk unks into knowable events in PM, which include, among many factors, early signs sensemaking. However, none of them focused on understanding how a project manager and team members act to create sensemaking of the early signs of unk unks in innovative projects.

Therefore, this study aims to understand the relevance of perception and making sense of early signs for identifying unk unks in innovative projects prior to change. As a secondary goal, we sought to: (a) identify and understand the causes of unk unks; (b) understand the motivating factors for identifying unk unks, recognized as types of previous management and early sign perception; (c) assess efficiency in identifying unk unks; and (d) assess the relationship between the efficiency in identifying unk unks and the causes and motivating factors.

Next, the article summarizes the theoretical foundation on the subject, highlighting the themes of uncertainty and risks, early signs in projects and sensemaking. Then there is a detailed description of the methodological procedures used in the data collection, and qualitative and quantitative analysis. Finally, we present the results and offer discussion. The authors’ expectation is to make a practical contribution by recommending some approaches to PM, processes and the behavior of team members to help make sense of early signs and decrease uncertainty in innovative projects.

Theoretical Background

Uncertainty and risk in projects

Several authors affirm that Risk Management [RM], which is responsible for managing uncertainty in PM, does not deal with high levels of uncertainty (Loch et al., 2006; Murray-Webster & Pellegrinelli, 2010; Pender, 2001; Perminova & Wikstro, 2008). RM requires clear identification of event likelihood, both for risk and for opportunity, and of the level of their consequences in the project plan. The absence of adequate information makes it impossible to generate the planning, execution and appropriate monitoring of projects (Pich, Loch, & Meyer, 2002).

Despite this, unk unks are precisely the lack of knowledge of future events, which make them unpredictable. Thus, RM “will neither cope with the unknown unknowns, nor with extreme complexity or ambiguity” (Murray-Webster & Pellegrinelli, 2010, p. 12). Lehtiranta (2014) suggests more research about unk unks, which she called unanticipated and unrealistic assumptions, aiming at helping build a holistic approach to RM. Regarding the treatment of this type of uncertainty, only PRINCE2 (Office of Government Commerce [OGC], 2009) cites the monitoring of early signs, calling them signals and early warnings. ISO 31000 (Purdy, 2009) highlights the perception of and response to changes, which may occur slowly or suddenly.

Early signs in PM

The studies about early signs in PM were related to project success (Kappelman, Mckeeman, & Zhang, 2006; Sánchez & Pérez, 2004), with issues (Nikander & Eloranta, 1997); with risks (Nikander & Eloranta, 2001); with causes (Nikander, 2002); with frameworks for environmental scanning (Kuosa, 2010); with barriers for detecting them (Williams et al., 2012); and with barriers against effective responses to them (Haji-Kazemi et al., 2015).

The taxonomy used by the authors is varied: preliminary signal (Nikander & Eloranta, 1997, 2001; Project Management Institute [PMI], 2013); early warning signs (Kappelman et al., 2006; Sánchez & Pérez, 2004; Williams et al., 2012); warning signs (OGC, 2009); future signs (Hiltunen, 2008). For this study, we adopted the early sign taxonomy. Its nature is characterized as anticipatory, qualitative, ambiguous, fragmented; it may have various formats and may come from several different sources (Blanco, Caron-Fasan, & Lesca, 2003). In this regard, we identified several groups of early signs (Nikander & Eloranta, 1997; Sánchez & Pérez, 2004): gut feelings, communication among areas, expression from the various stakeholders, project control and reports, documents, and external sources.

Early signs can be classified into early information and early symptoms (Hiltunen, 2008). Early information consists of a small number of signs with low visibility, making it difficult to interpret, as in the case of gut feelings. Although the first symptoms present a great number of signs, interpretation can also be hindered by the lack of clarity in meaning, such as in the case of a person’s behavior change. Williams, Klakegg, Walker, Andersen and Magnussen (2012) found that assessment tools (project review, health assessment, benchmarking, post-project assessment and audit) “are limited in their ability to pick up early warning signs” (p. 47). These authors considered gut feelings to be any perceived sign that was not formalized in an indicator, without a specific focus or issue in mind. Schoemaker and Day (2009) affirm that problems of excessive information, organization filters, and selective perception and people’s cognitive biases prevent managers from creating meaning from these signs. That is, even when the sign is perceived, it may not be interpreted correctly.

Kappelman, Mckeeman and Zhang (2006) claim that the signs can come from three sources, dealing with Information Technology projects: product, process and people; the latter two sources are the most prevalent. Corroborating this view, Nikander and Eloranta (2001) concluded that most of the signs are linked to human factors, and are likely to occur at any time in the project. They also noted that the most important characteristics of those who identify the early signs are experience and insight, and that the traditional methods of PM monitoring and control should be used. The person who identifies the sign is not always the decision maker, who may even be outside the project team. However, “we are not very good at picking up early warning signs” (Williams et al., 2012, p. 37). Even, when we can perceive them, we need to make sense of their meaning to the project objectives and requisites.

Sensemaking

Weick (1995) created the concept of sensemaking, involving a broad and consistent approach to information interpretation. In addition to the construction of meaning about something, sensemaking would include notions such as structuring the unknown, placement of stimuli to create models, thought processes using retrospective data to explain surprises, placement of significance from authorship on ideas, individual and collective understanding of situations, development of cognitive maps of the environment, and creation of filters and ways for interpretation.

Sensemaking activities are very close to the concept of organizing, because people organize information and ideas to make sense out of them (Weick, Sutcliffe, & Obstfeld, 2005). The probable flow for sensemaking can be (Weick, 1995, p. 18): “people concerned with identity in the social context of other players engage themselves in ongoing events from which they extract cues and make plausible sense retrospectively while enacting more or less order into those ongoing events”.

The author indicates that this sentence does not show feedback, continuity, simultaneous processing, and also that some steps may not occur. Thus, the creation of meaning has seven properties (Weick, 2001):

1. Social context: where the meaning is created, not discovered, with interaction among people, be it either personally or not. The meaning created tends to be one that has social support, validation and shared relevance. Even in creating individual sense, there is an interaction with what one imagines what the other one understands about it. Therefore, even without communication, people influence each other;

2. Personal identity: the creation of meaning starts at the individual level (sensemaker) with either maintaining or establishing a personal identity. In interaction with the environment, the result is observed, with the understanding affected by varying needs of individuals in organizations (Weick, 1995) such as belonging to, evolving, being effective, standing out, integrating, representing the institution etc.;

3. Retrospect: the present is always recognized and based on past experience and tacit knowledge, including past decisions on adapting plans and objectives. To turn the abstract into concrete, people often act and then try to identify the reason for their actions. Weick (1995) reinforces the idea that actions are recognized only after their full implementation;

4. Salient cues: where cues can be understood as early signs, because it is perception of change in the organizational environment, such as an anomaly or through oral communication from a trusted source, recognizing an ecological change (Weick et al., 2005). They are seen and extracted from the environment for a context within mental models, personal beliefs, rules, and procedures. The individual undergoes retrospective experiences as environmental conditions affects his/her perception and selection of cues. Since the construction of meaning is a continuous process, new signs will be observed;

5. Ongoing projects: the process premise is the refinement of understanding, the actions taken and the restoration of balance, continuously and dynamically in context. It can be said that the actions and decisions are treated as a cycle more than as a linear sequence (Weick et al., 2005). They become an event only when the limits within that flow are defined or when an interruption occurs (Weick, 2001). In organizations, people are always within projects; their visions are directed to the situations, information, emotions and interruptions that occur with them (Weick, 1995);

6. Enactment: the people involved try to place the information within a known flow or create a new one, acting and generating a number of possible alternatives. The creation of meaning is based on building a reality, just as the legislature makes the laws (enact), by designating authority to events or signs within a specific context. People always dynamically create the environment, with actions or reactions, in the same way that the environment itself does. People act and then evaluate the results of their actions on an ongoing basis (Weick, 1995);

7. Plausibility: attempts to reduce ambiguity and misunderstanding. These alternatives try to explain the facts and the meaning of the changes for the organization, using people’s knowledge and retrospective experiences (individual and organizational). The expected result is a plausible history. People expand their repertoire by discarding some of their tools and beliefs (Weick, 2006), and maintaining or establishing a personal identity (Weick, 1995). Thus, in this continuous cycle, the plausible history tends to become more and more substantial, turning into experience.

In PM, the creation of meaning for the practice rule is implemented through countless translations based on the context, the history, the authorized standards and the feedback processes (Christiansen & Varnes, 2009). This application is also influenced by senior managers’ interpretation, use and feedback. Even companies with patterns of extensive and elaborate rules apply them in a flexible manner, according to the projects and the functional manager’s understanding. In creative projects, as a sensemaker, the manager gives meaning to the team’s effort, turning the collective representation of the project into a shared meaning, through informal communication and formal meetings with all stakeholders, whether internal or external to the project, including those external to the organization (Simon, 2006). This action is critical to building new paradigms (Thiry, 2001).

Research Methods

This research uses a mixed method with a fully mixed sequential equal status design (Leech & Onwuegbuzie, 2009). The dimension of fully mixed research can be explained with the use of at least one of four components: “the research objective, type of data and operations, type of analysis, and type of inference” (Leech & Onwuegbuzie, 2009, p. 270). The objectives are different; an exploratory process was applied in the qualitative phase in order to understand the relevance of the early signs to identify unk unks on innovative projects, and in the quantitative phase, the nature of the study was descriptive, to meet the goal of contributing to the existing theories. The type of data was the same for both phases, because the first phase generates variables and categories used in the second phase.

The unit of the research analysis is innovative projects and to select the appropriate ones we use the classification of Gallouj and Weinstein (1997), which apply tree vectors, competence [C], technology [T], and characteristics of product or service [Y]. They defined six types, but we use the three most innovative: radical innovation when at least vectors C and T are new; incremental when the vectors system is marginally modified with addition or substitution of any components; and ad hoc when the innovation is built by social interaction with client or user.

A cross-field study was conducted in companies located in Brazil, from April 2011 until December 2011. The sample was intentional and by convenience, because the elements were related to the characteristics set out in the formulated plan. The data collection started with a questionnaire sent to 152 innovative projects with 33 respondents. 16 projects, from 15 different organizations, were selected because they were characterized as radical innovation (7 projects), incremental (5 projects), or ad hoc (4 projects).

The questionnaire and the interview protocol consisted of open and qualitative questions, based on the Sensemaking Methodology theory (Dervin, 2008). Two unpredictable events were required for each project, but, for some projects, the respondent reported additional unpredictable events. These were also accounted, resulting in 35 events, addressed as incorporated units of analysis.

We conducted 16 in-depth interviews, obtaining supplementary material whenever possible. Interviews were face-to-face, when possible, or virtual. They happened outside of the workplace and lasted about 1.5 hours. Recording and transcribing the interview yielded 626 double-spaced pages. The questionnaires answered produced 52 additional pages.

The majority of respondents were male (81%), ten were project managers, three were sponsors and three were members of the team or members of the Project Management Office. They all participated actively in the PM of the project analyzed. Almost all of the organizations were service suppliers, with a predominance of domestic companies (75%) and large companies (44%). The vast majority of events (83%) would have or did have either a high or a very high impact on the projects’ results, highlighting their importance.

The type of analyses and inference were different in the two phases. In the first phase, qualitative analysis, the authors followed the six steps indicated by Creswell (2014): organizing, reading, coding, describing categories, forecasting the representation of information and extracting the final interpretation. It should be clear that the first four steps were not sequential, as the analyses of each project and the respective unpredictable events were conducted immediately after the interview. There was also a maturation process of the categories within each variable, leading to a continuous refinement, through case reanalysis. Coding was performed by using NVivo9 software, based on the variables identified in the theoretical foundation. For both the interview and the questionnaire, phrases mentioned by respondents were coded, classifying them into categories and generating new categories whenever necessary. We prioritized grounded theory techniques (Strauss & Corbin, 2008), such as using questions and especially analysis of phrases and comparisons between events.

The independent variables elected were causes of the unk unks, and the motivating factor of identifying unk unks was addressed in two items: one related to the type of previous management and the other to the early sign perception. Three characteristics were evaluated for this sign: (a) its type, (b) who perceived it, and (c) the moment they perceived it. The level of efficiency in identifying unk unks was considered a dependent variable. This variable was split into: (a) anticipative identification level [AIL], classified as identified previously and identified subsequently to the occurrence of an unpredictable event; and (b) event impact level [EIL], which was ranked more impact when it negatively affected customers, the organization and / or the project, and less impact when there was no negative impact or when it was possible to mitigate it.

In the second phase, quantitative analysis, the chi-square test or Fisher’s test was applied for nominal and ordinal variables to evaluate the theoretical hypothesis of two groups that differ in certain characteristics, namely, the relative frequency (or column%) of each of the two groups. When the cells percentage with expected values smaller than five were greater than 25%, it was necessary to apply Fischer proofs that do not have this requirement, but the table must be square: in this case, 2x2. Therefore, the categories of some variables were adjusted in order to meet this requirement, as described in the relational analysis section.

A 10% significance level was established in the relational analysis, which is acceptable in view of the small sample size and because the study is in the Social Sciences (Siegel & Castellan, 2006). We highlight the following limitations: (a) the selected cases, because even if they met the established requirements, they cannot be considered representative of an innovative project population, by sample size or by scale, and cannot be considered for innovation significance for each organization; (b) the choice of one respondent in a field study restricted the answers to the respondent’s perceptions, hindering other strategies of information validation; (c) the transformation that some variables experienced in relational analysis so that statistical tests could be applied may have prevented the identification of other relevant relationships due to the sample small size. Therefore, the results obtained from this study are difficult to generalize onto the population of PM innovators, but they are of great importance in establishing management practices in order to minimize the occurrence and effects of unk unks.

Results

First, we present the qualitative and descriptive analysis of the independent and dependent variables, based on the data obtained in the survey and interviews. In the second part, we will analyze the association between the dependent and independent variables to understand the early signs’ sensemaking relevance to identify unknown unknowns in innovative projects.

Qualitative and descriptive analysis

Table 1 displays the information obtained in the questionnaire and interviews. They will be explained in the next items.

Table 1
Qualitative Study Results

Unpredictable event	1	2	3	4	3	6	7	8	9	10	11	12	13	14	13	16	17	18	19	20	21	22	23	24	23	26	27	25	29	30	31	32	33	34	33	Total
Generating factor of the unpredictable uncertainty
External cause				X	X	X	X	X		X	X	X	X			X	X	X		X		X	X	X	X	X	X	X		X	X				X	23
Organizational cause	X		X				X		X					X															X					X		7
Cause originated in PM		X		X	X								X		X				X		X					X						X	X			10
Total of causes per unpredictable event	1	1	1	3	2	1	3	1	1	1	1	1	2	1	1	1	1	1	1	1	1	1	1	1	1	2	1	1	1	1	1	1	1	1	1	40
Motivating factor to identify the unpredictable uncertainty
Type of previous management of the unpredictable uncertainty
Search for information										X						X		X		X														X		5
Preriously managed bv planning or RM								X				X			X		X										X			X	X					7
Without previous management	X	X	X	X	X	X	X		X		X		X	X					X		X	X	X	X	X	X		X	X			X	X		X	23
Moment of the project in which the first eariy sign was perceived
Begginmg	X									X		X				X		X	X	X	X					X	X							X		11
During		X	X		X		X	X	X					X								X	X	X				X	X	X	X	X	X		X	17
End				X		X					X		X		X		X								X											7
Early Signs of unk unks
Early Information
Communication																																PM		PM		2
Project team	PM																																			1
Client's expression					TE											TE																				2
Partner's expression											X																									1
Extemal sources																PM		TE																		2
Gut feelings										PM										PM										SP						3
First Symptoms
Project and report control								TM																			TM	X								3
Client's expression					TE											TE					TM															3
User's expression																							TE													1
Work within the project			PM		TE				TM					SP															TM		SP					6
Sudden event			TE	TE		TM	SP				TE	TE	TE		TE		PM		PM			TE		TE	PM	PM		TE		TE			TE		TE	18
Total early signs per unpredictable event			1	1	3	1	1	1	1	1		1	1	1	1	3	1	1	1	1	1	1	1	1	1	1	1	2	1	2	1	1	1	1	1	42
Level of Efficiency in identfying the unforesseeable uncertainty
AIL - Antecipath e Identification Level
Previous			X		X				X	X				X		X		X		X	X		X						X			X		X		14
Subsequent				X		X	X	X			X	X	X		X		X		X			X		X	X	X	X	X		X	X		X		X	21
EIL - Event Impact level
Less impact			X		X				X	X				X		X		X		X	X		X		X	X	X		X		X	X		X		18
More impact				X		X	X	X			X	X	X		X		X		X			X		X				X		X			X		X	17

• Note. Legend: SI - Search for information, RM - Risk Management, TE - Team, PM - Project Manager, TM - Team member, SP – Sponsor.

Causes of unk unks

In a small portion of unpredictable events, there were more than one cause for the occurrence of unk unks, totaling 40 causes for 35 events. The causes verified were: external to the organization, comprising the majority of cases (57%), and including customers, environments, marketing, partners and suppliers; organizational (18%), originating in the structural characteristics and organizational procedures, related to a lack of structure and lack of integration in the areas; and originating in the PM, mentioned in ten unpredictable events (25%), when there was either failure in communication, failure in planning, problems in the team or an emergency solution.

Motivating factor for identifying unk unks

Type of previous management of unk unks

This variable has three categories according to Table 1: search for information, with 14% of cases; previously managed by planning or RM, with 20% of cases; without previous management, which represent 66% of cases. Search for information to overcome unk unks is the most proactive type of management (Loch et al., 2008). An example of the five unpredictable events of this type occurred after monitoring the factors that affected the project, through sources external to the organization, which allowed for identification of a variable that had not been considered in any similar project in Brazil. Respondents mentioned this type of management when an unpredictable event was previously identified and this search generated less impact on the project goals, as shown in Table 2.

Table 2
Type of Previous Management by Level of Efficiency

Level of efficiency in identification	Identification Level	Subsequent			Previous		Total
	Impact Level	More impact	Less impact		More impact	Less impact
Type of previous management of unpredictable event	Search for information	0	0		0	5	5
	Management by planning or RM	5	2		0	0	7
	Not previously Managed	12	2		0	9	23
	Total	17	4		0	14	35

In seven unpredictable events, the uncertainty was being managed by planning or RM. One is the case of the project for generating a new product, which required a machine with an unusual technical ability. In the previous analysis, numerous companies in Brazil and abroad were identified, which reported on their websites that they would produce that specific machine if it did not exist. Thus, in their RM, a contingency action was generated to import such equipment, since it is not manufactured in the domestic market. However, even after searching abroad, they found that the equipment did not exist, nor would it be possible for the suppliers to manufacture it. Therefore, a design was needed, and the team would lead its manufacturing.

Early sign perception

Table 1 presents the three characteristics of the early sign perception identified in the interviews: the moment of the project in which the first early sign was perceived, the type of early sign, and who perceived it. More than one early sign may have been perceived for the same event, so there were 42 signs for 35 unpredictable events. The cells in Table 1, marked with an X, show that an early sign was mentioned during the interview; however, it was not considered before the event occurred. These cells were disregarded in the accounting.

Each type of project has a life cycle and phases, defined according to the project’s product. To identify the moment in which the early sign was perceived, we generated a basic nomenclature that could comprehend any project. Thus, we defined three moments, as can be seen in Table 1: the beginning, 11 events, when the project is in its early stage of planning and solution development; during, 17 events, when the product or service is in full development or performance, and the end, 7 events, when the product is already developed and about to go into operation or a start-up phase.

The early sign may have been perceived by, according to Table 1: the team, when the respondent did not specifically mention one person, representing 45% of cases; the project manager, 26%; a team member, who can be specified (14%); or the sponsor, in 10% of cases.

About the type of early sign, early information, which is more difficult to perceive, was identified in ten interviews (26%) marked in Table 1. The first symptom has a sign easier to identify, as twelve cases (31%) found. A third category was created to represent an unpredictable event not perceived in advance, called a sudden event (43%), and in most cases, there was no time for actions to minimize its impact. It must be considered that other signs may have existed, but they were not observed or considered relevant by respondents, as indicated by an X in Table 1.

Early information, such as communication, in two of the 42 cases, detailed in Table 1, represents the obtaining of a sign in communication between departments during the project. As in the case of the leader who sought a system with basic information for his project, and when another manager suggested that he seek a particular department, he found more detailed handwritten information than he had expected.

The only case in which early information came from the project team took place at the beginning of a project. The manager perceived it during a brainstorm session: "I felt the need for them to understand each other better... to be able to write". In this case, the new team understood well how to develop a product, but was completely unaware of this new product market.

The early sign is perceived by the customer’s expression at the interface with the project team. We found five classifications for this type of sign, two of which classified as early information and three as first symptoms. In the first case, the respondent affirmed: "I could tell that there were very few directors who were more interested..." in leading that project within the client organization. Soon after, clients “openly expressed in a workshop" on the subject, which was considered to be a first symptom. These cases demonstrate how the signs became stronger, going from early information to first symptoms.

The leader, who was responsible for new product distribution, in the partner organization showed the only occurrence of the partner's expression. The head partner had left the organization, surprising the team, which did not sense the earlier signs, because “He himself had already said that he was very tired, he was exhausted". According to the respondent, "we would probably not want to believe that scenario."

There were two cases of early information obtained from external sources. In one case, the project manager had noticed rejection of his product by some organizations, "There are congresses; there are events in which we participate... where, through contact, we could see that some [people] were reacting" negatively to the product.

Three unpredictable events were preceded by gut feelings. In the first case, the respondent said, “It is as if we were [from the beginning] making a product to attract these great players.”

In three cases, the first symptoms were obtained through project and report control. In one of them, the team member noticed, after a detailed assessment of a supplier's control report, that data was fraudulent, which was confirmed after an audit.

The first symptom perceived by the user’s expression occurred in only one case, when the team realized during the pilot phase that the level of consumers’ involvement with the product was not as they had predicted.

The type of early sign mentioned the most by the respondents, in six cases, was the first symptom that occurred while working within the project. In one case, the sponsor was trying to define when a support area would be available for the project, but he realized that many systems were to be implemented on the same date, which made the support area availability impossible for the project.

Relationship between the characteristics of the early sign perception

As shown in Table 3, once the first early sign to each unpredictable event is perceived, the authors observed that: most of the early signs occurred suddenly (51%), the other two types occurred almost at the same rate - early information (23%) and first symptoms (26%); when these two types occurred, most early signs were perceived first by the team (47%), next, by the project manager (41%) and, last, by the sponsor (11%); half of the early signs (49%) were perceived during the project, eleven cases (31%) at the beginning and the remainder (20%) at the end.

Table 3
Amount of Unpredictable Events by Type of Early Sign Related to the Moment of Their Perception and Who Perceived Them

Type of early sign	Moment of perception			Who perceived it			Total
	Beginning	During	End	Team	Project Manager	Sponsor
Early information	6	2	0	2	6	0	8
First symptoms	2	7	0	6	1	2	9
Sudden	3	8	7	12	4	2	18
Team	4	11	5				20
Project Manager	7	2	2				11
Sponsor	0	4	0				4
TOTAL	11	17	7	20	11	4	35

• Note. Occurrences related to a team member were added to the team.

Most cases of early information occurred in the beginning of the project and most first symptoms occurred during the project, as well as the sudden events, whose occurrence was higher during and at the end of the project. Project managers were responsible for most of the early information perceptions, while the team perceived the most of the first symptoms. The project team perceived most of the sudden events. It can also be seen in Table 3 that the project manager was the one who perceived most signs at the beginning of the project. The team perceived more signs during the project, as well as the sponsor, with few occurrences.

Level of efficiency in identifying unk unks

The level of efficiency was measured by AIL and EIL. Regarding the AIL of the unk unks, of the 16 respondents, three failed to report an unpredictable event with a previous level of anticipation, however it was possible to identify two events of this level in one of the projects, totaling 14 events previously identified (40%). In events identified after occurrence, two respondents were able to report a third event with this level of anticipation, totaling 21 events identified later (60%).

About EIL, respondents reported only five cases (14%) in which there was no negative impact. In 13 cases, (37%) there was impact, but it was still possible to mitigate it. However, in 17 cases (49%), the impact was strong. Thus, it can be said that almost half of unpredictable events had less impact - with or without mitigating the impact.

Relational analysis

As it can be seen in Figure 1, both the cause of unk unks and the motivating factor for identifying unk unks are related to the level of efficiency in identifying unk unks. There are three exceptions to this relationship: (a) it was not possible to verify the association between the external cause and EIL, or (b) between the cause within the PM and any of the variables relating to levels of efficiency, and (c) between who perceived the early sign and AIL. Because the type of previous management of unk unks has three categories with small number of events in each of them, it was not possible to group them to meet the conditions for the chi-square test or Fisher test; therefore, it was not possible to evaluate their relationship to the efficiency level.

Figure 1
Relationship between Causes and Motivating Factor of Identifying with Level of Efficiency in Identifying Unk Unks

Table 4 shows that the subsequent identification of the unpredictable event occurrence was caused mostly by external factors (81%) rather than by previous identification. The organizational cause is more frequent in previously identified unpredictable events (42.9%) with less impact (33.3%). In a comparison between the two types of cause, it can be said that external causes are more likely to be identified later, and as for the internal cause, it is more likely to be identified beforehand with less impact.

Table 4
Distribution of Occurrence of Cause, Type of Early Sign and of Who Perceived the Sign by AIL and EIL

Variables			AIL			EIL		Total
			Previous	Subsequent		Less impact	More impact
Occurrence of cause external to the organization	Without external cause	Freq.	8	4				12
		%	57.1%	19.0%				34.3%
	With externalcause	Freq.	6	17				23
		%	42.9%	81.0%				65.7%
Occurrence of organizational cause	Without organizational cause	Freq.	8	20		12	16	28
		%	57.1%	95.2%		66.7%	94.1%	80.0%
	With organizational cause	Freq.	6	1		6	1	7
		%	42.9%	4.8%		33.3%	5.9%	20.0%
Occurrence of early sign	Without early sign	Freq.	0	18		2	16	18
		%	0.0%	85.7%		11.1%	94.1%	51.4%
	With early sign	Freq.	14	3		16	1	17
		%	100.0%	14.3%		88.9%	5.9%	48.6%
Who perceived the early sign	Team	Freq.				7	13	20
		%				43.8%	86.7%	64.5%
	Project Manager	Freq.				9	2	11
		%				56.2%	13.3%	35.5%
Moment of the early sign perception	Beginning	Freq.	7	4		9	2	11
		%	50.0%	19.0%		50.0%	11.8%	31.4%
	During / End	Freq.	7	17		9	15	24
		%	50.0%	81.0%		50.0%	88.2%	68.6%
Total		Freq.	14	21		18	17	35

Based on the first sign perceived in each unpredictable event, and in order to perform the statistical test, we summed the quantities of early information and first symptom occurrences, creating a category called with early sign. Cases of sudden early sign were categorized as without early sign. As shown in Table 4, for all events identified prior to the unpredictable event occurrence, there was early sign perception; however, for subsequent identification, there tended to be proportionally more unpredictable events without early sign (85.7%). In turn, the vast majority of events with less impact (88.9%) were those with at least one early sign perceived, while in almost all unpredictable events with more impact (94.1%), the early sign was not perceived. The early sign perception was relatively higher in unpredictable events with previous identification and those with less impact; and the opposite occurred when the early sign is not perceived.

In Table 4, two topics deserve a more detailed analysis, as they can be considered anomalies. The first refers to the three cases of subsequent identification of unpredictable event occurrence and with early sign. These events were previously managed by planning or RM; in two of them, management was not sufficiently thorough, and unexpected factors arose. In the third case, despite the use of outsourcing for mitigating the risk and despite all the team’s efforts, they could not avoid a labor action, unexpected by the project manager. The second topic reveals two cases, in which there was less impact, although the early sign had not been perceived beforehand. In one of them, when the project manager perceived the unpredictable event occurrence, he negotiated with the client because he would not have human resources or funds to meet the legal change that occurred during the project. After some legal advice, the client accepted the proposal. In the second case, the expert team main members, assigned and approved by the customer, could not start the project because of a delay in the client’s approval, but it was possible to adopt workarounds, avoiding greater impact to the project.

Regarding the ones who perceived the early sign variable, also in order to enable statistical tests, unpredictable events identified by the sponsor were disregarded, because of the low number of occurrences (four). According to the distribution showed in Table 4, the events with less impact tended to be proportionally perceived more often by the project manager (56.2%), while the team perceived the vast majority of those with more impact (86.7%).

About the early sign perception moment, we summed the perception occurrences during and at the end of the project in order to allow for statistic test application. In Table 4, it can be noticed that, while the previous identification was divided in equal proportions between the beginning and during or at the end of the project, in most cases identified subsequently (81%), the perception of the early sign occurred at the final stages of the project.

Likewise, in cases with less impact, there was an equal ratio between the signs perceived in the beginning and those perceived later. However, in almost all events with more impact (88.2%), the early sign was perceived during or at the end of the project. The perception in the beginning of the project, relatively, is more frequent in events with less impact, while the perception during or at the end of the project was probably the cause of greater impact. It is possible to infer that, the later we perceive the unk unks, the greater the chance of an impact on the project, since it was also not previously identified.

Discussion

The qualitative analysis identified that the external cause was the most frequent one, followed by the cause originating in PM and, finally, the organizational cause. Besides this, there was more than one cause for some unpredictable events. Changes with the organization external factors were more likely to be identified than the organizational internal causes. In addition, the changes due to external factors caused more impact on the project. Despite what various RM standards indicate (Committee of Sponsoring Organizations of the Treadway Commission [COSO], 2007; HM Treasury, 2004), external cause can only be mitigated and not prevented. In this sample, there were six cases of previous identification characterized with external cause. In one of these, the client ended up solving the issue that had suffered procrastination and, on the other, unk unks could still be identified in the project pilot phase. The early signs perception in these cases have shown the relevance of the perception and action to identifying unk unks, as an effective mechanism to mitigate or to manage the external causes impact in innovative projects.

In this study’s sample, most unpredictable events had no previous unk unks management; however, there were five events started by an information search and RM previously managed seven events in a planning phase, allowing their further identification. Furthermore, inadequate PM decisions may cause unpredictable events, such as in the case of emergency solutions. Inadequate management can also make it difficult to overcome knowledge gaps by improperly interpreting the knowledge level about an uncertain project area. Thus, it is possible to say that RM in projects was not enough to appropriately identify and address unk unks, as stated by Nikander and Eloranta (2001). In such cases, unpredictable factors arose, from external causes, such as unrealistic information given by suppliers and partners, or in case of internal causes, by erroneous assessment of event likelihood and its consequence level.

Although more than one early sign was identified in some unpredictable events, the vast majority of events occurred suddenly; that is, without perception of any early sign. It is believed that behavioral traits, such as heuristics and biases, have hampered this perception: overconfidence and unrealistic optimism about what was happening in the project could work as a barrier to perceive and understand the changes in the environment (Haji-Kazemi et al., 2015). One among many examples of overconfidence in the projects studied concerned a project manager working in a small company, who was aware that the partner, a big multinational trying to initiate its business in Brazil, had had problems abroad in adapting its processes to local contexts. However, he believed that his partnership would be different. Unfortunately, the adaptation problem occurred again. In innovative projects there are gaps in knowledge that cannot be found in the individuals in retrospect (Weick, 1995), which can also hamper this perception. This retrospect is important to recognize the early signs, salient cues in change of environment, and the relevance of starting to work.

Besides the fact that in the vast majority of event no early signs was perceived, in less than a third of the events, an early information sign was mentioned and, almost in the same proportion, the first symptom of a problem was detected. We identified many types of early signs, using the classification of Sánchez and Pérez (2004). The majority of them can be considered gut feelings (Williams et al., 2012) or intuition and improvisation (Leybourne & Sadler-Smith, 2006), communication between areas, expression of clients, users and partners, and external sources. Just as how intuition and improvisation are almost more important than assessment methods in complex projects (Williams et al., 2012), these inferences show that they are almost more important in innovative projects too. In one case, the project manager knew that there was a non-government organization [NGO] working against his product for unfounded reasons. The project was directed to specialists in this area who were providing academic information; however, some of them agreed with the NGO, but he did not know how many and who they were. He started to implant the project very slowly with a simple approach process. He went along with the team in some visits together with these specialists, deeply analyzing the situation when they found a specialist against the product. Learning how to approach these specialists enabled them to create a complete process to avoid problems.

It is important to note that many early signs were perceived through stakeholders, like members of the areas in the organization, clients, partners and users. The stakeholder management is the area in PM responsible for engaging the positive and negative stakeholders in the project. Thus, we recommend that stakeholder monitoring, a process of stakeholder management, could incentivize the discussion of early sign perception, not only by the project manager, but also by the team. All members must feel responsible for anticipating the perception of problems and changes in the environment which impact the project objectives, discussing the stakeholder risks (Van Os, Van Berkel, De Gilder, Van Dyck, & Groenewegen, 2015). Thus, the process of sensemaking can be transformed into a collective and future-oriented model, as suggested by Stigliani and Ravasi (2012).

Another way to involve stakeholders and make sense of early signs in the beginning of a project, decreasing the uncertainty, is by using Design Thinking (Gruber, de Leon, George, & Thompson, 2015). This method is comprised of three phases: discovery, a human-centered approach to innovation that permits observing and discovering stakeholder needs; definition, which involves framing the problem, clearing the restrictions, and developing insights in a multidisciplinary team; and delivery, the iterative process which permits learning through visualizations and prototype tests. Although it has been cited as “the best way to be creative and innovate”, management discourse is not yet academically anchored (Johansson-Sköldberg, Woodilla, & Çetinkaya, 2013, p. 121), because the IDEO approach, created by Brown (2008), is grounded in experience, rather than research.

These unpredictable events occurred in all phases of the project, being more prevalent in the development stages and even at the end of the project. Although Williams et al. (2012) found that the signs could occur at any phase of the project, in this sample early signs sensemaking at the end of the project was not observed. This represents a greater impact, which could jeopardize the whole project. Probably, there was no time to reduce ambiguity and misunderstanding to create a plausible solution, since it requires people’s knowledge and retrospective experiences (individual and organizational).

The project manager tended to perceive the early information more often and at the beginning of the project. In such cases, it can be said that the manager performed as a sensemaker, through interaction with stakeholders and the environment. Thus, it can be said that he/she perceived environmental change more effectively, probably because he/she was closer to all project stakeholders and his/her surroundings, and when this occurred early in the project, the solution effectiveness was higher. This may indicate a tendency for efficiency decrease over time. Under these circumstances, the efficiency in identifying unk unks could be related to the continuity property of sensemaking. If the early sign perception occurs in the beginning of the project, there will be more time to improve the understanding of the actions taken. Thus, the sooner the early sign is perceived, the longer the time to understand and overcome the knowledge gap, and therefore the team is more likely to get a good result.

The fact that project manager perceives the unk unks more effectively than others involved shows to this study’s authors the opportunity to improve the perception and the analytical understanding of the entire team. Many studies have concentrated on project manager perception (Haji-Kazemi et al., 2015; Nikander, 2002; Williams et al., 2012). Personal identity (Weick, 1995) is primordial for anyone to be engaged in the early signs perception and their sensemaking; probably, the team members do not realize that giving meaning to early signs is their responsibility, or they do not have proper analytical skills or proactive behavior, which can be trained. However, in order to confirm these premises, more research is required.

On the other hand, perception of early information led managers to seek information about this kind of change. This search is the enactment that the manager applies, in order to understand and determine the significance of change, acting and creating a new reality. Within this new reality, alternatives were created and choices were made promptly in order to mitigate the likely impact. Considering that meaning creation is a social process, a manager acting as sensemaker becomes vital for the whole team to be alert about the environmental changes that may affect the project.

In two unpredictable events that occurred suddenly, the respondent reported an early signal but considered it unimportant. The exceptions in the impact in relation to the absence of perception of the early sign demonstrate the inclusion of luck, as in the case in which there was an agreement with the client, cited earlier. In a reactive attitude, some project managers think that perceiving the early signs is a matter of luck: “You are either lucky or alert when you are in face of a situation that is passing in front of you, and you had never thought about it..., having in view that people are always doing many things and have no time to perceive.... Your mind is always full of things, so it is hard to capture something out of nothing, just thinking like: ‘Oh, I think this project can have a problem here’”. However, we must deal with unk unks in innovative projects to advance our success indicators.

Conclusion

This study aimed to understand the relevance of perception and make sense of early signs to identifying unk unks in innovative projects prior to impact. To do this, we used a sample of 16 innovative projects and 35 events representative of unk unks. Based on a questionnaire and in-depth interviews, analysis of the data was initially qualitative and non-parametric statistical tests were applied later.

As secondary goals, grounded in our sample, we sought to:

1. identify and understand the causes of unk unks, when detecting: external cause, which included customer, environment, marketing, partner and supplier; organizational cause related to the lack of structure and lack of integration in the areas; and cause originating in PM, like communication, failure in planning, team problems or even an emergency solution;

2. understand the motivating factors for identifying unk unks, recognized as types of previous management and early sign perception, when we identified three types of previously management search for information, with five cases, four with external cause; seven cases previously managed by planning or RM, and the other events had no previous management. The vast majority of events occurred without perception of any early sign, probably because biases have hampered this perception, such as overconfidence. The majority of perceived early sign can be considered first symptoms of a problem and others are really early information;

3. assess the efficiency in identifying unk unks: 14 events were identified prior to the occurrence of unk unks and 21 were identified subsequently; while 17 had more impact when it negatively affected customers, the organization and / or the project; and 18 had less impact when there was no negative impact or when it was possible to mitigate it;

4. assess the relationship between identifying unk unks and the causes and motivating factors: we recognize that events with organizational causes were more likely to be identified earlier with less impact, and events with external causes have further identification, unless search for information began in the beginning phase of the project. In all events with more impact, the early sign was perceived during or at the end of the project. When project managers perceived early signs, they caused less impact, when others perceived the early signs, it was later and they cause more impact.

As a result of the investigation, it can be said that the study contributed to highlight the relevance of early sign perception and sensemaking to innovative projects, mainly because some unk unks with external causes can be detected in time to mitigate their impacts. We recommend the use of learning processes in project management in order to acquire knowledge and retrospect about the innovation in the project. As soon as possible, these methods should be applied, because the team’s social interpretation needs time to reduce ambiguity and to make sense of early signs; they must act to understand what the meanings of changes are. We also recommended to maintain or intensify sensemaking activities throughout the project, mainly in stakeholder management. The accountability of team members in this process must be improved, while the project manager acts as sensemaker inside and outside the organization. In addition, the learning process will help to support this perception more than assessment methods.

The joint study of unk unks, early signs, sensemaking and PM is still little explored, but it allows for new studies and for further development of the issues raised here. To develop a prescriptive theory with broad generalization, other in-depth statistical studies should be conducted, focusing on real participation of team members and the importance of processes in stakeholder management in projects in order to decrease uncertainty. The use of creative methods, such as Design Thinking, can be analyzed in case studies and design science research to assess their performance towards the sensemaking process of unk unks. Another point that emerged in this research concerns the decrease in the team project’s efficiency to perceive and address unk unks. Statistical and even behavioral studies could highlight factors that affect these performance issues.

• Editor’s note. Marcos André Mendes Primo served as Action Editor for this article.

APPENDIX

Statistical Results of the Relationship Between Independent and Dependent Variables

Statistical Results of the Relationship Between Independent and Dependent Variables

Independent Variables	Variables	Dependent variables
		AIL			EIL
		p	Statistic test		p	Statistic test
Cause of unk unks	Occurrence of external cause	3.1%	Fisher		19.3%	χ2
	Occurrence of organizational cause	1.0%	Fisher		8.8%	Fisher
	Occurrence of cause originating in PM	70.4%	Fisher		47.1%	Fisher
Motivating factor of identifying unk unks	Type of early sign	0.0%	χ2		0.0%	χ2
	Who perceived the early sign	12.8%	Fisher		1.3%	χ2
	Moment of the early sign perception	7.3%	Fisher		1.5%	χ2

References

• Ansoff, H. I. (1975). Managing strategic surprise by response to weak signals. California Management Review, 18(2), 21-34. https://doi.org/10.2307/41164635
  » https://doi.org/10.2307/41164635
• Blanco, S., Caron-Fasan, M.-L., & Lesca, H. (2003). Developing capabilities to create collective intelligence within organizations. Journal of Competitive Intelligence and Management, 1(1), 80-92.
• Brown, T. (2008). Design thinking. Harvard Business Review, 86 (6), 1-11. http://doi.org/10.5437/08956308X5503003
  » http://doi.org/10.5437/08956308X5503003
• Christiansen, J. K., & Varnes, C. J. (2009). Formal rules in product development: Sensemaking of structured approaches. Journal of Product Innovation Management, 26 (5), 502-519. https://doi.org/10.1111/j.1540-5885.2009.00677.x
  » https://doi.org/10.1111/j.1540-5885.2009.00677.x
• Committee of Sponsoring Organizations of the Treadway Commission. (2007). COSO gerenciamento de riscos corporativos - Estrutura integrada: Sumário executivo Retrieved from https://www.coso.org/Documents/COSO-ERM-Executive-Summary-Portuguese.pdf
  » https://www.coso.org/Documents/COSO-ERM-Executive-Summary-Portuguese.pdf
• Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods approaches (4th ed.). Thousand Oaks, California: Sage.
• De Meyer, A., Loch, C. H., & Pich, M. T. (2002). Managing project uncertainty: From variation to chaos. MIT Sloan Management Review, 43 (2), 60-67. http://doi.org/10.1109/EMR.2002.1032403
  » http://doi.org/10.1109/EMR.2002.1032403
• Dervin, B. (2008, July). Interviewing as dialectical practice: Sense-making methodology as exemplar Paper presented at International Association for Media and Communication Research (IAMCR), (pp. 1-34), Stockholm, Sweden.
• Fellows, R., & Liu, A. (2016). Sensemaking in the cross-cultural contexts of projects. International Journal of Project Management, 34 (2), 246-257. http://doi.org/10.1016/j.ijproman.2015.03.010
  » http://doi.org/10.1016/j.ijproman.2015.03.010
• Gallouj, F., & Weinstein, O. (1997). Innovation in services. Research Policy, 26 (4/5), 537-556. https://doi.org/10.1016/S0048-7333(97)00030-9
  » https://doi.org/10.1016/S0048-7333(97)00030-9
• Gruber, M., de Leon, N., George, G., & Thompson, P. (2015). Managing by design. Academy of Management Journal, 58 (1), 1-7. http://doi.org/10.5465/amj.2015.4001
  » http://doi.org/10.5465/amj.2015.4001
• Haji-Kazemi, S., Andersen, B., & Klakegg, O. J. (2015). Barriers against effective responses to early warning signs in projects. International Journal of Project Management, 33 (5), 1068-1083. http://doi.org/10.1016/j.ijproman.2015.01.002
  » http://doi.org/10.1016/j.ijproman.2015.01.002
• Haji-Kazemi, S., Andersen, B., & Krane, H. P. (2013). Identification of early warning signs in front-end stage of projects, an aid to effective decision making. Procedia - Social and Behavioral Sciences, 74 (1877), 212-222. http://doi.org/http://dx.doi.org/10.1016/j.sbspro.2013.03.011
  » http://doi.org/http://dx.doi.org/10.1016/j.sbspro.2013.03.011
• Hamel, G. (2007). The future of management Boston: Harvard Business School Publishing.
• Hiltunen, E. (2008). The future sign and its three dimensions. Futures, 40 (3), 247-260. http://doi.org/10.1016/j.futures.2007.08.021
  » http://doi.org/10.1016/j.futures.2007.08.021
• HM Treasury. (2004). The orange book management of risk - Principles and concepts London: HMSO publications.
• Johansson-Sköldberg, U., Woodilla, J., & Çetinkaya, M. (2013). Design thinking: Past, present and possible futures. Creativity and Innovation Management, 22 (2), 121-147. https://doi.org/10.1111/caim.12023
  » https://doi.org/10.1111/caim.12023
• Kappelman, L. A., Mckeeman, R., & Zhang, L. (2006). Early warning signs of IT project failure: The dominant dozen. Information Systems Management, 23 (4), 31-36. https://doi.org/10.1201/1078.10580530/46352.23.4.20060901/95110.4
  » https://doi.org/10.1201/1078.10580530/46352.23.4.20060901/95110.4
• Kuosa, T. (2010). Futures signals sense-making framework (FSSF): A start-up tool to analyse and categorise weak signals, wild cards, drivers, trends and other types of information. Futures, 42 (1), 42-48. http://doi.org/10.1016/j.futures.2009.08.003
  » http://doi.org/10.1016/j.futures.2009.08.003
• Leech, N. L., & Onwuegbuzie, A. J. (2009). A typology of mixed methods research designs. Quality & Quantity, 43 (2), 265-275. http://doi.org/10.1007/s11135-007-9105-3
  » http://doi.org/10.1007/s11135-007-9105-3
• Lehtiranta, L. (2014). Risk perceptions and approaches in multi-organizations: A research review 2000-2012. International Journal of Project Management, 32 (4), 640-653. http://doi.org/10.1016/j.ijproman.2013.09.002
  » http://doi.org/10.1016/j.ijproman.2013.09.002
• Leybourne, S., & Sadler-Smith, E. (2006). The role of intuition and improvisation in project management. International Journal of Project Management, 24 (6), 483-492. http://doi.org/10.1016/j.ijproman.2006.03.007
  » http://doi.org/10.1016/j.ijproman.2006.03.007
• Loch, C. H., DeMeyer, A., & Pich, M. (2006). Managing the unknown: A new approach to managing high uncertainty and risk in projects Hoboken, New Jersey: John Wiley & Sons, Inc.
• Loch, C. H., Solt, M. E., & Bailey, E. M. (2008). Diagnosing unforeseeable uncertaintly in a new venture. The Journal of Product Innovation Management, 25 (1), 28-46. https://doi.org/10.1111/j.1540-5885.2007.00281.x
  » https://doi.org/10.1111/j.1540-5885.2007.00281.x
• Mehta, N., Hall, D., & Byrd, T. (2014). Information technology and knowledge in software development teams: The role of project uncertainty. Information and Management, 51 (4), 417-429. http://doi.org/10.1016/j.im.2014.02.007
  » http://doi.org/10.1016/j.im.2014.02.007
• Murray-Webster, R., & Pellegrinelli, S. (2010). Risk management reconceived: Reconciling economic rationality with behavioural tendencies. Journal of Project, Program & Portfolio Management, 1 (1), 1-16.
• Nikander, I. O. (2002). Early warnings (Doctoral dissertation). Helsinki University of Technology, Espoo, Finland.
• Nikander, I. O., & Eloranta, E. (1997). Preliminary signals and early warnings in industrial investment projects. International Journal of Product Management, 15 (6), 371-376. https://doi.org/10.1016/S0263-7863(96)00087-7
  » https://doi.org/10.1016/S0263-7863(96)00087-7
• Nikander, I. O., & Eloranta, E. (2001). Project management by early warnings. International Journal of Project Management, 19 (7), 385-399. https://doi.org/10.1016/S0263-7863(00)00021-1
  » https://doi.org/10.1016/S0263-7863(00)00021-1
• Office of Government Commerce. (2009). Managing successful projects with PRINCE2 (5th ed.). London: Stationery Office Books.
• Pender, S. (2001). Managing incomplete knowledge: Why risk management is not sufficient. International Journal of Project Management, 19 (2), 79-87. http://doi.org/10.1016/S0263-7863(99)00052-6
  » http://doi.org/10.1016/S0263-7863(99)00052-6
• Perminova, O., & Wikstro, K. (2008). Defining uncertainty in projects - a new perspective. International Journal of Project Management, 26 (1), 73-79. http://doi.org/10.1016/j.ijproman.2007.08.005
  » http://doi.org/10.1016/j.ijproman.2007.08.005
• Pich, M. T., Loch, C. H., & Meyer, A. De. (2002). On uncertainty, ambiguity, and complexity in project management. Management Science, 48, 1008-1023. http://doi.org/10.1287/mnsc.48.8.1008.163
  » http://doi.org/10.1287/mnsc.48.8.1008.163
• Project Management Institute. (2013). A guide to the project management body of knowledge (PMBOK® Guide) Newtown Square, Pennsylvania: Author.
• Purdy, G. (2009). Raising the standard - the new ISO Risk Management Standard Retrieved from http://aen.org.au/wp-content/uploads/member-documents/rto-elearning/Knowledge%20Bank/Risk%20Management/Raising%20the%20Standard%20-%20the%20New%20ISO%20Risk%20Management%20Standard.pdf
  » http://aen.org.au/wp-content/uploads/member-documents/rto-elearning/Knowledge%20Bank/Risk%20Management/Raising%20the%20Standard%20-%20the%20New%20ISO%20Risk%20Management%20Standard.pdf
• Ramasesh, R. V, & Browning, T. R. (2014). A conceptual framework for tackling knowable unknown unknowns in project management. Journal of Operations Management, 32 (4), 190-204. http://doi.org/10.1016/j.jom.2014.03.003
  » http://doi.org/10.1016/j.jom.2014.03.003
• Sánchez, A. M., & Pérez, M. P. (2004). Early warning signals for R&D projects: An empirical study. Project Management Journal, 35 (1), 11-23.
• Schoemaker, P. J. H., & Day, G. S. (2009). Why we miss the signs. MIT Sloan Management Review, 50 (2), 43-44. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=ofm&AN=510949322&site=ehost-live
  » http://search.ebscohost.com/login.aspx?direct=true&db=ofm&AN=510949322&site=ehost-live
• Siegel, S., & Castellan Jr., N. J., (2006). Estatística não-paramétrica para as ciências do comportamento (2ª ed.). Porto Alegre: Artmed.
• Simon, L. (2006). Managing creative projects: An empirical synthesis of activities. International Journal of Project Management, 24 (2), 116-126. http://doi.org/10.1016/j.ijproman.2005.09.002
  » http://doi.org/10.1016/j.ijproman.2005.09.002
• Snowden, D. J., & Boone, M. E. (2007). A leader’s framework for decision making. Harvard Business Review, 85 (11), 68-76. Retrieved from https://hbr.org/2007/11/a-leaders-framework-for-decision-making
  » https://hbr.org/2007/11/a-leaders-framework-for-decision-making
• Stigliani, I., & Ravasi, D. (2012). Organizing thoughts and connecting brains: Material practices and the transition from individual to group-level prospective sensemaking. Academy of Management Journal, 55 (5), 1232-1259. http://doi.org/10.5465/amj.2010.0890
  » http://doi.org/10.5465/amj.2010.0890
• Strauss, A., & Corbin, J. (2008). Pesquisa qualitativa: técnicas e procedimentos para o desenvolvimento de teoria fundamentada (2a ed.). Porto Alegre: Artmed.
• Thamhain, H. (2013). Managing risks in complex projects. Project Management Journal, 44 (2), 20-35. http://doi.org/10.1002/pmj.21325
  » http://doi.org/10.1002/pmj.21325
• Thiry, M. (2001). Sensemaking in value management practice. International Journal of Project Management, 19 (2), 71-77. http://doi.org/10.1016/S0263-7863(00)00023-5
  » http://doi.org/10.1016/S0263-7863(00)00023-5
• Van Os, A., Van Berkel, F., De Gilder, D., Van Dyck, C., & Groenewegen, P. (2015). Project risk as identity threat: Explaining the development and consequences of risk discourse in an infrastructure project. International Journal of Project Management, 33 (4), 877-888. http://doi.org/10.1016/j.ijproman.2014.10.016
  » http://doi.org/10.1016/j.ijproman.2014.10.016
• Weick, K. E. (1995). Sensemaking in organizations London: Sage.
• Weick, K. E. (2001). Making sense of the organization Singapore: Blackwell Publishing.
• Weick, K. E. (2006). Faith, evidence, and action: Better guesses in an unknowable world. Organization Studies, 27 (11), 1723-1736. http://doi.org/10.1177/0170840606068351
  » http://doi.org/10.1177/0170840606068351
• Weick, K. E., Sutcliffe, K. M., & Obstfeld, D. (2005). Organizing and the process of sensemaking. Organization Science, 16 (4), 409-421. https://doi.org/10.1287/orsc.1050.0133
  » https://doi.org/10.1287/orsc.1050.0133
• Wideman, R. M. (1992). Project and program risk management: A guide to managing project risks and opportunities Newton Square: PMI - Project Management Institute.
• Williams, T., Klakegg, O. J., Walker, D. H. T., Andersen, B., & Magnussen, O. M. (2012). Identifying and acting on early warning signs in complex projects. Project Management Journal, 43 (2), 37-53. http://doi.org/10.1002/pmj
  » http://doi.org/10.1002/pmj

Publication Dates

• Publication in this collection
  2017

History

• Received
  17 May 2017
• Reviewed
  05 Sept 2017
• Accepted
  14 Sept 2017