Promoting and Selecting Strategy-Aligned Projects through Consensus in Universities - An Artifact

Abstract

Project portfolio management has been studied through diverse aspects. Different approaches have been used to improve project portfolio selection, but not all of them have been actually used. This study presents an artifact to improve project portfolio selection process in universities. The artifact considers strategic alignment and focuses on achieving consensus, considering university characteristics about strategic planning and decision making environment. The artifact was developed through an incremental approach using quasi-experiments in simulated situations to check and improve the accuracy of the solutions being developed. Finally, the artifact was implemented in a real situation inside a university, involving 185 projects, 15 criteria, and 12 people from different faculties and knowledge areas. Results show that the artifact is suitable to the university environment. Beyond the artifact itself and some highlights involving its development process, the multicriteria constructivist approach, the balanced scorecard, and the scoring technique, another contribution is an adapted AHP tool created to integrate traditional AHP from Saaty (1980) and a study from Bose (2015) to reduce cognitive conflict in group decision making. Other studies, in more diverse contexts, still need to be done, but the proposal seems to be promised.

Keywords:
project portfolio; university; consensus; strategic alignment; project selection

Introduction

Fostering projects is one of the activities carried out by universities to encourage faculty members through some direction. Doing this requires a method to select projects according to the desired direction. This kind of activities is addressed by project portfolio management, which has been one of the most researched themes in project management (Kwak & Anbari, 2009; Padalkar & Gopinath, 2016). Project selection research problem is about finite and insufficient resources to execute all possible projects. Dutra, Ribeiro, and Carvalho (2014) presented several methods that have been developed and proposed to improve project selection activities. Unfortunately, even considering the wide diversity among existing methods, not all of them have been actually used by organizations (Ghasemzadeh & Archer, 2000).

Organizations differ from each other, but some of them have intrinsic common characteristics. In universities, some of these intrinsic common characteristics are related to collegiate structure (Burnesa, Wend, & By, 2014; Falqueto & Farias, 2013; Meyer, 2007), where consensus-based decisions have a strength role. Another similar characteristic among universities is their planning environment, usually marked by diffuse objectives (Scharmach, Domingues, Carvalho, & Machado, 2012). This scenario is a challenge for the selecting strategy-aligned projects, and it leads to some additional issues to the general based solutions that have been proposed. The research problem of this study is building an artifact to improve project portfolio selection in that context. Second section presents the research method and third section presents an overview of project portfolio management and its context in the university environment. Fourth section presents problem classes and their requirements in such context, analyzing some existing artifacts against these requirements. Fifth section describes the development process that led to the final artifact. Sixth section details artifact main components and presents its implementations and evaluation process. Final section presents the conclusions.

Research Method

The study presented in this article is about developing an artifact to improve activities related to promote and select strategy-aligned project in university environment. According to van Aken and Rome (2009), design science is an option to this kind of research. Dresch, Lacerda, and Antunes (2015) proposed a research method to carry this kind of study, called Design Science Research method (DSR). They highlight the contributions of their method as a support to develop relevant and rigorous scientific research that are also useful to organizations. The method has twelve steps. First five steps are about to identify the research problem and create awareness about it through systematic literature review, analyzing related artifacts to propose a new one. This is used as a starting point to the next three steps, in which the artifact is designed, developed, and evaluated. According to the method, these steps should be done in an interactive way, leading to a cyclic improvement process. Once the results achieved are good enough, last four steps are taken to publicize the results and achieve new knowledge.

Third and fourth sections of this article cover the first five steps of the method and fifth and sixth sections cover the other ones. The systematic literature review and the artifacts identification presented in third section were carried out in two steps. First one was based on 71 articles mentioned by Dutra et al. (2014) in their study covering the period from 2000 to 2012. Second one was based in searching tools available at the most relevant international journals cited by Dutra et al. (2014). The keywords to identify related work covering the period from 2013 to 2017 are: project, portfolio, BSC, AHP, ANP, ELECTRE, MAUT, PROMETHEE. Putting it all together, 147 articles were analyzed during this process. This literature review was used as a guideline to carry out some quasi-experiments, which were developed incrementally to deepen the problem of awareness. Six versions of the artifact were developed and tested in that process. Table 1 presents these six versions, their approaches and results up to the final approved version. The two versions tested during the awareness problem phase were used to check the feasibility of some specific approaches. Since one of these approaches seemed to be feasible, it was taken as a reference to proceed with the artifact development process. Versions created during development and evaluation phases, on the other hand, focused on artifact refinements. Version 0.2 was considered a baseline proposal, and versions 0.3 to 1.1 focused on incremental refinements. Versions 1.0 and 1.1 were used as a reference to implementation and evaluation phases, which were carried out in a real project selection process situation.

Table 1
Artifact development process

Phase	Focus	Version	Test type	Action	Result
Problem awareness phase	Check feasibility of key-approaches	0.0	Quasi-experiment (rector and vice-rector)	Check feasibility of MCDA-C approach	Not feasible
		0.1	Quasi-experiment (head of undergraduate pro-rectory and her team)	Check feasibility of an AHP tool adapted by the authors to define criteria and their weights	Partially feasible
Artifact development phase	Incremental improvements	0.2 baseline proposal	Quasi-experiment (head of undergraduate pro-rectory and her team)	Confirm feasibility of version 0.1 and test specific score technique approaches to project evaluation	Partially feasible
		0.3	Quasi-experiment (undergraduate pro-rectory team)	Test new approaches to score technique project evaluation over version 0.2 baseline	Partially feasible
Artifact implementation and evaluation phase	Artifact evaluation and final adjustments	1.0	Implementation in a real environment and situation (committee with members from several faculties)	Test new adjustments made to version 0.3 score technique approaches	Feasible
		1.1	New implementation + Survey and interviews committee with members from several faculties)	Do final adjustment and publicize	Approved

The artifact development process as a whole and its implementation took place at the same university. This university is a Brazilian higher education institution (IFES) and its characteristics are aligned to the university environment requirements pointed out in fourth section. IFES is an acronym used to qualify a public university maintained by federal government in Brazil. The university academic staff is about 2,000 professors and it attends about 25,000 students. IFES also offers 130 undergraduate and 106 postgraduate courses (30 PhD) in different knowledge areas. This diversity qualifies the university as a good place to analyze the artifact regarding the heterogeneity of the decision-making group. Being a public university is also a qualifier in the context of the artifact evaluation. That is because, among Brazilian universities, public ones are those in which collegial structure role is more relevant (D. P. Lacerda, 2009), which enhances the importance of group discussion and the looking for consensus. Planning aspect can also be tested in this environment, since the higher education evaluation system used by Brazilian government indicates that, among Brazilian universities, public ones are those that have greater difficulty with the planning aspects (Pinto, Mello, & Melo, 2016).

Two real project selection processes were conducted in this university to evaluate the artifact. During this process, 185 projects were analyzed through 15 criteria by 12 people from different faculties and knowledge areas. Artifact final evaluation used data collected from these implementations, such as audio recording and transcription, as well as the support material used by participants. These materials were analyzed and four complementary evaluation strategies were also used: (a) a survey with people involved in the selection process; (b) an analysis about selected projects pertinence; (c) an interview with two committee members; and (d) an interview with the head of the institution’s undergraduate degree, responsible for the portfolio.

Interviews were carried out to deepen two important aspects covered by Cooper, Edgett, and Kleinschmidt (1999): relevance and adequacy of the project portfolio components and satisfaction with the method itself. Both aspects were addressed in the interviews. The interviews were conducted with two target-audiences, in order to get two complementary visions: from the portfolio manager and from members of the evaluation committee.

Project Portfolios and the University Environment

Portfolio management dates back to the 1950s, with the seminal work of Markowitz (Gomes, 2016), which laid the foundations for what nowadays is known as the modern portfolio theory. Markowitz’s portfolio concept has been adapted to different areas, including project management. Project portfolio management is one of the topics covered by Project Management Institute (PMI). PMI is a global organization that provides education and research about project management subjects. There are three main themes addressed by PMI in this context: project management, program management, and portfolio management. According to PMI, portfolio management is about coordinating programs and projects to achieve organizational goals, having its activities concerned with selection, prioritization, and governance of projects and programs (PMI, 2013).

This close relation between portfolio management and strategy is also mentioned by authors of other project portfolio models, such as Padovani and Carvalho (2016) and Archer and Ghasemzadeh (1999). Importance of portfolio theme on project management has been highlighted by different studies. Padalkar and Gopinath (2016) investigated articles published on the main project management journals around the world between 2000 and 2015. They identified twenty research themes and portfolio management ranks among the top five. Kwak and Anbari (2009) did similar investigation but took it of specific project management journals and focused on those with high impact factor in management area. They classified eight research themes or categories. Portfolio management and strategic alignment have been the most researched one since the 1970s.

As argued by Baldridge (1983 as cited in Scharmach et al., 2012), higher education institutions attend an audience with diverse and complex needs. Complexity has been studied in different ways and some methods have been used to deal with it. Kubler, Robert, Derigent, Voisin, and Traon (2016) categorized these different methods into two groups. The first one considers multi-attribute decision-making (MADM) methods, which involve intense people participation and judgments. The second one are goal-oriented methods (MODM), which are in the opposite way about people participation.

MODM methods (such as genetic algorithms, and linear or goal programming) are used when there is an infinite or very large number of options to consider in choosing an alternative. MADM methods are also known as multicriteria decision support tools (MCDA), and are most commonly used when there are a limited number of alternatives to consider. In a literature review to develop a project portfolio model, Dutra et al. (2014) show that both approaches have been used in project portfolio selection methods.

Decision making in universities is based on the consensus among peers, as argued by Millet (1962 as cited in D. P. Lacerda, 2009). This culture is materialized in a collegial structure, which has been studied through different countries and regions, such as England (Burnesa et al., 2014), New Zealand (Meyer, 2007), and South America (Falqueto & Farias, 2013). In such environment, a method to project portfolio selection is supposed to have a strong approach to achieve consensus through the different decision making steps involved. Although MODM project selection methods can be used for this, MCDA intensive people involvement approach tends to be more appropriate. Based on a similar categorization used by Dutra et al. (2014), the same can be said about qualitative and hybrid methods against quantitative methods.

Among the qualitative and hybrid methods presented in Dutra et al. (2014), the most used hybrid methods have been analytic network protocol (ANP) and analytic hierarchy protocol (AHP). Among the qualitative ones, the most used are fuzzy logic, score technique, balanced scorecard (BSC), Delphi, bubble, and multi-attribute utility (MAUT). AHP and ANP are both MCDA methods and, according to the study of Dutra et al. (2014), there are also some other MCDA methods been used, such as ELECTRE, TOPSIS, and PROMETHEE.

Problem Classes, Requirements, and Artifact Identification

According to Dresch et al. (2015), in Design Science Research it is important to delimit the field of an artifact being created. This is important to make clear the context in which the artifact will fit. This section presents the problem classes related to the scope of this work, as well as its requirements for university environment. The section also presents an analysis about some artifacts found in the literature from the point of view of these requirements. There is no established taxonomy that can be used in a uniform way to identify or categorize different approaches used in project portfolio management. Thus, it is necessary to define some concepts to delimit the field related to the artifact presented in this article. At least four concepts are important in the context of this research: model, methodology, method, and technique.

For the purpose of this research, a portfolio management model is considered to be a generalist approach that describes macroprocesses to be followed to carry out portfolio management as a whole. A portfolio management methodology is defined as the coordinated application of a set of methods and techniques capable of covering all the processes of a model. A method is similar to a methodology, but applicable to only some of the processes that make up the model. A technique is a procedure or a tool to be used in a process with the purpose of producing a certain output or part of it. Considering these concepts, the artifact presented in this study is a method. Among the existing methods in portfolio management, there are at least three problem classes, according to Table 2. Based on these problem classes, the artifact presented in this article refers to items 1 - project classification and 3 - final portfolio definition.

Table 2
Problem classes involving project portfolio selection methods

Problem subclass	Characteristics
1. Methods for classifying projects	To classify a set of projects according to their importance or alignment to the organizational strategy
2. Methods for portfolio optimization	To analyze available resources and, based on a pre-existing classification, suggest portfolios that optimize the use of resources
3. Methods for defining the final portfolio	Evaluate different portfolio options and select the one with the best balance according to organizational strategy or portfolio goal

Requirements of the artifact to the university environment

In the context of project portfolio management, resources are considered to be finite and insufficient to run all projects. Thus, there is an intrinsic decision-making process related to select and define the projects that will be part of the portfolio. On the other hand, project portfolio management is also based on strategic-alignment. This way, an organization’s strategic planning environment and its decision-making environment may influence the methods used to select its project portfolio. This section presents three requirements that must be taken into account by a project selection artifact when applied to the context of a university.

First requirement is group decision making and looking for consensus (req1). Decision making in university environment has a collegiate structure. The university collegiality has been analyzed in different countries, such as England (Burnesa et al., 2014) and New Zealand (Meyer, 2007), as well as Brazil and Argentina, which inherited their model from French universities (Falqueto & Farias, 2013). The study of Meyer (2007) has as background the case of a university in which the author herself participated in the (attempted) implementation of a policy to reduce the role of collegiate structures. Meyer analyzes the case itself and cites positive and negative aspects pointed out by studies by other authors, such as Welsh and Metcalf (2003), Scott (2002), Harloe and Perry (2004), and Doring (2002) to conclude that universities should look for decision structures that reconcile the benefits of a more managerial approach with the benefits of the representative collegiate structure. In line with this, D. P. Lacerda (2009) cites Meyer (2004) to argue that the university is an organization of academic professionals, where decision making is based on consensus among peers. So, the university collegiate structure is not only a part of the decision-making environment, but also of the looking for consensus. This way, the process of selecting a project portfolio in a university should, whenever possible, be based on group decision making and prioritizing to achieve a consensus decision.

The second requirement is the academic participation, equity, and heterogeneity of the decision-making group (req2). Group decision making and the looking for consensus by themselves do not define the characteristics of the decision-making environment in the university context. Due to the representative way in which collegiate structures are composed (Meyer, 2007), the group involved in decision making can be heterogeneous, adding complexity to the process. Beyond that, Meyer (2007) also cites Doring (2002) to argue that avoiding academic staff from decision-making process may lead to a loss of teaching quality because it can lead professors to a more conformist thinking. On the other hand, Fachin (2013) points out that Baldridge (1983) had already recognized the existence of a collegial and consensus model in universities, but preferred to use a concept called political model, in which power, conflict, and ideology play a prominent role. Hardy and Fachin (2000) argue that the concept of professional bureaucracy is present in universities. According to them, this is because authority does not come exclusively from hierarchical position, but also from knowledge and specialization, so that power is also decentralized, not being only at the top. Therefore, the project portfolio selection process in a university should keep in mind the political forces inside the institution, as well as take into account the participation of faculty members. This scenario can lead to a heterogeneous group involved in decision-making process.

The third requirement is the planning environment with diffuse goals and objectives (req 3). Strategic alignment is at the heart of project portfolio selection process (Archer & Ghasemzadeh, 1999; Padovani & Carvalho, 2016; PMI, 2013). A problem faced by universities in this subject is pointed out by Scharmach, Domingues, Carvalho and Machado (2012), who cited Baldridge (1983) to remind that universities are characterized by vague and ambiguous goals. On the other hand, Welsha, Nunez, and Petrosko (2006) studied some north-American universities and concluded that level of involvement of academic community in decision-making processes is one of the factors that may influence organizational support to the planning process. In Brazil, an analysis by Pinto, Mello, and Melo (2016) over ten years results (2004-2014) of 66 Brazilian universities in the country’s higher education evaluation process indicated that public universities have worst results in institutional planning. It should be remembered that in Brazil, public universities are those where collegiate structures are stronger (D. P. Lacerda, 2009). Also, according to Scott (2002 as cited in Meyer, 2007), this collegiate structure sometimes finds it difficult to reach a consensus. This context points to a scenario with extra difficulties in aligning project portfolio selection process with strategy in a university - either because of the difficulty in reaching a consensus or because of the ambiguity of organizational objectives.

Artifact identifying and analysis

This section presents an analysis of the strengths and weaknesses of different artifacts focused on project selection found in the literature and in consulting company websites. This analysis was based on the requirements presented in fourth section. Regarding Requirement 1, related to group decision making and looking for consensus, when looking among the artifacts surveyed, it is possible to identify some strategies that have being used. However, only few of them have actually been tested and documented. It cannot be said that there is a solution ready to be used in the university context. An example is the use of a group of experts to define or scoring criteria, which is cited by Asosheh, Nalchigar, and Jamporazmey (2010) and Almeida and Almeida (2013). Although mentioning a group of experts, neither case is clear about whether and how a consensus is reached among these experts.

The use of the Delphi technique, as in Khorramshahgol, Azni, and Gousty (1998), could be an alternative to seeking consensus. However, it can become difficult to implement it if the number of criteria or projects involved is too large. Computer simulation based on rating history, as in Bai, Li, Feng, and Guo (2010), is not exactly a group decision, although it takes into account opinions of different people to assemble the evaluation model. Nguyen, Nguyen, Le-Hoai, and Dang (2015) used semi-structured interviews with a group of people based on a pre-defined list of 50 topics, which were reduced to 36. They did not explain in their study how they reached consensus, but this approach can be an interesting alternative. However, it would be necessary to develop an approach to identify the starting topics, which can be difficult to do in the complex and diverse environment in which universities are involved. Finally, AHP and ANP techniques were also used, as in Kumar, Antony, and Cho (2009), Hsu, Tzeng, and Shyu (2003) and García-Melon, Poveda-Bautista, and Valle (2015). ANP is more robust and adapts to a larger variety of situations; however, its use is more complex when compared to AHP. Although both techniques have been used, none of these proposals cite the use of a consensus meeting among the participants to approve the final result.

Regarding requirement 2, about academic participation, equity and heterogeneity of decision-making group, two approaches were found. Although none of them explicitly address the issue of equity and academic participation, both presented solutions that help to solve the problem of heterogeneity and different point of views in decision-making group. Both alternatives comprise the use of fuzzy logic as a tool to minimize discrepancy among diversity in evaluator’s points of view. Hsu et al. (2003) used fuzzy AHP, a variation of traditional AHP that uses fuzzy logic. Bai et al. (2010) used fuzzy logic itself to assign variations of values to the natural language used to rate projects according to criteria. In that proposal, each evaluator has a personalized evaluation scale which makes them comparable to each other.

Requirement 3, related to diffuse goals in planning environment, has been less treated. Use of BSC is quite frequent, eventually integrating AHP or ANP. Among the artifacts that use BSC, the approach used by García-Melón et al. (2015) can be highlighted. It uses a group of people to establish which objectives are related to each other. This approach could be used to enable the use of AHP or ANP in cases where there are many objectives and where they are not arranged in BSC perspectives structure. Commercial strategy-focused solutions, such as Ernst & Young (http://www.ey.com/en_gl/transactions/growth-strategy, retrieved December 10, 2018), could be considered due to their focus on define a strategy. However, there is no information available on how this is done. Beyond that, defining a consensus strategy is one of the major challenges in university planning environment, as mentioned by Scharmach et al. (2012) citing Baldridge (1983). Using Delphi technique to define strategy is also an option, as in Khorramshahgol et al. (1998), but this can be hampered by the diversity and complexity of university environment. Finally, grouping a list of topics into factors, as in Nguyen et al. (2015), would be an option to simplify and align objectives that are scattered, although this was not their original proposal.

Artifact Development History

Artifact final version has seven stages and 22 steps, which were detailed through a continuous process of testing and refinement, considering different approaches involving the requirements presented in fourth section and literature review presented in third section. Table 3 presents these steps and, where applicable, the requirement they are related to. It also shows the evolution of the artifact and the different approaches used during the development process.

Table 3
Artifact stages, its evolution and relation to the environment requirements

Step and related requirement		Artifact evolution through development history
		Evaluation	Development		Awareness problem
		final (1.1)	evaluation (1.0)	baseline (0.3)	bsc/ahp (0.1)	mcda-c (0.0)
Stage 1 - Preparation
1.1. Team definition	-	-	-	-	-	-
1.2. Resource and budget
Stage 2 - Initial definitions
2.1. Transform BSC goals into criteria
2.2. Test criteria understanding/clarity	R3	BSC adapted criteria	BSC adapted criteria	BSC adapted criteria	BSC	MCDA-C
2.3. Approve criteria and evaluation method
Stage 3 – Engage evaluation committee
3.1. Nominate evaluation committee
3.2. Present method and criteria to evaluation committee	R2	-	-	-	-	-
Stage 4 - Define criteria and their weights
4.1. Criteria pairwise comparison (individually and taking notes)	R2	AHP group and individual + taking notes + midpoint	AHP group and individual + taking notes + midpoint	AHP group and individual + taking notes + midpoint	AHP group and individual + taking notes	-
4.2. Calculate midpoint	R1
4.3. Group meeting to consensus
Stage 5 - Identify projects
5.1. Review and approve criteria	R2	-	this step didn’t exist yet		-	-
5.2. Disclosure of selection process
5.3. Project elaboration by academics	-	-	-	-	-	-
5.4. Receiving projects
Stage 6 - Evaluate and prioritize projects
6.1. Assign projects to committee members	R2	Individual scoring + peer review + normalized mean + two-phase	Individual scoring + peer review + normalized mean	Individual scoring and group consensus	-	-
6.2. Individual project evaluation				steps 6.3 and 6.4 were used to achieve consensus; steps 6.5 and 6.6 didn’t exist yet	-	-
6.3. Calculate normalized mean and its inconsistencies
6.4. Check inconsistencies with evaluation committee (consensus)	R1				-	-
6.5. Reevaluate inconsistent projects listed by the Committee
6.6. Calculate final classification order
Stage 7 - Approve portfolio
7.1. Portfolio formalization meeting	R1	-	this step didn’t exist yet		-	-
7.2. Disclosure and resource allocation	-	-	-	-	-	-

Stages 2, 4, and 6 of the artifact are the trickiest, and the quasi-experiments that led the artifact development process were performed to test them. Steps that appear in Table 3 with a gray background are those which were modified or created during the refinement process occurred since the baseline proposal (version 0.3). Step 5.1, for instance, was added after first implementation of the artifact in the university (version 1.1), and is related to requirement 2, which is about academic participation, equity, and heterogeneity of the decision-making group. Adding this step was an example of the collegial power in university structure, since the initial criteria had been proposed by the head of undergraduate pro-rectory. Even so, the committee suggested removing some criteria and that was what really happened at the end.

There are other steps also related to the university environment requirements. All the steps in stage 2, for instance, are related to requirement 3, about a planning environment with diffuse objectives. Steps 4.2 and 4.3 are related do requirement 1, about group decision making and looking for consensus. There was also an adjustment during implementation phase, regarding the use of a two-phase project evaluation in which first activity uses an eliminatory conformity criterion. Actually, during the process, MCDA-C, BSC, AHP, or scoring technique were progressively tested, adapted and/or discarded. More about this process is presented in fifth section.

Evolution during problem awareness phase (versions 0.0 and 0.1)

At problem awareness phase, two different strategies were tested. First one (in version 0.0) was based on multicriteria constructivist approach (MCDA-C), which had been chosen for its potential to disseminate knowledge and support decision making (R. T. Lacerda, Ensslin, & Ensslin, 2010). Expectation was that these characteristics could help to achieve consensus about the criteria. To test the feasibility of this approach, a quasi-experiment with a focus group was conducted in the university using existing real projects as a basis for identifying topics and criteria. Two pre-tests were carried out involving two different focus groups with four members each, followed by a third pre-test with the rector and vice-rector. Conclusion was that the MCDA-C approach would be unfeasible due to the complexity and diversity of existing themes, which would require involvement of several people and a working time that did not fit the proposal under study. Thus, the MCDA-C approach of version 0.0 was discarded and a search for alternative solutions was carried out.

The alternative picked up to the next quasi-experiment was using AHP (Saaty, 1980) and BSC (Kaplan & Norton, 1992). AHP is one of several existing multicriteria analysis techniques, which are also known as MCDA. Although the constructivist approach had been considered unfeasible, considering its characteristics about involving people in the decision process (Kubler, Robert, Derigent, Voisin, & Traon, 2016), MCDA as a whole is an option to the university environment. Using BSC was an option to replace MCDA-C. BSC and AHP have already been used in this context, as demonstrated by Eilat, Golany, and Shtub (2008), Asosheh et al. (2010), Bai et al. (2010), Chan and Ip (2010), Hsu et al. (2003), Khorramshahgol et al. (1998), Kumar et al. (2009) and Nguyen et al. (2015). There are also studies mixing BSC and ANP (another multicriteria method), as in García-Melón et al. (2015), Chang (2015) and Yüksel and Dagdeviren (2010).

However, none of these studies were used in university environment and only a few of them were actually used in practice. This lack of practical evidence had been observed in other project selection methods (Ghasemzadeh & Archer, 2000) and, more recently, also in a risk management (Taroun, 2014). Thus, an improvement over traditional AHP implementations was developed to check if integrating BSC and AHP would be feasible in the university environment. The difference of this adapted AHP version from the traditional ones was a feature that allows members to report comments on each comparison. The use of comments came from Bose (2015), who used a similar strategy to reduce cognitive conflict in group decision making.

A new quasi-experiment involving this adapted AHP tool was carried out in the university, which had a BSC with seven strategic themes, each one with a three perspectives strategic map. Goals existing in one of these perspectives were used during the tests, which involved six participants who were divided into two groups. One group used an approach of direct weight assignment for each goal. Other one used the adapted AHP version to do the same job. Both groups made individual evaluations, meeting themselves after that to try to reach consensus. In the end, both groups were gathered in the same room to give their opinion. There was consensus about using AHP. The group also indicated that using original BSC goals had led to difficulties when comparing the criteria. In addition, it was quite difficult to reach consensus about comparison items with high divergence among the group. It was suggested to calculate and show a midpoint of individual responses to facilitate reaching consensus

Evolution during development and evaluation phases (versions 0.2, 0.3, 1.0, 1.1)

Development phase began at version 0.2, which was considered the artifact baseline. Feasibility of integrating BSC and AHP to reach consensus about criteria had already been tested. So, beyond a confirmatory test over this approach, next activities focused on how to achieve consensus in project evaluation process itself. Alternative chosen in this case was scoring technique. This technique is quite simple and has been used in different ways (Almeida & Almeida, 2013; Asosheh, Nalchigar, & Jamporazmey, 2010; Kumar, Antony, & Cho, 2009). Although the scoring technique is most commonly used with natural language scales, as in Almeida and Almeida (2013) or in the PMI guide (PMI, 2013), there is no consensus on which scale is most appropriate. Almeida and Almeida (2013) use a six-point scale with terms in natural language representing the values 0, 10, 40, 50, 60, 100, and PMI (2013) uses different terms and a three-point scale with values 0, 5, and 10. So, version 0.2 of the artifact was mainly developed to evaluate different scale possibilities, including the use of a rate from 0 to 10 instead of the natural language. Expectation was that, beyond being well accepted by professors because of its use in Brazilian student grades, a numeric scale could facilitate the understanding of an eventual midpoint calculation.

Version 0.2 was called the baseline proposal because it was the one in which the seven stages of the artifact were first stated. BSC, AHP, and scoring technique were used respectively in stages 2, 4, and 6 of the artifact. Thus, the baseline test focused on these stages. Stage 2, related to review and transform BSC objectives into criteria, was carried out by one of the authors and validated by the head of the undergraduate pro-rectory. As a result, besides being rewritten, some objectives were removed and some were splatted on. Resulting criteria from stage 2 were used in quasi-experiment of stage 4. Stage 4 uses an AHP adapted tool to reach consensus about criteria and their weights. Another adjustment made to AHP tool was showing the midpoint of individual analysis. Same procedures used in previous tests were used in this new quasi-experiment. Conclusion was that the adjustments made to the artifact would be enough to conduct a final consensus meeting involving the evaluators.

Stage 2 and stage 4 tests were confirmatory. Stage 6 test, at the other hand, involved evaluating different scoring technique approaches, in order to verify which one had more potential. To carry out the test, four projects that already had been executed by the pro-rectory were selected to simulate an evaluation process. While choosing the projects that would take part in the quasi-experiment, the head of pro-rectory raised the possibility of adding a criterion related to the comprehensiveness of the projects. This criterion would be used in order to differentiate projects with a more restricted scope from those with an institutional scope. This situation was incorporated into the quasi-experiment. Four participants were selected and then divided into two groups. They used different evaluation combinations involving natural language scales and scales from 0 to 10 for both BSC-driven criteria and comprehensiveness criteria. Initial tests indicated that none of the alternatives would be enough to support a final consensus meeting. Nevertheless, it was pointed out that maybe comprehensiveness criteria could be measured in a quantified way. So, a new version was built.

Version 0.3 was the first to involve improvements on the baseline. Steps 2 and 4 from version 0.2, involving BSC and the AHP adapted tool, were considered satisfactory. Thus, version 0.3 focused on improvements to stage 6, related to project evaluation. This new version used a quantitative approach to deal with comprehensiveness criteria, using measures such as number of students and teachers affected by the project, number of laboratories used, among others. For BSC-driven criteria, it was still necessary to find a solution to reach consensus over the score assigned by each evaluator to each project, so a new quasi-experiment was carried out using grades with a 0 to 10 scale. The use of this scale is due to the traditional Brazilian student grades (from 0 to 10) and because it is easier to understand a midpoint based on a numeric scale.

Results pointed that, even with measurable and quantifiable criteria for the comprehensiveness aspect, there were still issues on interpretation, which led to different results depending on the evaluator’s point of view. Although some divergences had been overcome and consensus had been reached sometimes, it took a relatively long time to be done and this approach seemed to be unfeasible in real world. Beyond that, another issue involving a discrepancy over evaluators’ grades emerged. The mean of evaluators’ grades ranged from 2.275 to 7.200. That could be a problem if different projects where evaluated by different people. Use of comprehensiveness criteria was also discarded: first, due to the difficulty of evaluate them in an objective measurable way; second, due to the focus, as they would be used in real-world implementation where the artifact would be evaluated. Since achieving consensus on individual projects seemed to be unfeasible, a contour solution had to be built. The alternative found was to minimize the damage effect caused by discrepancy in grades, as well as to change the final consensus meeting focuses. This was implemented on version 1.0 of the artifact.

Contour solution in version 1.0 was to normalize evaluators’ grades after finishing an individual project evaluation process. Normalizing was done considering the highest and the lowest marks of each evaluator as their 0 and 10 mark limits. Thus, at the end of the process, each evaluator ends up having his or her personalized value scale, which may be comparable to the others. This is a kind of personal preset scale reached by fuzzy logic strategy used in Bai et al. (2010). It is less precise, but it is simpler and can be done easily. Normalization is done just after individual evaluation procedures. As soon as the results are available, a meeting is held with evaluators to show the results. At this meeting, instead of looking for consensus on each project and criterion, projects with discrepancy between two evaluators are highlighted and presented to the group to decide what must be done.

Artifact, Its Implementation and Evaluation

This section presents more information about the artifact’s main components, as well as the results achieved with its implementation in a real environment. First subsection presents stages and steps of the artifact’s final version. Second subsection presents more information about stages 4 and 6 of the artifact. Third subsection presents implementation process and some relevant results achieved.

Artifact stages and steps

This section presents the artifact’s final version, describing, in Table 4, main activities and outputs related to each stage and step. First column of Table 4 presents the stages and steps of the artifact. For each stage is presented a brief description and the main outputs expected at the end of the stage.

Table 4
Description and outputs of the artifact stages and steps

Artifact stages and steps	Description	Outputs
Stage 1 – Preparation	Define team members who will carry out the project portfolio selection activities, as well as the budget and resources available	• Implementation team
1.1. Team definition		• Portfolio budget and resources
1.2. Resource and budget definition
Stage 2 – Initial definitions	Evaluate BSC goals, pick up selected ones and rewrite them to reduce ambiguity. Present and approve criteria and evaluation method with the head of portfolio	• Selected BSC goals
2.1. Transform BSC goals into criteria		• Initial criteria list
2.2. Test criteria understanding/clarity		• Method approval
2.3. Approve criteria and evaluation method
Stage 3 – Engage evaluation committee	Define the rules to be used to nominate committee members and do nominate them. After that, explain the process to them to get their engagement	• Committee member list
3.1. Nominate evaluation committee		• Committee member meeting
3.2. Present method and criteria to evaluation committee
Stage 4 – Define criteria and their weights	(Tool: adapted AHP)	• Consensus criteria weights according to AHP tool
4.1. Criteria pairwise comparison	Set up the AHP tool and conduct pairwise comparison meetings until consensus is reached among committee members
4.2. Calculate midpoint
4.3. Group meeting to consensus
Stage 5 – Identify projects	Review criteria and their weights, making adjustments if necessary. After that, disclosure the project selection process, informing the criteria, their weights and applicable project templates	• Final criteria list
5.1. Review and approve criteria		• Final criteria weights
5.2. Disclosure of selection process		• Project selection rules and templates
5.3. Project elaboration by academics		• Proposed projects
5.4. Receiving projects
Stage 6 – Evaluate and prioritize projects	(Tool: adapted scoring technique)	• Maximum and minimum committee member grades
6.1. Assign projects to committee members	Assign projects to committee members in a pairwise manner, setting up the tool. After member evaluations, conduct a group meeting with the members showing results and occasional discrepancies. Reevaluate discrepant projects until consensus is reached on their final grades	• Normalized grades
6.2. Individual project evaluation		• Project final grades
6.3. Calculate normalized mean and its inconsistencies
6.4. Check inconsistencies with committee
6.5. Reevaluate inconsistent projects listed by the committee
6.6. Calculate final classification order
Stage 7 – Approve portfolio	Conduct a meeting to define final project list and its budget being allocated to each project. After that, disclosure final result	• Project final list
7.1. Portfolio formalization meeting		• Budget allocation to each project
7.2. Disclosure and resource allocation

Information presented in Table 4 summarizes the artifact implementation process and can be used as a guide to implement it in a university. Second subsection presents more information about stage 4 and stage 6, which are the most complex ones.

Artifact’s main components (stage 4 and stage 6)

As mentioned earlier, stages 2, 4, and 6 are the trickiest and are those which passed through quasi-experiments during artifact’s development process. Stage 2, although important, is quite simple and was spelled out in the artifact just to reinforce that, due to planning ambiguity issue, it is quite possible that BSC goals cannot be used just as written in strategic plan. There is no relevant specific tool for that, so stage 2 is not detailed in this section. On the other hand, in stage 4, in which weight criteria are decided, there are some complexity involved and a specific tool was developed to improve traditional AHP applications. Traditional AHP tools use pairwise comparison based on a scale from 9 to 1/9, having a midpoint to indicate neutrality, and can use four or eight points from neutral comparison to highest comparison parameter. The tool developed for the artifact uses four points scale to individual comparison and eight points scale to group comparison. The main extra feature added in comparison with traditional AHP applications is a text field that must be used to write comments about each comparison item. This comment area was inspired on Bose (2015). His study improved a group decision process by adding a tool in which participants could send their comments to each other explaining the reason for their particular decisions, in order to reduce cognitive conflict.

Figure 1 shows part of the tool used to support group decision making during stage 4 of the artifact. The tool is filled with aleatory information from three hypothetical evaluators. Note, at left side, that there is an X marking the midpoint within the eight points scale. This X is marked by the tool according to individual comparisons made by each evaluator. Each pairwise item also has a row where a number is automatically set up to indicate how many evaluators marked a given position in the scale. There is also a text field close to each pairwise item. The other fields at the right side can be used as support to specific situations. The tool must be used by a moderator, who should conduct the group meeting and read the comments about each pairwise comparison item before starting group discussion, in which participants are stimulated to talk about their opinions and, if necessary, move the X mark. First pairwise comparison item in Figure 1 is an example of discrepancy among evaluators’ opinion. Third pairwise comparison item, in contrast, has low or none opinion discrepancy.

Figure 1
Artifact - Stage 4 - Component used as group decision support tool in pairwise comparison

Stage 6 approach is quite different from stage 4. It doesn’t focus on reaching consensus about each single situation. So, the tool developed to support stage 6 focus on highlight situations in which a project presents discrepancy considering grades from the two evaluators. Each evaluator assigns a grade from 0 to 10 to each criteria and each project. Theses grades are normalized by the tool, based on evaluator minimum and maximum grade. This is done to minimize the effects of one very strict evaluator compared to another very benevolent. Figure 2 shows a sample using the group decision part of stage 6 tool. The field at the top center in Figure 2 (with value 6) is used to set up the parameter considered relevant by the participants to mark a project evaluation as a discrepant one. At the top right it is presented how many projects are considered discrepant according to this parameter.

Figure 2
Artifact - Stage 6 - Tool used to identify discrepancy between two evaluator grades

Project list is showed ordered from first to last. There is a column with an OK or DIF mark. Projects marked with DIF are those in which evaluator’s grades are considered discrepant. In the Figure 2 example, the project with the fifth highest grade is marked as DIF because the difference between normalized grade awarded by first and second evaluators (EV1 and EV2) is greater than 6, which is the value assigned to discrepancy parameter (normalized grades from EV1 and EV2 are, respectively, 9.48 and 2.40).

Artifact implementation and evaluation

The artifact was used by undergraduate pro-rectory of the university to foster teaching and classroom projects to be carried out in a decentralized manner among the 14 teaching units of the institution. At the time this article is written, the artifact had been used by the institution in two editions of the same selection process, with an interval of one year between the first and second editions. First edition got 97 projects submitted for evaluation (about 5% of academic staff submitted a project). In the second edition, it was 88 projects. In both editions, projects were analyzed by a committee which members were appointed in a representative way by an invitation sent to each one of 14 academic units, asking them to nominate one representative. Not all of them made the nomination, and the first edition had representatives from six teaching units. Second edition had seven units represented. In the second edition, there were changes in the committee members. Even so, they decided to maintain criteria and weights used in previous edition. That way, steps 1 to 4 of the artifact were not implemented in the second edition.

Meetings with the committee to analyze the criteria and projects were done as predicted by artifact stages and steps presented in Table 3. Six committee members participated in stage 4 and seven participated in stage 6. There were two meetings in stage 4 and another two meetings in stage 6. Different points of view became evident in both stages and the artifact supported both situations properly due to consensus. Beyond the different point of view situations, the influence of the committee over the process as a whole also emerged in the end of stage 4.

In stage 4, committee members were divided into two groups with three members each. Each group got two sets of criteria to evaluate. A BSC perspective with only three criteria was assigned to both groups. Other two perspectives had six criteria each, and each one was assigned to a different group. Group members individually evaluated the criteria they were assigned to. After finishing their work, all group members were put together for group discussion. Figure 3 shows part of the results got from group decision. First pairwise comparison item had members with different points of view. One member marked position -3, indicating that criteria at right were most important, while two other members marked opposite positions. The midpoint of individual evaluations was initially marked on position 4, highlighted with a circle in Figure 3. After group discussion, they decided to move the X mark to position 5. This group discussion process proceeded among each pairwise comparison item.

Figure 3
Part of group decision results as showed in the pairwise comparison tool from stage 4

Time spent to reach consensus among all comparison items in each six criteria perspectives was about eighty minutes. These perspectives had 15 comparison items each (30 comparison items at all). The group decision process led to change the original midpoint in 16 comparison items. In the perspective containing three criteria, group members decided to assign weights directly to the criteria, instead of using pairwise comparison. Figure 4 shows resulting weights before and after group decision process, highlighting with a circle those criteria with higher before/after weight difference. Result obtained by automatically generated midpoint got a consistency ratio (CR) of 0.060. In final group decision, this ratio was 0.138. Saaty (1980) recommends a CR up to 0.10, but it is pointed out that some small variations can be tolerated.

Figure 4
Sample of results got before and after group decision during artifact’s implementation phase

Beyond the changes that occurred from reaching consensus over pairwise comparison items, committee member also decided to eliminate some criteria. They did so by considering that these criteria had reached a very low weight and so it would be better to put them apart from the process to avoid any noise among academic staff. So, two criteria from Figure 4 were removed and both weights and CR were recalculated. New calculated CR was 0.090. Considering the data collected during group decision meetings, it can be said that the artifact played an important role in the process as a whole. The comments feature added to traditional AHP approach were useful for reaching consensus among the group. This can be attested by a committee member statement just in the beginning of the process: “can we see the comments, please?” Another evidence was that consensus was reached even in cases with very different points of view. This can be observed in the first comparison item at Figure 3 and in these opposite statements made by three committee members during group discussion process:

(member 1): “Guys, probably I’m the first evaluator there. I don’t use labs.”

(member 2): “I think lab is not just to put a lab coat … it’s the hands on … that’s why I put high grades … Maybe you don’t have labs, but I have one.”

(member 1): “in that case, musical instrument lessons are 100% practical ….”

member 3): “We, in geography course, have a lot of field job and field lessons ….”

At stage 6, both the normalized grades approach and the restricted focus on projects with discrepant grades were important to reach consensus. Three projects were quickly reviewed by a third evaluator due to very discrepant grades among them. On the other hand, the range of grades assigned by committee members was quite diverse. Figure 5 was taken from the artifact evaluation tool and presents the five best rated projects (at left), their normalized grades, and evaluators who analyzed each project. The table on the right presents the evaluators maximum and minimum grades. Each participant analyzed at least fifteen projects. Although the maximum and minimum grades are quite different, among the top five there are projects evaluated by six different evaluators (av1, av2, av3, av4, av6, av7). Contrarily, if the normalization process were withdrawn, a single pair of evaluators would have been responsible for classifying four of their assigned projects in the first positions.

Figure 5
Part of project evaluation tool results got during artifact implementation phase

Beyond the project selection process results, information collected through the complementary evaluation strategy was also satisfactory. The means got from the questionnaire were acceptable, but only four committee members answered it. So, data collected from interviews is more important. Interview carried out with two committee members indicated they were satisfied with the method. Similarly, the head of undergraduate pro-rectory were also satisfied. She pointed out that she intends to keep using the artifact in the future, and her statement at the end of the interview is a good briefing about the results achieved: “So ... I think maybe that was the great merit of that work ... that we got the collegial vision ... because we had all the areas involved ... and we had the focus on planning ... I think that maybe this is the great conclusion.”

Conclusions

This article presented an artifact to improve strategy-aligned project selection in the university environment, which includes some intrinsic and common characteristics such as the collegial structure, the looking for consensus (Burnesa et al., 2014; D. P. Lacerda, 2009; Meyer, 2007), and lack of objectiveness in planning, as argued by Baldridge (1983 as cited in Scharmach et al., 2012). Artifact presented in this article was constructed through an evolutionary development process. Five different versions were developed and tested until the final sixth version, which was implemented and evaluated in a real world situation inside a university. The artifact itself and its development process, as well as the implementation and evaluation procedures carried out, are described in the article and can be useful for further researches on the theme.

The final version of the artifact has seven stages, and its main components are based on existing solutions such as BSC, AHP, and scoring technique. Balanced scorecard (BSC) is used as a support for preliminary criteria identification. An AHP tool adapted by the authors is used to define criteria and their weights. Adjustments made to scoring techniques are used to analyze, score, and select projects. Artifact procedures and tools were used in two real project selection processes, supporting the evaluation of 185 projects (97 projects in first implementation and 88 in the second one). The suitability of the artifact was evaluated by analyzing data collected during these implementations and by applying questionnaires and interviews to people involved. Results point that the artifact fits the expectations of people involved and is suitable to the university environment.

Other similar solutions had already been proposed by Eilat et al. (2008), Asosheh et al. (2010) , Bai et al. (2010), Chan and Ip (2010), García-Melón et al. (2015), Yüksel and Dagdeviren (2010), and Chang (2015). None of them, however, focused on intrinsic characteristics of university environment. Also, these proposals do not present an integrated stage or step based solution or even describe the development or evaluation process used over the artifact. Most of them, actually, doesn’t make it clear if proposals were used in practice, which is one of the problems in the literature on the theme (Ghasemzadeh & Archer, 2000).

Beyond generating an artifact which is suitable and applicable in a university environment, this study also contributes to related research field in different ways. First one is the role played by AHP adapted tool, especially with regard to the integration of traditional AHP with a proposal from Bose (2015) to reduce cognitive conflict in group decision making. This approach seems to be promising in scenarios involving this kind of decision-making approach. Other contributions are: issues and difficulties related to implement a multicriteria constructivist approach (MCDA-C) to select institutional projects inside the university environment; quasi-experiments highlighting different approaches over scoring technique; create awareness and take into account ambiguities in institutional strategic plan. Another contribution concerns DSR method itself, proposed by Dresch et al. (2015). Using exploratory studies just from the beginning, in DSR awareness problem stage, it seems to be promising, and including it as a formal recommendation in the DSR is a possibility.

Among the limitations of this study is the fact that all quasi-experiments that supported the artifact development process were performed in the same institution. Although it is an institution representative of the main problems faced by universities in general, each organization is unique and it would be important to test the artifact in other institutions and situations. There are some future enhancements involving the artifact. An open issue is about the selection of projects with the potential to become comprehensive and institutional projects. Artifact was developed to select local solutions, so comprehensiveness was not one of the criteria used. By focusing on that, selection process was unable to point out projects which were highly rated in a specific criterion and, as discovered in the artifact evaluation process, also had high potential to become an institutional initiative. So, it seems that the artifact is not enough for activities like that.

Another limitation is that the artifact does not achieve other related problem classes, such as the portfolio optimization process. In addition, there is a situation involving the formal portfolio definition process. Although the artifact provides a specific meeting for this purpose, it does not provide a precise guide on how to consider details about resource allocation. Other similar artifacts have this limitation too, and this is a good theme to be investigated in the future.

References

• Almeida, N., & Almeida, F. (2013). Metodologia para gerenciamento de portfólio: Teoria e prática Rio de Janeiro, Brazil: Brasport.
• Archer, N., & Ghasemzadeh, F. (1999). An integrated framework for project portfolio selection. International Journal of Project Management, 17(4), 207-216. https://doi.org/10.1016/s0263-7863(98)00032-5
  » https://doi.org/10.1016/s0263-7863(98)00032-5
• Asosheh, A., Nalchigar, S., & Jamporazmey, M. (2010). Information technology project evaluation: An integrated data envelopment analysis and balanced scorecard approach. Expert Systems with Applications, 37(8), 5931-5938. https://doi.org/10.1016/j.eswa.2010.02.012
  » https://doi.org/10.1016/j.eswa.2010.02.012
• Bai, S., Li, S., Feng, R., & Guo, Y. (2010). Organizational project selection based on fuzzy multi-index evaluation and BP neural network. International Conference on Management and Service Science, Wuhan, China. https://doi.org/10.1109/ICMSS.2010.5576020
  » https://doi.org/10.1109/ICMSS.2010.5576020
• Bose, U. (2015, March). Design and evaluation of a group support system supported process to resolve cognitive conflicts. Computers in Human Behavior, 49, 303-312. https://doi.org/10.1016/j.chb.2015.03.014
  » https://doi.org/10.1016/j.chb.2015.03.014
• Burnesa, B., Wend, P., & By, R. T. (2014). The changing face of English universities: Reinventing collegiality for the twenty-first century. Studies in Higher Education, 39(6), 905-926. https://doi.org/10.1080/03075079.2012.754858
  » https://doi.org/10.1080/03075079.2012.754858
• Chan, S., & Ip, W. (2010). A Scorecard‐Markov model for new product screening decisions. Industrial Management & Data Systems, 110(7), 971-992. https://doi.org/10.1108/02635571011069068
  » https://doi.org/10.1108/02635571011069068
• Chang, K.-L. (2015). A hybrid program projects selection model for nonprofit TV stations. Mathematical Problems in Engineering, 2015, 1-10. https://doi.org/10.1155/2015/368212
  » https://doi.org/10.1155/2015/368212
• Cooper, R. G., Edgett, S. J., & Kleinschmidt, E. J. (1999). New product portfolio management: Practices and performance. Journal of Product Innovation Management, 16(4), 333-351. https://doi.org/10.1111/1540-5885.1640333
  » https://doi.org/10.1111/1540-5885.1640333
• Dresch, A., Lacerda, D. P., & Antunes, J. A., Jr. (2015). Design science research - A method for science and technology advancement Porto Alegre, Brazil: Bookman.
• Dutra, C. C., Ribeiro, J. L., & Carvalho, M. M. de. (2014, August). An economic-probabilistic model for project selection. International Journal of Project Management, 32(6), 1042-1055. https://doi.org/10.1016/j.ijproman.2013.12.004
  » https://doi.org/10.1016/j.ijproman.2013.12.004
• Eilat, H., Golany, B., & Shtub, A. (2008). R&D project evaluation: An integrated DEA and balanced scorecard approach. Omega, 36(5), 895-912. https://doi.org/10.1016/j.omega.2006.05.002
  » https://doi.org/10.1016/j.omega.2006.05.002
• Fachin, R. C. (2013). Estratégias e gestão de universidades. Temas contemporâneos sobre gestão universitária Florianópolis, Brazil: Bookes.
• Falqueto, J. M. Z., & Farias, J. S. (2013). A trajetória e a funcionalidade da universidade pública brasileira. Revista Gestão Universitária na América Latina - GUAL, 6(1), 22-41. https://doi.org/10.5007/1983-4535.2013v6n1p22
  » https://doi.org/10.5007/1983-4535.2013v6n1p22
• García-Melón, M., Poveda-Bautista, R., & Valle, J. L. (2015). Using the strategic relative alignment index for the selection of portfolio projects application to a public Venezuelan Power Corporation. International Journal of Production Economics, 170, 54-66. https://doi.org/10.1016/j.ijpe.2015.08.023
  » https://doi.org/10.1016/j.ijpe.2015.08.023
• Ghasemzadeh, F., & Archer, N. (2000). Project portfolio selection through decision support. Decision Support Systems, 29(1), 73-88. https://doi.org/10.1016/s0167-9236(00)00065-8
  » https://doi.org/10.1016/s0167-9236(00)00065-8
• Gomes, A. de D. (2016, December 12). A representação social da gestão de portfólio de projetos (Dissertação de mestrado). Escola Brasileira de Administração Pública e de Empresas, Fundação Getúlio Vargas, Rio de Janeiro, RJ, Brazil.
• Hardy, C., & Fachin, R. (2000). Gestão estratégica na universidade brasileira -Teoria e casos (2a ed.). Porto Alegre, Brazil: Editora Universidade.
• Hsu, Y.-G., Tzeng, G.-H., & Shyu, J. Z. (2003). Fuzzy multiple criteria selection of government-sponsored frontier technology R&D projects. R&D Management, 33(5), 539-551. https://doi.org/10.1111/1467-9310.00315
  » https://doi.org/10.1111/1467-9310.00315
• Kaplan, R. S., & Norton, D. P. (1992, January/February). The balanced scorecard - Measures that drive performance. Harvard Business Review, 71-79. Retrieved from https://hbr.org/1992/01/the-balanced-scorecard-measures-that-drive-performance-2
  » https://hbr.org/1992/01/the-balanced-scorecard-measures-that-drive-performance-2
• Khorramshahgol, R., Azani, H., & Gousty, Y. (1998). An integrated approach to project evaluation and selection. IEEE Transactions on Engineering Management, 35(4), 265-270. https://doi.org/10.1109/17.7449
  » https://doi.org/10.1109/17.7449
• Kubler, S., Robert, J., Derigent, W., Voisin, A., & Traon, Y. L. (2016). A state-of the-art survey & testbed of fuzzy AHP (FAHP) applications. Expert Systems with Applications, 65, 398-422. https://doi.org/10.1016/j.eswa.2016.08.064
  » https://doi.org/10.1016/j.eswa.2016.08.064
• Kumar, M., Antony, J., & Cho, B. R. (2009). Project selection and its impact on the successful deployment of Six Sigma. Business Process Management Journal, 15(5), 669-686. https://doi.org/10.1108/14637150910987900
  » https://doi.org/10.1108/14637150910987900
• Kwak, Y. H., & Anbari, F. T. (2009). Analyzing project management research: Perspectives from top management journals. International Journal of Project Management, 27(5), 435-446. https://doi.org/10.1016/j.ijproman.2008.08.004
  » https://doi.org/10.1016/j.ijproman.2008.08.004
• Lacerda, D. P. (2009). A gestão estratégica em uma universidade privada confessional: Compreendendo se e como as intenções transformam-se em ações estratégicas (Tese de doutorado). Programa de Pós-graduação em Engenharia de Produção, Universidade Federal do Rio de Janeiro, Rio de Janeiro, RJ, Brazil.
• Lacerda, R. T. de O., Ensslin, L., & Ensslin, S. R. (2010). Um estudo de caso sobre gerenciamento de portfólio de projetos e apoio à decisão multicritério. Revista Gestão Industrial, 6(1), 01-29. https://doi.org/10.3895/s1808-04482010000100001
  » https://doi.org/10.3895/s1808-04482010000100001
• Meyer, L. H. (2007). Collegial participation in university governance: A case study of institutional change. Studies in Higher Education, 32(2), 225-235. https://doi.org/10.1080/03075070701267269
  » https://doi.org/10.1080/03075070701267269
• Nguyen, A. T., Nguyen, L. D., Le-Hoai, L., & Dang, C. N. (2015). Quantifying the complexity of transportation projects using the fuzzy analytic hierarchy process. International Journal of Project Management, 33(6), 1364-1376. https://doi.org/10.1016/j.ijproman.2015.02.007
  » https://doi.org/10.1016/j.ijproman.2015.02.007
• Padalkar, M., & Gopinath, S. (2016). Six decades of project management research: Thematic trends and future opportunities. International Journal of Project Management, 34(7), 1305-1321. https://doi.org/10.1016/j.ijproman.2016.06.006
  » https://doi.org/10.1016/j.ijproman.2016.06.006
• Padovani, M., & Carvalho, M. M. (2016). Integrated PPM process: Scale development and validation. International Journal of Project Management, 34(4), 627-642. https://doi.org/10.1016/j.ijproman.2016.01.006
  » https://doi.org/10.1016/j.ijproman.2016.01.006
• Pinto, R. S., Mello, S. P., & Melo, P. A. (2016). Meta-avaliação: uma década do Processo de Avaliação Institucional do SINAES. Avaliação: Revista da Avaliação da Educação Superior, 21(1), 89-108. https://doi.org/10.1590/s1414-40772016000100005
  » https://doi.org/10.1590/s1414-40772016000100005
• Project Management Institute. (2013). The standard for portfolio management (3rd ed.). Newtown Square, PA: Author.
• Saaty, T. L. (1980). The analytic hierarchy process New York, NY: McGraw-Hill.
• Scharmach, A. L., Domingues, M. J., Carvalho, C. E., & Machado, N. S. (2012). Avaliação das possibilidades de implantação do balanced scorecard em instituições de ensino superior. Revista Gestão Universitária na América Latina - GUAL, 5(1), 250-272. https://doi.org/10.5007/1983-4535.2012v5n1p250
  » https://doi.org/10.5007/1983-4535.2012v5n1p250
• Taroun, A. (2014). Towards a better modelling and assessment of construction risk: Insights from a literature review. International Journal of Project Management, 32(1), 101-115. https://doi.org/10.1016/j.ijproman.2013.03.004
  » https://doi.org/10.1016/j.ijproman.2013.03.004
• Van Aken, J. E., & Romme, G. (2009). Reinventing the future: Adding design science to the repertoire of organization and management studies. Organization Management Journal, 6(1), 5-12. https://doi.org/10.1057/omj.2009.1
  » https://doi.org/10.1057/omj.2009.1
• Welsha, J. F., Nunez, W. J., & Petrosko, J. (2006). Assessing and cultivating support for strategic planning: searching for best practices in a reform environment. Assessment and Evaluation in Higher Education, 31(6), 693-708. https://doi.org/10.1080/02602930600760892
  » https://doi.org/10.1080/02602930600760892
• Yüksel, I., & Dagdeviren, M. (2010). Using the fuzzy analytic network process (ANP) for Balanced Scorecard (BSC): A case study for a manufacturing firm. Expert Systems with Applications, 37(2), 1270-1278. https://doi.org/10.1016/j.eswa.2009.06.002
  » https://doi.org/10.1016/j.eswa.2009.06.002

Guest editors: Aline Dresch, Daniel Pacheco Lacerda, Paulo Augusto Cauchick-Miguel, Joan van Aken, and Raymond Opdenakker

Editor-in-chief: Carlo Gabriel Porto Bellini

Editorial assistant: Luciane Kato Kiwara

Publication Dates

• Publication in this collection
  08 May 2020
• Date of issue
  2020

History

• Received
  15 Apr 2019
• Accepted
  09 Mar 2020
• Published
  31 Mar 2020