The problem of research project portfolio selection in educational organizations: a case study

Abstract:

This paper proposes a model to support decisions in scientific-research project portfolio selection in an educational institution. We used the Analytic Hierarchy Process (AHP) method for development of this model, combining relative and absolute measurement techniques. That combination allowed us to handle a large number of projects and to measure, in an easier way, all variables in the selection. Our research procedure was a case study based on exploratory research and followed by a quantitative modeling approach. Additionally, we performed documental research and conducted non-structured interviews to better understand the research setting and institutional goals. This contributes to propose a more consistent tailoring grant-request and research-funding project selection criteria. This paper contributes to sort out research project proposals that best adhered to the organization's goals, allowing educational-organization managers to handle the problem of limited-resource allocation in the context of large number of project requests.

Keywords:
Multicriteria support to decision-making; AHP; Project portfolio management; Scientific research evaluation; Project selection

Resumo:

Este estudo propõe um modelo para apoiar o processo decisório na seleção de propostas de projetos de pesquisa científica em uma instituição de ensino e pesquisa. Na construção do modelo foi utilizado o método Analytic Hierarchy Process (AHP), combinando técnicas de mensuração relativa e absoluta. Tal combinação tornou possível tratar um grande número de projetos e mensurar, mais facilmente, todas as variáveis do problema de seleção. Como procedimento de pesquisa utilizou-se o estudo de caso, baseando-se em pesquisa exploratória, seguida de uma abordagem de modelagem quantitativa. Adicionalmente, empregaram-se pesquisa documental e entrevistas não estruturadas, a fim de compreender melhor o contexto da pesquisa e os objetivos institucionais e propor adaptações mais consistentes nos critérios de seleção de projetos de solicitação de bolsas e recursos para pesquisa na instituição. Este estudo contribui para a escolha de propostas de projetos de pesquisa mais aderentes aos objetivos institucionais, permitindo que gestores de instituições de ensino lidem com o problema de alocação de recursos limitados face o grande número de demandantes de projetos.

Palavras-chave:
Apoio multicritério à decisão; AHP; Gerenciamento de portfólio de projetos; Avaliação da pesquisa científica; Seleção de projetos

1 Introduction

As a result of the hectic and dynamic world of business, organizations are increasingly relying on projects to conduct the necessary actions to achieve their goals. As a result, projects have been growing in volume and complexity. To cope with the growing demand for projects, they devised the Project Portfolio, or Project Portfolio Management (PPM). This concept gives an array of tools to ensure that a collection of projects can be evaluated, aiming at prioritization in the allocation of resources and at aligning a project portfolio with organizational strategies (Project Management Institute, 2013). However, too many thoughts and projects that emerged in organizations outnumber the available financial resources. That requires a method to support the selection process, as it is the case of the multicriteria methods (Archer & Ghasemzadeh, 1999; Almeida & Duarte, 2011; Carvalho et al., 2013). This requirement is even more relevant to government entities which are accountable for managing public funds to provide society with significant results.

This challenge is faced by the chief manager of the Instituto Federal de Educação, Ciência and Tecnologia Fluminense, a vocational education institute sited in upstate Rio, also known as IFFluminense or simply IFF. IFF is a Brazilian organization that affords vocational education, research and outbound courses in the State of Rio de Janeiro. It has 13 campuses in upstate Rio, in the coastal, lowland towns of Western Rio and in Metropolitan Rio. The research department at IFF holds the duty to control the provision of research-encouraging grants to the students. The assignment of these grants involves distributing a reduced funding to meet the growing demand of research projects filed by the requesters. These projects are submitted and evaluated against multiple criteria.

The problem IFF faces is, therefore, the fact that there is not any systematized model for project selection in place to ensure the most efficient allocation of resources in those projects. Besides meeting the criteria required to generate, foster and disseminate the scientific knowledge, those projects must also comply with the institution’s purposes. This paper addresses the decision-making problem that emerges during the process of evaluating a high number of research projects involving multiple variables. Therefore, the challenge is to select those project proposals that best suit the institution's mission.

The overall aim of this paper is to propose a solution to scientific-research project selection problems in educational institutions using the method of multicriteria support to decision or Analytic Hierarchy Process (AHP), whose application combines both relative and absolute measurements. The specific goals of our paper are: a) identifying the state of the art in multicriteria support to decision that can steer the decision-makers in educational organizations working in the process of prioritizing and selecting research projects; b) restructuring the criteria used in evaluating scientific research and identifying the aspects considered in each criterion in order to comply with the goals of the institution and the quality of research; and c) characterizing the context under the multicriteria perspective to check the suitability of the used multicriteria method to the addressed problem.

This paper focuses on the selection of primary and applied research project proposal selection. The evaluation criteria must harmonize with the goals and mission of the institution. We considered an organizational environment in which the goals and the institutional mission had been previously set.

This paper contributes to encourage the use of multicriteria analysis methods to support decision-making by managers of government-run and private organizations, to handle the complex decision problem in research projects. The question this paper raises is this: how can one prioritize and select research projects that align the most with the institutional goals by using multicriteria support to decision-making methods (AMD)? The following sections describe the theoretical references that support the development of this research, the used methodology, the result of the application of this model and our final remarks.

2 Theoretical references

2.1 Multicriteria support to decision-making

The first decision-supporting scientific methods were created in the 70's. Those methods came into play from the need to incorporate a number of dimensions of a problem, which involved both the quantitative and qualitative aspects of the decision-making process. Those methods consider the subjectivity of the decision maker. They enable the evaluation of the alternatives based on the modeling of the preferences of decision makers to achieve a satisfactory solution (Gomes et al., 2004).

The use of this methodology helps managers to deal with the complexity of this problem in a simpler way. Besides, it favors the communication between the stakeholders and increases the credibility of the decision itself, thanks to the transparency added to the decision-making. This is so because the preferences of the decision makers are externalized, leading them to commit themselves in the decision-making process (Gomes et al., 2002).

As per Vincke (1992), experts in AMD classified AMD methods into three big families: the Multiatribute Utility Theory, the Subordination Methods and the Interactive Methods. The first two of these families are referred to as the American School and the French School respectively. One notices in Gomes et al. (2002) that those schools differentiate themselves by the preference structures they are based on. Such structures result from the combination of established preference relationships and their properties.

According to the previous authors, the methods of the American School rely on a relational structure of preference that does not admit incomparability and enables the transitivity in the preference relation. This makes it possible to add the performances of the alternatives obtained in the several criteria into a single value, single criterion and synthesis. Conversely, the methods used by the French School do support incomparability. Such methods apply to a situation in which one needs to set allowance limits to depict the hesitation of the decision maker in the process of issuing his/her preferences.

As for the Interactive Methods, on the other hand, Roy & Slowinski (2013) say that said methods include two steps: the calculation and the dialogue one. The dialogue stage allows decision makers to provide his/her preferences in case he/she does not agree with the outcome, repeating the cycle until a satisfactory outcome is achieved.

As per Roy (2005), in practice, the application of the AMD works out the following problems: the selection problem, the classification problem and the ordering problem. These problems may be mutually dependent (Gomes et al., 2004). This situation resembles this paper, which consisted of a problem to order the alternatives, from the best down to the worst, considering how research projects contributed to fulfill institutional goals and the research quality. The outcome enabled managers to make the best choices among an array of potential alternatives.

Several recent applications in project selection are found in the literature. One example is the work of Lima et al. (2014) who use the Preference Ranking Organization Method for Enrichment Evaluations I (PROMETHEE I) method aiming to prioritize projects in a water-treatment and sanitation company. Also, authors López & Almeida (2014) applied the PROMETHEE V method to select a project portfolio in the electrical industry, characterized by a problem of ordering. In both applications, they took into account the non-compensatory rationality of the decision in the problem.

As per Silva et al. (2014), compensatory methods allow a bad evaluation obtained by the alternative in a certain criterion to be offset in other criteria. Special attention shall be paid to the application of another method of the PROMETHEE family, the PROMETHEE II, used by Araújo & Almeida (2009) in the selection of investments in the O&G industry to handle an ordering problem. Such methods do not apply to this work, in which a compensatory approach is necessary in measuring the criteria.

One should also add the application of the Analytic Network Process (ANP) methods by Ivanović et al. (2013) in the selection of projects in the transportation industry considering the interactions and correlations between the elements in the decision-making problem. This method was used by Cheng & Li (2005) to order and select projects in the construction industry. Also, they took into account the mutual dependence existing in the problem irrespectively of the problem under analysis, in which the criteria are independent.

Neves et al. (2015) point out that AHP is the most often used method in the area of planning and management in the O&G industry. Also, Méxas et al. (2013) have used AHP to sort integrated management systems (ERP). They found this method to be one of the most often used in selecting ERP systems. Another sample is the application of AHP with integer programming by Padovani et al. (2010) in selecting and allocating resources to the project portfolio in the chemistry industry. Integrating both methods has helped to align the portfolio, to prioritize projects and to allocate resources. The previous authors warn about the unfeasibility of this method to problems showing a high number of alternatives present in the problem set of this work.

However, Saaty (2005) suggests to use AHP blending the relative and absolute measurements to tackle a large number of alternatives. This solution has suited the problems looked at by this work. It allowed us to measure qualitative criteria in a compensatory fashion and assess the alternatives in each criterion with the assignment of absolute values. The following sections shows a brief description of this method.

2.1.1 The AHP method

The AHP method was devised in the 70's by Thomas L. Saaty. It consisted of the creation of a model that reproduced the way the human mind works in the evaluation of alternatives before a complex decision-making problem. Moreover, this method allows one to deal with problems dealing with both tangible and intangible values, thanks to the capacity to create measures for the qualitative variables, based on subjective judgments issued by the decision makers (Saaty, 1991). According to Paoli & Moraes (2011), the AHP method allows one to measure the impact of the different criteria used in the decision-making problems relatively to the overall goal. It is one of the most known methods used in the American School (Gomes et al., 2002).

According to Saaty (1990), the AHP methods enables modeling the decision problem in an hierarchical structure. This model, starting from the major goal, drills down into various criteria making up a new layer. Each criterion of a single level can be broken into two or more criteria and so on, making it easier to treat and understand the problem. Each criterion placed in the lower end of the structure, named leaf criterion, is broken into alternatives. This allows each alternative to be evaluated in light of each leaf criterion. One recommends using a reduced number of levels and criteria to avoid compromising problem understanding (Gomes et al., 2004). Figure 1 shows AHP’s hierarchical structure.

Figure 1
The hierarchical structure of the AHP. Source: Adapted from Saaty (1990, 1991).

Once the problem is already structured, the judgment of preferences by the decision makers is collected. In this case, all criteria immediately subordinate to the same criterion are organized into a square matrix, to allow decision makers to compare them against each other. A template of this matrix is shown in Table 1.

Table 1
Judgment Matrix in light of a particular criterion lying immediately above.

	C1	C2	...	Cn
C1	a11	a12	...	a1n
...	...	...	...	...
Cn	an1	an2	...	ann

• Source: Adapted from Vargas (2010).

Table 1 shows a square matrix to the order n, whose criteria are represented by C₁ thru C_n. The judgments between the criteria make up matrix A = (a_ij), where the positions of row (i) and column (j) vary from 1 to n. The judgment a_ij follows these rules:

• Rule 1: If a_ij = α, then a_ji = 1/α, α≠0. where α is the numerical value of the judgment based on the Saaty scale (1991), as depicted in Chart 1. So, we have a_ji = 1/a_ij.

  Chart 1
  Scale of importance used in the judgment of the AHP.
  

  Degree of importance	Reasons
  Numerical scale	Conceptual Scale
  1	The same	Both compared elements equally contribute to achieving the goal.
  3	Moderate	The compared element is slightly more important than the other.
  5	Strong	Experience and the judgment strongly favors this element against the other.
  7	Very Strong	The element being compared is much stronger against the element in the comparison and this strength can be observed in practice.
  9	Absolute	The compared element has the highest level of evidence possible on its side.
  2,4,6,8	Intermediate values between both judgments, used when the decision maker is having a hard time to decide between two neighboring degrees of importance.

  • Source: Adapted from Saaty (1990, 1991).

• Rule 2: If C_i is judged to be equal to C_j in relative importance, then a_ij = 1 and a_ji = 1; and, particularly, a_ij = 1, i = j, i = 1, 2, ..., n.

When criteria C_i is compared against C_j, one should first spot which is the most important element and then how most important it is. If C_i outweighs C_j in importance, then a_ij= α and a_ji = 1/α. Otherwise, a_ij=1/α and a_ji= α.

As per Saaty (1991), after making the judgment, the local mean priority (PL) is calculated. The PL value determines the importance of each criterion against the one immediately above. The calculation of the elements' PLs (criteria or alternatives), represented in the judgment matrix, can be performed by means of either the exact or the approximate method (Saaty, 1991). As per Vargas (2010), the approximate method is as efficient as the exact method and it is simpler to apply. The calculation of the PLs using the approximate method is shown in Table 2. Note that the calculation assigned to w₁, w₂, ..., w_n represents the PL of criterion C₁, C₂, ..., C_n, respectively.

Table 2
Calculation of the local mean priorities against a particular criterion immediately above

	C1	C2	...	Cn	PLs
C1	w11= a11 / t1	w12=a12 / t2	...	w1n=a1n / tn	w1=(w11+...+w1n)/ n
C2	w21=a21 / t1	w22 =a22 / t2	...	w2n=a2n / tn	w2=(w21+...+w2n)/ n
...	...	...	...	...	...
Cn	wn1=an1 / t1	wn2 = an2 / t2	...	wnn=ann / tn	wn=(wn1+...+wnn)/ n

• t_j represents the total of column j of the source matrix of the judgments A=(a_ij), j, i=1, ...,n.
• Source: Adapted from Vargas (2010).

Saaty (1991) has found out that a reciprocal and positive judgment matrix A = (a_ij), whose judgments issued by the decision makers are considered perfect, generates a consistent matrix, where a_ik=a_ij.a_jk, i, j, k = 1, 2, …, n. The occorrence of this situation fulfills the Equation 1:

$Aw=nw$ (1)

where: w is the eigenvector of A, which corresponds to the vector of priorities (w₁, w₂, …, w_n);

n is the eigenvalue of A, whose value is the number of order of the judgment matrix;

However, as per Saaty (1991), in practice there can be inconsistent judgments. Said author has observed that slight variations in a_ij impact the eigenvalue. The deviation of the eigenvalue relatively to the order of the matrix is an indicator to determine the proximity of the consistency. This way, Saaty (1991) proposed the calculation of the Consistency Ratio (CR), admitting an allowance of 10%. For higher values, the judgment must be redone.

The following step consists of checking the consistency of judgments. Vargas (2010) uses Saaty's proposal to adopt the following steps to calculate the CR:

Using the following Equation 2, the highest eigenvalue of the judgment matrix (𝛌_Max) is calculated.

${\lambda }_{Max}={\displaystyle \sum }_{j=1}^n{t}_j.w_j$ (2)

where: t_j is the total of the criteria judgments represented by column j of the original matrix of judgments (Table 1) and w_j is a priority of the same criterion, associated to row j of the normalized matrix (See Table 2).

Using Equation 3, the Consistency Index (CI) is calculated as follows.

$CI=\frac{{\lambda }_{Max}-n}{n-1}$ (3)

where: n is the number of order of the judgment matrix.

Then, as Equation 4 shows, the Consistency Ratio (CR) is calculated.

$CR=\frac{CI}{RI}$ (4)

where the Random Index (RI) is the consistency rate of a reciprocal matrix that was randomly generated by Oak Ridge Laboratories, as you can see in Table 3. The order of the matrix (n) determines the index to be applied.

Table 3
Table of random consistency indexes.

n	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15
RI	0.00	0.00	0.58	0.90	1.12	1.24	1.32	1.41	1.45	1.49	1.51	1.48	1.56	1.57	1.59

• Source: Saaty (1991).

As per Vargas (2010), after calculating the PLs of the criteria, the global priority (PG) of each criterion is calculated. This is performed by multiplying its own PL by the other PLs of the criteria immediately above. The found PG determines the relative contribution to achieving the final goal.

As per Vargas (2010) the same procedure to calculate the PL for the alternatives is adopted. In this case, the degree of importance is compared against the alternatives relatively to each leaf criterion and their PLs are determined. Finally, the PG of each alternative is the result of the overall sum of the multiplication of the relative PL of each leaf criterion by the PG of this criterion. The values of the PG, generated by the AHP, determine the percentage importance factor of the criteria and alternatives to reach the overall goal based on the the decision makers' preferences. One example of the classical model of the AHP application can be seen in Vargas (2010).

2.1.2 The AHP method: relative measurement along with absolute measurement

As per Saaty (2005), the AHP method is feasible for problems involving a large number of alternatives, which means a higher number of comparisons between the alternatives for each criterion. To solve this problem, said author suggests that absolute measurement be used combined with the relative measurement of the AHP.

Also, as per Saaty (1990), the absolute measurement is applied when one intends to measure the elements in each criterion based on a conceptual scale. First, one should set what the conceptual scale is. For example: excellent, very good, good, average, fairly good, bad.

As the said author points out, after setting the nominal scale to be considered in each criterion, the scales should be measured against the importance of the related criterion. In measuring the scales, one adopts the same judgment and PL calculation procedure.

In this case, as per Saaty (1990), when showing the judgment matrix, the elements of the scale must be ranked from the best to the worst, into columns, from left to right, and into rows from up down, respectively. The PL generated for each element in the nominal scale is used to convert it into its numerical value. This is the absolute value to be assigned in evaluating the criterion for the alternative.

According to Saaty (1990), the overall measurement of each alternative results from the multiplication of the PG of the leaf criterion, given by the relative measurement of the AHP, by the absolute measurement of the alternative in the corresponding criterion. The performance of the alternative determines its priority. The higher the performance, the better the alternative is. That allows ranking the priorities of the alternatives in a top-down fashion.

2.2 Project portfolio management

The first projects date back to remote times. However, only after the 40's, the need to apply stronger efforts to plan projects was observed. That was due to the growing uncertainties and highers costs of opportunity in the allocation of capital to the new projects (Kerzner, 2009). Its use became even stronger as of the 90's, driven by the need to respond to the lower life cycle of products in a faster and more effective way, by technological breakthroughs and tougher market competition. By late 90's, with the rise in both the quantity and the complexity of the project scope, organizations started to be concerned about aligning projects to the organization's goals. That is when the importance assigned to the project portfolio management grew (Carvalho et al., 2013).

The Project Management Institute (2013) defines project portfolio as a collection of products or programs, to allow for an effective management, whose focus consists of the analysis and maintenance of the portfolio. Maintenance aims to identify, prioritize and authorize projects better allocate resources and ensure a consistent project that is very well aligned with organizational strategies. The objectives are driving sources in the decision-making process. They must be taken into account in making up the project portfolio (Meskendahl, 2010; Bond et al., 2008).

As stated by Meskendahl (2010), selecting projects to make up the portfolio has been growing aided by decision-supporting methods, given the complexity and broadness of the involved variables. Using those methods, as stated by Vargas (2010), ensures that the available financial resources be allocated into those proposals suiting the most the institutional objectives.

2.3 Evaluation criteria considered in scientific research projects

Scientific research is key to economic growth and social development. The Brazilian government, in its federal, state and local levels, is accountable for the scientific and technological research activities, whose actions are implemented by means of research promotion agencies (Alves et al., 2015).

However, the high rise in the research activities by both government-run and private organizations, requires using an evaluation system, as a result of the constraints in the available promotion funding. This system is good both to justify the choices of managers in terms of the allocation of investments in research and to ensure the proper use of the public funding, allowing higher transparency in the made decisions (Francisco, 2002). As per Rodrigues (2011) and Moreira & Velho (2008), these agencies have valued the quantitative aspects in the evaluation. It is, therefore, necessary to pay more attention to the qualitative aspects of the production of knowledge, enabling investing in projects of highest quality.

As per Francisco (2002), in selecting research projects one should consider as criteria for the submitted project what its impact in the productive sector is, motivated by some aspects such as the produced innovation and the potential to steer new public policies. Francisco (2002) does not ignore in his analysis, how important other criteria like the social impact, the capacity to generate new job opportunities, to develop new skills and to reuse the knowledge generated by the research in academic courses and in the organizations. As for the social aspect, Francisco (2002) also emphasizes the environmental aspects, as criteria having the capacity to adpot clean technologies.

This is a topic that is far from being a consensus. In less recent perspectives, one usually finds in authors like Tuncer (1975) the perception that even if it does not generate innovations for the industry, a scientific research is useful when it is the source of information to the technical and scientific community. Today, there are those authors who support this perspective. Andrade (2010), for example, remarks that even if the research is not innovative, it must bring up a new focus or clarify a certain subject matter. As per, Andrade (2010), one should consider the feasibility of the research, the ease of access to the addressed subject, the available financial resources and materials and the deadline to be met.

Miranda & Almeida (2004) introduce a list of the analyzed criteria in the evaluation of the minor master’s degrees by Brazilian agency Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) in the area of Engineering courses III, grouped into six criteria dimensions. One of these criteria is Research Activity. It includes these subcriteria: (i) compliance of the lines of research with the focus areas; link between lines of research and research projects; tailoring of the number of lines of research relatively to the size of Professor Reference Hub 6 (members of the faculty in Upper Degree Institutions having a significant involvement in teaching activities, research and tutoring, as per Smit et al. (2002)); and involvement of the faculty in research projects.

As per Santana (2009), in evaluating the research one should consider criteria such as: how it contributes to technological advances and development; how it solves nation-wide, state and local problems; the education of researchers for the market; the motivation for the search for scientific excellence, dissemination of science, guaranteeing the scientific hegemony of Brazil; influencing society’s conception of science, behaviors, values and culture. Luiz (2006), in his turn, takes into account the criteria related to the researcher’s profile and productive capacity, such as: titles, teaching seniority and number of advised students, awards and academic prizes, bibliographic production, impact of the bibliographic production, technical papers and participation in university benches.

This set of aspects has steered the tailoring of these criteria to the commission in charge of following up the modeling of the decision proposed in this article. The proposed model will be discussed next.

3 Methodology

The methodology used in this article is based on the relative-measurement multicriteria AHP method combined with the absolute measurement. One of the motivations to use AHP was how easy it is to measure all variables in the system with it. Another motivation was its capacity to structure problems hierarchically, which led us to measure the criteria in an offsetting manner based on how each of them contributed to the major goal. That has enabled us to better state the problem under analysis which consisted of prioritizing the projects that were most aligned with the institutional goal and with the requirements of each research project. The proposed model to select scientific research projects at IFFluminense included the following steps.

3.1 Criteria identification

Based on document survey, we identified the criteria already being used in the evaluation of the research. We observed that although the existing criteria were aligned with the objectives they did not present a clear definition of the evaluation parameters. Moreover, both the criteria related to the technical aspects of the research and the criteria related to the objectives of the institution were evaluated at the same level. That could cause conflicts and inconsistencies in the final evaluation of projects. To work out this problem, two criterion groups were created: criteria to align with institutional objectives and specific criteria to evaluate the scientific research. Also, the aspects to be evaluated for each criterion were determined.

It is important stressing the role of the Commission made up to monitor and validate the process of tailoring the criteria and their aspects to the AHP method. The data coming from the non-structured interviews were matched against other information collected from documentary research, which enabled the construction and the triangulation of the evidences translated by the case study. The members of the Internal Commission had access to the results to make comments and validate both the criteria-tailoring stage and its aspects and the findings of the research.

Both groups of criteria divided the selection process into two steps. The first stage, based on the first group, aimed to screen the projects that are aligned the most with the institutional objectives. The second stage aimed to select, among the projects approved in the first stage, those that suit the best the requirements of the scientific research.

3.2 Application of the AHP method to measure the criteria and the aspects

The measurement of the criteria and the aspects is performed by calculating the priorities of the AHP based on the judgment of the decision makers. To facilitate, we used a spreadsheet previously set for each stage of the selection process to process all the steps required by AHP.

First, we organized each group of criteria into the AHP hierarchy. To make it easier to perform comparisons during the course of the judgment, we have put together a three-layer structure and at most six elements subordinate to each criterion.

The collection of judgments was recorded in the spreadsheet itself. The recording was made by one of the authors of this paper, who played the facilitator, never intervening in the decision-making process. Each judgment of the calculation of the local, global and consistency priorities were calculated automatically. This enabled the group of decision makers to ponder and redo their judgments. The judgment was performed by means of a consensus obtained by the members of the Commission and, subsequently, approved by the Chamber of Research at IFFluminense.

3.3 Preparation of the projects' individual score sheet templates

The design of the score sheet of the research project for each stage has been based on the group of criteria. Each evaluation element corresponded to the criterion/leaf aspect of the AHP hierarchy. So the weight generated for the criterion/leaf aspect determined the weight of the question. In this stage, the nominal scales to be used by the advisors as optional answers for each item. For some aspects that had a quantitative nature, a conversion table matching the research setting of the Institution was created. This table was designed by the Commission and approved by the Chamber of Research. This table was attached to the score sheet to be used by the evaluator.

3.4 Application of the AHP in the measurement of the nominal scales

For each criteria structure, we applied the AHP method to determine the degree of importance of each nominal scale, used as an optional answer, relatively to the leaf criteria, corresponding to the item in the score sheet. The nominal values of each scale were entered into a square matrix, from the best down to the worst, in the rows, from top down, and in the columns, from the left to the right. Using the matrix, the judgment of the nominal values was performed in light of the related lead criterion and the local mean priorities and the consistency of the judgment were determined.

The judgment was performed by one of the authors of this article and subsequently approved by the Commission and by the Chamber of Research. Also, all the necessary procedures to calculate were developed and integrated into the electronic spreadsheet of data. The priority value calculated for each scale corresponds to the absolute value to be assigned to the criterion by the time of the project evaluation, based on the conceptual evaluation of the decision maker.

3.5 Individual evaluation of the project proposals

The evaluation of projects was carried out based on two evaluation cards, one for each stage. The evaluation procedure was managed by the research sector itself. It was in charge of sending the evaluation cards to the advisors by e-mail, along with the project proposal. Once the evaluation period was over, the research sector moved the answers of each evaluation stage into their respective electronic data spreadsheet. These spreadsheets were preset and integrated into the calculation of the criteria/aspects and the nominal scales generated by the AHP method.

The spreadsheets of the two steps were programmed to assign the associate absolute values to the advisors' nominal answers. This way, for each spreadsheet, they calculated the value of the project linked to each criterion automatically by multiplying its relative measurement by the absolute value assigned to each alternative in the criterion. Then, the project's global performance was calculated by adding the products obtained in each criterion.

3.6 Prioritizing and selecting projects aligned with institutional goals

Based on the performance of the projects calculated in the evaluation answer spreadsheet as for the institutional goals. They ordered the answers from top down performance wise. The projects topping the list were those that adhered the institutional goals the most, while the worst ones were at the bottom. Answers were ordered as per functionality in the data spreadsheet.

3.7 Prioritizing and selecting the projects that adhered the most to the scientific research requirements

In this stage, the worst projects in the previous ranking were eliminated. The performance obtained for the scientific research criteria was ranked from top down. Managers then chose those projects that fairly much fulfilled the Evaluation Criteria of the scientific research and the institutional alignment aiming to allocate research grants.

4 The results

The proposal model was applied to IFFluminense. IFFluminense provides teaching, research and outbound course to promote the social, economic development of the area. Its mission is to develop a schooling that is committed to society's needs and to promote scientific, technological advances in a sustainable fashion. Research activities is managed by the Supporting Office of Research and Innovation. It centralizes the management of the research policies and activities, which are articulated with the research and community-oriented education, and promote research, science, technology and innovation. The Supporting Office of Research and Innovation holds the duty to select research projects to run for the Young Scientist grants.

This selection process abides by the rules established by the Institutional Research and Entrepreneurship Promoting Grants at IFFluminense. This program aims to encourage research and entrepreneurship by giving out grants provided by either Brazilian research promotion bodies or IFFluminense itself.

The problem being addressed, however, is limited to selecting new basic and applied research projects. These projects run for the grants provided by the Institutional Research and Entrepreneurship Promoting Grants at IFFluminense. More specifically, they run for the grants of the Institutional Young Scientist Program (PIBIC in the Brazilian acronym) and of the Institutional Technological Development and Innovation Introduction Grants (PIBITI), awarded by IFF using its own resources or using funding from the Brazilian Council for Scientific and Technological Development (CNPq).

This is an yearly selection process that starts off with a Call. During the project-submission period, the researcher fills out the new project application form and sends it to the Supporting Office for Research and Innovation through the Electronic Research Project Submission System (SiSEP-IFF).

The projects are then evaluated by internal and external advisors, who are invited by the Chamber of Research. When the evaluation period is over, the Chamber chooses the projects to allocate the research grants, based on the total score resulting from the evaluation and the number of available grants, including them in the portfolio.

IFF, however, did not have a system-based way to support the selection process of a large number of research projects submitted by the researchers of the many campuses, whose projects run for a limited number of grants. To help solve this problem, we applied the AHP method, combining relative and absolute measurements.

The first stage of the application was the identification of the criteria, which resulted into two groups of criteria as shown in Charts 2 and 3. Both groups of criteria comply with the two steps of the screening. The first step is the screening of the projects that matched institutional goals the most. The second step aims to select the projects that aligned the most to the requirements of the scientific research at IFFluminense.

Chart 2
Representation in the AHP hierarchy of the used criteria to evaluate research projects in terms of how they contributed to achieving institutional goals.

Major goal (Obj#1)	Criterion	Aspect	Code
Research is committed to local and state development	Scientific and Technological Advance	Provides new scientific/technological knowledge (new laws, theories, concepts, models) or a new approach relying on previous knowledge.	Obj#1_1
		Provides the creation and/or improvement of methods, processes, materials and/or services potentially applicable to the economic, political and/or social sectors.	Obj#1_2
		Encourages the acquiring of research skills by the young scientist grantees, technical staff, college-degree staff, master degree and Ph.D. students, Masters' degree holders, Ph.D. degree holders and Post-Ph. D. degree holders.	Obj#1_3
	Social commitment	Encourages new job opportunities, opportunities for traineeship, training courses, products and services, develops professional skills of employees and students and contributes to the critical awareness of citizens.	Obj#1_4
		Cooperates direct and indirectly to sustainable development, adopting and/or encouraging sustainable practices from an environmental, economic and social perspective.	Obj#1_5
	Links to education, research and outbound courses	Plans to tie up the teaching, research and outbound courses activities at IFFluminense.	Obj#1_6
		Plans to review the application of knowledge resulting from the research to solve a social problem.	Obj#1_7
	Links to the research sectors at IFF	The project's research field corresponds to one of the priority research themes at IFFluminense.	Obj#1_8
		The theme being researched matches the lines of research in the research hub to which it is associated.	Obj#1_9

• Source: Designed by the authors.

Chart 3
Representation in the AHP hierarchy of the Criteria Evaluation of the projects in terms of the scientific research requirements.

Major goal (Obj#2)	Criterion	Aspect	Code
Evaluation of the scientific research	Originality	This is a new theme that debates a new approach to an existing topic.	Obj#2_1
	Technical quality of project	Problem is clearly stated	Obj#2_2
		Methodological matching	Obj#2_3
		Theoretical foundation	Obj#2_4
		Procedures to publicize and use the findings by society	Obj#2_5
	Quality wording and text organization	Spelling	Obj#2_6
		Grammar use	Obj#2_7
		Clarity	Obj#2_8
		Objectivity	Obj#2_9
		Formal Structure	Obj#2_10
	Grantees' plan of activities matches the goals and schedule of the project	Project's activities are aligned with its goals	Obj#2_11
		Students' activities match the schedule of the project	Obj#2_12
	Feasibility of carrying out the project	Availability of material resources	Obj#2_13
		Financial support from another funding entity	Obj#2_14
		Sizing-up of schedule	Obj#2_15
	Scientific and technological productivity in the thematic area of the project, as stated in the researcher's Lattes CV	Titles of the researcher	Obj#2_16
		Publication (article in indexed journals, book or special edition, book chapter) or filed patent.	Obj#2_17
		Publication in congress/events proceedings	Obj#2_18
		Advisor in Young Scientist Programs or in dissertations (undergraduation or minor master's)	Obj#2_19
		Advisor in Master's or Ph.D. degrees	Obj#2_20
		Participation in benches (graduation dissertations, Master's Degree or Ph. D. dissertations)	Obj#2_21

• Source: Designed by the authors.

Therefore, grouping the Evaluation Criteria together allowed us to make sure a quality decision was made. Also, the setting of the aspects for each criterion allowed us to standardize the process and have a shared understanding between different people, avoiding different opinions.

In the second stage of application, we measured the criteria and the aspects. The weights returned by the AHP for each group of criteria, based on the judgments issued by the decision-makers of the research field can be seen in Charts 4 and 5, as follows. Because of their experience and expertise, this judgment can be easily performed, allowing for the generation of a good outcome, considering the aimed objectives.

Chart 4
Weights of the criteria and scales used in evaluating projects in terms of how they contributed to achieving the institutional goal.

Major goal: Research is committed to state and local development (RC = 0.00)
Criterion / Aspects		Weight	Item under evaluation		Weight of the answer options
Technological and Scientific Advances Weight = 0.3507 (RC=0.03)
Obj#1_1 (RC=0.00)		0.1992	1.1	very much		fairly much	little	not at all
				0.5375		0.3027	0.1055	0.0543
Obj#1_2 (RC=0.00)		0.0344	1.2	very much		fairly much	little	not at all
				0.5375		0.3027	0.1055	0.0543
Obj#1_3 (RC=0.00)		0.1171	1.3	very much		fairly much	little	not at all
				0.5375		0.3027	0.1055	0.0543
Social commitment weight = 0.3507 (RC=0.00)
Obj#1_4 (RC=0.00)		0.2630	2.1	very much		fairly much	little	not at all
				0.5375		0.3027	0.1055	0.0543
Obj#1_5 (RC=0.00)		0.0877	2.2	very much		fairly much	little	not at all
				0.5375		0.3027	0.1055	0.0543
Linked to the Education, Research and Outbound Courses weight = 0.1892 (RC=0.00)
Obj#1_6 (RC=0.00)		0.1419	3.1	very much		fairly much	little	not at all
				0.5375		0.3027	0.1055	0.0543
Obj#1_7 (RC=0.00)		0.0473	3.2	very much		fairly much	little	not at all
				0.5375		0.3027	0.1055	0.0543
Linked to the lines of research at IFF weight=0.1093 (RC=0.00)
Obj#1_8 (RC=0.00)		0.0364	4.1	Yes		No
				0.9000		0.1000
Obj#1_9 (RC=0.00)		0.0729	4.2	High		Average	Low	None
				0.5375		0.3027	0.1055	0.0543
Total	1.0000

• Source: Designed by the authors.

Chart 5
Weights of the criteria and the scales of the evaluation of the projects in terms of the requirements for each scientific research.

Major goal: evaluation of the scientific research (RC=0.01)
Criterion/aspects	Weight	Item under evaluation	Weights of the answer options
Originality of the research Weight = 0.0580
Obj#2_1 (RC=0.00)	0.0580	1.1	yes	partially	No
			0.7606	0.1577	0.0817
Technical quality of the project weight = 0.2204 (RC=0.00)
Obj#2_2 (RC=0.00)	0.0773	2.1	Great	Good	Regular	Awful
			0.5375	0.3027	0.1055	0.0543
Obj#2_3 (RC=0.00)	0.0773	2.2	Great	Good	Regular	Awful
			0.5375	0.3027	0.1055	0.0543
Obj#2_4 (RC=0.00)	0.0417	2.3	Great	Good	Regular	Awful
			0.5375	0.3027	0.1055	0.0543
Obj#2_5 (RC=0.00)	0.0241	2.4	Great	Good	Regular	Awful
			0.5375	0.3027	0.1055	0.0543
Quality of wording and text organization weight = 0.0800 (RC=0.01)
Obj#2_6 (RC=0.00)	0.0114	3.1	Great	Good	Regular	Awful
			0.5375	0.3027	0.1055	0.0543
Obj#2_7 (RC=0.00)	0.0114	3.2	Great	Good	Regular	Awful
			0.5375	0.3027	0.1055	0.0543
Obj#2_8 (RC=0.00)	0.0314	3.3	Great	Good	Regular	Awful
			0.5375	0.3027	0.1055	0.0543
Obj#2_9(RC=0.00)	0.0186	3.4	Great	Good	Regular	Awful
			0.5375	0.3027	0.1055	0.0543
Obj#2_10 (RC=0.00)	0.0071	3.5	Great	Good	Regular	Awful
			0.5375	0.3027	0.1055	0.0543
Plan of activities for grantees matches the goals/schedule of the project. Weight=0.2106 (RC=0.00)
Obj#2_11(RC=0.00)	0.1580	4.1	Great	Good	Regular	Awful
			0.5375	0.3027	0.1055	0.0543
Obj#2_12 (RC=0.00)	0.0527	4.2	Great	Good	Regular	Awful
			0.5375	0.3027	0.1055	0.0543
Feasibility to carry out the project Weight= 0.2106 (RC=0.01)
Obj#2_13 (RC=0.00)	0.1135	5.1	High	Medium	Low
			0.7606	0.1577	0.0817
Obj#2_14 (RC=0.00)	0.0345	5.2	High	Enough	Little	None
			0.5375	0.3027	0.1055	0.0543
Obj#2_15(RC=0.00)	0.0626	5.3	Great	Good	Regular	Awful
			0.5375	0.3027	0.1055	0.0543
Scientific and technological productivity of the researcher in the thematic area the project belongs to, based on the researcher's Lattes CV Weight=0.2204 (RC=0.05)
Obj#2_16(RC=0.03)	0.0264	6.1	Post Doctor Degree	Ph.D.	Master's Degree	Minor Master's	Undergraduation
			0.4867	0.2720	0.1370	0.0661	0.0382
Obj#2_17 (RC=0.03)	0.0982	6.2	great	good	Regular	low	None
			0.4867	0.2720	0.1370	0.0661	0.0382
Obj#2_18 (RC=0.03)	0.0084	6.3	great	good	Regular	low	None
			0.4867	0.2720	0.1370	0.0661	0.0382
Obj#2_19 (RC=0.03)	0.0311	6.4	great	good	Regular	low	None
			0.4867	0.2720	0.1370	0.0661	0.0382
Obj#2_20 (RC=0.03)	0.0480	6.5	great	good	Regular	low	None
			0.4867	0.2720	0.1370	0.0661	0.0382
Obj#2_21 (RC=0.03)	0.0084	6.6	great	good	Regular	low	None
			0.4867	0.2720	0.1370	0.0661	0.0382
Total 1.0000

• Source: Designed by the authors.

Charts 4 and 5 show the criteria that are part of the first layer of criteria in AHP's hierarchical structure and the resulting PGs. Both Charts 4 and 5 show the identification code for each leaf criterion, that is, for each aspect considered in the evaluation of a certain criterion. The description of this code can be seen in previous charts, Charts 2 and 3, respectively. Column “Weight” shows the global priority of each leaf criterion. Chart 4 and Chart 5 also show the RC value that was calculated based on the judgment between the criteria/aspects that are immediately subordinate to the overhead criterion to calculate the weights.

After setting the criteria and the weights, we made up the individual score sheet of the projects for each stage of the selection process and we set the nominal scales for each item. The items in each score sheet that are associated to each leaf criterion can be seen in column “evaluation item” in Charts 4 and 5, respectively. For each item the elements of the chosen scales are listed. These scales allowed for the conceptual evaluation of the item by the evaluator, suiting well the qualitative criteria.

The next step was to determine the measurements of the nominal scales. For each group of criteria, Charts 4 and 5 show, respectively, the measurements of the nominal scales resulting from the AHP. They also show the RC value produced by the evaluation of the relative scales of each aspect, whose value is shown next to its identifier. Note that most of the judgments of the criteria/aspects and scales presented in Charts 4 and 5 have an RC value of zero. That shows a high degree of coherence in issuing the judgments by the participating decision makers.

From the answers of the advisors obtained from the score sheets of each step, we calculated the individual performance of each project. The following Chart 6 shows the ranking of the first stage of all projects. The projects considered to be the worst in each step were eliminated from the second step Lastly, we ordered projects in terms of performance from top down based on the criteria of the scientific research.

Chart 6
Ordering of projects based on the two goals.

Obj#1: alignment with institutional goals (first ranking)			Obj#2: fulfillment of the scientific research criteria (final ranking)
Project	Score	Ranking	Project	Score	Ranking
3	0.550684	1st	49	0.511931	1st
49	0.550684	2nd	5	0.509791	2nd
2	0.523192	3rd	7	0.481797	3rd
30	0.480854	4th	6	0.447083	4th
...	...	...	...	...	...
12	0.164049	58th	38	0.163276	51th
53	0.131822	59th	25	0.150343	52th
10	0.130795	60th	23	0.129835	53th
59	0.112414	61th	61	0.114163	54th

• Source: Designed by the authors.

The final ranking, as shown in Chart 6, supported managers to decide on the projects that could match the institutional interests and the scientific research requirements in an effective way. The choice of projects was limited to the number of grants to allocate.

5 Conclusions

This work contributed to support the decision-making by managers who had to screen a large number of projects to find those that best suited the institutional goals and specific research criteria. This allowed choosing the projects that best adhered to award a limited number of young scientist grants.

The review of the literature about multicriteria decision-support methods has provided us with the foundation to help them to better understand the problem and to tailor the used methods. We believe that both the conceptual basis and the model structure can aid managers in the education field in other Brazilian educational organizations in the decision making linked to selecting research projects involving multiple criteria.

The restructuring of the criteria used in the evaluation of projects has allowed us to choose projects of highest quality, provided that the used criteria harmonize with institutional goals, therefore linked to the needs of the stakeholders. The first step ensured choosing proposals that best adhered to the purposes of the institution, since those proposals stem from different thoughts of the applicants. The second step, based on the first selection, aimed to prioritize the projects that met the research requirements. That allowed choosing the projects that best matched the investments in research. Moreover, the setting of the aspects for each criterion has favored a shared understanding between advisors as for the aspects analyzed in each criterion. This awarded more consistency to the project evaluation process.

The importance given to the selection of projects aligned with the organization's goals, in order to make up a more effective portfolio is seen in literature. And so is the importance of using multicriteria support methods. This way, the application of the AHP method combined with both relative and absolute measurements has suited well the problem at issue. Besides allowing for the screening of a large number of projects, it made it easier to determine how the weights of the criteria and the aspects contributes to achieving the goal of each evaluation step. Considering that each aspect corresponds to an item of the evaluation, it made it easier to make the score sheet for each step, in which the measurement of each item is the aspect weight generated by the AHP. Using the absolute measurement simplified the evaluation process because nominal scales were used. That has allowed for a conceptual evaluation of the qualitative aspects. Moreover, applying this method has enabled all participants to know what the institutional objectives were, leading them to ponder about the criteria that best represented the goals, the culture and the values of the Institution. This resulted in a more precise determination of the criteria weights based on the managers' consensus.

The relative measurement of AHP enabled determining the weight of the contribution of each item of evaluation of the project to achieve the goal. Therefore, the evaluation of each project has reflected the interests of the Institution. This way, the ranking generated in the first step ensured choosing those projects that best aligned with the institutional goals for the second step. This enabled to list the projects of highest quality in the second step of the selection. This work has contributed to both simplifying and standardizing the evaluation process and improve the quality of the decision-making process in the field of research.

In spite of the ease to use the electronic-spreadsheet methods, the authors of this article suggest that a software be developed to allow combining the AHP method and the relative and absolute measurement integrated with the individual-evaluation collection of the projects. The ease to use this method in this software can allow new application at other levels of the R&D institution.

Because of the exploratory nature of this article, it has constraints. Considering that it focused on the selection of new projects submitted to the conceptual evaluation in multicriteria based on a case study, the umbrella statements stemming from such an article may not be applicable at all to similar contexts in other R&D institutions. However, because of its exploratory nature, one believes that such aspects do not lose their validity as an empirical-research effort targeted at the solution of a management problem involving the allocation of resources in an R&D institution.

• Financial support: None.

Referências

• Almeida, A. T., & Duarte, M. D. O. (2011). A multicriteria decision model for selecting project portfolio with consideration being given to a new concept for synergies. Pesquisa Operacional, 31(2), 301-318. http://dx.doi.org/10.1590/S0101-74382011000200006
  » http://dx.doi.org/10.1590/S0101-74382011000200006
• Alves, A. S., Quelhas, O. L. G., Silva, M. H. T., & Lameira, V. J. (2015). On the role of university in the promotion of innovation: exploratory evidences from a university-industry cooperation experience in Brazil. International Journal of Innovation and Learning, 17(1), 1-18. http://dx.doi.org/10.1504/IJIL.2015.066061
  » http://dx.doi.org/10.1504/IJIL.2015.066061
• Andrade, M. M. (2010). Introdução à metodologia do trabalho científico: elaboração de trabalhos na graduação (10. ed.). São Paulo: Atlas.
• Araújo, A. G., & Almeida, A. T. (2009). Apoio à decisão na seleção de investimentos em petróleo e gás: uma aplicação utilizando o método PROMETHEE. Gestão & Produção, 16(4), 534-543. http://dx.doi.org/10.1590/S0104-530X2009000400004
  » http://dx.doi.org/10.1590/S0104-530X2009000400004
• Archer, N. P., & Ghasemzadeh, F. (1999). An integrated framework for project portfolio selection. International Journal of Project Management, 17(4), 207-216. http://dx.doi.org/10.1016/S0263-7863(98)00032-5
  » http://dx.doi.org/10.1016/S0263-7863(98)00032-5
• Bond, S. D., Carlson, K. A., & Keeney, R. L. (2008). Generating objectives: can decision makers articulate what they want? Management Science, 54(1), 56-70. http://dx.doi.org/10.1287/mnsc.1070.0754
  » http://dx.doi.org/10.1287/mnsc.1070.0754
• Carvalho, M. M., Lopes, P. V. B. V. L., & Marzagão, D. S. L. (2013). Gestão de portfólio de projetos: contribuições e tendências da literatura. Gestão & Produção, 20(2), 433-453. http://dx.doi.org/10.1590/S0104-530X2013000200013
  » http://dx.doi.org/10.1590/S0104-530X2013000200013
• Cheng, E. W. L., & Li, H. (2005). Analytic network process applied to project selection. Journal of Construction Engineering and Management, 131(4), 459-466. http://dx.doi.org/10.1061/(ASCE)0733-9364(2005)131:4(459)
  » http://dx.doi.org/10.1061/(ASCE)0733-9364(2005)131:4(459)
• Francisco, L. T. S. T. (2002). (Indicadores para avaliação de resultados de projetos de pesquisa científica e tecnológicaDissertação de mestrado). Universidade Federal do Rio Grande do Sul, Porto Alegre.
• Gomes, L. F. A. M., Araya, M. C. G., & Carignano, C. (2004). Tomada de decisões em cenários complexos (M. C. G. Araya, Trad.). São Paulo: Pioneira Thomson Learning.
• Gomes, L. F. A. M., Simões Gomes, C. F., & de Almeida, A. T. (2002). Tomada de decisão gerencial: enfoque multicritério. São Paulo: Atlas.
• Ivanović, I., Grujičić, D., Macura, D., Jović, J., & Bojović, N. (2013). One approach for road transport project selection. Transport Policy, 25, 22-29. http://dx.doi.org/10.1016/j.tranpol.2012.10.001
  » http://dx.doi.org/10.1016/j.tranpol.2012.10.001
• Kerzner, H. (2009). Project Management: a systems approach to planning, scheduling, and controlling (10. ed.). New Jersey: John Wiley & Sons.
• Lima, M. T. A., Oliveira, E. C. B., & Alencar, L. H. (2014). Modelo de apoio à decisão para priorização de projetos em uma empresa de saneamento. Production, 24(2), 351-363.
• López, H. M. L., & Almeida, A. T. (2014). Utilizando PROMETHEE V para seleção de portfólio de projetos de uma empresa de energia elétrica. Production, 24(3), 559-571. http://dx.doi.org/10.1590/S0103-65132013005000045
  » http://dx.doi.org/10.1590/S0103-65132013005000045
• Luiz, R. R. (2006). Avaliação de produtividade acadêmica: uma proposta de quantificação. RBPG, 3(6), 300-312.
• Meskendahl, S. (2010). The influence of business strategy on project portfolio management and its success: a conceptual framework. International Journal of Project Management, 28(8), 807-817. http://dx.doi.org/10.1016/j.ijproman.2010.06.007
  » http://dx.doi.org/10.1016/j.ijproman.2010.06.007
• Méxas, M. P. P., Costa, H. G., & Quelhas, O. L. G. (2013). Avaliação da importância relativa dos critérios para a seleção de Sistemas Integrados de Gestão (ERP) para uso em empresas da construção civil. Gestão & Produção, 20(2), 337-356. http://dx.doi.org/10.1590/S0104-530X2013000200007
  » http://dx.doi.org/10.1590/S0104-530X2013000200007
• Miranda, C. M. G., & Almeida, A. T. (2004). Visão multicritério da avaliação de programas de pós-graduação pela CAPES: o caso da área engenharia III baseado no ELECTRE II e MAUT. Gestão & Produção, 11(1), 51-64. http://dx.doi.org/10.1590/S0104-530X2004000100005
  » http://dx.doi.org/10.1590/S0104-530X2004000100005
• Moreira, M. L., & Velho, L. (2008). Pós-graduação no Brasil: da concepção “ofertista linear” para “novos modos de produção do conhecimento” implicações para avaliação. Avaliação, 13(3), 625-645.
• Neves, R. B., Pereira, V., & Costa, H. G. (2015). Auxílio multicritério à decisão aplicado ao planejamento e gestão na indústria de petróleo e gás. Production, 25(1), 43-53. http://dx.doi.org/10.1590/S0103-65132013005000060
  » http://dx.doi.org/10.1590/S0103-65132013005000060
• Padovani, M., Carvalho, M. M., & Muscat, A. R. N. (2010). Seleção e alocação de recursos em portfólio de projetos: estudo de caso no setor químico. Gestão & Produção, 17(1), 157-180. http://dx.doi.org/10.1590/S0104-530X2010000100013
  » http://dx.doi.org/10.1590/S0104-530X2010000100013
• Paoli, D., & Moraes, L. A. F. (2011). Apoio Multicritério à decisão como subsídio à gestão ambiental: o caso da Aggreko Brasil. Gestão & Produção, 18(2), 379-390. http://dx.doi.org/10.1590/S0104-530X2011000200012
  » http://dx.doi.org/10.1590/S0104-530X2011000200012
• Project Management Institute – PMI. (2013). Um guia do conjunto de conhecimentos em gerenciamento de projetos: Guia PMBOK (5. ed.). Pennsylvania.
• Rodrigues, L. O. C., Fo. (2011). Publicar mais, ou melhor? O tamanduá olímpico. Psicologia USP, 22(2), 457-472. http://dx.doi.org/10.1590/S0103-65642011005000014
  » http://dx.doi.org/10.1590/S0103-65642011005000014
• Roy, B. (2005). Paradigms and challenges. In J. Figueira, S. Greco & M. Ehrgott (Eds.), Multiple criteria decision analysis: state of the art surveys (pp. 3-24). New York: Springer Sciense+Business Media.
• Roy, B., & Slowinski, R. (2013). Questions guiding the choice of a multicriteria decision aiding method. EURO Journal on Decision Processes, 1(1), 69-97. http://dx.doi.org/10.1007/s40070-013-0004-7
  » http://dx.doi.org/10.1007/s40070-013-0004-7
• Saaty, T. L. (1990). How to make a decision: The analytic hierarchy process. European Journal of Operational Research, 48(1), 9-26. http://dx.doi.org/10.1016/0377-2217(90)90057-I
  » http://dx.doi.org/10.1016/0377-2217(90)90057-I
• Saaty, T. L. (1991). Método de análise hierárquica (W. S. Silva, Trad.). São Paulo: McGraw-Hill.
• Saaty, T. L. (2005). The Analytic Hierarchy and Analytic Network Processes for the measurement of intangible criteria and for decision-making. I In J. Figueira, S. Greco & M. Ehrgott (Eds.), Multiple criteria decision analysis: state of the art surveys (pp. 345-408). New York: Springer Sciense+Business Media.
• Santana, M. G. H. (2009). Produção do conhecimento científico em transição: novas perspectivas para a avaliação das ações induzidas na área de saúde pelo CNPq (Tese de doutorado). Universidade de Brasília, Brasília.
• Silva, V. B. S., Schramm, F., & Carvalho, H. R. C. (2014). O uso do método PROMETHEE para seleção de candidatos à bolsa-formação do Pronatec. Production, 24(3), 548-558.
• Smit, J. W., Dias, E. W., & Souza, R. F. (2002). Contribuição da Pós-graduação para a Ciência da Informação no Brasil: uma visão. DataGramaZero: Revista de Ciência da Informação, 3(6).
• Tuncer, Y. (1975). Some aspects of science-based technological promotion. Policy Sciences, 6(3), 267-280. http://dx.doi.org/10.1007/BF00139970
  » http://dx.doi.org/10.1007/BF00139970
• Vargas, R. V. (2010). Utilizando a programação multicritério (Analytic Hierarchy Process – AHP) para selecionar e priorizar projetos na gestão de portfólio. In PMI Global Congress 2010. Washington. Recuperado em 12 de janeiro de 2016, de http://www.ricardo-vargas.com/articles/analytic-hierarchy-process/#portuguese
  » http://www.ricardo-vargas.com/articles/analytic-hierarchy-process/#portuguese
• Vincke, P. (1992). Multicriteria decision-aid. England: John Wiley & Sons.

Publication Dates

• Publication in this collection
  23 Feb 2017
• Date of issue
  Jan-Apr 2017

History

• Received
  24 Mar 2015
• Accepted
  03 Oct 2015