VALUE FOCUSED PERFORMANCE INDICATORS FOR GRADUATE PROGRAMS

ABSTRACT

Performance assessment is addressed in the literature as means for continuous improvement. The objective of this study was to identify the most valued performance criteria for a Graduate Program of Biotechnology in a University of Brazil through the Value Focused Thinking (VFT) approach. The choice of the VFT was due to the holistic vision provided by it and the focus on the identification of organizational values. Among the objectives found, it is possible to notice an expansion of the vision of performance that is more focused on publications regarding the regulatory body Coordination of Improvement of Higher Level Personnel (CAPES), and contemplating among others the quality in the student’s training as evidence of other stakeholders. The VFT approach met the established assumptions, efficiently allowing strategic, fundamental and media objectives to be identified, as well as measurable attributes to be used by the program to enable more efficient management.

Keywords:
performance; VFT; graduate studies; biotechnology

1 INTRODUCTION

Performance assessment is approached in the literature as an efficient way to seek continuous improvement and to help the competitiveness of organizations. This is not a trivial task, since it involves specific characteristics that each company or sector has. According to ^{Sobreira Netto (2007)} and ^{Neely (1999}, ²⁰⁰⁵), monitoring the performance of organizations is a recurring concern.

According to ^{Choong (2018}), few authors describe how measurement attributes were constructed. According to ^{Bititci et al. (2012}), a properly designed and structured performance measurement system provides an effective basis for the performance management system, the first being used as a management tool.

Internally, Higher Education Institutions - HEIs - generally do not use methods in order to promote the monitoring and improvement of their performance (^{BRESSIANI et al., 2001}, ^{MODELL, 2005} and UMASHANKAR and DUTTA, 2007). Given the importance of Graduate Programs - GP, both for universities and for society in general, it is essential that performance be measured so that it is possible to structure a management process in search of continuous improvement.

Moritz, Moritz and Melo (2011) add that the country is far from having an education system capable of sustaining the innovation needed to position itself in the world market. ^{Marttunen, Lienert and Belton (2017}) highlight the development of the VFT among the problems structuring methods (PSMs), since it is a holistic approach to support the identification and structuring of the values of decision makers and objectives, as well as the creative generation and evaluation of alternatives.

The objective of this work is to identify the most valued performance criteria for a Graduate Program of Biotechnology in Brazil through the VFT approach so that it can better manage its performance and promote the continuous improvement of the needs of all stakeholders involved. A GP in Biotechnology is multidisciplinary and has, according to the recommendations of ^{CAPES (2016)}, an emphasis on planning and management, which is in line with the proposal of a performance measurement system.

After the introduction, section two follows, with the theoretical framework on PSMs, the main existing methods and the VFT approach used as PSM. In section three, we approach the origins of Graduate Programs in Brazil, followed by the research-action in section four, applying the VFT in the GP in Biotechnology, object of this study, and followed lastly by the final considerations in the fifth and last section.

2 PROBLEM STRUCTURING METHODS AND MEASURING PERFORMANCE

According to Almeida (2013), a multicriteria decision problem is comprised of at least two alternatives of action to choose in a situation, where this choice is driven by the desire to meet multiple objectives, which may be conflicting and associated with the consequences of choice of the alternative to be followed. According to the same author, several decisions are taken daily in organizations with or without formal methods of decision support, and the main concern is the impact of such decisions, i.e. their consequences. The problem structuring methods meet this concern, aiming to provide better conditions for decision-making.

The research and practice of PSM have produced a series of approaches since the 1970s, such as Strategic Options Development and Analysis (SODA), Soft Systems Methodology (SSM), Strategic Choice Approach (SCA), Robustness Analysis,Drama Theory, Group Model Building and VFT (^{ACKERMANN et al., 2014}).

For ^{Mingers and Rosenhead (2004}) and ^{Mingers (2011)}, the most used PSMs are SODA, SSM and SCA. ^{Mingers (2011)} presents the main PSMs (SODA, SSM and SCA), mentioning the VFT and several others as approaches used for this purpose and adds that PSM are no longer only focused on structuring problems, there are many examples of soft methods also seeking to solve them.

In this research, we opted for using the VFT approach detailed in the sub item below. Studies published with the use of VFT have proved to be a more objective approach in structuring problems and encourage their use.

According to ^{Keisler et al. (2014}), in its design and application, VFT is consistent with the motivations and practice of PO soft. The emphasis on identifying organizational values is what differentiates VFT among the existing problem-structuring methods. Focusing on values not only broadens the decision scope and the number of alternatives to be considered but also promotes a more strategic thinking about how such decisions can influence the achievement of multiple goals. VFT goes beyond most PSMs, linking objectives to specific actions with measurable results (^{KEISLER, 2012}).

Although there is no shortage of publications on performance measurement systems, there is a paucity of research on how to measure them and, more precisely, what activities contribute to performance using an appropriate measurement theory (^{CHOONG, 2018}).

Performance measurement models give little guidance on how business performance indicators can be chosen and operationalized (^{SHAH et al., 2012}). Such models, according to ^{Van Looy and Shafagatova (2016)}, suffer from the lack of guidance on the performance indicators themselves and how they can be used in practice.

Choong (2013 and ²⁰¹⁴) conducted a scientific survey, such as: ABI / Inform ProQuest, Emerald Full Text, Science @ Direct and EBSCO from 1990 to 2012, and noticed a shortage of articles exploring the attributes required for a PMS.

2.1 Value Focused Thinking

For ^{Keeney (1996}), values are fundamental to everything we do, being the driving force for making a decision. Therefore, they should be the basis for the time and effort we spend thinking of decisions. However, in practice, decision-making often translates as a choice between alternatives. To consider that is to adopt the opposite path, since it ranks the identification of alternatives above the articulation of values. Values are fundamental in any decision situation; on the other hand, the alternatives are only relevant as a means to achieve their values. Thus, one must focus on the values and then the alternatives that can reach them.

A decision maker structures their goals, and is able not only to compare ready-made alternatives, but to create alternatives that were not apparent early in the process but that have become desirable and feasible within the framework of goals and values. In group decision situations, the creation of alternatives using a systematic approach is of great interest, since the alternatives must be based on stakeholder values, since their values are the reason for their interest in the matter of decision (^{HASSAN, 2004}).

^{Kim and Ahn (1999}) corroborate that it is very difficult for a single decision maker to consider all the important aspects of a problem in view of the complexity of socioeconomic environments. Therefore, most decisions are made by a group of people.

According to ^{Keeney (1996} and ²⁰⁰⁷), the VFT implementation procedures are based on the following concepts: Identification of objectives; Structure objectives; Creation of alternatives; and Decision opportunities. ^{Keeney (1992)} explains that the fundamental objectives are the essential reason of interest in the decision situation and the means objectives influence the degree to which the fundamental objectives can be achieved.

According to Almeida (2013), the VFT allows a clear structuring for the objectives, allowing them to be reached through three factors: decision context, an object and a direction of preference. He further emphasizes that the decision context is specified by the activity contemplated, and must be compatible with the fundamental objectives in the structuring of a decision situation. The values of the decision makers are contained in the objectives, the set of objectives are the basis of interest of any decision. The objectives condition what is of interest in the decision in a qualitative way and serve as a guide for any quantitative approach to be performed.

The Strategic Objectives, according to ^{Keeney (1992}), are fundamental objectives that guide the making of all the organizational decisions and are used to make decisions at the strategic level of an organization. According to ^{Keeney (2007)}, for an analysis in terms of objectives, an attribute to measure the achievement of each goal is necessary. Good attributes are essential for insightful analysis.

Attributes are a way to measure goals and see if they are being achieved. This is not an easy task in the use of VFT, requiring integration of decision makers to identify them. In examples presented by the author, this task is carried out after identifying the fundamental objectives, taking advantage of the knowledge of those involved to verify measurable attributes for the fundamental objectives listed. The terms performance measure, criterion, and metric are often used as synonyms (^{KEENEY, 2007}).

The attributes can be of three types, according to ^{Keeney (1992}): natural, constructed and proxy. The natural attributes are of general use and have a common interpretation, easily related to the objective one wishes to measure. Proxy attributes have characteristics similar to natural ones, usually involving a scale that is commonly used that can be counted or physically measured. The difference is that they do not directly measure the purpose of the concern. A constructed attribute, according to ^{Keeney (2007)}, is sometimes developed to directly measure the achievement of a goal when no natural attribute exists.

^{Parnell et al. (2013}) reviewed the scope and magnitude of VFT applications and the main developments in theory since the VFT was introduced in 1992 and found 89 articles written in 29 journals from 1992 to 2010. Of the articles studied, 66% are directed towards application, 34% towards theory and 18% towards case studies. Of these 89 articles, 66 were classified as decision ones, in which 45% of them pertained to opportunity for decision, 26% to a single decision and 29% to a portfolio. The applications were classified by domain of the problem, with defense being the largest with 46%, environment and energy was the second largest with 19%, and the corporate applications of VFT were only 9% of the articles. This can be explained by the disincentives to publish results that create value for the company, according to the authors.

The total sum of the citations was over 1600, with 42% of the citations for 10 articles by Keeney, and ^{Belton et. al. (1997)} is the most cited article except Keeney. VFT applications and several useful research contributions were found, as well as an increasing number of VFT papers written by international experts that encourage its use.

Table 1 presents publications on VFT applications obtained through a bibliographic review done between April 2017 and October 2017 on the main scientific bases (IEEE Xplore, Scielo, SpringerLink, Emerald insight, Science Direct, Wiley InterScience, SAGE Journals Online, GALE CENGAGE Learning) and in national publications on the subject. VFT is useful in many different decision contexts, helping to identify important goals, those previously unrecognized, and providing a logical and consistent way of identifying the relationships between objectives.

Table 1
- VFT applications

Year	Authors	VFT approach application
1992	Keeney and McDaniels	Selection of strategic objectives for BC Hydro.
1999	McDaniels and Trousdale	Tourism Planning in Guimaras, Philippines.
2001	Keeney	To build a Value Model for Telecommunication Management Decisions.
2001	Keeney and McDaniels	To develop a set of goals for the US government’s climate change policy decisions.
2001	Arvai, Gregory and McDaniels	The risks to the riparian area salmon habitat in a hydroelectric plant.
2001	Gregory, Arvai and McDaniels	Environmental risk management.
2004	Hassan	The comparative implications of building materials such as wood, masonry and concrete.
2004	Jurk,Chambal and Thal	To identify innovative ideas to improve the Air Force’s ability to perform its core competency.
2004	Merrick and Garcia	To improve river basin quality in Richmond, Virginia.
2004	Kajanus, Kangasb and Kurttilac	Tourism management: fundamental objectives and means for the vitality of rural areas.
2005	Sheng, Nah and Siau	Strategic implications of mobile technology in a leading publishing company.
2005	Merrick et al.	To understand the safety decisions made by domestic tanker operators.
2008	Peharda and Hunjak	Selection of an automatic rifle for the Croatian Armed Forces.
2010	Barclay and Osei-Bryson	Selection of important criteria for the development of Information Systems.
2010	Keeney and Winterfeldt	To identify and structure the goals of terrorists.
2010	Sheng, Nah and Siau	To understand the values of education available through mobile technology and use these values as guidelines for implementing use in education.
2011	Alencar, Mota and Alencar	Disposal of plaster waste in construction sites.
2011	Keeney and Winterfeldt	To reduce the costs of terrorism by developing a comprehensive set of internal security objectives.
2011	Selart and Johansen	Comparison of the VFT and AFT methods (Ideas Focused on Alternatives) to generate ideas,together with 70 Human Resources employees.
2012	Keeney	World Trade Center (Evacuation Area) Federal Safety Report.
2013	Keeney	Different internal security risks and to evaluate potential customers for American Express cards.
2013	Lopes and Almeida	Selection of portfolio projects in the oil and gas exploration area.
2013	May, Dhillon and Caldeira	Planning and implementation of Enterprise Resource Planning - ERP.
2013	Morais et al.	To discuss the application of VFT in Brazil to three problems in different contexts: water management, information/information technology (IS/IT), strategic planning, and the elimination of plaster waste.
2014	Almeida, Morais and Almeida	Sale price in a Manipulation Pharmacy - South of Brazil.
2014	Keisler et al.	Community Development Corporations.
2014	Simon, Regnier and Whitney	The US Department of Defense (DoD) has identified its energy requirements as a keyvulnerability, the article provides for the identification of goals and associated definitions to facilitate horizontaland vertical communications operations within the DoD.
2014	Vieira and Duarte	To propose alternatives that help the rural region of Pernambuco dairy to reach some level of economic, social and environmental sustainability.
2015	Poleto et al.	To identify and implement information security policies.
2015	Reichert, Langhans and Schuwirth	To support to the environmental decision, a didactic case study on the prioritization of the spatial rehabilitation of the rivers.
2015	Siebert and Keeney	Creation of quality alternatives, a study of five involving concrete decisions of substantial importance for the participants involved.
2015	Urtiga and Morais	Conflicts involving the use of water as a scarce resource. VFT used in the preliminary stagesof the negotiation to enable the creation of values among the negotiators.
2016	Bezerra Junior, Culha Filho and Cavalcante Junior	To structure the prioritization of which technical courses will be offered by a Technical and Vocational Education Institution of Rio Grande do Norte.
2016	Kunz, Siebert and Mütterlein	Method for strategic management based on the combination of the Balanced Scorecard with VFT, case study in Nordbayerischer Kurier, a German regional newspaper.
2016	Paiva and Daher	Cleaner production practices in a garment company in the rural region of Pernambuco.
2017	Alencar, Priori Jr and Alencar	Sustainability in the construction environment.
2017	Andrade et al.	Paraíba River Basin Committee, to mitigate problems of the water crisis.
2017	Coelho	To identify policies to promote the active aging of industrial workers.
2018	Bernardo, Gaspar and Antunes	To assess the energy efficiency of school buildings.

Specifically on performance assessment, ^{Barclay and Osei-Bryson (2010}) propose a method for developing a set of performance criteria or objectives based on the views of project stakeholders and associated measures aligned to those goals. The project performance framework is primarily based on the principles and advantages of Value-Oriented Thinking (VFT) and Goal Question Metrics (GQM) to obtain and develop these performance criteria and measures associated with the values of the project stakeholders. Three project cases are used to illustrate and evaluate the model.

^{Kibira et al. (2018}) address the issue of what to measure in order to assess environmental sustainability in the process-level industry. The scope of work includes identification, definition, selection and composition of environmental performance indicators for processes. The objectives of their work are to provide a guide for the industry to identify environmental indicators from existing sources, to define new environmental indicators, to select the most effective indicators based on criteria and to compose a final set of environmental indicators. The VFT is used in identifying values and experts on the subject matter build these. The assignment of the value function for quantification of values is also applied. A final set of criteria is obtained after further review by stakeholders.

3 GRADUATE PROGRAMS IN BRAZIL

According to ^{Balbachevsk (2005)}, graduate programs arose in Brazil in the early 1930s, when the first Brazilian universities attracted some foreign teachers who brought an institutional model with this level of education.

According to ^{Santos (2003)}, the term “graduate school” was used for the first time in the 1940s, in Article 71 of the Statute of the University of Brazil. In the 1950s, accords were signed between the United States and Brazil that involved a series of agreements between US and Brazilian schools and universities with the exchange of students, researchers and professors.

The National Campaign for the Improvement of Higher Education Personnel (CAPES), the regulatory body of the GP, was created in 1951, as it can be seem in its website (^{CAPES, 2017}) to “ensure the existence in sufficient quantity and quality of specialized personnel to meet the needs of public and private enterprises aimed at the development of the country”.

The activities of CAPES can be grouped in the following lines of action, each one developed by a structured set of programs: Evaluation of stricto sensu GPs; Access and dissemination of scientific production; Investments in the formation of high-level resources in the country and abroad; Promotion of international scientific cooperation; Induction and promotion of the initial and continuous training of teachers for basic education, in its classroom or distance formats.

According to the CAPES website (2017), the Evaluation of the National Graduate System is an essential activity to ensure and maintain the quality of Master’s and PhD courses in the country. The objectives of the Evaluation are: Certification of the quality of the Brazilian graduate course (reference for the distribution of grants and resources for the promotion of research); Identification of regional asymmetries and strategic areas of knowledge in the National Graduate System - SNPG - to guide induction actions in the creation and expansion of graduate programs in the national territory.

The goals of the National Graduate System - SNPG are: Graduate training of teachers for all levels of education; Training of qualified human resources personnel for the non-academic market; Strengthening the scientific, technological and innovation bases.

The Biotechnology area was created by CAPES in 2008, with the purpose of stimulating technological development and transferring generated knowledge, in order to contribute to the country’s increased competitiveness and to the generation of innovation products and processes in the areas of environment, health, agriculture and industry. It is a multidisciplinary area that generates opportunities for professionals to act both in scientific and technological fields (^{CAPES, 2016}).

According to ^{Pereira and Schenberg (2004}), the multidisciplinary and interinstitutional ethos that is particular to Biotechnology is not usual among graduate programs and, therefore, a program that contemplates it does not often fit into the current norms of evaluation and regulation of GPs.

^{Rossi (2012}) states that biotechnology can be defined as a technology that contemplates several productive segments with different areas of knowledge. The development of biotechnology is considered complex, and this stimulates the formation of cooperation networks and the involvement of several actors in a large part of the sectors that it comprises. Biotechnology is seen as a new paradigm and brings to the center of innovation process analysis interactive knowledge based on cooperation, competition, and feedback, through the simultaneous involvement of heterogeneous agents that seek to share, achieve and develop knowledge and may result in new products and processes.

According to the CAPES website (²⁰¹⁶), for Biotechnology the following weight structure is used for evaluation.

• Program proposal: compulsory, but weighing 0 in the evaluation.

• Teachers: 20% weight in the evaluation.

• Student Body, Thesis and Dissertations: 30% weight in the evaluation.

• Intellectual Production: 40% weight in the evaluation.

• Social Inclusion: 10% weight in the evaluation.

Considering that the object of this study is a GP in Biotechnology, it is of fundamental importance to verify what CAPES, as a stakeholder of this process, expects from the program, and how it is evaluated. It is possible to verify that the performance indicators of greater weight are linked to the intellectual production, with 40% of weight in the evaluation of CAPES, although several other items are contemplated in the area document and mentioned in the evaluation proposal.

Aspects such as the need for planning and management of the program aiming at future development, the international challenges of the area in the production of knowledge, as well as purposes in the better training of its students are contemplated in the proposal of the program and considered as prerequisite, having no weight in the evaluation.

In addition to the aforementioned aspects, it should be mentioned that the new CAPES evaluation proposal, highlighting aspects related to the strategic planning of the program and self-evaluation, already validates the need for an internal PMS and the characteristics of the proposed model differentiate it in the process, since they allow for an evaluation according to the peculiarities and size of the program, as well as a process prior to the request of CAPES, the reflection of a work valued by the coordination of the program, before becoming a requirement.

4 METHODOLOGY

As the objective of this work is to identify the most important performance criteria according to its stakeholders for GP through the application of the PSM of referenced use in the literature, the VFT approach, its application was structured following the steps below, according to figure 1.

Figure 1
- Research-action Steps.

The first step is to identify the stakeholders of the organization, the ones interested in the operation; and the decision-makers, that is, those who hold the power of decision, involved in the construction of the performance measurement system (PMS). It is important for everyone to be identified so that their needs and goals can be regarded in the process.

^{Ackermann and Eden (2011}) put forth three themes regarding the strategic management of decision-makers: stakeholder identification, stakeholder relationship dynamics and stakeholder strategies. They proposed an identification and classification of stakeholders in a diagram of power and interest, divided into four quadrants, as seen in Figure 2. With reference to such classification the stakeholders and decision makers will be selected for this work.

Figure 2
- Diagram for identification and classification of stakeholders.

According to ^{Coelho (2017}), the selection of stakeholders is an important step in the decision-making process, since it is necessary to identify people who have the power to define the fundamental and strategic objectives and still make the decision. Stakeholders will be involved in steps two (goal identification), three (goal structuring) and four (network building); decision makers, meanwhile, will be involved in step five (identification of measurable attributes).

In the second step, the application of the VFT with the Value Identification/Wish List is initiated, in which individual brainstorming takes place through the VFT approach with the stake-holders and decision makers, in which the objectives will be identified according to the values [200B?][200B?]of each stakeholder/decision maker.

The third step is to elaborate the Hierarchy of Objectives through the WITI test with the statements identified in the previous item (one by one), ranking the fundamental objectives.

The fourth step is building the Network of Goals. After the hierarchization of the objectives, the facilitator elaborates the list of congregated desires and structures a network of objectives with the average and fundamental objectives.

Finally, in the fifth step the list of measurable attributes that will be the performance criteria and subcriteria will be composed, validated by the decision makers.

5 APPLICATION OF THE PROPOSED METHOD

The object of study defined for this work is a Graduate Program in Biotechnology at an private HEI in the State of Sa˜o Paulo, Brazil. This program was implemented in 2015 and the first group of masters and doctoral candidates enrolled in the second semester of 2015. It has 11 permanent teachers and 4 collaborating teachers from other institutions.

The first step, as seen in Figure 2, aims to identify the stakeholders and decision makers involved in the construction of the PMS. According to a survey conducted by ^{Pereira e Silva (2003}), the most cited stakeholders in work on Higher Education are: students, organizations, educational institution and society/government. We used this survey and the diagram of ^{Ackermann and Eden, (2011}) as references for identification of stakeholders and decision makers for the Graduate Program.

According to the diagram of ^{Ackermann and Eden (2011}),“players”, “subjects”, “regulators” and “crowd” are all stakeholders involved (with a high degree of interest and power, high power and low interest and low interest and power, respectively), and again the “players” will be the decision makers who have a high degree of power to support or not the strategies of the HEI in relation to the GP.

With regard to stakeholders:

• Representing the “Organizations” group: a professor who also worked at a Biotechnology company and an entrepreneur in the Biotechnology area without relations with the IES were interviewed;

• Representing the “Educational Institution” group: for a broader vision of the PMS, between coordinator, teachers, student and management representative were placed in different groups;

• Representing the “Society/government” group: a person from the society linked to the academic environment was interviewed, working at a Graduate Program.

For the creation of the PMS, it was defined that the collegiate (composed of the coordinator, teachers and students of the program) is the main decision-maker, in view of the involvement of the members in the decision-making and specific connection to the program. These represent the “players” (according to ^{Ackermann and Eden, 2011}), that is, the larger stakeholders and with greater decision power within the GB and who are able to assist in building an PMS of effective utility to the program.

In order to identify the stakeholders’ criteria, representatives of the following groups were selected: Organizations, Institution of Education and Society were interviewed, totaling 8 interviews with an average duration of 30 minutes which were validated after transcription with the stakeholders. The steps are two, three and four.

The decision-makers, who make up the collegiate and are among the stakeholders, represented by 6 members (program coordination, teachers and student representative) are involved in stage five.

The VFT application process in this work was structured in steps two through five: Identification of values/List of objectives, Hierarchy of objectives, Network of objectives and List of measurable attributes.

The second step is the Identification of values/Wish List. The guiding question asked during the interviews was: What do you consider important to be measured/evaluated in a GP? Also asked were the questions from the ^{Keeney questionnaire (1992}), to stimulate the identification of objectives, in view of the purpose of the application is the construction of an PMS for a Graduate Program in Biotechnology.

1. A wish list. What do you want? What do you value? What should you want?

2. Alternatives. What constitutes a perfect alternative, a terrible alternative, or a reasonable alternative? What is good or bad about each one?

3. Problems and deficiencies. What is wrong or right with your organization? What needs to be adjusted?

4. Consequences. What happened that was good or bad? What can happen that worries you?

5. Goals. Restrictions and guidelines. What are your aspirations? What are the limitations?

6. Different perspectives. Who would your competitor or an interested party be? At some point in the future, what would interest you?

7. Strategic objectives. What are your ultimate goals? What are your absolutely fundamental values?

8. General objectives. What goals do you have for your customers, employees, shareholders, yourself? Which are: environmental, social, economic or health and safety are important?

9. Structuring of objectives. Follow the means-ends relationships: why is this goal important, how can you reach it? Use specification: what do you mean by this purpose?

10. Quantification of objectives. How will you measure the attainment of that goal? Why is goal A three times more important than goal B?

In the third step, individual hierarchies were developed for each interviewee and this was gathered in a single validated list with the stakeholders involved. The WITI test was applied with the statements identified in step 2 (one by one). An individual hierarchy of the objectives presented by each interviewee is structured for them. The strategic objective found is to fulfill the needs of those involved in the GP.

In the fourth step, the Network of objectives is presented. After the ranking of the objectives, according to table 2, in a single list, the network of strategic, fundamental and medium objectives was development and validated with the stakeholders.

Table 2
- Objectives Hierarchy

1.	To provide quality training to the student
1.1	To determine knowledge (the student)
1.2	To generate more ideas (the student)
1.3	To improve management skills (the student)
1.4	Improving Quality of Progress
1.5	To expand the teaching / guidance capacity (of the student)
1.6	To be able to conduct research (the student)
1.7	To meet deadlines (the student)
1.8	To increase the ability to improve one’s life (the student)
1.9	To improve collaboration in the organization in which it operates (the student)
2	To forge partnerships between GP and companies
2.1	To provide financial return to partner companies
2.2	To qualify employees of partner companies
2.3	To advise partner companies
3	To develop quality publications
3.1	To meet CAPES requirements for publications
3.2	To generate natural reflex of GP’s work with students and teachers
3.3	To develop useful research for society, with tangible results (products, publications or patents)
3.4	To develop research for problem solving
3.5	To develop research in internal partnerships
3.6	To broaden the generation of knowledge
4.	To build a quality teaching staff
5.	To have infrastructure for research development (internal and external investments)
6.	To establish a good relationship between those involved in GP
7.	To possess an assertive selection process
8.	Making the GP economically viable

It was verified that all the interviewees mentioned the Quality Training to the student as a fundamental objective. The publications that stand out as a requirement of CAPES, as a regulatory agency, also appear among the objectives presented, but in a more natural way, much more as a reflection of the work carried out by the GP than as a single goal to be achieved.

Objectives such as: the need for a partnership between the GP and quality companies and staff were also very important among the interviewees, followed by the economic viability and infrastructure for the GP.

The objectives presented demonstrate that the concerns and decisions in a GP reach far beyond simply cataloguing the publication indexes, they require the integration of the team involved and several different types of research in order for them to be published.

Then the network of objectives was elaborated with the strategic, fundamental and means objectives. This construction took place through analysis of the interviews and obtaining information from the decision-makers, who also validated it, as seen in Figure 3.

Figure 3
- Network of Objectives.

For the structured network, a list of measurable attributes was also elaborated, step five, through a discussion with the decision makers (Table 3) in function of the established fundamental objectives. The decision-makers were individually asked to indicate how we could measure the attributes presented, what measurable attributes would be used to monitor the goals pointed out in the network. A single list was drawn up and validated by the decision-makers. These represent the performance criteria pointed out by the decision makers and CAPES (following the requests of the area document) for the GP in Biotechnology object of this study.

Table 3
- Measurable attributes

Fundamental Objective/Attributes	Type of attribute	Formula to measure the attribute
1. Provide Education to the student
Meeting deadlines in a four-year period	Natural	Student Term for four years
Grades obtained in disciplines in a four-year period	Natural	Mentions attributed to students in a four-year period.
Participation / Presentation in Events, Lectures, External Workshops (National and International) per year	Natural	Number of student/teacher events per year
Masters and Doctors trained in a four-year period.	Natural	No. of masters and doctors trained in a four-year period.
Student awards received in a four-year period.	Natural	Numberofstudentawardsreceivedinafour-year period.(Qualitative)
Supervision/Co-supervision of undergraduate and graduate research in a four-year period	Natural	Number of student Supervision/Co-supervisions
2. Forge partnerships between GP and companies
Companies visited per year	Natural	Number of companies visited per year
Holding of Events, Workshops, lectures with companies per year	Natural	Number events with companies per year
Participation as an advisor or consultant per year	Natural	Number of participations as advisor/consultant per year
Students / teachers on projects involving company per year	Natural	Number of projects with companies per year
3. Develop quality publications
Publications in Impact Journals per year	Natural	Number of student/teacher publications per year
Patents/Products in a four-year period.	Natural	Number of student/teacher patents/products in a four-year period.
Books or book chapter in a four-year period	Natural	Number of student/teacher books or book chapter in a four-year period
Teacher publications in Impact Journals per year	Natural	Number of teacher publications per year
Teacher Patent/products per year	Natural	Number of teacher Patent/products per year
Books or book chapters written by Teachers in a four-year period	Natural	Number of books or book chapters written by Teachers in a four-year period
4. Build a Quality Faculty
Teachers integrated with the Undergraduate Department in a four-year period	Natural	Number of teachers integrated with the Undergraduate Department in a four-year period
Integrationprojectswithhighschoolsina four-year period	Natural	Number of integration projects with high schools in a four-year period
Number of students supervised in a four-year period	Natural	Number of students supervised in a four-year period
Number of teachers with grants in a four-year period	Natural	Number of Teachers with grants in a four-year period
Teachers’ courses offered in a four-year period	Natural	Number of teachers’ courses offered in a four-year period
Students’ Evaluation per year	Built	Student Evaluation Form regarding.
Projects aimed at social insertion, entrepreneurship and solidarity (university extension) in a four-year period	Natural	Number of extension projects in a four-year period
Projects or collaborations geared towards internationalization in a four-year period	Natural	Number of internationalization projects in a four-year period
Teacher awards received in a four-year period	Natural	Number of teacher awards received in a four-year period
5. H infrastructure for research development
Number of accessible scientific bases by the program	Natural	Number of accessible scientific bases by the program
Number of projects supported by research agencies (FAPESP, CNPq) and by companies in a four-year period	Natural	Number of projects with external support in a four-year period
6. Establish a good relationship between those involved in the GP
In-house Events, Lectures, and Workshops per year	Natural	Number of in-house Events per year
Joint productions/year	Natural	Number of Teacher joint productions/year per year
Joint supervisions/year	Natural	Number of teacher joint supervisions per year
Participations in Thesis/Dissertation Panels per year	Natural	Number of participations in Thesis/Dissertation Panels per year
Collaborators’ satisfaction	Built	Satisfaction level of GP collaborators per year
7. Harbor an assertive selection process
Number of enrollments/year	Natural	Number of enrollments per year
Graduates / New students (rate)	Natural	Number of graduates / Number of new students (four year period)
8. Make the GP economically viable
Total annual revenue (including grants and fees) x Annual total expenses	Natural	Tevenue x Expenses

Most measurable attributes are natural. Natural is a direct measure, Built is a defined scale for the attribute and Proxy is an indirect attribute that is either natural or built for a medium purpose (^{KEENEY, 1992}).

The practical and multidisciplinary nature of Biotechnology requires the development of products and patents, and this leads to the need to approach companies for partnerships. That allows for making applied research more efficient and able to give back more affordable products to society. This also applies to publications. Not that the “pure” searches are not necessary and valid, but, for those involved in the construction of this PMS, having more applied results was paramount.

It is necessary to make the goals measurable in order to check whether they are being reached and to draw action plans for that to happen. It should be noted that the measurement of performance is cyclical and that, with some objectives being achieved, others may arise, so the application of the VFT should occur also periodically to ensure that new objectives and alternatives can be identified.

Measuring is inherent in the continuous improvement process and will give the GP a better idea of “where it is” and “where it wants to go”. Just considering the requirements of the regulatory agency is not enough for the results needed for a GP, the work is broader. Listening to the stakeholders involved and verifying the performance objectives most valued by them, through the VFT, is the beginning of the construction of a PMS capable of attending to the particularities of each business, that team at that moment in time.

The built-in PMS encompasses the CAPES evaluation criteria and is complemented by others valued by the stakeholders / decision makers. It is possible to monitor the performance of the program over the four-year periods, outlining actions that reflect results within the evaluation period. The specific criteria pointed out in the PMS aid in the overall result of the program, and can indirectly generate positive impact on the CAPES criteria. Nevertheless, program coordination should be attentive to the CAPES performance criteria (weighting sheet) and their respective weightings, as they are frequently changed.

FINAL REMARKS

Proposing a method to design PMSs for Graduate Programs, in order to assist in their management and continuous improvement, is more than creating indicators, since CAPES already has them - although only partially, and, as a regulator, it dictates the “rules of the game”- but rather looking inside the program that proposes to go through this exercise (since it is a cyclical process) to check what their needs are, their choices and how to act so as not to be just another program in order to meet the requirements required, but different, by essence in pursuing its results as the reflection of a work based on strategy, planning and actions.

The use of the VFT approach was efficient in the identification of the performance criteria for the GP in Biotechnology, object of this study. The approach was able to meet PSM premises, allowing the identification of the strategic, fundamental and means objectives, assisting the GP in its action plan, as well as in the structuring of the measurable attributes capable of providing more support to the decision process, since each attribute aims at meeting the objectives set by the stakeholders themselves.

The application of the VFT as a PSM was encouraged by the authors cited in the theoretical framework section and, in an applied fashion, it was useful. For some of the interviewees the questions provided a deeper approach than for others, even though the questions designed to stimulate were carried out. There were difficulties among the decision-makers in understanding what the other had pointed out as a metric. This also allowed for debate on the qualitative and quantitative aspects of the criteria. Some of these interviewees informally justified their training in health and life sciences and did not feel at all comfortable to discuss the performance issues for a GP, which was something new, but also stimulated the need to seek a greater understanding with the proposal of this work.

It was possible to verify, by reviewing the CAPES area document, its prioritization with regard to publications/scientific production. The other stakeholders involved in the process highly value the Quality Education to the student, which is presented by all respondents in the application of the VFT; the respondents also mention publications, albeit with the argument that they should be useful for society with tangible results.

Objectives such as the need for partnerships with companies as a mutual form of assistance were also mentioned by several of the respondents, followed by quality faculty. The possibility of the GP being able to ascertain the results of such attributes provides a panorama of the reality of the program and opportunity for actions in search of continuous improvement, acting as a management tool.

The suggestion of this paper, already expected to be carried out by 2019, is the combination of the multicriteria decision support method Analytic Network Process - ANP, and through modeling multi-methodologies, identifying dependency and feedback relationships among such performance criteria, measure performance and management priorities - as proposed by ^{Piratelli and Belderrain (2010}).

ACKNOWLEDGEMENTS

This research has been financed in part by FUNADESP (Fundação Nacional de Desenvolvimento do Ensino Superior Particular). The first author would also thanks to CAPES (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior) for her PhD scholarship.

References

• ¹
  ACKERMANN F & EDEN C. 2011. Strategic management of stakeholders: Theory and practice. Long range planning, 44(3): 179-196.
• ²
  ACKERMANN F, FRANCO LA, ROUWETTE E & WHITE L. 2014. Special issue on problem structuring research and practice.
• ³
  ALENCAR LH, DE MIRANDA MOTA CM & ALENCAR MH. 2011. The problem of disposing of plaster waste from building sites: problem structuring based on value focus thinking methodology. Waste management, 31(12): 2512-2521.
• ⁴
  ALENCAR MH, PRIORI JR L & ALENCAR LH. 2017. Structuring objectives based on value-focused thinking methodology: Creating alternatives for sustainability in the built environment. Journal of cleaner production, 156: 62-73.
• ⁵
  ALMEIDA S, MORAIS DC & DE ALMEIDA AT. 2014. Agregação de pontos de vista de stakeholders utilizando o Value-Focused Thinking associado à mapeamento cognitivo. Production, 24(1): 144-159.
• ⁶
  ARVAI JL, GREGORY R & MCDANIELS TL. 2001. Testing a structured decision approach: value-focused thinking for deliberative risk communication. Risk Analysis, 21(6): 1065-1076.
• ⁷
  BALBACHEVSKY E. 2005. A pós-graduação no Brasil: novos desafios para uma política bem-sucedida. Os desafios da educação no Brasil Rio de Janeiro: Nova Fronteira, 1: 285-314.
• ⁸
  BARCLAY C & OSEI-BRYSON KM. 2010. Project performance development framework: An approach for developing performance criteria & measures for information systems (IS) projects. International Journal of Production Economics, 124(1): 272-292.
• ⁹
  BELTON V, ACKERMANN F & SHEPHERD I. 1997. Integrated support from problem structuring through to alternative evaluation using COPE and VISA. Journal of Multi-Criteria Decision Analysis, 6(3): 115-130.
• ¹⁰
  BERNARDO H, GASPAR A & HENGGELER ANTUNES C. 2018. A Combined Value Focused Thinking-Soft Systems Methodology Approach to Structure Decision Support for Energy Performance Assessment of School Buildings. Sustainability, 10(7): 2295.
• ¹¹
  BEZERRA JR G, CUNHA FILHO W & CAVALCANTE JR JRO. 2016. Estruturação e análise multiatributo para priorização da oferta de cursos técnicos em uma instituição de ensino profissionalizante.
• ¹²
  BITITCI U, GARENGO P, DÖRFLER V & MENDIBIL K. 2012. Performance measurement: questions for tomorrow. Advanced Production Management Systems, 14(3): 305-327.
• ¹³
  BRASIL, MINISTÉRIO DA EDUCAÇÃO. 2016. Sobre as áreas de avaliação. CAPES - Coordenação de Aperfeiçoamento de Pessoal de Nível Superior. Available at: Available at: http://www.capes.gov.br/images/documentos/Documentos_de_area_2017/BIOT_docarea_2016.pdf Accessed: 1 ago 2017.
  » http://www.capes.gov.br/images/documentos/Documentos_de_area_2017/BIOT_docarea_2016.pdf
• ¹⁴
  BRASIL, MINISTÉRIO DA EDUCAÇÃO. 2017. História e missão. CAPES - Coordenação de Aperfei`çoamento de Pessoal de Nível Superior. Available at: Available at: http://www.capes.gov.br/images/documentos/Documentos_de_area_2017/BIOT_docarea_2016.pdf Accessed: 1 ago 2017.
  » http://www.capes.gov.br/images/documentos/Documentos_de_area_2017/BIOT_docarea_2016.pdf
• ¹⁵
  BRESSIANI F, ALT PRC & MASSOTE AA. 2001. O Uso do Balanced Scorecard como Instrumento de Melhoria de Desempenho de uma Instituição de Ensino Superior. In: Congresso Brasileiro de Ensino de Engenharia
• ¹⁶
  CHOONG KK. 2014. Understanding the features of performance measurement system: a literature review. International Journal of Production Research, 52(14): 4174-4197.
• ¹⁷
  CHOONG KK. 2018. Use of mathematical measurement in improving the accuracy (reliability) & meaningfulness of performance measurement in businesses & organizations. Measurement, 129: 184-205.
• ¹⁸
  COELHO G. 2017. Identificação de políticas de seguranc¸a e saúde para a promoção do envelhecimento ativo dos trabalhadores da indústria utilizando value focused thinking Master’s thesis. Curso de Engenharia Aeronáutica e Mecânica, Área de Produção - Instituto Tecnológico de Aeronáutica.
• ¹⁹
  DE ALMEIDA AT. 2000. Processo de decisão nas organizações: construindo modelos de decisão multicritério Editora Atlas SA.
• ²⁰
  DE ANDRADE MRB, DE QUEIROZ NEIVA MT, SCHRAMM VB & SCHRAMM F. 2017. O uso do método VFT para apoiar um comitê de bacia hidrográfica na discussão de um problema complexo. XLIX Simpósio Brasileiro de Pesquisa Operacional,.
• ²¹
  DOS SANTOS CM. 2003. Tradições e contradições da pós-graduação no Brasil. Educação & sociedade, 24(83): 627-641.
• ²²
  EDEN C & ACKERMANN F. 2013. Making strategy: The journey of strategic management Sage.
• ²³
  GREGORY R, ARVAI J & MCDANIELS T. 2001. Value-focused thinking for environmental risk consultations. In: Environmental risks: Perception, evaluation and management pp. 249-273. Cambridge: Emerald Group Publishing Limited.
• ²⁴
  HASSAN OA. 2004. Application of value-focused thinking on the environmental selection of wall structures. Journal of environmental management, 70(2): 181-187.
• ²⁵
  JURK DM, CHAMBAL SP & THAL JR AE. 2004. Using value-focused thinking to select innovative force protection ideas. Military Operations Research, pp. 31-43.
• ²⁶
  KAJANUS M, KANGAS J & KURTTILA M. 2004. The use of value focused thinking and the A’WOT hybrid method in tourism management. Tourism management, 25(4): 499-506.
• ²⁷
  KEENEY GL & VON WINTERFELDT D. 2010. Identifying and structuring the objectives of terrorists. Risk Analysis: An International Journal, 30(12): 1803-1816.
• ²⁸
  KEENEY RL. 1996. Value-focused thinking: Identifying decision opportunities and creating alternatives. European Journal of operational research, 92(3): 537-549.
• ²⁹
  KEENEY RL. 2001. Modeling values for telecommunications management. IEEE Transactions on Engineering Management, 48(3): 370-379.
• ³⁰
  KEENEY RL. 2007. Developing objectives and attributes. Advances in decision analysis: From foundations to applications, pp. 104-128.
• ³¹
  KEENEY RL. 2012. Value-focused brainstorming. Decision Analysis, 9(4): 303-313.
• ³²
  KEENEY RL. 2013. Identifying, prioritizing, and using multiple objectives. EURO Journal on Decision Processes, 1(1-2): 45-67.
• ³³
  KEENEY RL & MCDANIELS TL. 1992. Value-focused thinking about strategic decisions at BC Hydro. Interfaces, 22(6): 94-109.
• ³⁴
  KEENEY RL & MCDANIELS TL. 2001. A framework to guide thinking and analysis regarding climate change policies. Risk Analysis, 21(6): 989-1000.
• ³⁵
  KEENEY RL & VON WINTERFELDT D. 2011. A value model for evaluating homeland security decisions. Risk Analysis: An International Journal, 31(9): 1470-1487.
• ³⁶
  KEISLER J. 2012. Is Value Focused Thinking a Problem Structuring Method or Soft OR or what? University of Massachusetts Boston. Available at: http://works.bepress.com/jeffrey_keisler/47
  » http://works.bepress.com/jeffrey_keisler/47
• ³⁷
  KEISLER J, TURCOTTE DA, DREW R & JOHNSON MP. 2014. Value-focused thinking for community-based organizations: objectives and acceptance in local development. EURO Journal on Decision Processes, 2(3-4): 221-256.
• ³⁸
  KIBIRA D, BRUNDAGE MP, FENG S & MORRIS K. 2018. Procedure for selecting key performance indicators for sustainable manufacturing. Journal of Manufacturing Science and Engineering, 140(1): 011005.
• ³⁹
  KIM SH & AHN BS. 1999. Interactive group decision making procedure under incomplete information. European Journal of Operational Research, 116(3): 498-507.
• ⁴⁰
  KUNZ RE, SIEBERT J & MÜTTERLEIN J. 2016. Combining value-focused thinking and balanced scorecard to improve decision-making in strategic management. Journal of Multi-Criteria Decision Analysis, 23(5-6): 225-241.
• ⁴¹
  LOPES YG & DE ALMEIDA AT. 2013. A multicriteria decision model for selecting a portfolio of oil and gas exploration projects. Pesquisa Operacional, 33(3): 417-441.
• ⁴²
  MARTTUNEN M, LIENERT J & BELTON V. 2017. Structuring problems for Multi-Criteria Decision Analysis in practice: A literature review of method combinations. European Journal of Operational Research, 263(1): 1-17.
• ⁴³
  MAY J, DHILLON G & CALDEIRA M. 2013. Defining value-based objectives for ERP systems planning. Decision Support Systems, 55(1): 98-109.
• ⁴⁴
  MERRICK JR & GARCIA MW. 2004. Using value-focused thinking to improve watersheds. Journal of the American Planning Association, 70(3): 313-327.
• ⁴⁵
  MERRICK JR, GRABOWSKI M, AYYALASOMAYAJULA P & HARRALD JR. 2005. Understanding organizational safety using value-focused thinking. Risk Analysis: An International Journal, 25(4): 1029-1041.
• ⁴⁶
  MINGERS J. 2011. Soft OR comes of age-but not everywhere! Omega, 39(6): 729-741.
• ⁴⁷
  MINGERS J & ROSENHEAD J. 2004. Problem structuring methods in action. European journal of operational research, 152(3): 530-554.
• ⁴⁸
  MODELL S. 2005. Students as consumers? An institutional field-level analysis of the construction of performance measurement practices. Accounting, Auditing & Accountability Journal, 18(4): 537-563.
• ⁴⁹
  MORAIS DC, ALENCAR LH, COSTA AP & KEENEY RL. 2013. Using value-focused thinking in Brazil. Pesquisa Operacional, 33(1): 73-88.
• ⁵⁰
  MORITZ GO, MORITZ MO & MELO PA. 2013. A Pós-Graduação brasileira: evolução e principais desafios no ambiente de cenários prospectivos. Future Studies Research Journal: Trends and Strategies, 5(2): 03-34.
• ⁵¹
  NEELY A. 1999. The performance measurement revolution: why now and what next? International journal of operations & production management, 19(2): 205-228.
• ⁵²
  NEELY A. 2005. The evolution of performance measurement research: developments in the last decade and a research agenda for the next. International Journal of Operations & Production Management, 25(12): 1264-1277.
• ⁵³
  PAIVA MLDUG & DAHER SDFD. 2016. Abordagem VFT para estruturação de problema para melhoria da produção mais limpa em empresa de confecções do agreste pernambucano. XLVIII Simpósio Brasileiro de Pesquisa Operacional, pp. 437-447.
• ⁵⁴
  PARNELL GS, HUGHES DW, BURK RC, DRISCOLL PJ, KUCIK PD, MORALES BL & NUNN LR. 2013. Invited Review-Survey of Value-Focused Thinking: Applications, Research Developments and Areas for Future Research. Journal of Multi-Criteria Decision Analysis, 20(1-2): 49-60.
• ⁵⁵
  PEHARDA I & HUNJAK T. 2008. Selecting an Automatic Rifle using the Valuefocused Thinking Approach. Military Operations Research, 13(3): 19-26.
• ⁵⁶
  PEREIRA CA & SCHENBERG ACG. 2004. Pós-graduação em Biotecnologia: êxitos e dificuldades de uma experiência interinstitucional. RBPG - Revista Brasileira de Pós-Graduação, 1(1): 101-110.
• ⁵⁷
  PEREIRA MAC & DA SILVA MT. 2003. A key question for higher education: Who are the customers. In: 31st Annual Conference of the Production and Operations Management Society
• ⁵⁸
  PIRATELLI C & BELDERRAIN MCN. 2010. Supporting the Design of a Performance Measurement System with the Analytic Network Process. International Journal of Analytic Hierachy Process, 2(1): 44-65.
• ⁵⁹
  POLETO T, SILVA LC, CARVALHO VDH, MOURA JA & COSTA APCS. 2003. Modelo de decisão para identificação e priorização de políticas de segurança de informação em um hospital público. In: Encontro Nacional de Engenharia de Producão - Perspectivas Globais para a Engenharia de Produção ABEPRO.
• ⁶⁰
  REICHERT P, LANGHANS SD, LIENERT J & SCHUWIRTH N. 2015. The conceptual foundation of environmental decision support. Journal of environmental management, 154: 316-332.
• ⁶¹
  ROSSI GM ET AL. 2012. Biotecnologia no Brasil: uma análise empírica a partir dos dados da PINTEC Master’s thesis. Universidade Federal de Uberlândia.
• ⁶²
  SELART M & JOHANSEN ST. 2011. Understanding the role of value-focused thinking in idea management. Creativity and Innovation Management, 20(3): 196-206.
• ⁶³
  SHAH L, ETIENNE A, SIADAT A & VERNADAT FB. 2012. (Value, Risk)-based Performance Evaluation of Manufacturing Processes. IFAC Proceedings Volumes, 45(6): 1586-1591.
• ⁶⁴
  SHENG H, NAH FFH & SIAU K. 2005. Strategic implications of mobile technology: A case study using Value-Focused Thinking. The Journal of Strategic Information Systems, 14(3): 269-290.
• ⁶⁵
  SHENG H, SIAU K & NAH FFH. 2010. Understanding the values of mobile technology in education: a value-focused thinking approach. ACM SIGMIS Database: the DATABASE for Advances in Information Systems, 41(2): 25-44.
• ⁶⁶
  SIEBERT J & KEENEY RL. 2015. Creating more and better alternatives for decisions using objectives. Operations Research, 63(5): 1144-1158.
• ⁶⁷
  SIMON J, REGNIER E & WHITNEY L. 2014. A value-focused approach to energy transformation in the United States Department of Defense. Decision Analysis, 11(2): 117-132.
• ⁶⁸
  SOBREIRA NETTO F. 2007. Medição de desempenho organizacional: um estudo das vantagens e desvantagens dos principais sistemas sob as óticas teórico-acadêmica e de práticas de mercado. In: Encontro da Anpad, 31 ., 2007, Rio de Janeiro ANPAD.
• ⁶⁹
  URTIGA MM & MORAIS DC. 2015. Pre-negotiation framework to promote cooperative negotiations in water resource conflicts through value creation approach. EURO Journal on Decision Processes, 3(3-4): 339-356.
• ⁷⁰
  VAN LOOY A & SHAFAGATOVA A. 2016. Business process performance measurement: a structured literature review of indicators, measures and metrics. SpringerPlus, 5(1): 1797.
• ⁷¹
  VIEIRA DT & DUARTE MDO. 2014. Value-focused thinking: encontrando alternativas sustentáveis para pequenos e médios laticínios do agreste pernambucano. In: Encontro Nacional de Engenharia de Producão - Engenharia de Produção, Infraestrutura e Desenvolvimento Sustentável: a Agenda Brasil+10, 2014 ABEPRO.

Publication Dates

• Publication in this collection
  2 Dec 2019
• Date of issue
  Sep-Dec 2019

History

• Received
  21 Feb 2019
• Accepted
  21 Oct 2019