ABSTRACT
Performance assessment is addressed in the literature as means for continuous improvement. The objective of this study was to identify the most valued performance criteria for a Graduate Program of Biotechnology in a University of Brazil through the Value Focused Thinking (VFT) approach. The choice of the VFT was due to the holistic vision provided by it and the focus on the identification of organizational values. Among the objectives found, it is possible to notice an expansion of the vision of performance that is more focused on publications regarding the regulatory body Coordination of Improvement of Higher Level Personnel (CAPES), and contemplating among others the quality in the student’s training as evidence of other stakeholders. The VFT approach met the established assumptions, efficiently allowing strategic, fundamental and media objectives to be identified, as well as measurable attributes to be used by the program to enable more efficient management.
Keywords:
performance; VFT; graduate studies; biotechnology
1 INTRODUCTION
Performance assessment is approached in the literature as an efficient way to seek continuous improvement and to help the competitiveness of organizations. This is not a trivial task, since it involves specific characteristics that each company or sector has. According to Sobreira Netto (2007)68 SOBREIRA NETTO F. 2007. Medição de desempenho organizacional: um estudo das vantagens e desvantagens dos principais sistemas sob as óticas teórico-acadêmica e de práticas de mercado. In: Encontro da Anpad, 31 ., 2007, Rio de Janeiro. ANPAD. and Neely (199951 NEELY A. 1999. The performance measurement revolution: why now and what next? International journal of operations & production management, 19(2): 205-228., 200552 NEELY A. 2005. The evolution of performance measurement research: developments in the last decade and a research agenda for the next. International Journal of Operations & Production Management, 25(12): 1264-1277.), monitoring the performance of organizations is a recurring concern.
According to Choong (201817 CHOONG KK. 2018. Use of mathematical measurement in improving the accuracy (reliability) & meaningfulness of performance measurement in businesses & organizations. Measurement, 129: 184-205.), few authors describe how measurement attributes were constructed. According to Bititci et al. (201212 BITITCI U, GARENGO P, DÖRFLER V & MENDIBIL K. 2012. Performance measurement: questions for tomorrow. Advanced Production Management Systems, 14(3): 305-327.), a properly designed and structured performance measurement system provides an effective basis for the performance management system, the first being used as a management tool.
Internally, Higher Education Institutions - HEIs - generally do not use methods in order to promote the monitoring and improvement of their performance (BRESSIANI et al., 200115 BRESSIANI F, ALT PRC & MASSOTE AA. 2001. O Uso do Balanced Scorecard como Instrumento de Melhoria de Desempenho de uma Instituição de Ensino Superior. In: Congresso Brasileiro de Ensino de Engenharia., MODELL, 200548 MODELL S. 2005. Students as consumers? An institutional field-level analysis of the construction of performance measurement practices. Accounting, Auditing & Accountability Journal, 18(4): 537-563. and UMASHANKAR and DUTTA, 2007). Given the importance of Graduate Programs - GP, both for universities and for society in general, it is essential that performance be measured so that it is possible to structure a management process in search of continuous improvement.
Moritz, Moritz and Melo (2011) add that the country is far from having an education system capable of sustaining the innovation needed to position itself in the world market. Marttunen, Lienert and Belton (201742 MARTTUNEN M, LIENERT J & BELTON V. 2017. Structuring problems for Multi-Criteria Decision Analysis in practice: A literature review of method combinations. European Journal of Operational Research, 263(1): 1-17.) highlight the development of the VFT among the problems structuring methods (PSMs), since it is a holistic approach to support the identification and structuring of the values of decision makers and objectives, as well as the creative generation and evaluation of alternatives.
The objective of this work is to identify the most valued performance criteria for a Graduate Program of Biotechnology in Brazil through the VFT approach so that it can better manage its performance and promote the continuous improvement of the needs of all stakeholders involved. A GP in Biotechnology is multidisciplinary and has, according to the recommendations of CAPES (2016)13 BRASIL, MINISTÉRIO DA EDUCAÇÃO. 2016. Sobre as áreas de avaliação. CAPES - Coordenação de Aperfeiçoamento de Pessoal de Nível Superior. Available at: Available at: http://www.capes.gov.br/images/documentos/Documentos_de_area_2017/BIOT_docarea_2016.pdf
. Accessed: 1 ago 2017.
http://www.capes.gov.br/images/documento...
, an emphasis on planning and management, which is in line with the proposal of a performance measurement system.
After the introduction, section two follows, with the theoretical framework on PSMs, the main existing methods and the VFT approach used as PSM. In section three, we approach the origins of Graduate Programs in Brazil, followed by the research-action in section four, applying the VFT in the GP in Biotechnology, object of this study, and followed lastly by the final considerations in the fifth and last section.
2 PROBLEM STRUCTURING METHODS AND MEASURING PERFORMANCE
According to Almeida (2013), a multicriteria decision problem is comprised of at least two alternatives of action to choose in a situation, where this choice is driven by the desire to meet multiple objectives, which may be conflicting and associated with the consequences of choice of the alternative to be followed. According to the same author, several decisions are taken daily in organizations with or without formal methods of decision support, and the main concern is the impact of such decisions, i.e. their consequences. The problem structuring methods meet this concern, aiming to provide better conditions for decision-making.
The research and practice of PSM have produced a series of approaches since the 1970s, such as Strategic Options Development and Analysis (SODA), Soft Systems Methodology (SSM), Strategic Choice Approach (SCA), Robustness Analysis,Drama Theory, Group Model Building and VFT (ACKERMANN et al., 20142 ACKERMANN F, FRANCO LA, ROUWETTE E & WHITE L. 2014. Special issue on problem structuring research and practice.).
For Mingers and Rosenhead (200447 MINGERS J & ROSENHEAD J. 2004. Problem structuring methods in action. European journal of operational research, 152(3): 530-554.) and Mingers (2011)46 MINGERS J. 2011. Soft OR comes of age-but not everywhere! Omega, 39(6): 729-741., the most used PSMs are SODA, SSM and SCA. Mingers (2011)46 MINGERS J. 2011. Soft OR comes of age-but not everywhere! Omega, 39(6): 729-741. presents the main PSMs (SODA, SSM and SCA), mentioning the VFT and several others as approaches used for this purpose and adds that PSM are no longer only focused on structuring problems, there are many examples of soft methods also seeking to solve them.
In this research, we opted for using the VFT approach detailed in the sub item below. Studies published with the use of VFT have proved to be a more objective approach in structuring problems and encourage their use.
According to Keisler et al. (201437 KEISLER J, TURCOTTE DA, DREW R & JOHNSON MP. 2014. Value-focused thinking for community-based organizations: objectives and acceptance in local development. EURO Journal on Decision Processes, 2(3-4): 221-256.), in its design and application, VFT is consistent with the motivations and practice of PO soft. The emphasis on identifying organizational values is what differentiates VFT among the existing problem-structuring methods. Focusing on values not only broadens the decision scope and the number of alternatives to be considered but also promotes a more strategic thinking about how such decisions can influence the achievement of multiple goals. VFT goes beyond most PSMs, linking objectives to specific actions with measurable results (KEISLER, 201236 KEISLER J. 2012. Is Value Focused Thinking a Problem Structuring Method or Soft OR or what? University of Massachusetts Boston. Available at: http://works.bepress.com/jeffrey_keisler/47.
http://works.bepress.com/jeffrey_keisler...
).
Although there is no shortage of publications on performance measurement systems, there is a paucity of research on how to measure them and, more precisely, what activities contribute to performance using an appropriate measurement theory (CHOONG, 201817 CHOONG KK. 2018. Use of mathematical measurement in improving the accuracy (reliability) & meaningfulness of performance measurement in businesses & organizations. Measurement, 129: 184-205.).
Performance measurement models give little guidance on how business performance indicators can be chosen and operationalized (SHAH et al., 201263 SHAH L, ETIENNE A, SIADAT A & VERNADAT FB. 2012. (Value, Risk)-based Performance Evaluation of Manufacturing Processes. IFAC Proceedings Volumes, 45(6): 1586-1591.). Such models, according to Van Looy and Shafagatova (2016)70 VAN LOOY A & SHAFAGATOVA A. 2016. Business process performance measurement: a structured literature review of indicators, measures and metrics. SpringerPlus, 5(1): 1797., suffer from the lack of guidance on the performance indicators themselves and how they can be used in practice.
Choong (2013 and 201416 CHOONG KK. 2014. Understanding the features of performance measurement system: a literature review. International Journal of Production Research, 52(14): 4174-4197.) conducted a scientific survey, such as: ABI / Inform ProQuest, Emerald Full Text, Science @ Direct and EBSCO from 1990 to 2012, and noticed a shortage of articles exploring the attributes required for a PMS.
2.1 Value Focused Thinking
For Keeney (199628 KEENEY RL. 1996. Value-focused thinking: Identifying decision opportunities and creating alternatives. European Journal of operational research, 92(3): 537-549.), values are fundamental to everything we do, being the driving force for making a decision. Therefore, they should be the basis for the time and effort we spend thinking of decisions. However, in practice, decision-making often translates as a choice between alternatives. To consider that is to adopt the opposite path, since it ranks the identification of alternatives above the articulation of values. Values are fundamental in any decision situation; on the other hand, the alternatives are only relevant as a means to achieve their values. Thus, one must focus on the values and then the alternatives that can reach them.
A decision maker structures their goals, and is able not only to compare ready-made alternatives, but to create alternatives that were not apparent early in the process but that have become desirable and feasible within the framework of goals and values. In group decision situations, the creation of alternatives using a systematic approach is of great interest, since the alternatives must be based on stakeholder values, since their values are the reason for their interest in the matter of decision (HASSAN, 200424 HASSAN OA. 2004. Application of value-focused thinking on the environmental selection of wall structures. Journal of environmental management, 70(2): 181-187.).
Kim and Ahn (199939 KIM SH & AHN BS. 1999. Interactive group decision making procedure under incomplete information. European Journal of Operational Research, 116(3): 498-507.) corroborate that it is very difficult for a single decision maker to consider all the important aspects of a problem in view of the complexity of socioeconomic environments. Therefore, most decisions are made by a group of people.
According to Keeney (199628 KEENEY RL. 1996. Value-focused thinking: Identifying decision opportunities and creating alternatives. European Journal of operational research, 92(3): 537-549. and 200730 KEENEY RL. 2007. Developing objectives and attributes. Advances in decision analysis: From foundations to applications, pp. 104-128.), the VFT implementation procedures are based on the following concepts: Identification of objectives; Structure objectives; Creation of alternatives; and Decision opportunities. Keeney (1992)33 KEENEY RL & MCDANIELS TL. 1992. Value-focused thinking about strategic decisions at BC Hydro. Interfaces, 22(6): 94-109. explains that the fundamental objectives are the essential reason of interest in the decision situation and the means objectives influence the degree to which the fundamental objectives can be achieved.
According to Almeida (2013), the VFT allows a clear structuring for the objectives, allowing them to be reached through three factors: decision context, an object and a direction of preference. He further emphasizes that the decision context is specified by the activity contemplated, and must be compatible with the fundamental objectives in the structuring of a decision situation. The values of the decision makers are contained in the objectives, the set of objectives are the basis of interest of any decision. The objectives condition what is of interest in the decision in a qualitative way and serve as a guide for any quantitative approach to be performed.
The Strategic Objectives, according to Keeney (199233 KEENEY RL & MCDANIELS TL. 1992. Value-focused thinking about strategic decisions at BC Hydro. Interfaces, 22(6): 94-109.), are fundamental objectives that guide the making of all the organizational decisions and are used to make decisions at the strategic level of an organization. According to Keeney (2007)30 KEENEY RL. 2007. Developing objectives and attributes. Advances in decision analysis: From foundations to applications, pp. 104-128., for an analysis in terms of objectives, an attribute to measure the achievement of each goal is necessary. Good attributes are essential for insightful analysis.
Attributes are a way to measure goals and see if they are being achieved. This is not an easy task in the use of VFT, requiring integration of decision makers to identify them. In examples presented by the author, this task is carried out after identifying the fundamental objectives, taking advantage of the knowledge of those involved to verify measurable attributes for the fundamental objectives listed. The terms performance measure, criterion, and metric are often used as synonyms (KEENEY, 200730 KEENEY RL. 2007. Developing objectives and attributes. Advances in decision analysis: From foundations to applications, pp. 104-128.).
The attributes can be of three types, according to Keeney (199233 KEENEY RL & MCDANIELS TL. 1992. Value-focused thinking about strategic decisions at BC Hydro. Interfaces, 22(6): 94-109.): natural, constructed and proxy. The natural attributes are of general use and have a common interpretation, easily related to the objective one wishes to measure. Proxy attributes have characteristics similar to natural ones, usually involving a scale that is commonly used that can be counted or physically measured. The difference is that they do not directly measure the purpose of the concern. A constructed attribute, according to Keeney (2007)30 KEENEY RL. 2007. Developing objectives and attributes. Advances in decision analysis: From foundations to applications, pp. 104-128., is sometimes developed to directly measure the achievement of a goal when no natural attribute exists.
Parnell et al. (201354 PARNELL GS, HUGHES DW, BURK RC, DRISCOLL PJ, KUCIK PD, MORALES BL & NUNN LR. 2013. Invited Review-Survey of Value-Focused Thinking: Applications, Research Developments and Areas for Future Research. Journal of Multi-Criteria Decision Analysis, 20(1-2): 49-60.) reviewed the scope and magnitude of VFT applications and the main developments in theory since the VFT was introduced in 1992 and found 89 articles written in 29 journals from 1992 to 2010. Of the articles studied, 66% are directed towards application, 34% towards theory and 18% towards case studies. Of these 89 articles, 66 were classified as decision ones, in which 45% of them pertained to opportunity for decision, 26% to a single decision and 29% to a portfolio. The applications were classified by domain of the problem, with defense being the largest with 46%, environment and energy was the second largest with 19%, and the corporate applications of VFT were only 9% of the articles. This can be explained by the disincentives to publish results that create value for the company, according to the authors.
The total sum of the citations was over 1600, with 42% of the citations for 10 articles by Keeney, and Belton et. al. (1997)9 BELTON V, ACKERMANN F & SHEPHERD I. 1997. Integrated support from problem structuring through to alternative evaluation using COPE and VISA. Journal of Multi-Criteria Decision Analysis, 6(3): 115-130. is the most cited article except Keeney. VFT applications and several useful research contributions were found, as well as an increasing number of VFT papers written by international experts that encourage its use.
Table 1 presents publications on VFT applications obtained through a bibliographic review done between April 2017 and October 2017 on the main scientific bases (IEEE Xplore, Scielo, SpringerLink, Emerald insight, Science Direct, Wiley InterScience, SAGE Journals Online, GALE CENGAGE Learning) and in national publications on the subject. VFT is useful in many different decision contexts, helping to identify important goals, those previously unrecognized, and providing a logical and consistent way of identifying the relationships between objectives.
Specifically on performance assessment, Barclay and Osei-Bryson (20108 BARCLAY C & OSEI-BRYSON KM. 2010. Project performance development framework: An approach for developing performance criteria & measures for information systems (IS) projects. International Journal of Production Economics, 124(1): 272-292.) propose a method for developing a set of performance criteria or objectives based on the views of project stakeholders and associated measures aligned to those goals. The project performance framework is primarily based on the principles and advantages of Value-Oriented Thinking (VFT) and Goal Question Metrics (GQM) to obtain and develop these performance criteria and measures associated with the values of the project stakeholders. Three project cases are used to illustrate and evaluate the model.
Kibira et al. (201838 KIBIRA D, BRUNDAGE MP, FENG S & MORRIS K. 2018. Procedure for selecting key performance indicators for sustainable manufacturing. Journal of Manufacturing Science and Engineering, 140(1): 011005.) address the issue of what to measure in order to assess environmental sustainability in the process-level industry. The scope of work includes identification, definition, selection and composition of environmental performance indicators for processes. The objectives of their work are to provide a guide for the industry to identify environmental indicators from existing sources, to define new environmental indicators, to select the most effective indicators based on criteria and to compose a final set of environmental indicators. The VFT is used in identifying values and experts on the subject matter build these. The assignment of the value function for quantification of values is also applied. A final set of criteria is obtained after further review by stakeholders.
3 GRADUATE PROGRAMS IN BRAZIL
According to Balbachevsk (2005)7 BALBACHEVSKY E. 2005. A pós-graduação no Brasil: novos desafios para uma política bem-sucedida. Os desafios da educação no Brasil. Rio de Janeiro: Nova Fronteira, 1: 285-314., graduate programs arose in Brazil in the early 1930s, when the first Brazilian universities attracted some foreign teachers who brought an institutional model with this level of education.
According to Santos (2003)21 DOS SANTOS CM. 2003. Tradições e contradições da pós-graduação no Brasil. Educação & sociedade, 24(83): 627-641., the term “graduate school” was used for the first time in the 1940s, in Article 71 of the Statute of the University of Brazil. In the 1950s, accords were signed between the United States and Brazil that involved a series of agreements between US and Brazilian schools and universities with the exchange of students, researchers and professors.
The National Campaign for the Improvement of Higher Education Personnel (CAPES), the regulatory body of the GP, was created in 1951, as it can be seem in its website (CAPES, 201714 BRASIL, MINISTÉRIO DA EDUCAÇÃO. 2017. História e missão. CAPES - Coordenação de Aperfei`çoamento de Pessoal de Nível Superior. Available at: Available at: http://www.capes.gov.br/images/documentos/Documentos_de_area_2017/BIOT_docarea_2016.pdf
. Accessed: 1 ago 2017.
http://www.capes.gov.br/images/documento...
) to “ensure the existence in sufficient quantity and quality of specialized personnel to meet the needs of public and private enterprises aimed at the development of the country”.
The activities of CAPES can be grouped in the following lines of action, each one developed by a structured set of programs: Evaluation of stricto sensu GPs; Access and dissemination of scientific production; Investments in the formation of high-level resources in the country and abroad; Promotion of international scientific cooperation; Induction and promotion of the initial and continuous training of teachers for basic education, in its classroom or distance formats.
According to the CAPES website (2017), the Evaluation of the National Graduate System is an essential activity to ensure and maintain the quality of Master’s and PhD courses in the country. The objectives of the Evaluation are: Certification of the quality of the Brazilian graduate course (reference for the distribution of grants and resources for the promotion of research); Identification of regional asymmetries and strategic areas of knowledge in the National Graduate System - SNPG - to guide induction actions in the creation and expansion of graduate programs in the national territory.
The goals of the National Graduate System - SNPG are: Graduate training of teachers for all levels of education; Training of qualified human resources personnel for the non-academic market; Strengthening the scientific, technological and innovation bases.
The Biotechnology area was created by CAPES in 2008, with the purpose of stimulating technological development and transferring generated knowledge, in order to contribute to the country’s increased competitiveness and to the generation of innovation products and processes in the areas of environment, health, agriculture and industry. It is a multidisciplinary area that generates opportunities for professionals to act both in scientific and technological fields (CAPES, 201613 BRASIL, MINISTÉRIO DA EDUCAÇÃO. 2016. Sobre as áreas de avaliação. CAPES - Coordenação de Aperfeiçoamento de Pessoal de Nível Superior. Available at: Available at: http://www.capes.gov.br/images/documentos/Documentos_de_area_2017/BIOT_docarea_2016.pdf
. Accessed: 1 ago 2017.
http://www.capes.gov.br/images/documento...
).
According to Pereira and Schenberg (200456 PEREIRA CA & SCHENBERG ACG. 2004. Pós-graduação em Biotecnologia: êxitos e dificuldades de uma experiência interinstitucional. RBPG - Revista Brasileira de Pós-Graduação, 1(1): 101-110.), the multidisciplinary and interinstitutional ethos that is particular to Biotechnology is not usual among graduate programs and, therefore, a program that contemplates it does not often fit into the current norms of evaluation and regulation of GPs.
Rossi (201261 ROSSI GM ET AL. 2012. Biotecnologia no Brasil: uma análise empírica a partir dos dados da PINTEC. Master’s thesis. Universidade Federal de Uberlândia.) states that biotechnology can be defined as a technology that contemplates several productive segments with different areas of knowledge. The development of biotechnology is considered complex, and this stimulates the formation of cooperation networks and the involvement of several actors in a large part of the sectors that it comprises. Biotechnology is seen as a new paradigm and brings to the center of innovation process analysis interactive knowledge based on cooperation, competition, and feedback, through the simultaneous involvement of heterogeneous agents that seek to share, achieve and develop knowledge and may result in new products and processes.
According to the CAPES website (201613 BRASIL, MINISTÉRIO DA EDUCAÇÃO. 2016. Sobre as áreas de avaliação. CAPES - Coordenação de Aperfeiçoamento de Pessoal de Nível Superior. Available at: Available at: http://www.capes.gov.br/images/documentos/Documentos_de_area_2017/BIOT_docarea_2016.pdf
. Accessed: 1 ago 2017.
http://www.capes.gov.br/images/documento...
), for Biotechnology the following weight structure is used for evaluation.
-
Program proposal: compulsory, but weighing 0 in the evaluation.
-
Teachers: 20% weight in the evaluation.
-
Student Body, Thesis and Dissertations: 30% weight in the evaluation.
-
Intellectual Production: 40% weight in the evaluation.
-
Social Inclusion: 10% weight in the evaluation.
Considering that the object of this study is a GP in Biotechnology, it is of fundamental importance to verify what CAPES, as a stakeholder of this process, expects from the program, and how it is evaluated. It is possible to verify that the performance indicators of greater weight are linked to the intellectual production, with 40% of weight in the evaluation of CAPES, although several other items are contemplated in the area document and mentioned in the evaluation proposal.
Aspects such as the need for planning and management of the program aiming at future development, the international challenges of the area in the production of knowledge, as well as purposes in the better training of its students are contemplated in the proposal of the program and considered as prerequisite, having no weight in the evaluation.
In addition to the aforementioned aspects, it should be mentioned that the new CAPES evaluation proposal, highlighting aspects related to the strategic planning of the program and self-evaluation, already validates the need for an internal PMS and the characteristics of the proposed model differentiate it in the process, since they allow for an evaluation according to the peculiarities and size of the program, as well as a process prior to the request of CAPES, the reflection of a work valued by the coordination of the program, before becoming a requirement.
4 METHODOLOGY
As the objective of this work is to identify the most important performance criteria according to its stakeholders for GP through the application of the PSM of referenced use in the literature, the VFT approach, its application was structured following the steps below, according to figure 1.
The first step is to identify the stakeholders of the organization, the ones interested in the operation; and the decision-makers, that is, those who hold the power of decision, involved in the construction of the performance measurement system (PMS). It is important for everyone to be identified so that their needs and goals can be regarded in the process.
Ackermann and Eden (20111 ACKERMANN F & EDEN C. 2011. Strategic management of stakeholders: Theory and practice. Long range planning, 44(3): 179-196.) put forth three themes regarding the strategic management of decision-makers: stakeholder identification, stakeholder relationship dynamics and stakeholder strategies. They proposed an identification and classification of stakeholders in a diagram of power and interest, divided into four quadrants, as seen in Figure 2. With reference to such classification the stakeholders and decision makers will be selected for this work.
According to Coelho (201718 COELHO G. 2017. Identificação de políticas de seguranc¸a e saúde para a promoção do envelhecimento ativo dos trabalhadores da indústria utilizando value focused thinking. Master’s thesis. Curso de Engenharia Aeronáutica e Mecânica, Área de Produção - Instituto Tecnológico de Aeronáutica.), the selection of stakeholders is an important step in the decision-making process, since it is necessary to identify people who have the power to define the fundamental and strategic objectives and still make the decision. Stakeholders will be involved in steps two (goal identification), three (goal structuring) and four (network building); decision makers, meanwhile, will be involved in step five (identification of measurable attributes).
In the second step, the application of the VFT with the Value Identification/Wish List is initiated, in which individual brainstorming takes place through the VFT approach with the stake-holders and decision makers, in which the objectives will be identified according to the values [200B?][200B?]of each stakeholder/decision maker.
The third step is to elaborate the Hierarchy of Objectives through the WITI test with the statements identified in the previous item (one by one), ranking the fundamental objectives.
The fourth step is building the Network of Goals. After the hierarchization of the objectives, the facilitator elaborates the list of congregated desires and structures a network of objectives with the average and fundamental objectives.
Finally, in the fifth step the list of measurable attributes that will be the performance criteria and subcriteria will be composed, validated by the decision makers.
5 APPLICATION OF THE PROPOSED METHOD
The object of study defined for this work is a Graduate Program in Biotechnology at an private HEI in the State of Sa˜o Paulo, Brazil. This program was implemented in 2015 and the first group of masters and doctoral candidates enrolled in the second semester of 2015. It has 11 permanent teachers and 4 collaborating teachers from other institutions.
The first step, as seen in Figure 2, aims to identify the stakeholders and decision makers involved in the construction of the PMS. According to a survey conducted by Pereira e Silva (200357 PEREIRA MAC & DA SILVA MT. 2003. A key question for higher education: Who are the customers. In: 31st Annual Conference of the Production and Operations Management Society.), the most cited stakeholders in work on Higher Education are: students, organizations, educational institution and society/government. We used this survey and the diagram of Ackermann and Eden, (20111 ACKERMANN F & EDEN C. 2011. Strategic management of stakeholders: Theory and practice. Long range planning, 44(3): 179-196.) as references for identification of stakeholders and decision makers for the Graduate Program.
According to the diagram of Ackermann and Eden (20111 ACKERMANN F & EDEN C. 2011. Strategic management of stakeholders: Theory and practice. Long range planning, 44(3): 179-196.),“players”, “subjects”, “regulators” and “crowd” are all stakeholders involved (with a high degree of interest and power, high power and low interest and low interest and power, respectively), and again the “players” will be the decision makers who have a high degree of power to support or not the strategies of the HEI in relation to the GP.
With regard to stakeholders:
-
Representing the “Organizations” group: a professor who also worked at a Biotechnology company and an entrepreneur in the Biotechnology area without relations with the IES were interviewed;
-
Representing the “Educational Institution” group: for a broader vision of the PMS, between coordinator, teachers, student and management representative were placed in different groups;
-
Representing the “Society/government” group: a person from the society linked to the academic environment was interviewed, working at a Graduate Program.
For the creation of the PMS, it was defined that the collegiate (composed of the coordinator, teachers and students of the program) is the main decision-maker, in view of the involvement of the members in the decision-making and specific connection to the program. These represent the “players” (according to Ackermann and Eden, 20111 ACKERMANN F & EDEN C. 2011. Strategic management of stakeholders: Theory and practice. Long range planning, 44(3): 179-196.), that is, the larger stakeholders and with greater decision power within the GB and who are able to assist in building an PMS of effective utility to the program.
In order to identify the stakeholders’ criteria, representatives of the following groups were selected: Organizations, Institution of Education and Society were interviewed, totaling 8 interviews with an average duration of 30 minutes which were validated after transcription with the stakeholders. The steps are two, three and four.
The decision-makers, who make up the collegiate and are among the stakeholders, represented by 6 members (program coordination, teachers and student representative) are involved in stage five.
The VFT application process in this work was structured in steps two through five: Identification of values/List of objectives, Hierarchy of objectives, Network of objectives and List of measurable attributes.
The second step is the Identification of values/Wish List. The guiding question asked during the interviews was: What do you consider important to be measured/evaluated in a GP? Also asked were the questions from the Keeney questionnaire (199233 KEENEY RL & MCDANIELS TL. 1992. Value-focused thinking about strategic decisions at BC Hydro. Interfaces, 22(6): 94-109.), to stimulate the identification of objectives, in view of the purpose of the application is the construction of an PMS for a Graduate Program in Biotechnology.
-
A wish list. What do you want? What do you value? What should you want?
-
Alternatives. What constitutes a perfect alternative, a terrible alternative, or a reasonable alternative? What is good or bad about each one?
-
Problems and deficiencies. What is wrong or right with your organization? What needs to be adjusted?
-
Consequences. What happened that was good or bad? What can happen that worries you?
-
Goals. Restrictions and guidelines. What are your aspirations? What are the limitations?
-
Different perspectives. Who would your competitor or an interested party be? At some point in the future, what would interest you?
-
Strategic objectives. What are your ultimate goals? What are your absolutely fundamental values?
-
General objectives. What goals do you have for your customers, employees, shareholders, yourself? Which are: environmental, social, economic or health and safety are important?
-
Structuring of objectives. Follow the means-ends relationships: why is this goal important, how can you reach it? Use specification: what do you mean by this purpose?
-
Quantification of objectives. How will you measure the attainment of that goal? Why is goal A three times more important than goal B?
In the third step, individual hierarchies were developed for each interviewee and this was gathered in a single validated list with the stakeholders involved. The WITI test was applied with the statements identified in step 2 (one by one). An individual hierarchy of the objectives presented by each interviewee is structured for them. The strategic objective found is to fulfill the needs of those involved in the GP.
In the fourth step, the Network of objectives is presented. After the ranking of the objectives, according to table 2, in a single list, the network of strategic, fundamental and medium objectives was development and validated with the stakeholders.
It was verified that all the interviewees mentioned the Quality Training to the student as a fundamental objective. The publications that stand out as a requirement of CAPES, as a regulatory agency, also appear among the objectives presented, but in a more natural way, much more as a reflection of the work carried out by the GP than as a single goal to be achieved.
Objectives such as: the need for a partnership between the GP and quality companies and staff were also very important among the interviewees, followed by the economic viability and infrastructure for the GP.
The objectives presented demonstrate that the concerns and decisions in a GP reach far beyond simply cataloguing the publication indexes, they require the integration of the team involved and several different types of research in order for them to be published.
Then the network of objectives was elaborated with the strategic, fundamental and means objectives. This construction took place through analysis of the interviews and obtaining information from the decision-makers, who also validated it, as seen in Figure 3.
For the structured network, a list of measurable attributes was also elaborated, step five, through a discussion with the decision makers (Table 3) in function of the established fundamental objectives. The decision-makers were individually asked to indicate how we could measure the attributes presented, what measurable attributes would be used to monitor the goals pointed out in the network. A single list was drawn up and validated by the decision-makers. These represent the performance criteria pointed out by the decision makers and CAPES (following the requests of the area document) for the GP in Biotechnology object of this study.
Most measurable attributes are natural. Natural is a direct measure, Built is a defined scale for the attribute and Proxy is an indirect attribute that is either natural or built for a medium purpose (KEENEY, 199233 KEENEY RL & MCDANIELS TL. 1992. Value-focused thinking about strategic decisions at BC Hydro. Interfaces, 22(6): 94-109.).
The practical and multidisciplinary nature of Biotechnology requires the development of products and patents, and this leads to the need to approach companies for partnerships. That allows for making applied research more efficient and able to give back more affordable products to society. This also applies to publications. Not that the “pure” searches are not necessary and valid, but, for those involved in the construction of this PMS, having more applied results was paramount.
It is necessary to make the goals measurable in order to check whether they are being reached and to draw action plans for that to happen. It should be noted that the measurement of performance is cyclical and that, with some objectives being achieved, others may arise, so the application of the VFT should occur also periodically to ensure that new objectives and alternatives can be identified.
Measuring is inherent in the continuous improvement process and will give the GP a better idea of “where it is” and “where it wants to go”. Just considering the requirements of the regulatory agency is not enough for the results needed for a GP, the work is broader. Listening to the stakeholders involved and verifying the performance objectives most valued by them, through the VFT, is the beginning of the construction of a PMS capable of attending to the particularities of each business, that team at that moment in time.
The built-in PMS encompasses the CAPES evaluation criteria and is complemented by others valued by the stakeholders / decision makers. It is possible to monitor the performance of the program over the four-year periods, outlining actions that reflect results within the evaluation period. The specific criteria pointed out in the PMS aid in the overall result of the program, and can indirectly generate positive impact on the CAPES criteria. Nevertheless, program coordination should be attentive to the CAPES performance criteria (weighting sheet) and their respective weightings, as they are frequently changed.
FINAL REMARKS
Proposing a method to design PMSs for Graduate Programs, in order to assist in their management and continuous improvement, is more than creating indicators, since CAPES already has them - although only partially, and, as a regulator, it dictates the “rules of the game”- but rather looking inside the program that proposes to go through this exercise (since it is a cyclical process) to check what their needs are, their choices and how to act so as not to be just another program in order to meet the requirements required, but different, by essence in pursuing its results as the reflection of a work based on strategy, planning and actions.
The use of the VFT approach was efficient in the identification of the performance criteria for the GP in Biotechnology, object of this study. The approach was able to meet PSM premises, allowing the identification of the strategic, fundamental and means objectives, assisting the GP in its action plan, as well as in the structuring of the measurable attributes capable of providing more support to the decision process, since each attribute aims at meeting the objectives set by the stakeholders themselves.
The application of the VFT as a PSM was encouraged by the authors cited in the theoretical framework section and, in an applied fashion, it was useful. For some of the interviewees the questions provided a deeper approach than for others, even though the questions designed to stimulate were carried out. There were difficulties among the decision-makers in understanding what the other had pointed out as a metric. This also allowed for debate on the qualitative and quantitative aspects of the criteria. Some of these interviewees informally justified their training in health and life sciences and did not feel at all comfortable to discuss the performance issues for a GP, which was something new, but also stimulated the need to seek a greater understanding with the proposal of this work.
It was possible to verify, by reviewing the CAPES area document, its prioritization with regard to publications/scientific production. The other stakeholders involved in the process highly value the Quality Education to the student, which is presented by all respondents in the application of the VFT; the respondents also mention publications, albeit with the argument that they should be useful for society with tangible results.
Objectives such as the need for partnerships with companies as a mutual form of assistance were also mentioned by several of the respondents, followed by quality faculty. The possibility of the GP being able to ascertain the results of such attributes provides a panorama of the reality of the program and opportunity for actions in search of continuous improvement, acting as a management tool.
The suggestion of this paper, already expected to be carried out by 2019, is the combination of the multicriteria decision support method Analytic Network Process - ANP, and through modeling multi-methodologies, identifying dependency and feedback relationships among such performance criteria, measure performance and management priorities - as proposed by Piratelli and Belderrain (201058 PIRATELLI C & BELDERRAIN MCN. 2010. Supporting the Design of a Performance Measurement System with the Analytic Network Process. International Journal of Analytic Hierachy Process, 2(1): 44-65.).
ACKNOWLEDGEMENTS
This research has been financed in part by FUNADESP (Fundação Nacional de Desenvolvimento do Ensino Superior Particular). The first author would also thanks to CAPES (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior) for her PhD scholarship.
References
-
1ACKERMANN F & EDEN C. 2011. Strategic management of stakeholders: Theory and practice. Long range planning, 44(3): 179-196.
-
2ACKERMANN F, FRANCO LA, ROUWETTE E & WHITE L. 2014. Special issue on problem structuring research and practice.
-
3ALENCAR LH, DE MIRANDA MOTA CM & ALENCAR MH. 2011. The problem of disposing of plaster waste from building sites: problem structuring based on value focus thinking methodology. Waste management, 31(12): 2512-2521.
-
4ALENCAR MH, PRIORI JR L & ALENCAR LH. 2017. Structuring objectives based on value-focused thinking methodology: Creating alternatives for sustainability in the built environment. Journal of cleaner production, 156: 62-73.
-
5ALMEIDA S, MORAIS DC & DE ALMEIDA AT. 2014. Agregação de pontos de vista de stakeholders utilizando o Value-Focused Thinking associado à mapeamento cognitivo. Production, 24(1): 144-159.
-
6ARVAI JL, GREGORY R & MCDANIELS TL. 2001. Testing a structured decision approach: value-focused thinking for deliberative risk communication. Risk Analysis, 21(6): 1065-1076.
-
7BALBACHEVSKY E. 2005. A pós-graduação no Brasil: novos desafios para uma política bem-sucedida. Os desafios da educação no Brasil Rio de Janeiro: Nova Fronteira, 1: 285-314.
-
8BARCLAY C & OSEI-BRYSON KM. 2010. Project performance development framework: An approach for developing performance criteria & measures for information systems (IS) projects. International Journal of Production Economics, 124(1): 272-292.
-
9BELTON V, ACKERMANN F & SHEPHERD I. 1997. Integrated support from problem structuring through to alternative evaluation using COPE and VISA. Journal of Multi-Criteria Decision Analysis, 6(3): 115-130.
-
10BERNARDO H, GASPAR A & HENGGELER ANTUNES C. 2018. A Combined Value Focused Thinking-Soft Systems Methodology Approach to Structure Decision Support for Energy Performance Assessment of School Buildings. Sustainability, 10(7): 2295.
-
11BEZERRA JR G, CUNHA FILHO W & CAVALCANTE JR JRO. 2016. Estruturação e análise multiatributo para priorização da oferta de cursos técnicos em uma instituição de ensino profissionalizante.
-
12BITITCI U, GARENGO P, DÖRFLER V & MENDIBIL K. 2012. Performance measurement: questions for tomorrow. Advanced Production Management Systems, 14(3): 305-327.
-
13BRASIL, MINISTÉRIO DA EDUCAÇÃO. 2016. Sobre as áreas de avaliação. CAPES - Coordenação de Aperfeiçoamento de Pessoal de Nível Superior. Available at: Available at: http://www.capes.gov.br/images/documentos/Documentos_de_area_2017/BIOT_docarea_2016.pdf Accessed: 1 ago 2017.
» http://www.capes.gov.br/images/documentos/Documentos_de_area_2017/BIOT_docarea_2016.pdf -
14BRASIL, MINISTÉRIO DA EDUCAÇÃO. 2017. História e missão. CAPES - Coordenação de Aperfei`çoamento de Pessoal de Nível Superior. Available at: Available at: http://www.capes.gov.br/images/documentos/Documentos_de_area_2017/BIOT_docarea_2016.pdf Accessed: 1 ago 2017.
» http://www.capes.gov.br/images/documentos/Documentos_de_area_2017/BIOT_docarea_2016.pdf -
15BRESSIANI F, ALT PRC & MASSOTE AA. 2001. O Uso do Balanced Scorecard como Instrumento de Melhoria de Desempenho de uma Instituição de Ensino Superior. In: Congresso Brasileiro de Ensino de Engenharia
-
16CHOONG KK. 2014. Understanding the features of performance measurement system: a literature review. International Journal of Production Research, 52(14): 4174-4197.
-
17CHOONG KK. 2018. Use of mathematical measurement in improving the accuracy (reliability) & meaningfulness of performance measurement in businesses & organizations. Measurement, 129: 184-205.
-
18COELHO G. 2017. Identificação de políticas de seguranc¸a e saúde para a promoção do envelhecimento ativo dos trabalhadores da indústria utilizando value focused thinking Master’s thesis. Curso de Engenharia Aeronáutica e Mecânica, Área de Produção - Instituto Tecnológico de Aeronáutica.
-
19DE ALMEIDA AT. 2000. Processo de decisão nas organizações: construindo modelos de decisão multicritério Editora Atlas SA.
-
20DE ANDRADE MRB, DE QUEIROZ NEIVA MT, SCHRAMM VB & SCHRAMM F. 2017. O uso do método VFT para apoiar um comitê de bacia hidrográfica na discussão de um problema complexo. XLIX Simpósio Brasileiro de Pesquisa Operacional,.
-
21DOS SANTOS CM. 2003. Tradições e contradições da pós-graduação no Brasil. Educação & sociedade, 24(83): 627-641.
-
22EDEN C & ACKERMANN F. 2013. Making strategy: The journey of strategic management Sage.
-
23GREGORY R, ARVAI J & MCDANIELS T. 2001. Value-focused thinking for environmental risk consultations. In: Environmental risks: Perception, evaluation and management pp. 249-273. Cambridge: Emerald Group Publishing Limited.
-
24HASSAN OA. 2004. Application of value-focused thinking on the environmental selection of wall structures. Journal of environmental management, 70(2): 181-187.
-
25JURK DM, CHAMBAL SP & THAL JR AE. 2004. Using value-focused thinking to select innovative force protection ideas. Military Operations Research, pp. 31-43.
-
26KAJANUS M, KANGAS J & KURTTILA M. 2004. The use of value focused thinking and the A’WOT hybrid method in tourism management. Tourism management, 25(4): 499-506.
-
27KEENEY GL & VON WINTERFELDT D. 2010. Identifying and structuring the objectives of terrorists. Risk Analysis: An International Journal, 30(12): 1803-1816.
-
28KEENEY RL. 1996. Value-focused thinking: Identifying decision opportunities and creating alternatives. European Journal of operational research, 92(3): 537-549.
-
29KEENEY RL. 2001. Modeling values for telecommunications management. IEEE Transactions on Engineering Management, 48(3): 370-379.
-
30KEENEY RL. 2007. Developing objectives and attributes. Advances in decision analysis: From foundations to applications, pp. 104-128.
-
31KEENEY RL. 2012. Value-focused brainstorming. Decision Analysis, 9(4): 303-313.
-
32KEENEY RL. 2013. Identifying, prioritizing, and using multiple objectives. EURO Journal on Decision Processes, 1(1-2): 45-67.
-
33KEENEY RL & MCDANIELS TL. 1992. Value-focused thinking about strategic decisions at BC Hydro. Interfaces, 22(6): 94-109.
-
34KEENEY RL & MCDANIELS TL. 2001. A framework to guide thinking and analysis regarding climate change policies. Risk Analysis, 21(6): 989-1000.
-
35KEENEY RL & VON WINTERFELDT D. 2011. A value model for evaluating homeland security decisions. Risk Analysis: An International Journal, 31(9): 1470-1487.
-
36KEISLER J. 2012. Is Value Focused Thinking a Problem Structuring Method or Soft OR or what? University of Massachusetts Boston. Available at: http://works.bepress.com/jeffrey_keisler/47
» http://works.bepress.com/jeffrey_keisler/47 -
37KEISLER J, TURCOTTE DA, DREW R & JOHNSON MP. 2014. Value-focused thinking for community-based organizations: objectives and acceptance in local development. EURO Journal on Decision Processes, 2(3-4): 221-256.
-
38KIBIRA D, BRUNDAGE MP, FENG S & MORRIS K. 2018. Procedure for selecting key performance indicators for sustainable manufacturing. Journal of Manufacturing Science and Engineering, 140(1): 011005.
-
39KIM SH & AHN BS. 1999. Interactive group decision making procedure under incomplete information. European Journal of Operational Research, 116(3): 498-507.
-
40KUNZ RE, SIEBERT J & MÜTTERLEIN J. 2016. Combining value-focused thinking and balanced scorecard to improve decision-making in strategic management. Journal of Multi-Criteria Decision Analysis, 23(5-6): 225-241.
-
41LOPES YG & DE ALMEIDA AT. 2013. A multicriteria decision model for selecting a portfolio of oil and gas exploration projects. Pesquisa Operacional, 33(3): 417-441.
-
42MARTTUNEN M, LIENERT J & BELTON V. 2017. Structuring problems for Multi-Criteria Decision Analysis in practice: A literature review of method combinations. European Journal of Operational Research, 263(1): 1-17.
-
43MAY J, DHILLON G & CALDEIRA M. 2013. Defining value-based objectives for ERP systems planning. Decision Support Systems, 55(1): 98-109.
-
44MERRICK JR & GARCIA MW. 2004. Using value-focused thinking to improve watersheds. Journal of the American Planning Association, 70(3): 313-327.
-
45MERRICK JR, GRABOWSKI M, AYYALASOMAYAJULA P & HARRALD JR. 2005. Understanding organizational safety using value-focused thinking. Risk Analysis: An International Journal, 25(4): 1029-1041.
-
46MINGERS J. 2011. Soft OR comes of age-but not everywhere! Omega, 39(6): 729-741.
-
47MINGERS J & ROSENHEAD J. 2004. Problem structuring methods in action. European journal of operational research, 152(3): 530-554.
-
48MODELL S. 2005. Students as consumers? An institutional field-level analysis of the construction of performance measurement practices. Accounting, Auditing & Accountability Journal, 18(4): 537-563.
-
49MORAIS DC, ALENCAR LH, COSTA AP & KEENEY RL. 2013. Using value-focused thinking in Brazil. Pesquisa Operacional, 33(1): 73-88.
-
50MORITZ GO, MORITZ MO & MELO PA. 2013. A Pós-Graduação brasileira: evolução e principais desafios no ambiente de cenários prospectivos. Future Studies Research Journal: Trends and Strategies, 5(2): 03-34.
-
51NEELY A. 1999. The performance measurement revolution: why now and what next? International journal of operations & production management, 19(2): 205-228.
-
52NEELY A. 2005. The evolution of performance measurement research: developments in the last decade and a research agenda for the next. International Journal of Operations & Production Management, 25(12): 1264-1277.
-
53PAIVA MLDUG & DAHER SDFD. 2016. Abordagem VFT para estruturação de problema para melhoria da produção mais limpa em empresa de confecções do agreste pernambucano. XLVIII Simpósio Brasileiro de Pesquisa Operacional, pp. 437-447.
-
54PARNELL GS, HUGHES DW, BURK RC, DRISCOLL PJ, KUCIK PD, MORALES BL & NUNN LR. 2013. Invited Review-Survey of Value-Focused Thinking: Applications, Research Developments and Areas for Future Research. Journal of Multi-Criteria Decision Analysis, 20(1-2): 49-60.
-
55PEHARDA I & HUNJAK T. 2008. Selecting an Automatic Rifle using the Valuefocused Thinking Approach. Military Operations Research, 13(3): 19-26.
-
56PEREIRA CA & SCHENBERG ACG. 2004. Pós-graduação em Biotecnologia: êxitos e dificuldades de uma experiência interinstitucional. RBPG - Revista Brasileira de Pós-Graduação, 1(1): 101-110.
-
57PEREIRA MAC & DA SILVA MT. 2003. A key question for higher education: Who are the customers. In: 31st Annual Conference of the Production and Operations Management Society
-
58PIRATELLI C & BELDERRAIN MCN. 2010. Supporting the Design of a Performance Measurement System with the Analytic Network Process. International Journal of Analytic Hierachy Process, 2(1): 44-65.
-
59POLETO T, SILVA LC, CARVALHO VDH, MOURA JA & COSTA APCS. 2003. Modelo de decisão para identificação e priorização de políticas de segurança de informação em um hospital público. In: Encontro Nacional de Engenharia de Producão - Perspectivas Globais para a Engenharia de Produção ABEPRO.
-
60REICHERT P, LANGHANS SD, LIENERT J & SCHUWIRTH N. 2015. The conceptual foundation of environmental decision support. Journal of environmental management, 154: 316-332.
-
61ROSSI GM ET AL. 2012. Biotecnologia no Brasil: uma análise empírica a partir dos dados da PINTEC Master’s thesis. Universidade Federal de Uberlândia.
-
62SELART M & JOHANSEN ST. 2011. Understanding the role of value-focused thinking in idea management. Creativity and Innovation Management, 20(3): 196-206.
-
63SHAH L, ETIENNE A, SIADAT A & VERNADAT FB. 2012. (Value, Risk)-based Performance Evaluation of Manufacturing Processes. IFAC Proceedings Volumes, 45(6): 1586-1591.
-
64SHENG H, NAH FFH & SIAU K. 2005. Strategic implications of mobile technology: A case study using Value-Focused Thinking. The Journal of Strategic Information Systems, 14(3): 269-290.
-
65SHENG H, SIAU K & NAH FFH. 2010. Understanding the values of mobile technology in education: a value-focused thinking approach. ACM SIGMIS Database: the DATABASE for Advances in Information Systems, 41(2): 25-44.
-
66SIEBERT J & KEENEY RL. 2015. Creating more and better alternatives for decisions using objectives. Operations Research, 63(5): 1144-1158.
-
67SIMON J, REGNIER E & WHITNEY L. 2014. A value-focused approach to energy transformation in the United States Department of Defense. Decision Analysis, 11(2): 117-132.
-
68SOBREIRA NETTO F. 2007. Medição de desempenho organizacional: um estudo das vantagens e desvantagens dos principais sistemas sob as óticas teórico-acadêmica e de práticas de mercado. In: Encontro da Anpad, 31 ., 2007, Rio de Janeiro ANPAD.
-
69URTIGA MM & MORAIS DC. 2015. Pre-negotiation framework to promote cooperative negotiations in water resource conflicts through value creation approach. EURO Journal on Decision Processes, 3(3-4): 339-356.
-
70VAN LOOY A & SHAFAGATOVA A. 2016. Business process performance measurement: a structured literature review of indicators, measures and metrics. SpringerPlus, 5(1): 1797.
-
71VIEIRA DT & DUARTE MDO. 2014. Value-focused thinking: encontrando alternativas sustentáveis para pequenos e médios laticínios do agreste pernambucano. In: Encontro Nacional de Engenharia de Producão - Engenharia de Produção, Infraestrutura e Desenvolvimento Sustentável: a Agenda Brasil+10, 2014 ABEPRO.
Publication Dates
-
Publication in this collection
2 Dec 2019 -
Date of issue
Sep-Dec 2019
History
-
Received
21 Feb 2019 -
Accepted
21 Oct 2019