Application of the enterprise management tools Lean Six Sigma and PMBOK in developing a program of research management

Abstracts

OBJETIVO: Implementar um programa de gestão da pesquisa científica em um hospital geral aplicando as ferramentas de gestão empresarial Lean Seis Sigma e PMBOK no gerenciamento de projetos nessa área. MÉTODOS: Foi utilizada a metodologia Lean Seis Sigma para melhoria do processo de gestão da pesquisa científica institucional por meio de ferramenta específica (DMAIC) para identificação, implementação e posterior análise das soluções encontradas, tendo como base as boas práticas descritas no PMBOK. RESULTADOS: São apresentadas as soluções implementadas na Sociedade Beneficente Israelita Brasileira Albert Einstein para o gerenciamento dos projetos de pesquisa institucionais. As soluções foram categorizadas em quatro instâncias: pessoas, processos, sistema e cultura organizacional. Uma análise preliminar das soluções implementadas mostra que essas são, total ou parcialmente, aderentes às preconizadas no Guia PMBOK. CONCLUSÃO: Neste estudo de caso post facto, verificou-se que as soluções implementadas a partir do projeto Lean Seis Sigma e baseadas no PMBOK permitiram a melhoria de processo da gestão da pesquisa científica institucional, constituindo um modelo que pretende contribuir com a busca de soluções inovadoras na gestão da pesquisa pelas diferentes instituições com atividade científica no Brasil.

Instrumentos para a gestão da atividade científica; Gestão do conhecimento para a pesquisa em saúde

HEALTH ECONOMICS AND MANAGEMENT

Application of the enterprise management tools Lean Six Sigma and PMBOK in developing a program of research management

Cora Hors^I; Anna Carla Goldberg^I; Ederson Haroldo Pereira de Almeida^II; Fernando Galan Babio Júnior^I; Luiz Vicente Rizzo^I

^IHospital Israelita Albert Einstein - HIAE, São Paulo (SP), Brazil

^IISociedade Beneficente Israelita Brasileira Albert Einstein - SBIBAE, São Paulo (SP), Brazil

^{Corresponding author}

ABSTRACT

OBJECTIVE: Introduce a program for the management of scientific research in a General Hospital employing the business management tools Lean Six Sigma and PMBOK for project management in this area.

METHODS: The Lean Six Sigma methodology was used to improve the management of the institution's scientific research through a specific tool (DMAIC) for identification, implementation and posterior analysis based on PMBOK practices of the solutions found.

RESULTS: We present our solutions for the management of institutional research projects at the Sociedade Beneficente Israelita Brasileira Albert Einstein. The solutions were classified into four headings: people, processes, systems and organizational culture. A preliminary analysis of these solutions showed them to be completely or partially compliant to the processes described in the PMBOK Guide.

CONCLUSION: In this post facto study, we verified that the solutions drawn from a project using Lean Six Sigma methodology and based on PMBOK enabled the improvement of our processes dealing with the management of scientific research carried out in the institution and constitutes a model to contribute to the search of innovative science management solutions by other institutions dealing with scientific research in Brazil.

Keywords: Instruments for management of scientific activity; Knowledge management for health research

INTRODUCTION

Management of projects and research in Brazil

The areas of biological sciences and health show low levels of compliance with modernization of science and technology management. One of the causes raised is the fear of the researcher that structured management of scientific research implies interference in the researcher's autonomy⁽¹⁾. It is important to understand this culture in Brazil within the material conditions that fed it. In contrast with the private scenario, in which investments must be focused on the results obtained, the "personalistic" environment of academic research in health generated a division between academic investigation and the development of technology and products strategic to the national reality.

This scenario has been changing over the last ten years. In the report just recently released by The State of Sao Paulo Research Foundation (FAPESP)⁽²⁾, the migration of research conduction and financing by the public sector (formerly centralized primarily in universities) to the private sector, when comparing the growth of scientific production in different Brazilian institutions, has followed a worldwide tendency.

Recognizing the need to promote scientific research in the area of health, an instrument of acquisition of technology and strategy knowledge, the Ministries of Health and of Science and Technology have increasingly jointly adopted measures that include the creation of the Department of Science and Technology (DECIT) in 2000; the Secretariat for Science, Technology and Strategic Inputs (SCTIE) in 2003; the preparation of an agenda of priorities in health research; the performance of national and regional public announcements, among them, the Research Program for the Unified Healthcare System - PPSUS (DECIT), in 2006⁽³⁾.

This is a significant cultural change: the "property" of research becomes, at least in part, shared with an institution that clearly defines its priorities. Additionally, in this association, there is an implicit contract for search of socially relevant, and preferentially measurable, results. The distance between academic research and the "development of products" is diminished, which was previously almost exclusive reserved for the private institutions.

Management tools: Lean Six Sigma and Project management

Six Sigma is a structured methodology that seeks excellence in competitiveness by means of continuous improvement of the processes involved in the production of goods or services, taking into consideration all important aspects of a business⁽⁴⁾. The Lean system, whose origins stem from the Toyota System of Production, seeks to eliminate surplus, i.e., exclude what has no value for the client and impose swiftness and efficiency to the company⁽⁵⁾. The integration of Six Sigma to the Lean system allows the company to benefit from both strategies. The program that results from this integration is called Lean Six Sigma (LSS) and constitutes a comprehensive, powerful, and effective tactic for solving problems as well as for creating new processes and products⁽⁵⁾. The relation between processes and projects is in the organization of the activities involved. The project is necessary when this organization cannot be carried out at the operational limit⁽⁶⁾.

Within the scope of projects, during the 1960s, two not-for-profit institutions were created to standardize management of projects and advance in the state-of-theart in this area of knowledge: the Institute for Project Management Association (IPMA), in 1965, in Europe, as the first international organization created with this focus, and the Project Management Institute (PMI), in 1969, in the United States. The PMI is dedicated to the advancement of project management, promoting activities, studies, events, training, and bibliography specialized in this area of knowledge, with PMBOK⁽⁷⁾ as its publication prepared to standardize and disseminate good practices in project management.

Project management is the application of knowledge, skills, tools, and techniques to the project's activities in order to meet its requirements. The different needs of each project are extremely important for the adaptation of the various tools, making them appropriate for each one.

PMBOK⁽⁷⁾ describes 42 processes in five groups (Initiation; Planning; Execution; Monitoring and Control; and Conclusion), dividing these processes into nine areas of knowledge, as is described below:

- Integration: defines the processes and activities that integrate the various elements necessary for project management;

- Scope: defines the processes and activities in order to guarantee that the project includes all the work necessary (and only the work necessary) for the project to be concluded successfully;

- Time: describes the processes and activities for the project to be concluded within the correct timeline;

- Costs: describes the processes and activities of planning, estimating costs, determining the budget, and expenditure control of the project;

- Quality: describes the processes and activities in reference to planning, monitoring, and quality control and assurance of the project;

- Human resources: describes the processes and activities related to planning, hiring, mobilization, development, and management of the project team;

- Communications: describes the processes and activities related to the generation, collection, distribution, storage and final destination of the project's information in an opportune and appropriate manner;

- Risks: describes the processes and activities related to identification, analysis, and control of the risks of the project;

- Acquisitions: describes the processes and activities related to the purchase or acquisition of products, services, or results for the project.

An effective management is based on this guide, but the PMI recommends that the project management team understand and use the knowledge and skill of at least four other areas of specialization: (1) knowledge, norms, and regulations of the area of application;

(2) understanding of the project environment; (3) general management knowledge and skills; and (4) interpersonal abilities. It is the responsibility of the team project to apply the knowledge it deems necessary, in such a way as to adjust to the specific project and to the organization⁽⁸⁾.

Initially introduced in sectors such as Information Technology, Engineering, and Industry, the use of these management tools is under expansion to other areas, such as the Third Sector and Health.

OBJECTIVE

To implement a scientific research management program in a general hospital, applying the business management tools, LSS and PMBOK, in managing projects in this area.

METHODS

The qualitative exploratory-descriptive case study was used as a method of analysis⁽⁹⁾, widely used until today in the social sciences and in qualitative organizational research⁽¹⁰⁾. Using qualitative analysis⁽¹¹⁾, this case study was structured in the following way: (a) bibliographic review; (b) selected case report; (c) results obtained; and (d) qualitative preliminary analysis of compliance with the solutions implemented in PMBOK^®.

Short case description

The Sociedade Beneficente Israelita Brasileira Albert Einstein (SBIBAE) is a not-for-profit civil society, active in the healthcare sector. It was founded in 1955, and in 1971, it inaugurated the Hospital Israelita Albert Einstein in the municipality of Sao Paulo (SP). The hospital was consolidated, over the years, as one the most advanced in Brazil and became an international reference, having as it mission "to offer excellence in quality in the realm of healthcare, generation of knowledge and social responsibility, as a form of evidencing the contribution of the Jewish community to Brazilian society." In accordance with this mission, the institution extended its activities and became an ample healthcare system, creating the Instituto Israelita de Ensino e Pesquisa (IIEP), in order to generate and share knowledge in the area of health by means of teaching and research activities; the Instituto Israelita de Responsabilidade Social (IIRS), to take Einstein Hospital's care to the most needy population; and Medicina Diagnóstica e Preventiva (MDP), in order to offer prevention tests and programs to hospital patients and those outside.

The IIEP was inaugurated in 1998 and since then its activities have grown without interruption. Despite the fact that institutional scientific research has been monitored by the IIEP since 2005 by means of a research project leader, it was noted that the process showed significant deficiencies that precluded a reliable costeffectiveness analysis of the research, compromising scientific and financial management. Various factors led to the identification of the need to improve research management at SBIBAE, among them, internal audit, which revealed inconsistencies in the database; increase in research activities at the institution, not in line with the funding resources; available monitoring and control; and direct questioning of the leadership as to results of investments in research over the last years, in other words, the cost-effectiveness relationship of research at the institution.

Application of the LSS tool

Characterizing the problem, the proposal for improvement in the research management process was presented to the Einstein Program of Continuous Process Improvement [Programa Einstein de Melhoria Contínua de Processos], LSS.

The LSS program is well-known and important in the progression of quality management processes of companies from different sectors. There is a series of available tools; the one most used at SBIBAE is DMAIC methodology, in which each one of the five letters indicates one of the five phases that characterize it: definition, measurement, analysis, implementation of solutions, and control of the process⁽¹²⁾. After identification of the problem and approval of the project during portfolio prioritization analysis, the leader of the project received the institutional training necessary. During the first phase of the project, mapping was performed of the main entities interested and involved (also known as stakeholders) in the processes related to research management, as per DMAIC methodology, for the identification of the needed improvements. Concomitantly, alignment was sought of solutions with the good practices of project management, based on the PMBOK Guide, for integrated incorporation of the processes into a computerized platform in research project management, as is shown in figure 1.

Definition phase (D)

In this phase, the problem was characterized and the project contract was prepared. Definition of the team took into consideration the inclusion of representatives from the different interested parties, such as specialists in the areas related to research management, besides the choice of a project sponsor that could act as a facilitator.

The project contract contains the main information that characterized the improvement project, such as is shown on chart 1.

Measurement phase (M)

From its submission to its conclusion, a research project benefits from various controls, and frequently needs status updates in its different stages of development. Procedures such as document adaptation to make the project viable, peer evaluation of merit, approval of ethics bodies (Ethics in Research Committee and National Committee of Ethics in Research), purchase and/or importation of materials, progress reports, and the record of processes, innovative product, or others) are aspects with their own levels of complexities, and unless they are well integrated into the management process, they tend to be outdated and out of the control of the institution.

Thus, during this phase, the design for macro flow of scientific research at the institution was prepared, which allowed the comprehension of the complexity of research management with the objective of gradual elimination of parallel controls, of not yet defined policies, or yet, the identification of the need for revision and validation of the operational flows (Figure 2).

Analysis phase (A)

Various tools, indicated by DMAIC methodology, were used. In order to illustrate this phase, Ishikawa's diagram was used (Figure 3A) with the objective of mapping the difficulties in information input in the current SGPP system, and an Effort x Impact Matrix (Figure 3B) for the prioritization of the problems identified. Also part of the analysis was the determination of the desired levels of efficiency (sigma level) for the financial control processes of the investments made in the research and for scientific production, used as marker of conclusion and submission of scientific project results to SGPP. Practically all the problems identified were capable of being addressed in the search for solutions for management improvement. Therefore, at the end of the analysis phase, there was a list of the primary problems to be addressed, seeking the implementation of improvements and control of the solutions incorporated into the research management process, as outlined in chart 2.

Implementation of the improvement phase (I)

All improvements were implemented during this phase according to an action plan that resulted from the analysis phase. In order to assure the expected benefits, communication actions at the institution were fundamental.

With the sponsors of the project and in a planned and validated manner, information was sent to all institution collaborators informing and explaining the improvements incorporated into research management. Additionally, all the information was made available online on the institutional website, including an instruction manual and models of the documentation required for project submission. An electronic address was generated for consultation and problem solution, as well as a direct consultation canal (FAQ - frequently asked questions). There was also a series of meetings with the different types of users. These actions were vital for introducing, into the institutional culture, the changes needed, directly linked to research activities.

Control phase (C)

The control phase of the solutions implemented occurred during the period of development of the new Research Project Management System prototype, designed and approved at the end of the LSS project. In this phase, monthly collections were manually made of data, which were analyzed as the indicators defined in the project contract. When the new management system, developed and completely implemented in the following months, became effective, the automated mechanisms allowed the system itself to furnish the reports for management and control of all the processes involved in management of scientific research projects at the institution.

RESULTS

Based on DMAIC methodology, research management was structured at SBIBAE, and the solutions occurred in four equally important areas: people, processes, system (the tool developed), and organization culture. Total duration of the project was 12 months: 6 months for application of the DMAIC methodology (design of processes and prototype of the new system) and another 6 months for the development and implementation of the new research management system at the institution.

Human resources dedicated to research management

During theLSS project, the absence of a person in charge of research management was identified. Due to the impossibility of hiring a full-time professional for the activity, the group of research managersdecided to share the task with the hired researcherswith the formation of a Scientific Committee of IIEP. Thememebers of this group of professionals committed to researchand the evaluation process alternate among them in a rotation regime, each one responsible for the evaluation of one project submitted for evaluation. This activity showed it had a direct and favorable impact on the effectiveness of the institutional research.

Another committee established was the Management Committee of I-Search, responsible for monitoring and controlling processes of the computerized management system. The committee was constituted by professionals with different types of expertise, including an information technology analyst, a researcher, a clinical trial coordinator, a research manager, an administrator, and an administrative coordinator. Since its implementation, improvementshave been continuously introduced tending to the institutional demands generated by the information made available by the system itself.

The scientific evaluation process of the research projects was also reviewed, seeking a greater involvement on the part of the body of reviewers. For this, the group was renovated and new reviewers were invited by means of communications and meetings in groups, besides promotion of participation in the clinical body of SBIBAE as return for the use of the institution's research structure, as per models of requirements made by FAPESP. We must point out that the hired clinical body is composed of a specialized group of professionals who many times have been involved in responsibilities of teaching and research in the great universities of the country, and for this reason, are interested in developing research at the institute. Finally, the evaluation process came to be monitored by the management system and messages started to be sent out to remind the reviewer of the deadlines for delivery of the evaluations. All the evaluation processes became monitored by centralized control of the management system, enabling, therefore, the follow-up of pending matters, effectiveness, and performance of the people involved in each one of the processes.

As a result, the involvement of diverse professionals, from different areas and with different roles in research activities, has contributed towards a culture favorable for effective, transparent, and integrative research. Today, the coordinating bodies may monitor these results by means of process indicators incorporated into the reports from the areas and from the General Board, evaluated by the technique of the balanced score card (BSC).

Mapping and designing the processes

All the processes involved in research management had their flows designed and validated, namely: submission of new research project; evaluation of the project at the different stages of analysis, i.e., initial screening and adaptation of documentation, scientific management committee, official declaration of scientific merit (two independent evaluations), ethics committees, viability approved by the area manager; analysis and approval of the budget; accompaniment of the project with analysis of partial scientific reports, and final delivery of the project; automated flows of request for evaluations by specialists, of delivery and requests for scientific reports, responses of confirmation and acknowledgment, among others. Figure 4 illustrates the flow chart of the followup process of performance of the research projects.

Development of a new research project management system

Mapping the processes and the desired functions of a computerized system resulted in the I-Search software⁽¹³⁾, that manages the multiple aspects raised during the LSS project. Chart 3 lists some of the functions offered to the software users.

Based on a database rich in information as to research projects, detailing of the proponent and the members of each research team member, partner institutions, origin of financing, and deliveries, the strategic management of this portfolio of projects may be performed. Examples of indicators obtained by means of I-Search: -Quality of the proposals and rates of approval,

according to the proponent area, to the professional's training, and to the year of approval;

- Cost-effectiveness of the research: percentage of research projects published, values and origin of financing of the approved projects;

- Growth of research activities: evolution of the number of projects submitted and approved each year, per area, specialty, for definition of the strategy in research and short-, medium-, and long-term action plans;

- Costs of publication: number of articles published/ sum of investments in research.

Organizational culture

The various actions orchestrated by the research managers, with the support of their teams and of the General Board, were fundamental for divulging the improvements implemented and the initiation of a new organizational culture (Chart 4).

Compliance with the solutions implemented in PM-BOK

Once the LSS project was finalized and the new management system, I-Search, had been developed, a qualitative preliminary analysis was made of the solutions implemented as to their compliance with good practices and the processes described in the PMBOK Guide (Chart 5). This preliminary analysis was designed based on controls, reports, and indicators of performance, process, and finances generated by the system. In this way, for each of the 42 processes recommended in the Guide, there was an assessment as to whether the solution implemented met (green) or failed to meet (red), or partially met (yellow) the recommendations of the Guide. The result is shown on the PMBOK table, which summarizes the processes and their association with each one of the 9 areas of knowledge in project management. The colors green, yellow, and red, indicate the consensus in compliance analysis, carried out in the five groups of processes in each area of knowledge. The use of colors has the purpose of acting as a signal of possible points needing attention in process areas and/ or groups when evaluating the situation of research project management at the institution.

The last column, the consensus as to compliance with the processes for each area, defined in colors green, yellow, or red, indicates which areas of knowledge need more attention, acting as an alert signal (especially if the objectives of the projects are not being reached). The analysis shows that the solutions implemented cover, partially or totally, eight of the nine areas of knowledge. It also shows that none of the processes recommended by good practices is adopted for risk management, and that only a few processes are used in the management of the areas of communication, quality, and scope, making these points that need to receive more attention. This result shows that some processes can still be improved as to the success of the research, and that they should be prioritized in the subsequent improvements of the research management program.

This type of analysis may serve as a model in the evaluation of the maturity of the institutional research management program when compared to other areas of SBIBAE (for example, work and information technology), or even when used at other institutions with scientific research activities.

DISCUSSION

Scientific research projects share multiple stakeholders with different objectives, understandings, and needs, creating a challenge to institutional management of the strategy adopted by a company. At the same time that the researcher commits himself to research-fostering institutions, with specific deliveries for each one of them, he should also answer to the institution where he carried out his work, to the interests of the sponsors, and to the competition among groups for obtaining resources. To manage these projects in a cohesive manner and aligned with the strategic initiative of the company is usually a challenge conducted in a poorly systematized way. This contrasts with the competitive potential associated with effective management of a scientific program.

It is important to understand this culture within the material conditions that fed it. In contrast with the private scenario, not linked to the university, where investments should focus on the results obtained, the "personalistic" environment of academic research in healthcare has cause a rift between academic investigation and the development of technology and products strategic for national needs.

In Brazil, factors that contribute significantly to this result are the actions that have been adopted jointly between the Ministry of Health and the Ministry of Science and Technology, acknowledging the need to make scientific research in the area of heath an instrument for acquisition of technology and strategic knowledge⁽³⁾.

To deepen these changes requires a new approach in scientific project management. Even recognizing its particularities, such as the elevated risks and uncertainties inherent to this type of activity, the constant need for review of the scope due to progress of scientific knowledge, efficiency in allocation of resources may be significantly increased by the use of structured management forms. This ongoing change encounters a certain resistance in the area of healthcare. Even so, the experiences of other locations where these changes have been made^(14,15), as well as the experience of SBIBAE reported here, indicate is usefulness and viability in the administration and management of research projects.

The preliminary analysis of compliance with the solutions implemented to the processes described in PMBOK for scientific research management showed that these satisfy, at least to some extent, the good practices recommended for project management of projects. In agreement with this result, the analysis of maturity of institutional project management performed by Prado's methodology⁽¹⁶⁾ evidenced an equivalent result regarding research management, suggesting that the model herein proposed may serve as a basis for other institutions as well.

It was possible to note that, despite the fact that I-Search was developed as an internal solution for research management, centralizing all the processes of administration and quality control, in the present study an analysis of the specific compliance of the software as a project management tool was not one of the objectives. Nevertheless, the system allows documents (plans, reports, and approvals) to be attached; it makes available access and editing of the documents; it allows the maintenance of the history of changes made to the scope, the timeline, and the budget; it provides alert mechanisms for deliveries by means of e-mails sent automatically; it records messages exchanged among the participants and specialists involved in the various projects; it has Internet access, among other functions.

A detailed analysis, therefore, remains open, and consolidated methodologies may arise for the evaluation of project generator systems, such as that used by Fornari⁽¹⁷⁾. In this type of analysis, the software in question should have at least one tool to manage each area of knowledge, and compliance is measured according to the processes taken into account.

The change in organizational culture in favor of effective management of research projects is a slower process, but vitally important. Change is a direct consequence of the improvements and functions implemented that generate a series of benefits throughout the institution. There are many examples, such as: research management based on rich and reliable information, transparency of the information, contribution to the decision-making process of the institution, furnishing information on the results of the research and the potential to generate products/ innovation, as well as information on the persons involved (human capital). As to the strategies adopted for management of the research portfolio, the maintenance of a database is fundamental for management of the scientific knowledge produced (theses, articles, patents, and new processes such as delivery of the research projects), to support the definition of policies that favor "effective" research, increased productivity, improved cost-benefit relationship of the scientific research projects, promotions of external capture of resources, international collaboration, and partnerships for R&D.

The dimension of value that project management reaches in management of the organization is evident, taking on the fundamental role in prioritization and in follow-up of the strategic actions associated with scientific research. Among them, it enables the conduction of projects in a way aligned with the interests of high-level management, the unfolding of the Strategic Planning into a Portfolio of Projects, the improvement of the capacity and quality of the deliveries, the compliance with deadlines and the intended timeline, and the dissemination of information on the performance of the projects. All these actions contribute towards the dynamic identification of deficiencies and best practices for the organization complex, allowing the implantation of support services for better management of research projects.

CONCLUSION

The solutions implemented during and after the LSS project enabled the correct structuring of research management of SBIBAE. Knowledge in scientific research management is still a field under construction within the reality of project management of research institutions in the area of healthcare. In parallel, society has become ever more critical and demanding as to investments made in research, expecting the rational use of resources and the guarantee of socially relevant answers. Effective and efficient management in scientific research already is, in this way, a need. With this article we hope to help in the construction of knowledge of this type of management.

Received on: Jun 30, 2012

Accepted on: Nov 26, 2012

Grants: none.

Conflict of interest: none.

Study carried out at Hospital Israelita Albert Einstein - HIAE, São Paulo (SP), Brazil.

• 1. Pereira JC, Saes SG, Escuder MM. Definindo prioridades de gestão de ciência e tecnologia em saúde. Rev Saúde Pública. 1997;31(6):624-31.
• 2. Fundaira JC, Saes SG, Escuder MM. Definindo prioridades de gest necessidade. Esperamos, com este trabalho, auxilia010 [Internet]. São Paulo; 2011 [citado 2012 Out 25]. Disponível em: http://www.fapesp.br/6479
• 3. Departamento de Ci/6479 e Tecnologia, Secretaria de Ciência, Tecnologia e Insumos Estratégicos, Ministério da Saúde. Programa de fomento à pesquisa para os sistemas e serviços locais de saúde: gestão compartilhada em saúde. Rev Saúde Pública. 2006;40(6):1131-6.3.3.
• 4.Rotondaro RG. Seis Sigma: estratégia gerencial para a melhoria de processos, produtos e serviços. São Paulo: Atlas; 2002.
• 5. Werkema MC. Lean Seis Sigma Introdugma: Estratégia gerencial manufacturing. Rio de Janeiro: Wekema; 2006.
• 6. Valle AB, Soares CA, Finocchio Junior J, Silva LS. Fundamentos do gerenciamento de projetos. Rio de Janeiro: FGV Management; 2007.
• 7. Project Management Institute (PMI). Um guia do conhecimento em gerenciamento de projetos (Guia PMBOK). 4a ed. Atlanta: Global Standard; 2008.
• 8. Sampaio M. Metodologia de gerenciamento de projetos e melhoria do desempenho. São Paulo:FGV; 2011.
• 9. Yin RK. Case study research: design. Thousand Oaks: Sage Publications; 1989.
• 10. Yin RK. Case study research: design and methods. 3rd ed. Thousand Oaks: Sage Publications; 2009.
• 11. Arbache AP. Metodologia científica para TCC. São Paulo: Fundação Getúlio Vargas; 2010.
• 12. Marshall Junior I, Cierco AA, Rocha AV, Mota EB, Leusin S. Gestão da qualidade. 10a ed. Rio de Janeiro: FGV; 2010.
• 13. Sociedade Beneficente Israelita Brasileira Hospital Albert Einstein. I-Search - Sistema Gerenciador de Projetos de Pesquisa, 2010. Sigilo até 29/12/2020. Registro no INPI: 11563-6. Rev Bras Prop Int. 2011;2012. [criado por Holthausen AC, Goldberg AC, Hors CP, Babio Junior FG, Gouveia MR, Vinuela RG, Silva SM]
• 14. Pinheiro AA, Siani AC. Guilhermino JF, Henriques MG, Quental CM, Pizarro AP. Metodologia para gerenciar projetos de pesquisa e desenvolvimento com foco em produtos: uma proposta. Rev Admin Pública. 2006;40(3):457-78.
• 15. Barreto Junior JT. Indicadores e métricas: ferramentas para avaliação de resultados de P&D visando a inovação no setor elétrico [dissertação]. Rio de Janeiro: Pontifica Universidade Católica do Rio de Janeiro; 2009 [citado 2012 Out 24]. Disponível em: http://www2.dbd.puc-rio.br/pergamum/tesesabertas/0611878_09_pretextual.pdf
• 16. Prado D. Maturidade em gerenciamento de projetos. 2a ed. Novo Lima: INDG; 2010.
• 17. Fornari AR. Estudo comparativo da aderência de ferramentas livres ao PMBOK (2004). [dissertação]. Rio de Janeiro; 2009 [citado: 2012 Dez 2]. Disponível em: http://www.bcc.ufla.br/monografias/2010/Estudo_Comparativo_da_Aderencia_de_Ferramentas_Livres_ao_PMBOK.pdf

Publication Dates

• Publication in this collection
  22 Jan 2013
• Date of issue
  Dec 2012

History

• Received
  30 June 2012
• Accepted
  26 Nov 2012