Acessibilidade / Reportar erro

DEVELOPMENT OF METRICS FOR PROTOCOLS AND OTHER TECHNICAL PRODUCTS

Abstracts

Objective:

To develop a proposal for metrics for protocols and other technical products to be applied in assessing the Postgraduate Programs of Medicine III - Capes.

Methods:

The 2013 area documents of all the 48 Capes areas were read. From the analysis of the criteria used by the areas at the 2013's Triennal Assessment, a proposal for metrics for protocols and other technical products was developed to be applied in assessing the Postgraduate Programs of Medicine III. This proposal was based on the criteria of Biological Sciences I and Interdisciplinary areas.

Results:

Only seven areas have described a scoring system for technical products. The products considered and the scoring varied widely. Due to the wide range of different technical products which could be considered relevant, and that would not be punctuated if they were not previously specified, it was developed, for the Medicine III, a proposal for metrics in which five specific criteria to be analyzed: Demand, Relevance/Impact, Scope, Complexity and Adherence to the Program. Based on these criteria, each product can receive 10 to 100 points.

Conclusion:

This proposal can be applied to the item Intellectual Production of the evaluation form, in subsection "Technical production, patents and other relevant production". The program will be scored as Very Good when it reaches mean ≥150 points/permanent professor/quadrennium; Good, mean between 100 and 149 points; Regular, mean between 60 and 99 points; Weak mean between 30 and 59 points; Insufficient, up to 29 points/permanent professor/quadrennium.

Protocols; Handbooks; Classification; Education, graduate


Objetivo:

Desenvolver proposta de métricas para protocolos e outras produções técnicas a serem aplicadas na avaliação dos Programas de Pós-Graduação da Área Medicina III da Capes.

Métodos:

Foram lidos os documentos de área de 2013 de todas as 48 Áreas da Capes. A partir da análise dos critérios utilizados por elas na avaliação trienal 2013, foi desenvolvida uma proposta de métricas para protocolos e outras produções técnicas. Esta proposta foi baseada nos critérios adotados pelas Áreas Ciências Biológicas I e Interdisciplinar.

Resultados:

Apenas sete áreas descreveram sistema de pontuação para produtos técnicos, e as produções consideradas e a pontuação variaram amplamente. Dada à imensa gama de produções técnicas diferentes que podem ser consideradas relevantes, e que não seriam contempladas em sistema de pontuação caso fossem especificadas, foi desenvolvida para a Medicina III uma proposta de métricas em que são analisados cinco critérios específicos para cada produção: Demanda, Relevância/Impacto, Abrangência, Complexidade e Aderência ao Programa. Com base nestes critérios, cada produção pode receber de 10 a 100 pontos.

Conclusão:

Esta proposta poderá ser aplicada ao item Produção Intelectual da Ficha de Avaliação, subitem "Produção técnica, patentes e outras produções consideradas relevantes". Será considerado Muito Bom o Programa que obtiver média ≥150 pontos/docente permanente/quadriênio; Bom, média entre 100 e 149 pontos; Regular, média entre 60 e 99 pontos; Fraco, média entre 30 e 59 pontos; e Deficiente média ≤29 pontos/docente permanente/quadriênio.

Protocolos; Manuais; Classificação; Educação de pós-graduação


INTRODUCTION

The Professional Master's (MP) is strictly speaking postgraduate modality focused on the training of professionals in the various fields of knowledge, through the study of techniques, processes or issues that meet any demand of the labor market11. Fundação CAPES - Ministério da Educação. Mestrado Profissional: o que é? Disponível em: http://www.capes.gov.br/avaliacao/sobre-a-avaliacao/mestrado-profissional-o-que-e. Acesso em 14/02/2015.
http://www.capes.gov.br/avaliacao/sobre-...
. Its objectives include professionals training for the exercise of professional practice and advanced manufacturing procedures, to improve the effectiveness and efficiency of public and private organizations through the troubleshooting and the generation and application of innovation processes; to transfer knowledge to society; to contribute to aggregate competitiveness and to increase productivity in companies, public and private organizations, among others22. Ministério da Educação. Portaria Normativa MEC nº 17, de 28 de dezembro de 2009. Disponível em: https://www.capes.gov.br/images/stories/download/legislacao/Revogada-Portaria-Normativa-n_7-22-de-junho-2009-Mestrado-Profissional.pdf. Acesso em 14/02/2015.
https://www.capes.gov.br/images/stories/...
.

Given to the identity of the MP, the technical production is fundamental to this mode; can be inferred that the technical production is for the MP as well as the scientific production is for academic programs.

Nevertheless, as the number of MP is still very low in most of Capes areas, even absent in some, most of them do not have specific metrics and criteria for technical productions.

Among the technical products are the following protocols, manuals and booklets. According to the dictionary of the Portuguese language, protocol is characteristic of rigid rules for procedures; manual, as the work of small format that contains notions or guidelines relating to discipline or technical book that guides the implementation or the particular task improvement; it is a practical guide; and primer can be applied to any instructional elementary text33. Houaiss A, Villar MS, Franco FMM. Dicionário Houaiss da língua portuguesa. 1ª ed. Rio de Janeiro: Objetiva; 2009. 1986 p..

Specifically in the health area, this type of technical production is very important because it can have a major impact on population health; therefore, should be encouraged, particularly in MP courses, to have direct application in society. However, were not found in the 2013 area of ​​documents in any of the areas of Capes, definitions or metrics for evaluation protocols, manuals or booklets.

The objective of this study was to develop a proposal for metrics to protocols and other technical productions, such as manuals and brochures, to be applied in Medicine III.

METHODS

This study was conducted with the objective of developing a proposal to be presented and discussed with coordinators, teachers and students of the postgraduate programs of Medicine III, during the Fifth National Meeting of Postgraduate - Medicine III of CAPES, held on 8 and December 9, 2014, in São Paulo, Brazil.

Were read and analyzed the 2013 year the area documents from all 48 areas of Capes, and it was found that the technique production is covered predominantly in qualitative manner, and often generically. In addition, it was found that the item "Other techniques productions" of the evaluation form can cover wide range of technical production, which would not be tackled in a scoring system if they failed to be specified.

So, was developed for Medicine III a proposal for metrics that considers specific aspects of each production considered "other", that is, not one of those that have defined metrics such as patents, books and events, for example. This proposal was based on the criteria adopted by areas Biological Sciences I and Interdisciplinar44. Documento de área e Comissão da Trienal 2013. Área Ciências Biológicas I. Disponível em: http://www.capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Ciencias_Biologicas_I_doc_area_e_comiss%C3%A3o_block.pdf. Acesso em 18/11/2014.
http://www.capes.gov.br/images/stories/d...
,55. Documento de área e Comissão da Trienal 2013. Área Interdisciplinar. Disponível em: http://capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Interdisciplinar_doc_area_e_comiss%C3%A3o_block.pdf. Acesso em 18/11/2014.
http://capes.gov.br/images/stories/downl...
.

The proposed metrics may be included in the Evaluation Form of Medicine III programs in the item Intellectual Production, subsection "Technical productions and other productions considered relevant" excluding patents (which must have another classification), assigning to this subsection percentages 40% and 10% of the weight assigned to the item Intellectual Production in MP and academic programs, respectively66. Documento de área e Comissão da Trienal 2013. Área Medicina III. Disponível em: http://capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Medicina_III_doc_area_e_comiss%C3%A3o_att08deoutubro.pdf. Acesso em 18/11/2014.
http://capes.gov.br/images/stories/downl...
.

RESULTS

Were not found in the evaluation forms presented in all area documents definitions, criteria or metrics for protocols, manuals or booklets. The technical production is covered predominantly in qualitative manner, and largely in generic way .

Most do not adopt specific metrics for valuing technical products in general, with the exception of patents, for which the Biotechnology, Food Science, Biological Sciences III, Physical Education, Engineering I, III and IV and Interdisciplinary adopt punctuation system 55. Documento de área e Comissão da Trienal 2013. Área Interdisciplinar. Disponível em: http://capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Interdisciplinar_doc_area_e_comiss%C3%A3o_block.pdf. Acesso em 18/11/2014.
http://capes.gov.br/images/stories/downl...
,77. Documento de área e Comissão da Trienal 2013. Área Biotecnologia. Disponível em: http://capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Biotecnologia_doc_area_e_comiss%C3%A3o_block.pdf. Acesso em 18/11/2014.
http://capes.gov.br/images/stories/downl...

8. Documento de área e Comissão da Trienal 2013. Área Ciência de Alimentos. Disponível em: http://capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Ci%C3%AAncia_de_Alimentos_doc_area_e_comiss%C3%A3o_21out.pdf . Acesso em 18/11/2014.
http://capes.gov.br/images/stories/downl...

9. Documento de área e Comissão da Trienal 2013. Área Ciências Biológicas III. Disponível em: http://www.capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Ciencias_Biologicas_III_doc_area_e_comiss%C3%A3o_block.pdf. Acesso em 18/11/2014.
http://www.capes.gov.br/images/stories/d...

10. Documento de área e Comissão da Trienal 2013. Área Educação Física. Disponível em: http://capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Educa%C3%A7%C3%A3o_F%C3%ADsica_doc_area_e_comiss%C3%A3o_att08deoutubro.pdf. Acesso em 18/11/2014.
http://capes.gov.br/images/stories/downl...

11. Documento de área e Comissão da Trienal 2013. Área Engenharias I. Disponível em: http://capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Engenharias_I_doc_area_e_comiss%C3%A3o_16out.pdf. Acesso em 18/11/2014.
http://capes.gov.br/images/stories/downl...

12. Documento de área e Comissão da Trienal 2013. Área Engenharias III. Disponível em: http://capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Engenharias_III_doc_area_e_comiss%C3%A3o_16out.pdf. Acesso em 18/11/2014.
http://capes.gov.br/images/stories/downl...
-1313. Documento de área e Comissão da Trienal 2013. Área Engenharias IV. Disponível em: http://capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Engenharias_IV_doc_area_e_comiss%C3%A3o_16out.pdf. Acesso em 18/11/2014.
http://capes.gov.br/images/stories/downl...
.

The vast majority of areas, including Medicine III currently considered qualitatively technical publications for international, national, state or municipal bodies, papers published in technical journals, participation in technical committees (international, national, state or local), publishing of technical journals (scientific editor, associate editor or reviewer), development of protocols, standards or programs, consultancy or technical assistance, prototyping, and training courses, training or expertise of area professionals66. Documento de área e Comissão da Trienal 2013. Área Medicina III. Disponível em: http://capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Medicina_III_doc_area_e_comiss%C3%A3o_att08deoutubro.pdf. Acesso em 18/11/2014.
http://capes.gov.br/images/stories/downl...
.

Only seven areas described scoring system for technical products: Biological Sciences I, Applied Social Sciences I, Physical Education, Engineering I and III, Interdisciplinary and Psicology44. Documento de área e Comissão da Trienal 2013. Área Ciências Biológicas I. Disponível em: http://www.capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Ciencias_Biologicas_I_doc_area_e_comiss%C3%A3o_block.pdf. Acesso em 18/11/2014.
http://www.capes.gov.br/images/stories/d...
,55. Documento de área e Comissão da Trienal 2013. Área Interdisciplinar. Disponível em: http://capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Interdisciplinar_doc_area_e_comiss%C3%A3o_block.pdf. Acesso em 18/11/2014.
http://capes.gov.br/images/stories/downl...
,1010. Documento de área e Comissão da Trienal 2013. Área Educação Física. Disponível em: http://capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Educa%C3%A7%C3%A3o_F%C3%ADsica_doc_area_e_comiss%C3%A3o_att08deoutubro.pdf. Acesso em 18/11/2014.
http://capes.gov.br/images/stories/downl...

11. Documento de área e Comissão da Trienal 2013. Área Engenharias I. Disponível em: http://capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Engenharias_I_doc_area_e_comiss%C3%A3o_16out.pdf. Acesso em 18/11/2014.
http://capes.gov.br/images/stories/downl...
-1212. Documento de área e Comissão da Trienal 2013. Área Engenharias III. Disponível em: http://capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Engenharias_III_doc_area_e_comiss%C3%A3o_16out.pdf. Acesso em 18/11/2014.
http://capes.gov.br/images/stories/downl...
,1414. Documento de área e Comissão da Trienal 2013. Área Ciências Sociais Aplicadas I. Disponível em: http://www.capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Ciencias_Sociais_Aplicadas_doc_area_e_comiss%C3%A3o_16out.pdf. Acesso em 18/11/2014.
http://www.capes.gov.br/images/stories/d...
,1515. Documento de área e Comissão da Trienal 2013. Área Psicologia. Disponível em: http://www.capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Psicologia_doc_area_e_comiss%C3%A3o_21out.pdf. Acesso em 18/11/2014.
http://www.capes.gov.br/images/stories/d...
. Yields and the score vary widely depending on the characteristics of each area.

Given the wide range of different technical productions that may be considered relevant, and would not be included in the scoring system if they were specified, this proposal for metrics analyze specific aspects of each production envisaged "other" (that is, not one of those that have defined metrics such as patents, books and events, etc.).

This proposal is based on the criteria adopted by Biological Sciences I and Interdisciplinary areas44. Documento de área e Comissão da Trienal 2013. Área Ciências Biológicas I. Disponível em: http://www.capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Ciencias_Biologicas_I_doc_area_e_comiss%C3%A3o_block.pdf. Acesso em 18/11/2014.
http://www.capes.gov.br/images/stories/d...
,55. Documento de área e Comissão da Trienal 2013. Área Interdisciplinar. Disponível em: http://capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Interdisciplinar_doc_area_e_comiss%C3%A3o_block.pdf. Acesso em 18/11/2014.
http://capes.gov.br/images/stories/downl...
with adaptations, and considers five broad criteria: Demand, Relevance or Impact (economic, social, inclusion in basic education and health system), Scope, Complexity (innovative feature, preparation time, size) and adherence to the program (consistency with lines of research/practice, student participation, linking the paper for completion of the course).

Each production would be classified as T1 to T3, in descending order of importance. Figure 1 shows the classification proposed by score.

FIGURE 1
- Proposal for scoring protocols and other technical productions Medicine III - Capes

Multiplying the score by five criteria, each technique production could receive from 10 to 100 points. From this score, we elaborated a proposal of metrics for evaluation of Medicine III programs, shown in Figure 2.

FIGURE 2
- Proposal for allocation of concepts according to the scores of Other Techniques Productions Medicine III - Capes

The proposed metrics could be applied to item 4 of the Evaluation Form of Medicine III programs on the Intellectual Production, subsection "Technical production and other productions considered relevant." This subsection keeps the percentage of 10% for academic programs and 40% for MP programs respectively66. Documento de área e Comissão da Trienal 2013. Área Medicina III. Disponível em: http://capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Medicina_III_doc_area_e_comiss%C3%A3o_att08deoutubro.pdf. Acesso em 18/11/2014.
http://capes.gov.br/images/stories/downl...
.

DISCUSSION

Various spheres of government, trade associations and organizations provide professionals and public protocols, manuals and booklets in the most different areas of health. At the federal level, for example, the Ministry of Health offers and constantly updated Clinical Protocols and Therapeutic Guidelines - PCDT, which aims to establish the criteria for diagnosis, treatment algorithm, and mechanisms for clinical monitoring of different diseases1616. Portal da Saúde - SUS. (PCDT) Protocolos Clínicos e Diretrizes Terapêuticas. Disponível em: http://portalsaude.saude.gov.br/index.php/component/content/article?id=9315. Acesso em 18/11/2014.
http://portalsaude.saude.gov.br/index.ph...
.

Another example of protocols with national scope is the designed guidelines established by the National Agency of Health Insurance and the Brazilian Medical Association. This project consists of a number of protocols to aid medical decision-making, which were developed by different specialty societies1717. Agencia Nacional de Saúde Suplementar (ANS), Associação Médica Brasileira (AMB). Projeto Diretrizes. Disponível em: http://www.projetodiretrizes.org.br/. Acesso em 18/11/2014.
http://www.projetodiretrizes.org.br/...
. The Federal Medical Council also distributes, on a national level, a number of protocols, manuals and booklets for professionals in the medical area1818. Conselho Federal de Medicina - CFM Publicações. Manuais, Protocolos e Cartilhas. Disponível em: http://portal.cfm.org.br/index.php?option=com_content&view=article&id=22044:protocolos-do-cfm&catid=3 /. Acesso em 18/11/2014.
http://portal.cfm.org.br/index.php?optio...
. They are also available to professionals and public in general, many other protocols, manuals and booklets, with national, state, regional or municipal affairs, in many different areas of health.

This type of technical production comes with the mission of the MP to transfer knowledge to society and should be included in Medicine III. However, were not found in the 2013 area documents in any of the areas of Capes, definitions or metrics for protocols, manuals or booklets evaluation.

In the Capes programs Evaluation Form, technical output is valued in subsection "Technical productions and other productions considered relevant" contained in "Intellectual Production". It was observed that among the areas, relevant productions were considered varying widely, depending on the characteristics of each. For example, areas of Anthropology/Archaeology, Physical Education, Nursing, Engineering III, Education, Interdisciplinary and Urban and Regional Planning/Demography considered possible the organization of events and the presentation of papers at events and publications in annals55. Documento de área e Comissão da Trienal 2013. Área Interdisciplinar. Disponível em: http://capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Interdisciplinar_doc_area_e_comiss%C3%A3o_block.pdf. Acesso em 18/11/2014.
http://capes.gov.br/images/stories/downl...
,1010. Documento de área e Comissão da Trienal 2013. Área Educação Física. Disponível em: http://capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Educa%C3%A7%C3%A3o_F%C3%ADsica_doc_area_e_comiss%C3%A3o_att08deoutubro.pdf. Acesso em 18/11/2014.
http://capes.gov.br/images/stories/downl...
,1212. Documento de área e Comissão da Trienal 2013. Área Engenharias III. Disponível em: http://capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Engenharias_III_doc_area_e_comiss%C3%A3o_16out.pdf. Acesso em 18/11/2014.
http://capes.gov.br/images/stories/downl...
,1919. Documento de área e Comissão da Trienal 2013. Área Antropologia/Arqueologia. Disponível em: http://www.capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Antropologia_Arqueologia_doc_area_e_comiss%C3%A3o_21out.pdf. Acesso em 18/11/2014.
http://www.capes.gov.br/images/stories/d...

20. Documento de área e Comissão da Trienal 2013. Área Enfermagem. Disponível em: http://www.capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Enfermagem_doc_area_e_comiss%C3%A3o_att08deoutubro.pdf. Acesso em 18/11/2014.
http://www.capes.gov.br/images/stories/d...

21. Documento de área e Comissão da Trienal 2013. Área Ensino. Disponível em: http://www.capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Ensino_doc_area_e_comiss%C3%A3o_block.pdf. Acesso em 18/11/2014.
http://www.capes.gov.br/images/stories/d...
- 2222. Documento de área e Comissão da Trienal 2013. Área Planejamento Urbano e Regional/Demografia. Disponível em: http://www.capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Planejamento_Urbano_Demografia_doc_area_e_comiss%C3%A3o_16out.pdf. Acesso em 18/11/2014.
http://www.capes.gov.br/images/stories/d...
. Dentistry valued ​​the presentation of papers at events and publications in annals only when referred to student production2323. Documento de área e Comissão da Trienal 2013. Odontologia. Disponível em: http://www.capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Odontologia_doc_area_e_comiss%C3%A3o_att08deoutubro.pdf. Acesso em 18/11/2014., and most areas, including the Medicines I, II and III did not consider in academic master and doctoral programs such productions66. Documento de área e Comissão da Trienal 2013. Área Medicina III. Disponível em: http://capes.gov.br/images/stories/download/avaliacaotrienal/Docs_de_area/Medicina_III_doc_area_e_comiss%C3%A3o_att08deoutubro.pdf. Acesso em 18/11/2014.
http://capes.gov.br/images/stories/downl...
. So, perhaps the greatest challenge in the evaluation of MP is to define the technical productions that should be considered, as this definition will induce demand for this type of production among courses.

The MP is very new modality in Medicine III. The area has not yet assessed MP courses; therefore, there is not yet well established parameters for the assessment of the technical production, essential in this type of master's degree. However, the recent increase in the number of MP recommended by Capes and evaluated by the Medicine III makes essential the development of metrics for the assessment of the technical production.

The purpose of this proposal metrics for protocols and other technical production was not to analyze exclusively numbers. Rather, the idea is to allow qualitative assessment of each production, based on its scope, its impact, its relevance, its complexity and its relevance in program lines. The proposed figures have the sole purpose to establishment evaluation parameters, and must be analyzed together with the rest of the production.

CONCLUSION

A proposal of metrics for protocols and other technical output, which can be applied to the item Intellectual Production of the Evaluation Form of Medicine III programs, in subsection "Productions and other techniques considered relevant production" was developed. Should be maintained in this subsection the percentage of 10% for academic programs and 40% for MP. Will be considered Very Good program that gets average greater than 150 points for permanent teachers in four years; Good, between 100 and 149; Regular, between 60 and 99 points; Poor, between 30 and 59 points; Insuficient equal to or less than 29 points for permanent teachers in four years.

REFERENCES

Publication Dates

  • Publication in this collection
    2015

History

  • Received
    19 Feb 2015
  • Accepted
    12 Sept 2015
Colégio Brasileiro de Cirurgiões Rua Visconde de Silva, 52 - 3º andar, 22271- 090 Rio de Janeiro - RJ, Tel.: +55 21 2138-0659, Fax: (55 21) 2286-2595 - Rio de Janeiro - RJ - Brazil
E-mail: revista@cbc.org.br