The technical report of the CAPES’ technical production working group: a critical view

Abstract

This paper attempted to perform an analysis of the technical report for assessment of technical and technological products born from working group by Capes, in 2019. We aimed evaluate its instrumental adequacy. Our analysis applaied the literature about characteristics that must necessarily be present in technical or technological reports to support our conclusions. As a result, we found evidence that the Capes' technical report about assessment technical and technological products doesn't suit the necessary characteristics to be established as applicable assessment instrument over other technical reports.

Keywords:
technical report; gray literature; assessment

Resumo

O presente trabalho buscou realizar uma análise do relatório técnico da Capes, sobre avaliação de produtos técnicos e tecnológicos, produzido pelo grupo de trabalho responsável, em 2019, com o objetivo de avaliar sua adequação instrumental. Para nossas análises, utilizamos a literatura que aborda as características que, necessariamente, devem estar presentes em relatórios técnicos ou tecnológicos, dando suporte às nossas conclusões. Como resultado, identificamos evidências de que o relatório técnico da Capes não reúne as condições necessárias para figurar como um instrumento viável à uma aplicação na análise de outros relatórios técnicos.

Palavras-chave:
relatório técnico; literatura cinza; avaliação

Resumen

El presente trabajo buscó realizar un análisis del informe técnico Capes, sobre la evaluación de productos técnicos y tecnológicos, elaborado por el grupo de trabajo responsable, en el año 2019. Nos propusimos evaluar su adecuación instrumental. Para nuestro análisis y nuestras conclusiones, utilizamos la literatura que aborda los características que, obligatoriamente, deberán estar presentes en los informes técnicos o tecnológicos. Como resultado, identificamos evidencias de que el informe técnico Capes no reúne las condiciones necesarias para convertirse en un instrumento viable para su aplicación en el análisis de otros informes técnicos.

Palavras clave:
relato técnico; literatura gris; evaluación

1 Introduction

A report is a concise and organized document in which reasoning and activities related to a subject are recorded, with the fundamental objective of communicating them, either to the author themselves at future moments, or to third parties. Reports, technical or technological reports, academic-scientific or academic-pedagogical reports are, among other types of documents, part of the category known as "grey literature," characterized as informal manuscripts with limited circulation and not subject to peer review (or in a broader sense, not edited or published through conventional public access channels). They are understood as elements capable of effectively contributing to the dissemination of knowledge and, as such, require clear criteria to enable their evaluation and an acceptable quality of the final text, as a form of selection (Botelho; Oliveira, 2015).

Concerned with issues related to the proper communication and evaluation of this type of literature within Graduate Programs, a agency of the Brazilian Ministry of Education, “Coordenação de Aperfeiçoamento de Pessoal de Nível Superior” (Capes), in 2019, established the Working Group on Technical Production (WGTP), with representatives from all Capes Evaluation Areas, aiming to standardize criteria for the evaluation of what they referred to as the "Conclusive Technical Report - CTR," defined as follows:

A concisely prepared text containing information about the project/activity carried out, from its planning to its conclusions. It indicates in its content the relevance of the results and conclusions in terms of social and/or economic impact and the application of the knowledge produced (Brasil, 2019, p. 52; translation ours).

In this regard, several meetings were held for deliberations and decisions regarding the WGTP Report (Brasil, 2019), which included the following stages:

•Stage 1 – Evaluation of the results achieved in the study previously conducted by GT 06 CAPES (2015/2016); • Stage 2 – Consultation with all evaluation areas regarding the importance of each of the 62 technical productions listed in Table 1, conducted through an electronic form (Annex 1); •Stage 3 – Analysis of the relevance attributed by the areas to each technical production (Annex 2), and grouping of similar productions; •Stage 4 – Application of concepts to the Study and conversion of the most relevant technical production into Products and not the processes involved; •Stage 5 – Preparation of definitions, descriptive fields, and examples for each product, using the stratification criteria established in a previous study as a basis; •Stage 6 – Evaluation of the results achieved and preparation of the final report (Brasil, 2019, p. 10; translation ours).

Table 1
Quality Criteria for a Technical Report

QUALITY CRITERIA	Introduction	Theoretical Framework		Development		Formal Aspects
			Method	Results and Discussion	Conclusion
	Presentation of the topic Presence of practical and theoretical arguments on the problem to be addressed/ reported or the improvement opportunity Clear presentation of the study Objective brief account of the practical intervention	Presentation of the literature that enables the understanding of the study topic Existence of theoretical arguments that support the intervention proposal	Theoretical foundation of the methodology Methodology consistency with the study objective Detailed description of data collection Detailed description of data analysis	Characterization of the organization and the problem in the investigated context Presentation of the developed intervention Description of the activities undertaken to solve the problem Description of the achieved results Analysis of the results considering the literature presented in the theoretical foundation	Presentation of evidence that the proposed objectives were achieved Presentation of the study's contributions to professional practice and the field of knowledge Explanation of the lessons learned from the study Demonstration of the relevance of the study's results for other similar cases Declaration of the study's limitations and their implications Presentation of a suggestion for a future research agenda	All references cited in the text must be listed in the References section at the end of the document. All references listed at the end must be cited in the text. Publication guidelines should be adjusted according to the communication venue. Adherence to grammatical rules.

• Source: Biancolino; Kniess; Maccari; Rabechini Jr. (2012); Martens; Pedron; Oliveira (2021).

In light of this context and considering the importance of enhancing the production of grey literature within Graduate Programs, the present study aimed to analyze the WGTP Report in relation to the existing literature, in order to assess its suitability as a guiding instrument in establishing criteria and indicators that support the evaluation of technical reports as products suitable for qualified dissemination of technical and technological knowledge, with a scientific approach.

2 Literature review

According to Soubhia, Ruffino, and Dessunti (2005), a report is a document in which systematic and organic descriptions of reasoning and activities about a topic are presented, requiring complex mental operations such as organizing, synthesizing, arguing, analyzing, contrasting, justifying, composing, and proposing. Martens, Pedron, and Oliveira (2021) propose the term "technological article" or "technical report" as

[...] works with a practical problem-solving approach, while maintaining scientific rigor [...] technological articles are professional productions aimed at presenting solutions to problems [...] They should also be written in language accessible to their audience, which is generally composed not only of academics but also of market professionals [...] (Martens; Pedron; Oliveira, 2021, p. 143; translation ours).

Biancolino et al (2012) define a technical report as the final descriptive product of applied research or technical production, prepared with scientific and methodological rigor. They further elaborate, particularly in the organizational field:

The technical report follows the format of a short scientific article (7 to 10 pages, including references). Therefore, the technical report should: (1) present the basic differentiation of describing an intervention instead of analyzing a conventional theoretical/empirical object; and (2) propose practical and concrete improvements/results that can be adopted in the future[...]with the support of theoretical references from the field (Biancolino et al, 2012, p. 299; translation ours).

Moreover, as outlined by Martens, Pedron, and Oliveira (2021), we understand that a manuscript containing known solutions to known problems, without apparent innovation, does not qualify as an academic-scientific technical report, but rather as an academic-pedagogical technical report, similar to the proposal of Soubhia, Ruffino, and Dessunti (2005). In this context, Biancolino et al (2012) and Martens, Pedron, and Oliveira (2021) present the necessary characteristics of a technical report, characteristics that align it with a scientific article and the propositions presented by Savickas (2009), Motta (2017), and Souza (2022), as depicted in Table 1.

As Savickas (2009) and Motta (2017) make clear, an article is a more rigorous type of report, originating from academic research and, we can add, subjected to peer review.

The scientific quality of a Technological Article does not differ from traditional articles[...] the difference between scientific and technological articles lies in the approach to research and, in some cases, the audience (Motta, 2017, p. 03; translation ours).

As a rule, a report consists of pre-textual elements (cover, title page, and verso of the title page), textual elements (introduction, development, and conclusion), and post-textual elements (references, appendix, and glossary), varying in form and content of these elements in different sections of the manuscript, according to their purposes, intent, content, target audience, and communication medium (Soubhia; Ruffino; Dessunti, 2005; Savickas, 2009; Martens; Pedron; Oliveira, 2021). Whatever the type of report, it has a fundamental purpose: communication, and a basic requirement: presentation of useful and pertinent information, written in a clear, concise, organized, and substantiated manner. As a result, the report aggregates information from various consultable sources, for the knowledge of third parties, and enables, through cognitive effort in synthesizing, the enrichment of the personal underlying concepts of the author.

The underlying concept, according to some authors(1-3), is the concept that serves as an anchor for new information that is incorporated into the individual's cognitive structure over the course of their history through processes of progressive differentiation and integrative reconciliation. As new information is assimilated, the underlying concept becomes stronger and more comprehensive (Soubhia; Ruffino; Dessunti, 2005, p. 270; translation ours).

In this direction, Soubhia, Ruffino, and Dessunti (2005), in a study analyzing 83 academic reports from students in two different groups of the Nursing course, used a methodology involving two readings (the first aimed to familiarize themselves with the content of the work, and the second involved highlighting and annotating the margins of the pages to identify the ideas of each topic). They organized raw data into a frequency table indicating the proportion of correct responses and the degree of significance and compared the performance between the two groups using the MannWhitney Test (x²) or Fisher's test. Relevant to our discussion, in summary, the authors propose, as part of the results, the need for a report to possess the characteristics presented in Table 2.

Table 2
Structure and Quality Criteria of an Academic-Pedagogical Technical Report

QUALITY CRITERIA	Title	Introduction	Development	Conclusions	References
	Clarity Conciseness Concreteness Creativity	Presentation of the topic Conceptual definitions Problem delimitation Encouragement for reading Justification of the work Definition of objectives Introduction to the development	Literature contrast Proactive critical argumentation Assertive theoretical foundation Justification of positions Theoretical-practical dialectic Description of processes and results	Brevity Clarity Objectivity Closure of reasoning Proposal of developments Connection between literature, objectives, and results	Inclusion in the text Specificity Relevance Currency Adherence to technical standards

• Source: Soubhia; Ruffino; Dessunti (2005).

The postulations of Soubhia, Ruffino, and Dessunti (2005) find resonance in Pereira (2012), Lima (2013), and Porto and Gurgel (2018) when discussing elements related to the writing of an article. They are also echoed in Martens, Pedron, and Oliveira (2021), who present characteristics necessary for a technical report, closely aligned with expectations for a scientific article. According to Lima (2013, p. 51; translation ours), "A good article should be written with clarity, precision, and fluency, essential conditions for readers to feel interested and capable of understanding its content." Pereira (2012, p. 26; translation ours) warns that "Scientific writing constitutes a complex presentation of facts and arguments, guided by an elaborate process of reasoning."

Echoing the attributes proposed by Savickas (2009) in relation to the scientific report, Motta (2017) and Martens, Pedron, and Oliveira (2021) further emphasize the necessary scientific foundation and the concern for clear and objective writing that is suitable for their audience, which largely consists of non-academic professionals.

The key criteria for evaluating a Technological Article are clarity and objectivity. Articles that get straight to the point certainly have a greater chance of advancing through desk review. Thus, it is important to make it clear in the title and abstract what the manuscript is about[...] (Motta, 2017, p. 03; translation ours).

Additionally, according to Savickas (2009), a good research report must possess clear and concise language that is accessible to non-experts, portray the state of the art of the addressed topic, and demonstrate an appropriate level of research maturity to constitute a significant contribution to the field. In summary, Savickas' propositions can be observed in Table 3 (above), which contains the necessary criteria for constructing a scientific report. Furthermore, Savickas (2009) highlights some characteristics that are commonly seen as mistakes, compromising the quality of the report. Despite these characteristics potentially being transformed into positive criteria, alongside others, we will follow the author's inclination and emphasize them as aspects to be avoided.

Table 3
Structure and Quality Criteria of a Scientific Report

QUALITY CRITERIA	Title	Introduction	@	Development	@	References
	@	@	Method	Results	Discussion	@
	Presence of keywords Conciseness and Objectivity Informativen ess of the article's content and nature Absence of rhetorical questions and jokes	Work justification Definition of objectives and purposes Description of the hypothesis/questi on and its origin Context and origins of the problem Problem delimitation Expectations about problem resolution Brief presentation of previous research	Clarity of the method, subject, and research variables Explanation of ethical aspects, conveniences, biases, and conflicts of interest Validity and reliability of instruments and procedures Consistency between hypotheses and research measures Significance of the sample/research material Description and rationale of data collection	Comprehensive description of results1 Inclusion of responses to all formulated questions Clear and appropriate data illustration for understanding Presentation of significance tests Explanation of the importance and impact of findings Indication of method limitations	Literature contrast Emphasis on significant findings Presentation of alternative explanations for results Connection between results and hypotheses Theoretical-practical implications of the outcome Presentation and interpretation of conclusions Study limitations in relation to the ideal Estimated confidence and limitations of conclusions Suggestion of methodological procedures and research agenda	Conciseness Necessity Relevance Utility Accessibility Legitimacy Reliability Precision Currency

• Source: Savickas (2009).

In this regard, Savickas (2009) points out errors such as: excessive empiricism devoid of robust theoretical foundation; inversion of syntactic and semantic relationships², inclusion of content from one section into another; textual prolixity with irrelevant, repeated, or extraneous content; baseless and unfounded speculations; reductionist argumentation omitting relevant content; use of primary expressions and clichés³, and underestimation of the reader's interpretive capacity, explicitly stating what can be easily inferred from the text⁴, terminological confusion; lack of acceptance or refutation of proposed hypotheses; insufficient information for assessment and replication of the work; excessive detailing of standardized and published methods; excessive heterogeneity in the studied sample; lack of clarity regarding instrument scores; inadequate or insufficient description of statistical analyses; unclear figures and tables with insufficient information for interpretation; absence of promised information in the method section; conclusions based solely on figures and tables without adequate textual description; and redundancy of information between text, figures, and tables. Porto and Gurgel (2018, p. 115; translation ours) also offer a warning about this last error: "Tables summarize a set of observations and should be self-explanatory, without repeating information already contained in the text."

Motta (2017) presents additional criteria for the evaluation of technical reports:

In the introduction, clearly state the problem situation and/or opportunity for improvement related to the analyzed context (organization/government/social actors involved)[...]Include a brief diagnostic text about the situation/problem and/or opportunity, demonstrating mastery of the subject under study and the theoretical-scientific foundations supporting this diagnosis[...]Provide a concise description of the procedures used to gather data and relevant information for analyzing the situation[...]Present as an article a text that analyzes the problem situation and discusses possible alternatives for its resolution, innovation, improvement, or extrapolation[...]Conclude the text by demonstrating the proposal's contribution to organizations and/or society (Motta, 2017, p. 03; translation ours).

In line with this, Lima (2013) presents a reference on what not to do when writing an article and defines a technical report as a manuscript that "Reports on an investigation already carried out and specifies the steps taken, the results obtained, the analysis/interpretation of data, and the established conclusions" (Lima, 2013, p. 58; translation ours), corroborating many elements presented by Soubhia, Ruffino, and Dessunti (2005), Savickas (2009), Motta (2017), and Martens, Pedron, and Oliveira (2021), indicating the similarities between a report and an article regarding various aspects. Thus, a technical report, not unlike a scientific article (Pereira, 2012; Lima, 2013; Porto; Gurgel, 2018), should be capable of convincing the reader of its truth by describing and arguing with clarity and objectivity, either through logical concatenation of its ideas or by supporting these ideas with practical experience or appropriate literature that can provide substantiation for the assertions.

In this sense, borrowing from the contributions of Latour (1987) when discussing the production of scientific articles, the text of a report, regardless of its type, needs to incrementally and parsimoniously "pile up" information to create a positive impact on the reader. To achieve this, the text should be friendly in its dialogue, allowing the author to dissolve into its lines while the reader recognizes themselves within it, desiring to have written it. The digestibility and fluidity of the text are directly linked to the style and language used. The use of syntax, jargon, and technical terms suitable for the target audience should contribute to clarity while avoiding incomprehensibility for other audiences. Furthermore, the text should anticipate possible objections from the reader, constructing an argumentation that can challenge any counterarguments through a clear, coherent, and well-structured reasoning.

Special attention should be given to the title of a manuscript due to its seemingly simple yet significant role in captivating the reader. According to ABNT NBR 6023 (2018), the title is a word, expression, or phrase that expresses the constituent content of a document, while the subtitle, presented immediately after the title, provides extra clarification or supplementation to the title, aligned with the document's content. Souza (2022), in a study aiming to answer the question "How have authors presented the titles of scientific works published in the proceedings of the XXI National Meeting of Research and Graduate Studies in Information Science (ENANCIB)?" analyzed 342 titles of scientific articles, of which 67.3% were found to be appropriate according to the adopted theoretical framework. This framework enabled the stratification of analyzed works into nine classes: 1) nonspecific title; 2) affirmative title; 3) interrogative title; 4) title with formulas or symbols; 5) title with abbreviations; 6) sensationalist titles; 7) titles with excessive use of terms; 8) titles containing the methodology; and 9) themes in place of titles.

In general, we can observe that the expectations for titles in scientific articles align with the propositions for titles in academic-scientific and academic-pedagogical technical reports, as observed in Soubhia, Ruffino, and Dessunti (2005), Savickas (2009), Biancolino et al (2012), Martens, Pedron, and Oliveira (2021), and Souza (2022). The consensus lies in the requirement for titles to be precise, complete, concise, objective, informative, and specific, consisting of ten to twenty words, without sensationalism, power words, methodological description, illusions, abbreviations, equations, symbols, questions, affirmations, exclamations, periods, commas, or quotation marks.

Regarding the characteristics of writing articles, Souza (2022) also emphasizes the need for standardization, clarity, objectivity, precision, accessibility, and simplicity, aligning with what can be expected for report writing according to Motta (2017). Consequently, we can infer from Souza (2022) that the function of writing in a scientific article also coincides with the function of a report: useful, clear, and direct communication.

Additionally, it's worth noting that according to Savickas (2009), the "Conclusion" section, or what some may better define as "Final Considerations," is subsumed within the "Development/Discussion" section, while for Soubhia, Ruffino, and Dessunti (2005), it constitutes a separate section. On the other hand, Soubhia, Ruffino, and Dessunti (2005) present the "Development" section as a single block, whereas Savickas (2009) breaks it down into "Method," "Results," and "Discussion," delving into the details of each of these subsections. Both authors, with minor differences, align with the propositions of Martens, Pedron, and Oliveira (2021) for the structure and content of technical reports. However, more than differences, we observe similarities and complementarities among these authors regarding the expectations for a report, with a more detailed element in Savickas (2009), albeit with limitations in his original text. Thus, based on the literature presented, we embark on an analysis of the WGTP Report, the reflections of which we shall discuss next.

3 Discussion of Results

As a result of the Working Group on Technical Production (WGTP) within Capes, the Technical Conclusive Report (TCR) was defined as a product derived from activities carried out within Postgraduate Programs and subject to evaluation, according to the guidelines provided in the report produced by the WGTP (Brasil, 2019). The term TCR encompasses various types, subtypes, and purposes of reports, all of which can be considered under the umbrella term of gray literature (Botelho; Oliveira, 2015). Excerpts from Table 4 illustrate this typology.

Table 4
Types and Subtypes of Reports, According to the WGTP Report

TYPES OF REPORTS
Subtypes	"Technical conclusive report per se; Prepared management processes; Prepared market research; Applied simulations, scenario planning, and games; Prepared technology valuation; Prepared innovative business model; Prepared managerial tool; Technical opinions and/or notes on the validity, application, or interpretation of regulations" (Brasil, 2019, p. 19-20; translation ours).
Types	"Research project report; Technical advisory and consulting report and contract auditing; Environmental impact report or civil engineering work; Physical-chemical testing report for engineering, veterinary, chemistry, agronomy, etc.; Inspection/evaluation report in public and private institutions, agencies, or services" (Brasil, 2019, p. 52; translation ours).

• Source: Brasil (2019).

Although the types and subtypes of reports subject to evaluation were explicitly defined within the context of the WGTP Report, there is no detailed clarification of how these typologies interact as elements capable of guiding the assessment of this category of literature. The interconnection is left to the subjective deduction of the user of the WGTP Report when used as a guiding tool for the evaluation of technical products by Postgraduate Programs in Brazil. Such a characteristic contrasts with the propositions of Soubhia, Ruffino, and Dessunti (2005), Martens, Pedron, and Oliveira (2021), and Souza (2022), where textual clarity, including proper organization, cohesion, and coherence, is an essential element in a report. Without such clarity in the evaluation instrument, doubts arise about its existence in the results derived from its use, casting doubts on its validity.

When we compare the consulted literature (Savickas, 2009; ABNT, 2018; Souza, 2022) with the results obtained from the analysis of the WGTP Report, it is possible to observe a failure to meet the assumption of clear communication, starting with the title and subtitle of the latter, "Technical Production - Working Group", which presents two shortcomings: it is a non-specific title and has a theme in place of the title⁵. This lack of clarity extends to the body of the text. Upon analyzing its content, we notice that the enrichment of the underlying concepts (Soubhia; Ruffino; Dessunti, 2005) may not have been effective, given the conceptual mistakes and confusion throughout the text. In this sense, the Report exhibited shortcomings in terms of section organization, content conciseness, information clarity, theoretical accuracy, argumentative coherence, technical formalism, linguistic-grammatical rigor, validity of conclusions, and usefulness of results, contrary to assumptions outlined in Soubhia, Ruffino, and Dessunti (2005), Savickas (2009), Motta (2017), and Martens, Pedron, and Oliveira (2021).

As an example of the lack of organization, technical formalism, and linguistic-grammatical rigor, we have the proposed criteria for the evaluation of TCR in the section "Product Details according to Table 2” (our emphasis). In this section, the WGTP lists, through a form, criteria such as: the purpose of the Report, the degree of innovation in its content, the individual or collective nature of the report, and the nature and stage of its connection (if any) with research projects and scientific production of the author(s). They also request the non-obligatory declaration of resources, links, and applicability related to the technological product that led to TCR (Brasil, 2019). This section does not effectively align with the section "Stage 4 - Concepts Applied to the Study and Conversion of Technical Production into Products" (our emphasis), as it is inconsistent with the reference table for works, which became "Table 3" in the same section, since the 185th meeting of the Technical-Scientific Council of Higher Education, as mentioned in the document itself. Furthermore, the titles of each section, up to section 5, start with the expression "Stage," in reference to the methodological stages. However, this formal pattern is not followed in sections 6 and 7, without apparent reason. According to Pereira (2012), Lima (2013), and Porto and Gurgel (2018), this indicates a potential flaw in the manuscript, compromising its objectives.

Additionally, the fields of the TCR evaluation form, found in the section "Product Details (according to Table 2)," do not have a clear correspondence with the important information listed in the section "Stage 5 - Definitions, descriptive fields, and Description for each Product":

•Definition; •Examples; •Mandatory descriptive fields*; •Optional descriptive fields*; •Correlate with the options and fields available in the Sucupira and Lattes platforms; •Does the production need to be in the repository? •Classify and justify the productions and subtypes as technical or technological; •Determine if the production is the result of work carried out by the graduate program or if it is the result of individual work by the teacher, which would be done independently of whether the teacher is part of a program or not (Brasil, 2019, p. 21; translation ours).

As an example of the lack of theoretical precision and argumentative coherence, we can mention the attempt to differentiate technical from technological products. In order to delimit the content relevant to TCR, the WGTP sought to "establish the differences between a technical product and a technological product for the purpose of evaluating program production” (Brasil, 2019, p. 22; translation ours). To do so, they used criteria such as social impact, applicability, innovation, and complexity. It is worth noting the discriminatory potential attributed to the criterion of "innovation": "A product derived from the adaptation of existing knowledge will be considered a technical product and not technological" (Brasil, 2019, p. 2; translation ours). Also noteworthy is the definition assigned to technological product:

A technological product is a 'tangible object' with a high degree of novelty resulting from the application of new scientific knowledge, techniques, and expertise developed in the context of research in graduate programs, used directly to solve problems of companies producing goods or providing services to the population for the benefit of society (Brasil, 2019, p. 22; translation ours).

In both cases, the definition of a technical product and a technological product, it is possible to perceive a theoretical gap concerning the definitions used, which equate expressions like "innovation" and "novelty" and use the degree of innovation (incremental versus radical) to differentiate technical from technological products, without a strong theoretical basis, something unsupported by the literature in the field (Wolfe, 1994; Gregor; Hevner, 2013; Plonski, 2017; Lu; Matui; Gracioso, 2019). In the same line of reasoning, the WGTP presents criteria for evaluating the "innovation" criterion:

•Production with a high level of innovation: Development based on unpublished knowledge; •Production with a medium level of innovation: Combination of pre-established knowledge; •Production with a low level of innovation: Adaptation of existing knowledge; •Production with no apparent innovation: Technical production (Brasil, 2019, p. 24; translation ours).

These criteria have problems with their conceptual boundaries, as it becomes impractical to distinguish "Combination" from "Adaptation," as well as "pre-established knowledge" from "existing knowledge," or even to accurately and objectively distinguish high, medium, and low levels of innovation. Criteria, as analytical categories, need to have well-established boundaries in order to allow for the proper delineation of the object of evaluation without ambiguities (Laville; Dionne, 1999; Moraes; Galiazzi, 2006; Mamede, 2016; Carlomagno; Rocha, 2016).

In this regard, we may find an alternative to the current classification in Gregor and Hevner (2013), where they propose that invention corresponds to new solutions to new problems, improvement represents a new solution to a known problem, exaptation⁶ relates to a new use of known solutions for new problems, and application⁷ means the use of known solutions in known problems⁸, which blurs innovation within the concepts of improvement and exaptation. For Martens and Pedron (2019) and Martens, Pedron, and Oliveira (2021), of these four categories, only the first three relate to research problems at the stricto sensu level, as they require a systematic review of scientific and technical literature or even gray literature, and therefore have the potential to be included in a technical report, as they understand it.

In addition to the conceptual weakness of the criteria provided by the WGTP (Brasil, 2019), there is a contradiction in the statement that "A production with no apparent innovation" constitutes a "Technical production" when, in a previous section, it was stated that "A product derived from the adaptation of existing knowledge will be considered a technical product and not technological" (Brasil, 2019, p. 2; translation ours), as another example of argumentative and theoretical weakness. Such characteristics in a report compromise its clarity and usefulness, limiting its ability to convince the reader, as required by this type of work, as warned by Pereira (2012), Porto and Gurgel (2018), and Souza (2022).

To conclude, the usefulness of the results is an element that directly derives from the validity of the conclusions, which is dependent on theoretical precision and argumentative coherence, both of which are absent in most of the text, compromising validity and, consequently, usefulness. The validity of an instrument is related to its ability to measure and exactly delimit what it proposes to measure (Martins, 2006; Souza; Alexandre; Guirardello, 2017), which is reflected in the validity of the conclusions drawn from it. If there are inconsistencies in the conceptual boundaries of the criteria or categories proposed as references for analysis, a portion of the object under study may overlap with another analytical cut of the same object, creating noise in the interpretation of the other. Thus, an evaluation instrument with such characteristics will be a source of bias for the conclusions derived from its use in the analysis of other objects. Given the findings listed so far, the final considerations follow.

4 Final considerations

We understand that a technical-scientific report and a technical-pedagogical report - even though there may be areas of intersection between them, producing a hybrid type, as the TCR seems to be - differ from each other in terms of the origin of the content (the first, systematic research; the second, the account of an academic experience, which may or may not result from research) and their purpose (the first, the communication of research findings to third parties; the second, the documentation of an activity, for oneself or others, and the personal cognitive enrichment of the author). Within the limits established by the scientific culture within the field in question, both differ little from the structure and content envisaged for a scientific article, and there should be similarity in evaluating the quality of the three genres in terms of criteria such as objectivity, conciseness, clarity, parsimony, precision, organization, usefulness, ethics, technical formalism, theoretical-practical correspondence, argumentative coherence, linguistic-grammatical rigor, and validity of conclusions. In this sense, the main difference between them is the absence, in the first two (technical-scientific report and technical-pedagogical report), of the rigor of peer review, which provides prior qualification and relieves the evaluator of the need to evaluate and relativize the quality of the manuscript according to their own criteria, relying on the credibility of the criteria that have previously endorsed it.

Based on the findings obtained from the analysis of the WGTP Report, we conclude that it does not possess the necessary characteristics to serve as a reliable reference for the evaluation of technical reports, given the numerous identified flaws that can be summarized as follows:

• Section organization issues, with information and argumentation mismatches between them;

• Content conciseness issues, due to content repetition throughout the text;

• Information clarity issues, as necessary information for understanding statements and utterances is absent;

• Theoretical precision and argumentative coherence issues, as definitions of concepts oscillate between different sections or even within the same section, adopting definitions foreign to the field's literature, compromising the document's comprehensibility;

• Technical formalism issues, with text format inconsistencies and non-compliance with writing standards present in manuals such as ABNT or similar, or even those proposed by the WGTP itself, including the absence of bibliographic references;

• Linguistic-grammatical rigor issues, with numerous errors in writing standardization according to grammatical and writing rules;

• Validity of conclusions issues, resulting from weak argumentation and theory within the manuscript;

Furthermore, the findings demonstrate that the WGTP Report does not allow for a proper understanding of its theoretical assumptions and how to effectively and objectively apply its guidelines, as it unintentionally and excessively delegates to its users - according to subjective criteria of their own and potentially divergent from the desired references by the creators of the WGTP Report - the filling of the theoretical, conceptual, and methodological gaps it intended to present in its guidelines.

Given the widespread flaws identified in the structure and content of the object studied here, this study raises suspicions that the other products presented within the WGTP Report may also have similar issues to those identified here for the TCR. We believe that there is ample room for further empirical and specific studies on the broader theme of this work, in order to establish precise quality criteria and indicators capable of guiding the task of evaluating reports, as a gray literature element capable of effectively contributing to the dissemination of knowledge in a secure, clear, and useful manner.

• 1
  Significant and non-significant, positive or negative.
• 2
  Literally, "'Possibly relates' when it should be 'relates possibly'". (p. 08)
• 3
  Literally, “'Reinvention of the wheel’”. (p. 08)
• 4
  Literally, "’The objective of this study is [...]’". (p. 08)
• 5
  For unknown reasons, the report referred to here as the WGTP Report has commonly been endorsed in practice as a report on Technical and Technological Production (TTP), even though its official designation is "Technical Production - Working Group", which may indicates a misconception in the appropriation of the title, possibly due to its inadequacy in its own preparation (cf. the references in the Evaluation Reports for the 2017-2020 Quadrennial).
• 6
  Unlike authors such as Martens and Pedron (2019) and Martens, Pedron, and Oliveira (2021), we chose to adopt the literal concept "exaptation" here, not "extrapolation", as we believe it is more appropriate to the original sense introduced by Gregor and Hevner (2013). Exaptation is a term initially used in Paleontology, referring to a kind of deviation from the previously established function towards an unexpected function for which there was no prior prediction. It differs from adaptation in that adaptation evolves as a resource to meet a specific need that required it from the outset. "Adaptation" is typically understood as a function, while "exaptation" is seen as an incidental effect. For more details, refer to Gould and Vrba (1982) and Bryant (2014).
• 7
  To maintain syntactic parallelism of the terms, we adopt the meaning provided by the author, as when he states: “In this quadrant is work that would not normally be thought of as contributing to research because existing knowledge is applied in familiar problem areas in a routine way” (GREGOR; HEVNER, 2013, p. 347), we adopt "application", without detriment to the understanding of the original concept "routine design".
• 8
  Literally, in Gregor and Hevner (2013), we find: “Invention: New Solutions for New Problems” (p. 345); “Improvement: New Solutions for Known Problems” (p. 346); “Exaptation: Known Solutions Extended to New Problems” (p. 347); “Routine Design: Known Solutions for Known Problems” (p. 347).

Referências

• ABNT- ASSOCIAÇÃO BRASILEIRA DE NORMAS TÉCNICAS. NBR 6023. Rio de Janeiro, 2018. Disponível em: https://www.normasabnt.org/abnt-nbr-6023 Acesso em: 27 out. 2022.
  » https://www.normasabnt.org/abnt-nbr-6023
• BIANCOLINO, César Augusto et al Protocolo para elaboração de relatos de produção técnica. Revista de Gestão e Projetos -GeP, São Paulo, v. 3, n. 2, p. 294-307, 2011. Disponível em: https://periodicos.uninove.br/gep/article/view/9467/4224 Acesso em: 18 fev. 2023.
  » https://periodicos.uninove.br/gep/article/view/9467/4224
• BOTELHO, Rafael Guimarães; OLIVEIRA, Cristina da Cruz de. Literaturas branca e cinzenta: uma revisão conceitual. Ci.Inf., Brasília, DF, v. 44 n. 3, p. 501-513, 2015. Disponível em: https://www.brapci.inf.br/_repositorio/2017/06/pdf_856f1f5dcd_0000023339.pdf Acesso em: 13 nov. 2022.
  » https://www.brapci.inf.br/_repositorio/2017/06/pdf_856f1f5dcd_0000023339.pdf
• BRASIL. Ministério da Educação. Coordenação de Aperfeiçoamento de Pessoal de Nível Superior. Produção Técnica-Grupo de Trabalho: relatório. Brasília: MEC, 2019. Disponível em: https:/www.gov.br/capes/pt-br/centrais-de-conteudo/10062019-producao-tecnica-pdf Acesso em: 01 dez. 2022.
  » https:/www.gov.br/capes/pt-br/centrais-de-conteudo/10062019-producao-tecnica-pdf
• BRYANT, Levi. Onto-cartography: an ontology of machines and media. Minneapolis: University of Minnesota Press, 2014.
• CARLOMAGNO, Márcio C., ROCHA, Leonardo Caetano da. Como criar e classificar categorias para fazer análise de conteúdo: uma questão metodológica. Revista Eletrônica de Ciência Política, Paraná, v. 7, n. 1, p. 173-188, 2016. Disponível em: https://revistas.ufpr.br/politica/article/view/45771/28756 Acesso em: 05 nov. 2022.
  » https://revistas.ufpr.br/politica/article/view/45771/28756
• GOULD, Stephen J.; VRBA, Elisabeth S. Exaptation: a missing term in the science of form. Paleobiology, Pelotas, v. 8, n. 1, p. 4-15, 1982. Disponível em: https://wp.ufpel.edu.br/cdrehmer/files/2019/05/Gould-and-Vrba1982.pdf Acesso em: 13 fev. 2023.
  » https://wp.ufpel.edu.br/cdrehmer/files/2019/05/Gould-and-Vrba1982.pdf
• GREGOR, S.; HEVNER, A. R. Positioning and presenting design science research for maximum impact. MIS Quarterly, EUA, v. 37, n. 2, p. 337-355, 2013. Disponível em: https://dl.acm.org/doi/10.25300/MISQ/2013/37.2.01 Acesso em: 18 fev. 2023.
  » https://dl.acm.org/doi/10.25300/MISQ/2013/37.2.01
• LATOUR, Bruno. Ciência em ação: como seguir cientistas e engenheiros sociedade afora. São Paulo: Unesp, 2011 [1987].
• LAVILLE, Christian; DIONNE, Jean. A construção do saber: manual de metodologia de pesquisa em ciências humanas. Belo Horizonte (MG): UFMG, 1999.
• LIMA, Margarida. Conceção, redação e publicação de artigos científicos: conceção de artigos científicos. Nascer e Crescer: Revista de Pediatria do Centro Hospitalar do Porto, Portugal, v. 22, n. 01, p. 50-59, 2013. Disponível em: 00_NascerCrescer_22-1_PROVA-11.pdf (scielo.pt). Acesso em: 05 set. 2023.
  » scielo.pt
• LU, Yi Chieh; MATUI, Natália; GRACIOSO, Luciana. Definição da inovação no âmbito da pesquisa brasileira: uma análise semântica. RDBCI - Revista Digital Biblioteconomia e Ciência da Informação, Campinas, v. 17, n. e019023, p. 1-22, 2019. Disponível em https://brapci.inf.br/index.php/res/download/122560 Acesso em: 30 jan. 2020.
  » https://brapci.inf.br/index.php/res/download/122560
• MAMEDE, Walner. Modelo para a avaliação de mestrados profissionais orientados à formação de recursos humanos para o SUS: um estudo de caso. 2016. Tese (Doutorado) - Instituto de Psicologia, Universidade de Brasília, Brasília, 2016. Disponível em: https://repositorio.unb.br/handle/10482/19784 Acesso em: 05 set. 2023.
  » https://repositorio.unb.br/handle/10482/19784
• MARTENS, Cristina Dai Prá; PEDRON, Cristiane Drebes. A disseminação da produção técnica/tecnológica gerada em programas stricto sensu profissionais. Revista Inovação, Projetos e Tecnologias, São Paulo, v. 7, n. 1, p. 1-3, 2019. Disponível em https://periodicos.uninove.br/iptec/article/view/14712/7029 Acesso em: 18 fev. 2023.
  » https://periodicos.uninove.br/iptec/article/view/14712/7029
• MARTENS, Cristina Dai Prá; PEDRON, Cristiane Drebes; OLIVEIRA, Jairo Cardoso. Diretrizes para elaboração de artigos tecnológicos, artigos aplicados ou relatos técnicos de produção com ênfase profissional. Revista Inovação, Projetos e Tecnologias, São Paulo, v, 9, n. 2, p. 143-147, 2021. Disponível em: https://periodicos.uninove.br/iptec/article/view/21117/0 Acesso em: 18 fev. 2023.
  » https://periodicos.uninove.br/iptec/article/view/21117/0
• MARTINS, Gilberto de Andrade. Sobre confiabilidade e validade. RBGN, São Paulo, v. 08, n. 20, p. 1-12, 2006. Disponível em: https://rbgn.fecap.br/RBGN/article/download/51/272 Acesso em: 13 nov. 2022.
  » https://rbgn.fecap.br/RBGN/article/download/51/272
• MORAES, Roque; GALIAZZI, Maria do C. Análise textual discursiva Ijuí: Unijuí, 2005.
• MOTTA, Gustavo. D. S. Como escrever um bom artigo tecnológico? Revista de Administração Contemporânea, Rio de Janeiro, v. 21, n. 5, p. 04-08, 2017. Disponível em: https://www.scielo.br/j/rac/a/9fWvtsnTR6nNGNtn4MM7Z3h/?format=pdf⟨=pt Acesso em: 18 fev. 2023.
  » https://www.scielo.br/j/rac/a/9fWvtsnTR6nNGNtn4MM7Z3h/?format=pdf⟨=pt
• PEREIRA, Maurício Gomes. Artigos científicos: como redigir, publicar e avaliar. Rio de Janeiro: Guanabara Koogan, 2012.
• PLONSKI, Guilherme A. Inovação em transformação. Estudos Avançados, São Paulo, v. 31, n. 90, p. 7-21, 2017. Disponível em: https://www.scielo.br/j/ea/a/3Vmk8zqHbrVcgBwhMTyTC7d/?format=pdf⟨=pt Acesso em: 30 jan. 2023.
  » https://www.scielo.br/j/ea/a/3Vmk8zqHbrVcgBwhMTyTC7d/?format=pdf⟨=pt
• PORTO, Flávia; GURGEL, Jonas Lírio. Sugestão de roteiro para avaliação de um artigo científico. Ver. Bras. Ciênc. Esporte, Brasília, v. 40, n. 02, p. 111-116, 2018. Disponível em: https://www.scielo.br/j/rbce/a/R9tpwBqLnsZt3kmr6w6hRtR/?lang=pt Acesso em: 13 nov. 2022.
  » https://www.scielo.br/j/rbce/a/R9tpwBqLnsZt3kmr6w6hRtR/?lang=pt
• SAVICKAS, Mark L. Como redigir um relatório de pesquisa: finalidades e problemas em artigos científicos. Revista Brasileira de Orientação Profissional, São Paulo, v. 10, n. 1, p. 7-10, 2009. Disponível em: https://www.redalyc.org/pdf/2030/203014934003.pdf Acesso em: 03 out. 2022.
  » https://www.redalyc.org/pdf/2030/203014934003.pdf
• SOUBHIA, Zeneide; RUFFINO, Márcia Caron; DESSUNTI, Elma Mathias. Relatório de Atividade Acadêmica como Recurso de Aprendizagem da Pesquisa. Ver. Latino-Am. Enfermagem, São Paulo, v. 13, n. 2, p. 269-73, 2005. Disponível em: http://www.eerp.usp.br/rlae Acesso em: 03 out. 2022.
  » http://www.eerp.usp.br/rlae
• SOUZA, Ana Cláudia de; ALEXANDRE, Neusa Maria Costa; GUIRARDELLO, Edinêis de Brito. Propriedades psicométricas na avaliação de instrumentos: avaliação da confiabilidade e da validade. Epidemiol. Serv. Saúde, Brasília, v. 26, n. 3, p. 649-659, jul-set 2017. Disponível em: https://www.scielo.br/j/ress/a/v5hs6c54VrhmjvN7yGcYb7b/?format=pdf⟨=pt Acesso em: 03 out. 2022.
  » https://www.scielo.br/j/ress/a/v5hs6c54VrhmjvN7yGcYb7b/?format=pdf⟨=pt
• SOUZA, Marcos de. A pesquisa científica e seus títulos. Revista Conhecimento em Ação, Rio de Janeiro, v. 7, n. 01, p. 3-23, 2022. Disponível em: https://revistas.ufrj.br/index.php/rca/article/view/52267 Acesso em: 13 nov. 2022.
  » https://revistas.ufrj.br/index.php/rca/article/view/52267
• WOLFE, Richard A. Organizational innovation: review, critique and suggested research directions. Journal of Management Studies, Wiley Blackwell, v. 31, n. 03, p. 405-431, 1994. Disponível em: https://ideas.repec.org/a/bla/jomstd/v31y1994i3p405-431.html Acesso em: 30 jan. 2023.
  » https://ideas.repec.org/a/bla/jomstd/v31y1994i3p405-431.html

Publication Dates

• Publication in this collection
  10 Nov 2023
• Date of issue
  2023

History

• Received
  16 Feb 2023
• Accepted
  18 Aug 2023
• Reviewed
  05 Sept 2023