Profiles not metrics: the case of Brazilian universities

Abstract

Public universities, and science in general, in Brazil, are under attack from key persons of the government in interviews and articles published in non-scientific journals. Here we look at bibliography data from international science metric platforms (Scival® and Incites®) and official Brazilian agencies such as CAPES and CNPq to reach some conclusions based on scientific analysis. Brazilian Science has shown a steady improvement in quantity and quality over the last 20 years but has recently suffered (since 2015) under severe financial restrictions. An increase in international collaboration also increased citation impact, reaching almost five times the world average. While the medical and natural sciences show the highest impact and prominence, social sciences and the humanities also have spotlight areas with international excellence. Different research institutions and universities offer a variety of production profiles and impacts. This diagnosis shows the need for universities and research institutes in Brazil and funding agencies to undergo strategic planning for definition of mission/vision, goals to be reached, and areas for priority development. Continued support of public universities by the government is necessary for Brazilian autonomy in science and technology and its full integration in the world scientific community.

Key words
citation impact; collaboration; quantity; quality

INTRODUCTION

Adams et al. (2019) highlight the dangers of using simplified metrics or league tables to compare universities or researchers. Research institutions teach and interact with society, and research can generate a series of products, not only scientific papers (McManus et al. 2020a, McManus & Baeta Neves 2021a, b). Recent events in Brazil have brought investments in public university education and the impact of research under scrutiny (e.g., Angelo 2019). Financing for scientific projects and scholarships has seen severe restrictions since 2015 from the federal government, especially for research and postgraduate education, through budgets for the federal funding agencies (CAPES, CNPq, FINEP). This is not unusual in Brazil (Escobar 2019). Still, some of the arguments put forward include low-quality research, especially in the humanities and social sciences, and low overall productivity of the universities, with the president stating that 95% of research in Brazil is carried out in private universities (Leal 2019). These restrictions do not consider the strengths in Brazilian Science as a whole (Chinese Academy of Sciences & Clarivate Analytics 2019) or where each institution impacts locally, regionally or nationally.

Brazil has, by legal definition, three types of higher education establishments: public (including federal, State and municipal), non-profit (confessionary and local, including those linked to religious orders such as Catholic, Lutheran, Presbyterian etc.) as well as private for-profit universities (Durham 2003). While the private universities, which large for-profit groups may own, have the most undergraduate students (74%), almost all research in the country is carried out in the other two sectors, especially the public universities. At present, the public universities carry out about 95% of research and, contrary to the declarations above, publish higher-quality research (Kinser & Levy 2006, Lloyd 2013).

Publishing constitutes a baseline activity in science worldwide (Blind et al. 2018). The public and decision-makers should be aware that scientific discoveries are the basis for innovation, cultural advances, health care, defence, economic and political decisions in government, among numerous other uses. Basic research is a prerequisite and catalyst of application studies, and thus, essential for technological innovation. So, although we understand that economic and financial situation may trigger discussion about public support for the sector, it is necessary to keep the focus on empirical data on scientific performance, taking into account historical and geographical constraints of the different institutions in Brazil, as well as the overall socio-economic profile of the population and the need to improve higher education, with a focus on science and technology, as a way to improve the economy and long term social development (Baeta Neves et al. 2020).

Thus, our main research question here, in a scientometric context, is to evaluate historical trends of scientific production in Brazil, overall and in different areas, and how they differ among high education and research institutions countrywide.

MATERIALS AND METHODS

Data

Data from this study were obtained from SciVal (Elsevier.com); Incites (https://incites.clarivate.com/), as well as data from federal financing agencies Capes (Coordination for the Improvement of Higher Education Personnel - www.capes.gov.br) and CNPq (National Agency for Scientific and Technological Development - www.cnpq.br). There are 478 higher education institutions registered on the Capes database, out of which 223 are private, 89 are State, 155 federal (including universities and technological institutes), and a small number of municipality institutions. Data from Incites®, which were used as variables in our analyses (below), included number of scientific papers, Category Normalised Citation Index (CNCI - dividing the actual count of citing items by the expected citation rate for documents with the same document type, year of publication and subject area), Journal Normalized Citation Impact (JNCI - instead of normalising per subject area/field, it normalises the citation rate for the journal where the document is published), number and percentages of Times Cited, documents cited, documents in Q1 to Q4 journals (% of documents that appear in a journal in a particular Journal Impact Factor Quartile in a given year), documents in top 1% and 10% (top 1 or 10 % most cited documents in a given subject category, year and publication type divided by the total number of documents in a given set of documents, displayed as a percentage), highly cited, documents in JIF journals (Documents published in a journal found in Journal Citation Reports in a given year), industry and international collaboration (http://help.prod-incites.com/inCites2Live/indicatorsGroup/aboutHandbook/usingCitationIndicatorsWisely/impactRelativeToWorld.html). Other data included % of DOAJ (Directory of Open Access Journals), Gold, Bronze and Green documents (https://incites.help.clarivate.com/Content/open-access.htm). Data from SciVal included field-weighted citation index (FWCI), % international collaboration, number of documents, and value of awarded grants. The data allows comparing Brazilian indicators in a global context. Both CNCI (Web of Science) and FWCI (Scopus) theoretically have a world mean of one (1) for each knowledge field.

The Organisation for Economic Co-operation and Development (OECD, 2007) categories were used in this study for classification purposes. This scheme uses six broad subject categories for publishing data: (1) natural sciences, (2) engineering and technology, (3) medical and health sciences, (4) agricultural sciences, (5) social sciences, and (6) humanities. The discussion here is based on these broad fields.

Data for the analyses consisted thus in trend data for all variables in 178 Brazilian Universities and Research Institutes or Centres, in distinct OECD categories, as universities or research centres with less than 500 publications in 15 years were removed from these analyses.

Analysis

A Principal Component Analysis based on a correlation matrix among all indicators was used to synthesise information and evaluate the relationship between different indicators of productivity and better understand its structure. Regression analyses were used to assess temporal trends in scientific indicators, overall and for different research fields according to OECD categories, and test some specific relationships among variables.

The Brazilian institutions in the analyses were divided into two broad categories: i) research institutions with a specific mission such as those controlled by specific ministries in Brazil (Defence, Health, Agriculture or Science, Technology, Innovation and Communications as well as research hospitals; and ii) universities and University centres.

A Cluster analysis based on quality indicators was used to distinguish groups of research institutions and universities. These quality indicators included CNCI, % of documents in Q1 journals, % documents in top 1%, % highly cited papers, % hot papers, impact relative to the world, JNCI, % international collaborations, % DOAJ Gold documents, as well as % industry collaborations for the university analysis. A discriminant analysis allowed the evaluation of which quality indicators differentiated these clusters. A canonical analysis was then performed to evaluate the relationships between the clusters.

We then examined publishing patterns within the clusters using the OECD and FWCI indicators from SciVal®. FWCI by research topic cluster (Scival®) is given by university/research institution and word clouds constructed by Brazilian region (North (N), Northeast (NE), South (S), Southeast (S) and Center-West (CW)) for these topic clusters. Patent citations of scholarly output and media exposure of the research institutions per region were also examined (Scival®). All statistical analyses were performed using SAS® (Statistical Analysis System Inc, Cary, North Carolina).

Word clouds were built from words identified by regions in Topic-Clusters in SciVal®, by evaluating publishing linkages within the universities (https://service.elsevier.com/app/answers/detail/a_id/28428/supporthub/scival/).

RESULTS

Overall view

Brazil has a large and diverse scientific community (Figure 1). From 2000 up to the May 2019, Brazil has published 889,443 papers registered in Scopus® (accessed on 01/05/2019), with an increase from 15,067 papers per year in 2000, to 83,103 in 2019 (Figure 2). This increase had been accompanied by an equivalent increase in quality, with the Field Weighted Citation Index (FWCI) increasing from 0.78 in 2000 up to 0.90 in 2018 (Figure 3). The highest FWCI (0.96) was registered in 2016 (slightly lower than the world average). One factor of note is the steady decline in impact since this date (currently 0.84 for 2019: incomplete year).

Figure 1
a) Brazilian publications in Scopus® (2010 - 2019) by knowledge area; b) Areas of prominence in Brazilian Science; c) Top 10% prominence worldwide. AS: Agricultural Sciences; HUM: Humanities; NS: Natural Sciences; MS: Medical Sciences; ET: Engineering and Technologies; SS: Social Sciences.

Figure 2
Number of Brazilian documents in Scopus® from 2010 to 2019 by knowledge area (a) and region (b) of the country and Field Weighted Citation Index (FWCI) by knowledge area (c) and region (d) of the country. (All R² >0.93 for regressions).

Figure 3
a) Relationship between international collaboration and impact in Brazilian Science in different knowledge areas in Brazil; b) The effect of international collaboration on Field Weighted Output in Top 10% Citation Percentiles for Brazilian publications; and c) Trends in international collaboration by area of knowledge.

Areas of world prominence and impact (Figure 1c) in Brazilian Science include areas related to dentistry (bone and teeth, dentin), STEM (electric batteries, solar cells, lithium alloys, graphene, plasmons, nanotechnology, mathematics, astronomy), medicine (immunotherapy, neoplasms, women’s health care, viruses, obesity), social sciences (mortality, education in engineering, sustainable development), among others (Supplementary Material – Table SI). These areas also reflect the contribution of science and technology in different areas that are important for the country.

The increases in the number of papers per area per year (Figure 2) have been constant over the last 18 years (2,200/yr in natural sciences; 1,415 in medical sciences; 676 in engineering and technology; 783 in agricultural sciences; 576 in social sciences and 118 in humanities, from regression analysis (not shown)). Quality also improved in most areas (Figure 2) until 2016, when the mean FWCI was 0.97. From there was a decrease of FWCI to 0.85 in 2020.

The participation of Brazilian researchers in collaboration with scientists worldwide has also increased over the last ten years in all areas (Figure 3). It is important to note that 30% of the papers published in the previous five years (Table I) (141,702) were with foreign partners, primarily North America (61,675) and Europe (82,789), with a corresponding increase in impact (4.61), almost five times the world mean. The level of international collaboration reached 37.4% in 2019. An increase in international collaboration is accompanied by an overall increase in impact (Figure 3a), i.e., an increase of 2% impact with an increase of 1% in international collaboration.

Table I
Publications in co-authorship with non-Brazilian researchers (2013-2019) SciVal.com.

Region Totals	Collaborating Countries	Co-Authored Publications	%
Worldwide	206	141,702
Africa	54	6,789	4.79
Asia Pacific	47	23,556	16.62
Europe	48	82,789	58.42
Middle East	18	6,853	4.83
North America	23	61,675	43.52
South America	16	18,880	13.32

Multivariate analyses

Principal component analyses (Figure 4) are similar for the research institutions (Figure 4a) and universities (Figure 4b). In general, the more a university or research institute produces, the more documents it has in open access and the higher its H-index. This is also accompanied by a higher impact factor and a higher percentage of documents in better quality journals. Nevertheless, as seen in the second principal component, a subset of institutions produces a high number of papers, but not open access and with a lower impact factor.

Figure 4
First two principal components for quantity and quality indicators for Brazilian research institutes (a) and Universities (b).

Research institutions and research centers

Only six of the 38 research centres in this study are outside the southeastern region of the country. The discriminant analysis showed that 100% of institutions were correctly identified within their respective cluster. The main factors that discriminated between clusters included % International collaborations (R² = 0.81), % Q1 journals (0.74), % Q1 journals (0.67).

Both clusters 1 and 2 (Table II) represent institutions with high impact and high international collaboration activity that tend to publish in high impact journals. This is confirmed in the canonical analysis (Figure 5) with the distribution of institutions in Figure 6.

Figure 5
Canonical analysis for means cluster and variables (a) and separation by cluster (b) for research institutions in Brazil (Cluster 1 has highest impact factor and Cluster 5 lowest).

Figure 6
Proportion of documents (a and b) and Field Weighted Citation Index (FWCI) (c and d) for four different hospitals (Cluster 2) and three different engineering institutions (Cluster 5). (Cluster 1 has highest impact factor and Cluster 5 lowest).

Table II
Means for selected quality indicators for research institutions and centres in Brazil.

Cluster	CNCI	% Q1	% 1%	% High	%Hot Papers	Impact	JNCI	% Int	% DOAJ Gold
1	1.34	52.33	2.17	1.30	0.02	1.03	1.15	63.10	10.34
2	1.23	53.32	1.54	1.12	0.08	1.26	0.96	41.36	16.77
3	0.90	28.49	1.10	0.85	0.04	0.98	1.01	34.03	30.34
4	0.77	30.23	0.75	0.68	0.03	0.80	0.96	19.26	44.48
5	0.71	37.13	0.42	0.12	0.00	0.82	0.79	26.46	12.08

• Clusters: 1 = Highest Impact ; 5 – Lowest Impact ; CNCI - Category Normalized Citation Impact; JNCI - The Journal Normalized Citation Impact; DOAJ – Directory of Open Access Journals; Q1 1^st Quartile.

The better-classified research institutions (Figure 7) showed a high impact in their field of expertise. At the same time, those with a lower classification had a lower impact in their field of expertise and published in a broader range of knowledge areas. In this case, the medical sciences show the highest percentage of papers in the top 1%, averaging 1.27% in the last three years, followed by Social Sciences (0.98%), Natural Sciences (0.95%), Engineering (0.72%), Humanities (0.68%) and Agricultural Sciences (0.67%).

Figure 7
Canonical analyses for university classification in Brazil showing means cluster and variables (a) and individual universities (b). (Cluster 1 has highest impact factor and Cluster 5 lowest).

Universities

Most universities in clusters 1 and 2 were public federal, or private philanthropic (Table III), while most private for-profit institutions were in cluster 4, along with municipal universities. Most federal and State universities were in clusters 3 and 4. Of the 20 institutions in Cluster 5, 16 (80%) are either State or Federal. Most of these are relatively new and have only entered the postgraduate system recently. Figures 7 and 8 look at the distribution of universities with clusters and publication profiles of their clusters.

Figure 8
Proportion of Documents published in each OECD area of knowledge (a,b, and c) and impact (Field Weighted Citation Index - FWCI) of these publications (d, e and f) from randomly selected universities (different colours) in clusters 1, 3 and 4, respectively. (Cluster 1 has highest impact factor and Cluster 5 lowest).

Table III
Type of Brazilian Higher Education Institutions and region by cluster (Priv – private; Pub – Public).

Cluster	1		2		3		4		5		Total
Higher Education Institution Type
	Priv	Pub	Priv	Pub	Priv	Pub	Priv	Pub	Priv	Pub
Not for Profit	3		2		5		13		4		27
For Profit	1		4		3		9				17
Federal		3		5		15		33		10	66
Municipal								3			3
State						10		11		6	27
Total	4	3	6	5	8	25	22	47	4	16	140

Country Region
	1	2	3	4	5	Total
CW			1	6	5	12
N		2	2	5	3	12
NE	1	1	7	12	5	26
S	3	3	8	18	3	35
SE	3	5	15	28	4	55
Total	7	11	33	69	20	140

• Clusters: 1 = highest Impact ; 5 – Lowest Impact; CW – Center-west; N- North; NE – Northeast; S – South; SE – Southeast.

Considering now the institutional origin of the scientific production, most of Science in Brazil is carried out in higher education establishments, primarily public. 50% of all publications from Brazil come from 14 institutions, with the University of São Paulo accounting for 18% of documents. Eighty-six institutions produce 90% of the scientific documents, out of which six universities maintained by religious orders (Catholic, Lutheran) account for 2.6%, 58% were accounted by Federal Universities linked to the Ministry of Education, 14.5% come from state universities and only 1.8% from private institutions (SciVal®). Significant impact factors are seen mostly for public institutions (only two private hospitals and two catholic universities appear in the list of institutions with a mean impact factor greater than the world mean).

The clusters were efficient in describing differences between the different groups of universities (Table IV). Variables that discriminated between the different clusters of universities included CNCI (R² = 0.91), H-Index (0.23), % Green Documents (0.21); % Top 10 % (0.19); % Q4 journals (0.11); % hot papers (0.10) all with P<0.01.

Table IV
Percentage of correct designation of Higher Education Institutions to clusters .

	Cluster
	1	2	3	4	5
1	85.71	14.29	0	0	0
2	0	100	0	0	0
3	0	0	96.97	3.03	0
4	0	0	1.45	95.65	2.9
5	0	0	0	0	100
Total	7	11	33	69	20

• Clusters: 1 - Highest Impact ; 5 – Lowest Impact.

Clusters 1 and 2, as well as 4 and 5, were smaller institutions. In contrast, cluster 3 was composed mainly of larger institutions (Table V), as seen by the Number of WoS documents and institutional H-index. The citation index decreased with an increase in the cluster’s number, while the percentage of documents cited remained relatively stable across groups. The percentage of papers in higher quality journals was higher in lower-numbered clusters and vice versa. Higher ranked institutions published a higher percentage of papers as gold or green open access. Clusters 4 and 5 publish a high percentage of papers in open access journals but of lower quality. These also have significantly lower international collaborations. The larger universities (Cluster 3) have higher % industrial collaborations than the other clusters. Figure 9 reinforces this idea of different profiles for different institutions, looking at % Documents published in each OECD area of knowledge and the impact of these publications from randomly selected universities (different colours) in clusters 2, 3 and 4.

Figure 9
Word Clouds for Major Topics (measured by number of publications in SciVal®) studied in Brazil and by Region.

Table V
Average performance per Higher Education Institute cluster.

WOS Doc			Cluster
	1	2	3	4	5	Overall
	1899.00	2988.09	17849.09	3937.29	1280.15	6660.41
CNCI	1.43	1.08	0.85	0.68	0.53	0.77
% Docs Cited	72.54	73.20	70.80	71.25	66.82	70.73
% Q1Journals	36.78	35.28	33.32	27.35	20.26	28.84
% Q2Journals	23.54	25.23	25.32	25.60	21.71	24.85
% Q3Journals	18.75	19.19	20.39	21.95	22.65	21.31
% Q4Journals	20.93	20.30	20.97	25.10	35.38	25.01
% Top 1%	1.89	1.35	0.89	0.46	0.23	0.67
% Top 10 %	9.67	9.34	7.14	5.25	3.24	5.95
H-Index	57.71	57.55	94.00	51.59	29.60	59.22
% Other Gold Doc	5.02	2.37	2.31	1.65	1.71	2.04
% Green Accepted Doc	7.02	3.06	2.27	1.02	0.54	1.71
% All Open Access Doc	43.28	40.18	38.98	41.28	49.49	41.92
% Green Published Doc	20.83	18.15	17.65	16.65	17.21	17.29
% Bronze Doc	6.36	7.83	7.49	6.65	5.89	6.82
% DOAJ Gold Doc	29.11	27.53	26.68	30.83	40.51	30.89
% Highly Cited Papers	1.37	1.03	0.53	0.26	0.13	0.42
% Industry Collaborations	0.72	0.85	1.03	0.57	0.30	0.67
% International Collaborations	31.72	28.68	28.31	22.02	16.71	23.75
% HotPapers	0.09	0.11	0.04	0.01	0.01	0.03
Impact Relative to World	1.28	1.01	0.84	0.70	0.46	0.75
JNCI	1.09	1.03	0.95	0.89	0.85	0.92

• Clusters: 1 - Highest Impact ; 5 – Lowest Impact; WOS – Web of Science ; Doc – Documents, CNCI – Category Normalised Citation Impact; JNCI - Journal Normalized Citation Impact; DOAJ – Directory of Open Access Journals.

Figure 10 looks at the main research topics per region. While the South and Southeast show a high number of publications in the health areas, the northeast and centre-west show a higher number of topics related to agriculture and the environment. Some subjects can be identified as being relevant worldwide such as black holes, dentin, models, education, etc. Nevertheless, some topics are relative to Brazil or Brazilian situations, such as Jatropha curcas (Pinhão manso, used for biodiesel), prominent in the Centre-west and North-eastern regions, or themes related to the Amazon Forest and river in the North. Some of these subjects are primarily of interest to Brazil may influence the choice of journal for publishing and impact factor. The predominance of subjects linked to the South and Southeast in the overall picture for Brazil reflects the number of publications in these two regions.

Figure 10
Percentage of types of Media exposure of Brazilian universities and research institutions from 2014 to date (SciVal®) both online (a) and in print (b) and number of publications by region (c).

Tables SIIa and SIIb look at the impact of research topics defined by SciVal® by institution and university. All institutions in the country show at least one, but typically more than one, topic with FWCI above the world mean. Therefore, while the overall average FWCI of the institution may be low, each one has some area of quality. The research institutions published, on average, in 13% of the 782 topics, showing focus in their areas of action, with Embrapa and Fiocruz publishing in approximately 44% of the topics. On the other hand, the universities published on average in 37% of the topics, with the larger universities (such as USP, Unicamp, Unesp, UFRJ, UnB, UFMG, UFRGS, among others) publishing in approximately 63% of the topic clusters. About 75% of the topic clusters in the universities and 69% of the research institutions showed impact < 1. This represents a lower % of the actual publications as there is a low, positive but significant correlation (0.17) between FWCI and Number of papers per topic.

Table VI shows counts of patents citing knowledge produced by Brazilian universities by region and area of knowledge. The southeast produced most patent citations as expected as more institutions are in this region and they tend are older. Nevertheless, more average Patent-Citations were received per 1,000 Scholarly Outputs for the Engineering & Technologies in the Center-West region and the social services in the southern region. Patent citations tended to be lower in the social sciences and humanities, as expected. The INPI (National Institute for Industrial Property: https://gru.inpi.gov.br/pePI/jsp/Base_pesquisa.jsp) database has 12,913 patents deposited (not conceded) since 1979 with a university in the name of the depositor, 409 trademarks, 400 industrial designs, and 4033 computer programs. Data from WIPO (https://www.wipo.int/ipstats/en/statistics/country_profile/profile.jsp?code=BR ) show that most of these are from non-residents.

Table VI
Economic impact of Brazilian Universities and Research Institutions (SciVal ®) from all patent offices worldwide (2009-2018).

	Number	Agriculture	Engineering & Technologies	Humanities	Medical	Natural	Social
Brazil
Citing Patents Count1	13508	1413	4037	26	6819	8401	156
Patent-Cited scholarly Output2	5562	595	1541	13	2800	3328	82
Patent-citations count3	15715	1572	4401	26	7843	9243	162
Patent-citations per scholarly output4	24	13.8	34.7	1.8	33.3	25.7	2.2
North
Citing Patents Count	185	29	44		87	115	3
Patent-Cited scholarly Output	89	16	21		40	57	1
Patent-citations count	190	29	44		90	118	3
Patent-citations per scholarly output	9.8	4.5	19.1		20.6	8.9	1.7
Northeast
Citing Patents Count	1309	117	474		540	767	21
Patent-Cited scholarly Output	528	57	183		223	331	10
Patent-citations count	1354	124	482		549	788	21
Patent-citations per scholarly output	14.6	6.5	29.2		19.0	14.3	2.3
South
Citing Patents Count	2779	238	843	3	1372	1710	53
Patent-Cited scholarly Output	1087	119	349	3	521	648	26
Patent-citations count	2985	256	876	3	1470	1804	53
Patent-citations per scholarly output	21.4	8.6	31.4	1	30.9	23.8	3.1
Southeast
Citing Patents Count	9683	1077	2806	22	5054	5967	87
Patent-Cited scholarly Output	3939	423	1022	9	2052	2376	45
Patent-citations count	11184	1186	3033	22	5762	6576	90
Patent-citations per scholarly output	27.5	18.4	38.2	2.9	37.1	28.9	2.2
Centre-west
Citing Patents Count	809	152	263		309	566	7
Patent-Cited scholarly Output	359	67	106		136	246	3
Patent-citations count	861	152	276		318	588	7
Patent-citations per scholarly output	16.3	8.2	40.7		24.3	18.7	1.2

• ¹count of patents citing the Scholarly Output published in the given region; ² count of Scholarly Outputs in the given region that have been cited in patents; ³count of Patent-Citations received in the region; ⁴average Patent-Citations received per 1,000 Scholarly Outputs published in the region.

Media exposure can help increase the visibility of an institution. Figure 10 shows that, in general, media exposure of Brazilian research is low and needs to be improved, especially at international and regional levels. The southeast dominates this item, especially online. Most of these publications are in the USA (> 50%) and most for Petrobras (48%), not necessarily related to R&D. UFRJ has the second most media exposure in this region. Nevertheless, the University of Brasilia dominates the CW region, INPA and Museu Paraense Emilio Goeldi in the Northern Region, UFPE and UFBA in the Northeast, as well as UFRGS and UFPel in the South. The southeast and centre-west show the highest international exposure.

DISCUSSION

Brazilian Science has expanded in recent years, showing impact and relevance in many areas of knowledge (McManus et al. 2020a). This paper looks at this growth from several aspects, including region, area of knowledge and type of indicator.

The results here show that different institutions show different publishing and impact profiles. Brazil produces science in quantity and quality, which should be reinforced by institutional and national development and funding policies. What is expected of the researchers also needs to be clear and the indicators that will be used in evaluating the research.

The relationship between impact and international collaboration is well-documented internationally (Jeong et al. 2014, McManus et al. 2020a) and, according to these authors, the marginal increase in the quality of a team’s research is proportional to the academic excellence of its participants (see also Fortunato et al. 2018). This is in line with that found here. The areas with higher internationalisation (medical and natural sciences) showed faster increases in impact (Figure 3c), which was also seen in other countries (Bornmann & Leydesdorff 2013).

While quality has been improving in most areas, some still lag behind the world mean (Figure 2). This can be due to several factors. Bornmann & Leydesdorff (2013) showed that even highly-developed countries such as Japan perform below or approximately at the world average in every subject area. Oliveira (2016) states that publishing high-impact work requires very demanding conditions, including choice of appealing scientific/technological problems, skilled scientists, adequate infrastructure, and ability to communicate the results and new concepts. This is reinforced by studies such as Jordan et al. (2003).

Universities in Brazil are expected to carry out three important functions: form human resources in mass (USP for example, the largest public university in the country, has almost 100,000 students), produce excellent research, and transfer technology. In other countries happens in different types of institutions. To expect universities to show similar performance as top universities worldwide under restricted budgets seems counterproductive and meaningless. International rankings are used as one of the excuses for cutting funding. In a study published by FAPESP (2019), costs per postgraduate student in São Paulo State universities were radically lower than those that they are supposed to emulate (MIT, Oxford, Harvard etc.). This is in line with information from the World Bank (Salmi 2009) and other sources (DGEI 2012). The Mexican higher education budget in 2009, which accounts for the vast amount of spending in the sector, was equivalent to ¼ of the budget of the University of California at Berkeley (Muñoz García, 2009). According to Sandström & Besselaar (2016), quantity is necessary to have highly cited papers. Diniz-Filho et al. (2016) also showed that, in a Brazilian University, high correlations were found between the total number and the number of high-quality articles published (ranging from 0.75 to 0.95) by university faculty. Data in this study showed similar trends (Table II and Figure 6), with high correlations across universities between the number of papers published and papers in high-quality journals.

Financing for research in Brazil is described in Fonseca & Veloso (2018) and is highly variable. The leading investors (https://www.access4.eu/brazil/330.php, McManus & Baeta Neves 2021b) are Brazilian National Council for Scientific and Technological Development (CNPq), Finep, Federal Agency for Support and Evaluation of Graduate Education (CAPES), National fund for scientific and technological development (FNDCT), as well as state financing agencies (FAPs). These agencies have all undergone financial restrictions in recent years, under different governments. Government programs have also affected science financing, such as the Science without Borders Program (McManus & Nobre 2017), which saw mainly CAPES and CNPq redirect resources from research and postgraduate studies to pay scholarships for undergraduate students abroad (2015-2017 mostly). CAPES’ budget also includes resources for teacher training. This may, in part, explain part of the fall in citation impact from 2016. To be highly cited, there is a need to publish open access in high impact journals, which is a limitation under current budgetary constrictions. This is reflected by the decrease of Brazilian authors in the major international journals since 2016 (PLOSONE: 813 in 2016 to 472 in 2019; Physical Review D: 481 to 404; Nature 66 to 21). Many researchers have to pay for these publications out of their own pockets if the university does not cover the costs, which is usually the case (Pavan & Barbosa 2018, McManus et al. 2020a).

The fact that Brazilian science is recognised worldwide in a wide range of research areas is dependent on good infrastructure for research within the country (Sennes & Britto Filho 2012). With the current reduction in financing available to universities, the maintenance and upgrading of this science park are under threat, which has severe implications for the future of Science in Brazil. FINEP, which traditionally finances large-scale installations, and CNPq, which funds individual projects to researchers, have not given significant financing since 2016.

The universities and institutions in this study form over 60,000 masters and 21,000 doctors per year (https://www.gov.br/capes/pt-br/acesso-a-informacao/dados-abertos), with an increase of 52% in the number of postgraduate courses from 2010 (2840) to 2018 (4320). Most of this expansion has occurred in less well-developed regions such as the North (78%), Northeast (64%) and centre-west (70%). It cannot be expected that these institutions that recently entered into the postgraduate system produce the same quality science as the older, more established universities and research institutions in other regions of the country (which also have enjoyed more stable funding, such as those in São Paulo state). Nevertheless, data shows that public higher education intuitions in these regions have shown continued improvements in quality and quality of research over the last ten years (Figure 2). This is reflected in the increased number of well-evaluated postgraduate courses in these regions (https://sucupira.capes.gov.br/sucupira/), with two institutions from the northeast region appearing in the Nature Index Top Ten institutions (https://www.natureindex.com/country-outputs/brazil) for Brazil (Federal universities of Paraiba and Rio Grande do Norte). Overall, public universities in Brazil are young by world standards (Balbachevsky & Quinteiro 2003, Durham 2003), with formal founding for the oldest institutions in the early 1900s. Some in these less privileged areas date from the 1960s and 1970s (1975 – Federal University of Ceará; 1966 – Federal University of Maranhão; 1960 – University of Brasília, Federal University of Rio Grande do Norte and Federal University of Paraíba).

In Brazil, there is a culture for democratic distribution of resources for R&D. In the latest evaluation of the INCTs (National Institutes for Science and Technology), which is the highest-level supporting program of the CNPq (Conselho National de Desenvolvimento Científico & Tecnológico, associated to the Ministry of Science, Technology, Innovation and Communication - MCTIC) in 2016, 252 of the 345 projects were approved. This can be compared to 57 clusters of excellence in Germany (which could be considered analogous, although Brazilian financing is heavily reduced). There is low competitivity and disproportional valorisation of the curriculum instead of the overall concept of the proposal. There is also little stimulation for basic high-risk science, the capacity for error, disruptive or “useless science” (Flexner 1939), whereby science can show tremendous advances from the most unexpected sources.

Almost all regions showed low impact in the humanities and social sciences. McManus & Baeta Neves (2021a) look at this more closely, but it should be remembered that these databases do not cover these areas well (Lariviere et al. 2006). Both these papers show that local journals and books and book chapters are more important in these areas and explain these trends. Another factor related to HSS is that Google Scholar (not used here) may be a more appropriate source of information in this area (Meho & Yang 2007). Performing normalisation at the level of these broad fields studied here means that a high/low citation impact of a country in a particular area may be a consequence of the fact that most of the activity of the country in this field takes place in subfields with a relatively high/low citation density (Bornmann & Leydesdorff 2013). These authors state that differences in citation impact reflect differences in the research portfolios (rather than differences in actual scientific impact). Furthermore, a shift of the country’s research activities from subfields may lead to an increase (or decrease) of a country’s citation impact in a particular field as a consequence of a lower (higher) citation density to subfields with a relatively higher (or lower) citation density.

The low number of patents may reflect a difficulty in registering patents or intellectual property, as well as low innovation rates reflected in the Global Innovation Index (Dutta et al. 2018). Esteves & Feldmann (2016) state that patents are not the most relevant indicator linked with innovation development. They show that Brazil relies on importing rather than producing innovation, thereby limiting home-grown solutions to problems, that tax incentives are insufficient and legislation limiting. They show that relevant innovation is strongly linked to GDP per capita, public expenditure on education and R&D, exports of high-tech products and the Number of large companies in the country, and the need for higher integration between companies and universities. This also may reflect the fact that Brazil has shown difficulty in transforming results from scientific work into innovation and technology (Oliveira 2016; https://pubs.acs.org/doi/full/10.1021/acs.jpcc.6b01958), thus leading to a large trade deficit in technological products.

De Sousa et al. (2015) point to the lack of outside private funding and venture capital investment for R&D in Brazil as a whole and the perception of high risks associated with the innovative activity. However, the R&D cooperation of Brazilian innovative manufacturing firms with universities and research institutes leads to higher success rates for new products launched by these firms. These are especially low if compared with universities abroad (Brito Cruz 2019). Frischtak (2019) notes that 99% of Brazilian firms fail to innovate and fail to transfer and absorb new technologies. This is in line with that discussed by Fabiani & Sbragia (2014) on the Good Law (11196/05), which aimed to create incentives for private companies to invest in R&D but has shown below expectation impacts mainly because of bureaucracy associated with university partnerships or fiscal obligations to implement the law. The annual investment of the private sector in public research tends to be a useful measure of the interaction and knowledge transfer between business and higher education (King 2004).

Brazilian legislation can also impede scientific progress. The constitutional amendment “Roof” on public spending (PEC 241/55) limits the university’s capability to raise funds from outside the federal government, since, if the approved annual limit is exceeded, these resources are deposited in a shared government fund, and inaccessible to the university. Contracting foreigners, even on short-time- contracts, is very bureaucratic and takes a long time, with interaction from the Ministry of Justice, Ministry of Work, Federal Police and the National Council for Immigration, and even invited lecturers must go through the same process as itinerant workers from other countries. Brazilian institutions, therefore, hire mostly Brazilians (Oliveira 2016). A more recent Endowment Law (13800/2019) has yet to be regimented, but many universities have still to open endowment funds. The Legal Framework for Science, Technology and Innovation (Law 13243/16) changed the relationship of the universities with society and public policies. Still, its implementation is challenged by internal bureaucracy in the universities, public defender’s office, national control agencies such as TCU/CGU, Planning and Budget Ministry, lack of funding and judicial uncertainties (Miranda et al. 2019).

Other laws that affect the universities directly include: the Single Judicial Regiment (Regime Jurídico Único - RJU 8112/90), whereby all public servants (including federal university staff and technicians) are regulated by the same legislation as to hiring and firing, promotion, working hours, salaries, stability, etc. This limits the universities’ ability to, for example, attract highly prolific researchers from other institutions, as all must enter at the lowest level of the career, and they cannot legally offer pay differentiated by productivity. Therefore, the career as a university lecturer is based on that of a public servant, with salary equality between and within universities, independent of living costs of the city or individual productivity, not being based on merit. Almost all faculty enter as doctors and have exclusive dedication to the position, not exercising their profession outside the university, receiving tenure after only three years. This lack of merit-based promotion and recognition tends not to stimulate competition and merit.

The Law of Directives and Bases for National Education (9394/96) states that lecturers must spend eight hours a week in class. Although not stated in the law, many universities have tended to interpret this as undergraduate teaching. On the one hand, this has led to a proliferation of classes by lecturers not involved in postgraduate education to meet their class load, with an excessive workload for lecturers also associated with the postgraduate system (Diniz-Filho et al. 2016). These authors, along with Baccini et al. (2014), showed that teaching load does not significantly affect scientific production and that low scientific output cannot be attributed to a high undergraduate teaching load.

Acquiring a research grant or scholarship from a Federal or State funding agency can also is a heavy load for researchers as errors in spending (for example, on capital instead of consumables) means that this has to be paid out of his/her pocket. Even in the case of death, the onus passes on to the family to present accounts and research reports and devolution of funds if necessary.

A large section of the federal academic community recognises the need for change and strategic planning to amplify and strengthen the universities as an essential factor in national development. This was seen within the reactions to the “Print” Program (Programme for institutional internationalisation) recently proposed by CAPES (Oliveira 2019), as well as the proposed changes to the postgraduate evaluation process (Barata 2019).

Direct engagement in the generation of knowledge is required for sustainable economic development in highly competitive world markets (King 2004). Much needs to be done within the Brazilian university system to improve its efficiency and effectiveness. Recognition of, and building on, individual university/research institution strengths are recommended. Programs with specific goals need to be created to spur on development in areas and regions of interest for the country and increase collaboration with strong and effective research partners abroad.

CONCLUSIONS

There are more than 20,000 universities worldwide (Webometrics 2019). Despite all the challenges mentioned above, Brazilian university still manages to produce relevant research for the country’s development. Public universities, in particular, have played a key role in the economic and social development in developing countries (Ordorika & Lloyd 2013): by training a majority of the professional workforce; designing state institutions; tackling pressing development problems; and providing a wide array of community service (mainly health) and cultural programs. This tradition has come under attack in recent years. Governments throughout Latin America have used the region’s relatively poor showing in the international tables to justify accelerating reforms to their higher education systems, or, in the case of Chile, to conserve the existing model, in which students and their families bear the majority of the cost of their education. Press (2013) warns about the lack of public funding for research and shows that nations that spend close to 3% of their GDP on R&D are those that compete most successfully in the modern world.

Future generations depend on these institutions to secure Brazil’s autonomy in creating knowledge, innovation, solving its problems, and preserving its culture. For decades, Brazilian society has invested in building this institutional framework. While it may be time to question the fundamentals for its change, we need to strengthen it to face new challenges. It is not time to destroy it. Private funding for research, research carried out within businesses and more flexible, and up to date legislation are all needed to improve research infrastructure within the country. For this to occur, a nationwide discussion on university reform is necessary, since the last was in 1968 (over 50 years ago). The current system was essential to ensure the increase in size and dispersion of the system. Today’s challenges for the country are different and need a more efficient and responsive design. Instead of amplifying the threats, strengths should be highlighted and strategies formed for a healthy future.

ACKNOWLEDGMENTS

To the Coordenação de Aperfeiçoamento de Pessoa de Nível Superior (CAPES) and the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq) for scholarships and funding. AGSF acknowledges funding from Fundação Cearense de Apoio ao Desenvolvimento Científico e Tecnológico (FUNCAP) through the Cientista Chefe program.

REFERENCES

• ADAMS J, MCVEIGH M, PENDLEBURY D & SZOMSZOR M. 2019. Profiles not metrics. Clarivate Analytics. Clarivate.com/products/web-of-science.
• ANGELO C. 2019. Brazil’s government freezes nearly half of its science spending. Nature 568: 155-156.
• BACCINI A, BARABESI L, CIONI M & PISANI C. 2014. Crossing the hurdle: The determinants of individual scientific performance. Scientometrics 101: 2035-2062.
• BAETA NEVES AA, MCMANUS C & DE CARVALHO CH. 2020. The Impact of Graduate Studies and Science in Brazil: an analysis in the light of the indicators. Revista NUPEM 12(27): 254-276.
• BALBACHEVSKY E & DA QUINTEIRO MC. 2003. The Changing Academic Workplace in Brazil. In: Altbach PG (Ed), The Decline of the Guru. New York, Palgrave Macmillan, p. 75-106.
• BARATA RB. 2019. Mudanças necessárias na avaliação da pós-graduação brasileira. Interface (Botucatu) 23: e180635. http://dx.doi.org/10.1590/interface.180635.
• BLIND K, POHLISCH J & ZI A. 2018. Publishing, patenting, and standardisation: Motives and barriers of scientists. Res Pol 47: 1185-1197. https://doi.org/10.1016/j.respol.2018.03.011.
• BORNMANN L & LEYDESDORFF L. 2013. Macro-Indicators of Citation Impacts of Six Prolific Countries: InCites Data and the Statistical Significance of Trends. PLoS ONE 8: e56768. https://doi.org/10.1371/journal.pone.0056768.
• BRITO CRUZ CH. 2019. Benchmarking university/ industry research collaboration in Brazil. In: Reynolds ER, Schneider BR & Zylerberg E (Eds), Innovation in Brazil Advancing Development in the 21st Century. Routledge, London & New York. https://www.ifi.unicamp.br/~brito/artigos/chbc-chapter-5-Innovation-in-Brazil_final-20190401.pdf
  » https://www.ifi.unicamp.br/~brito/artigos/chbc-chapter-5-Innovation-in-Brazil_final-20190401.pdf
• CHINESE ACADEMY OF SCIENCES & CLARIVATE ANALYTICS. 2019. Research Fronts: Active Fields, Leading Countries, 24 p. http://english.casisd.cn/research/rp/201911/P020191127529769546193.pdf
  » http://english.casisd.cn/research/rp/201911/P020191127529769546193.pdf
• DE SOUSA AG, BRAGA MJ & MEYER LFF. 2015. Impact of cooperation on the R&D activities of Brazilian firms. Proced Econ Finan 24: 172-181.
• DGEI. 2012. La complejidad del logro académico: Estudio comparativo sobre la Universidad Nacional Autónoma de México y la Universidad de Sao Paulo [The Complexity of Academic Achievement: Comparative Study of the National Autonomous University of Mexico and the University of Sao Paulo]. Preliminary report. Mexico: DGEI/UNAM.
• DINIZ-FILHO JAF, FIORAVANTI MCS, BINI LM & RANGEL TF. 2016. Drivers of academic performance in a Brazilian university under a government-restructuring program. J Informetrics 10: 151-161.
• DURHAM ER. 2003. Higher education in Brazil: Public and Private. In: Brock C & Schwartzman S (Eds), The Challenges of Education in Brazil. Oxford Studies in Comparative Education. Oxford, UK: Triangle Journals, Ltd., 2004, p. 147-178.
• DUTTA S, LANVIN B & WUNSCH-VINCENT S. 2018. Global Innovation Index 2018. Energising the world with innovation. 11th edition. Cornell University, INSEAD, and WIPO (2018): Ithaca, Fontainebleau, and Geneva.
• ESCOBAR H. 2019. Brazilian scientists lament ‘freeze’ on research budget. Science 364: 111. DOI: 10.1126/science.364.6436.11.
• ESTEVES K & FELDMANN PR. 2016. Why Brazil Doesn’t Innovate: A Comparison among Nations. INMR - Innovation & Management Review 13: 63-82.
• FABIANI S & SBRAGIA R. 2014. Tax Incentives for Technological Business Innovation in Brazil: The Use of the Good Law - Lei do Bem (Law No. 11196/2005). J Techn Manag Innov 9: 53-63. https://dx.doi.org/10.4067/S0718-27242014000400004.
• FAPESP. 2019. Dispêndios de universidades intensivas em pós-graduação. https://revistapesquisa.fapesp.br/2019/07/10/dispendios-de-universidades-intensivas-em-pos-graduacao/?fbclid=IwAR2IFC93jjcaDaEWAO-LX_bXDSs6xr5BdFE1ABTQKosZXDRFu92YBoPy3b0
  » https://revistapesquisa.fapesp.br/2019/07/10/dispendios-de-universidades-intensivas-em-pos-graduacao/?fbclid=IwAR2IFC93jjcaDaEWAO-LX_bXDSs6xr5BdFE1ABTQKosZXDRFu92YBoPy3b0
• FLEXNER A. 1939. The usefulness of useless knowledge. Harpers 179: 544-552.
• FONSECA RS & VELOSO AP 2018. The practice and future of financing Science, Technology and Innovation. Foresight STI Govern 12: 6-22.
• FORTUNATO S ET AL. 2018. Science of science. Science 359(6379): 0185. https://science.sciencemag.org/content/359/6379/eaao0185.
• FRISCHTAK C. 2019. Science and Innovation in Brazil: where to now? In: Reynolds EB, Schneider BR & Zylberberg E. Innovation in Brazil: Advancing development in the 21st century. Routledge, Oxon and New York.
• JEONG S, CHOI JY & KIM J-Y. 2014. On the drivers of international collaboration: The impact of informal communication, motivation, and research resources. Sci Public Pol 41: 520-531. https://doi.org/10.1093/scipol/sct079.
• JORDAN GB, STREIT LD & MATIASEK J. 2003. Attributes in the Research Environment That Foster Excellent Research: An Annotated Bibliography. 34 p. SAND report 2003-0132. https://pdfs.semanticscholar.org/3f36/01690081727f115e21b766a7932cdcb767ac.pdf
  » https://pdfs.semanticscholar.org/3f36/01690081727f115e21b766a7932cdcb767ac.pdf
• KING DA. 2004. The scientific impact of nations. Nature 430: 311-316. https://www.nature.com/articles/430311a.
• KINSER K & LEVY DC. 2006. The for-Profit Sector: US Patterns and International Echoes in Higher Education. Albany, NY: PROPHE Working Paper Series.
• LARIVIERE V, GINGRAS Y & ARCHAMBAULT E. 2006. Canadian collaboration networks: A comparative analysis of the natural sciences, social sciences and the humanities. Scientometrics 68: 519-533.
• LEAL F. 2019. Walking toward the cliff. https://www.insidehighered.com/blogs/world-view/walking-toward-cliff
  » https://www.insidehighered.com/blogs/world-view/walking-toward-cliff
• LLOYD M. 2013. Las Políticas de Fomento a la Ciencia y Tecnología en México y Brasil: Un Estudio de Caso de la Universidad Autónoma Nacional de México y la Universidad de Saõ Paulo [Science and Technology Policies in Mexico and Brazil: A Case Study of the National Autonomous University of Mexico and the University of Sao Paulo]. Master´s thesis in Latin American Studies, National Autonomous University of Mexico.
• MASSARANI L & MOREIRA IC. 2016. Science communication in Brazil: A historical review and considerations about the current situation. An Acad Bras Cienc 88: 1577-1595. https://dx.doi.org/10.1590/0001-3765201620150338.
• MCMANUS C & BAETA NEVES AA. 2021a. Production Profiles in Brazilian Science, with special attention to social sciences and humanities. Scientometrics 126: 2413-2435.
• MCMANUS C & BAETA NEVES AA. 2021b. Funding research in Brazil. Scientometrics 126(1): 801-823.
• MCMANUS C, BAETA NEVES AA, MARANHÃO AQ, SOUZA FILHO AG & SANTANA JM. 2020b. International collaboration in Brazilian science: financing and impact. Scientometrics 125(3): 2745-2772.
• MCMANUS C & NOBRE CA. 2017. Brazilian Scientific Mobility Program-Science without Borders*-Preliminary Results and Perspectives. An Acad Bras Cienc 89: 773-786.
• MCMANUS CM, BAETA NEVES AA & MARANHAO AQ. 2020a. Brazilian publication profiles: Where and how Brazilian authors publish. An Acad Bras Cienc 89: 773-786.
• MEHO LI & YANG K. 2007. Impact of Data Sources on Citation Counts and Rankings of LIS Faculty: Web of Science Versus Scopus and Google Scholar. J Amer Soc Info Sci Tech 58: 2105-2125.
• MIRANDA ALBB, ARAUJO IT, FREIRE BGO & FERNANDES AJ. 2019. Inovação nas universidades: uma análise do novo marco legal. Rev Eniac Pesq 8: 85-98.
• MUÑOZ GARCÍA H. 2009. Introducción [Introduction]. In: La Universidad Pública en México [The Public University in Mexico], Seminário de Educación Superior, 5-22. Miguel Ángel Porrúa, Mexico. https://www.ses.unam.mx/integrantes/uploadfile/hmunoz/Munoz_UniversidadPublicaEnMexico.pdf
  » https://www.ses.unam.mx/integrantes/uploadfile/hmunoz/Munoz_UniversidadPublicaEnMexico.pdf
• OECD - ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT. 2007. Revised field of science and technology (FOS) classification in the Frascati manual. Paris, France: Working Party of National Experts on Science and Technology Indicators, Organisation for Economic Co-operation and Development (OECD).
• OLIVEIRA CS. 2019. A internacionalização do ensino superior no Brasil por meio da ação da CAPES: A Cocriação do Programa CAPES-PrInt. Doctor´s Thesis, University of Brasilia, 253 p.
• OLIVEIRA ON. 2016. Research Landscape in Brazil: Challenges and Opportunities. J Phy Chem 120: 5273-5276. https://doi.org/10.1021/acs.jpcc.6b01958.
• ORDORIKA I & LLOYD M. 2013. A Decade of International University Rankings: a Critical Perspective from Latin America. In: Marope PT, Wells P & Hazelkorn E (Eds). Rankings and Accountability in Higher Education Uses and Misuses. Paris: UNESCO Publishing, p. 210-231.
• PRESS WH. 2013. What’s So Special About Science (And How Much Should We Spend on It?). Science 342: 817-822. 10.1126/science.342.6160.817.
• SALMI J. 2009. The Challenge of Establishing World Class Universities. Washington, DC: The World Bank.10.1596/978-0-8213-7865-6.
• SANDSTRÖM U & BESSELAAR P. 2016. Quantity and/or Quality? The Importance of Publishing Many Papers. PLoS ONE 11(11): e0166149. https://doi.org/10.1371/journal.pone.0166149.
• SENNES RU & BRITTO FILHO A. 2012. Technological innovations in Brazil: performance, policies and potential. São Paulo: Cultura Acadêmica, 2012. 368 p. https://www.interfarma.org.br/public/files/biblioteca/technological-innovations-in-brazil-interfarma.pdf
  » https://www.interfarma.org.br/public/files/biblioteca/technological-innovations-in-brazil-interfarma.pdf
• WEBOMETRICS. 2019. Ranking Web de Universidades. Avaliable in http://www.webometrics.info
  » http://www.webometrics.info

SUPPLEMENTARY MATERIAL

Tables SI, SIIa, b

Publication Dates

• Publication in this collection
  03 Sept 2021
• Date of issue
  2021

History

• Received
  24 Feb 2020
• Accepted
  21 May 2020