Challenges and perspectives of academic evaluation

Academic evaluation has been an essential component of modern science since its inception, as science has moved away from personalized patronage toward its contemporary role as an essential enterprise of contemporary, democratic societies. In recent years, Brazil has experienced sustained growth in its scientifi c output, which is nowadays fully compatible with its status as a high middle-income country striving to become a fully developed, more equitable country in the years to come. Growth usually takes place amidst challenges and dilemmas and, in Brazil as elsewhere, academic evaluation is not exempt from such diffi culties. In a large, profoundly heterogeneous country with a national evaluation system and nationwide on-line platforms disseminating information on the most disparate fi elds of knowledge, the main challenges refer to how to pay attention to detail without losing sight of comprehensiveness and how to handle social and regional diversity while preserving academic excellence as the fundamental benchmark. DESCRIPTORS: Researcher Performance Evaluation Systems. Scientifi c Publication Indicators. Periodicals as Topic. Science, Technology and Innovation Indicators.

The need to evaluate what has been produced under the aegis of different funding sources is as old as the emergence of Modern Science itself.In its beginnings, evaluation was basically defi ned as a system of obscure (in the eyes of the public) rewards and vetoes applied to fi ndings that might please or challenge the status quo and the points of view of nobles and priests, as clearly illustrated by Galileo Galilei's trial.a The long and winding road between the emergence of Modern Science in Galileo's times and our own age is characterized by: 1. the defi nition of people who carry out research as "scientists" as late as 1834 (in the case of the English language) by the Rev. W. Whewell (1794-1866).
Whewell was an infl uential thinker in the era of Charles Darwin (1809-1882) and John Snow (1813-1858), so the very word "scientist" emerged at the same time as modern biology and epidemiology;

THE DISUNITY OF SCIENCE AS A CHALLENGE TO INTEGRATED EVALUATIONS
We are used to speaking of Science with a capital S, as such capitalized Science could muster aims, methods and procedures across the most different disciplines."Disunity of science" is an expression coined by the American physicist and historian of science Peter Galison d to document the fact different disciplines have boundaries, are propelled or hindered by different, idiosyncratic forces, and use methods and procedures tailored to their specifi c aims.
Brazil has a national evaluation system and a nationwide on-line curricula vitae (CV) platform (unifi ed as the Lattes platform e ), unlike other large federative countries, such as the US and Germany.The Brazilian system roughly resembles Canadian funding institutions in terms of profi ting from unifi ed, on-line CV platforms.f But key differences should be highlighted here: Canada is much less populated and heterogeneous (in both social and geographical terms) than Brazil, and its on-line systems cover specifi c areas, such as research on health.f On the other hand, Canada is a bilingual country, what asks for English and French versions of every single document.Both differ markedly from the topical nature of US-based institutions and agencies (compare, for instance, the specifi city and conciseness of the National Institutes of Health "eRA Commons" biosketches g with Brazilian or Canadian comprehensive CVs).
Brazil's challenge is enormous and comprises the need to accommodate the demands of the most different sciences and disciplines, varying from the most abstract ones, such as philosophy and theoretical physics, to empirical sciences, such as most research carried out in the fi eld of public health or engineering, as well as properly handle the pronounced heterogeneity of its institutions, society and geographic regions.

FOCUSING ON PUBLIC HEALTH
In a critical review of scientometrics in public health, Coimbra Jr. highlighted its comparatively low impact (vis-à-vis frontier science) and its contextual and applied nature (his analysis focused on Latin American scientifi c output). 3 The author is right in his criticism and his proposals toward broader and more "applied" criteria of scientifi c relevance.Not as a coincidence, fi erce battles shake every single fi eld of knowledge from time to time over the attribution of "Qualis scores" to journals, as listed at the CAPES webpage.h From my own perspective, any stable consensus regarding such scores would rather refl ect coercion than consensus.So, controversy (sometimes harsh) tend to (re)emerge here for ever and ever, but this is the price Brazilian academy will always pay to have a unifi ed national evaluation system across different disciplines, graduate programs, and localities.
In a similar way, on-line CV platforms seem to be condemned to be on the wing, pulled from one side towards conciseness and practicality and from other side towards comprehensiveness and capacity to accommodate different activities and demands.The recent incorporation by CNPq of a brand new module to Lattes platform on the dissemination of science seems to be a compromise between the straightjacket of peer review and the need for scientists to disseminate to the society at large using language as free of jargon and technicalities as possible.
The many different books written by Richard Feynman on physics and computation for non-specialists remain paradigmatic in terms of scientifi c soundness combined with clarity of exposition and conciseness, even when he focused on what he called "not-so-easy" pieces.Of course, to demand from all of us to be as talented and didactic as Feynman would push it too far.But maybe the sharp criticisms he published on teaching and doing physics in Brazil, as he observed fi rst-hand during his sabbatical at the Centro Brasileiro de Pesquisas Físicas (CBPF), in the early 1950s, i may help us to better integrate academic excellence, academic teaching and the dissemination of science to the public at large.
"I don't think non peer-reviewed publications (such as reports in newspapers or television, blogs or posts in the web) could ever replace peer-reviewed articles.They should, rather, complement and foster a dynamic dialogue with them.Peer-reviewing is far from perfect, as has been discussed in the most different forums, many of them launched by the editors of peer-reviewed journals themselves.But I do agree with a recent comment by the German researchers and editors Maerthens & Baethge 6 [that] peer-review is "fl awed and under-researched, but the best we have".In the same way, democracy has many fl aws and caveats, but for those who once lived in dictatorships it seems to be well defi ned in Winston Churchill's famous quotation: "[…] democracy is the worst form of government except all the others that have been tried".
Examples of proper and improper uses of the most different media in their attempts to disseminate science are so numerous, confl icting and confusing that any attempt to base evaluations on such examples are not likely to reach any valid conclusion or even a provisional agreement.On a personal note, I remember here the positive infl uence of the Brazilian journal on popular science "Ciência Hoje" j (launched in 1982), from which I have learned so many wonderful things about science (especially about fi elds I will never have the expertise, time or talent to explore), over many years.
On the negative side, I think any Brazilian citizen minimally concerned with science and their country should read about the defamatory newspaper campaign launched by a powerful group of Academia Nacional de Medicina against Carlos Chagas in the early 1920s.Lacking any compromise with scientifi c integrity, a group of renowned physicians launched against Chagas -the most distinguished biomedical scientist of that period, as well as against his landmark discovery (later on, named after him as Chagas' disease) -personal insults as well as biased criticisms.k Such examples, so close to our own efforts as biomedical researchers, is a clear warning that any evaluation should be as comprehensive as possible, always comparing and contrasting different evidence and perspectives.There is no magical solution, no easy fi x… as regarding civilization itself (as pointed by Sigmund Freud).Any evaluation will generate as a necessary corollary "its discontents".l I would like to conclude this article with an example from a fi eld of research which is situated so far from my own fi eld of expertise and from research published by Revista de Saúde Pública that at fi rst sight it may seem pure nonsense.Anyway, without some unusual ideas science remains stalled.
Living and working in the UK in the period I submitted these comments to Revista de Saúde Pública, I began to read in that same period a book published by the American theoretical physicist Lisa Randall.After reading the accolades that usually embellish the back covers of books and are so many times mistaken, I decided to browse her homepage at Harvard and then entered the Scopus database to double-check what Lisa Randall designated in her CV as "recent and highly cited papers" m (after all, epidemiologists are a kind of scientifi cally-endorsed gossipers).In the Scopus database, I realized that, as of May 30, Dr. Randall had been cited by 14,511 indexed papers.n Curiously, such a hard-nosed scientist, dealing with abstract concepts and sophisticated mathematics has authored two books on popular science (one of them, the book that launched my quest for additional information [Knocking on Heaven's Door]), contributed to many different newspapers and blogs, radio broadcasts, and -most surprising -wrote a libretto for an Opera.Defi nitively, Dr. Randall never attended the classic debates which took place repeatedly in our graduate forums, where wars are waged against "the highly cited scientists who have alienated themselves from real life problems".
I think it's time to turn old prejudices upside down and realize that old dichotomies (such as the one between "ivory tower intellectuals vs. people who care about life as it really is") are not as simple as people would like to think.Maybe such overly simplistic contrasts should instead be viewed upside-down, as in the funny sentence from Feynman (who was a bongo drummer and played in Rio de Janeiro's samba schools), introducing one collection of his lectures on physics: "It is odd, but on the infrequent occasions when I have been called upon to play the bongo drums in a formal place, the introducer never seems to fi nd it necessary to mention I also do theoretical physics."(Feynman, 1992, p. 13).

its major achievements in its fi rst 350 years in Bryson). 2 SCIENCE AFTER WORLD WAR II
2. the gradual replacement of doing research under the aegis (patronage) of a given member/circle of the clergy/nobility or on behalf of a given state b by the alternative perspective of doing science "supported" by public agencies, understood, to a large extent, as an independent task pursued by individual scientists/ research groups; 3. the establishment of societies, institutions and agencies with the explicit aim of promoting science and debating its concepts and fi ndings.One cannot imagine French or British science without institutions like the French Academy of Science, founded in 1666, or the Royal Society, in 1660 (about the latter, see a review of Education Personnel (CAPES -Coordenação de Aperfeiçoamento de Pessoal de Nível Superior), until then defi ned as a "campaign" c (what in English would be roughly the equivalent of a taskforce) became a permanent institution and launched its regular evaluations of graduate courses.WHAT DO EVALUATIONS MEAN, WHAT DON'T THEY Evaluation of academic work aims to assess the intrinsic value of research from the perspective of peers; to make science and its technological applications and social impacts accountable to governments and tax payers; and to provide guidance for policymaking in any area of societal life which may interact with science & technology (what might be roughly coincident with contemporary life in a broad sense; but see Feynman 5 on what he called our "unscientifi c" age).Some of the problems, frustrations and anxiety scholars and students feel respecting evaluation are secondary to (mis-)expectations.For instance, as a teacher and former coordinator of a graduate program, I heard many times that evaluations at the national level should include assessments of individual classes/ teachers.Desirable and sorely needed as such specifi a Such historical facts will not be analyzed here.Interested readers may consult Galileo Galilei biographies such as the recent one written by John L. Heilbron.Heilbron JL.Galileo.New York: Oxford University Press; 2012.Available from: http://www.amazon.com/Galileo-John-L-Heilbron/dp/0199655987/ref=sr_1_2?s=books&ie=UTF8&qid=1377282006&sr=1-2 b The latter idea was aggressively promoted by the most different dictatorial regimens and peaked on controlling and intolerance in the socalled "Gleichschaltung" [forcible coordination], imposed by the Nazis upon the 3 centuries-old Prussian Academy in June 1939.c Historical facts about CAPES are available at: Coordenação de Aperfeiçoamento de Pessoal de Nível Superior -CAPES.História e missão.Brasília (DF); 2013 [cited 2013 Jul 23].Available from: http://www.capes.gov.br/sobre-a-capes/historia-e-missao. 4 j Instituto Ciência Hoje.Rio de Janeiro; 2013 [cited 2013 Jul 23].Available from: http://cienciahoje.uol.com.brk Biblioteca Virtual Carlos Chagas.Trajetória.Rio de Janeiro: Casa de Oswaldo Cruz; 2013.A polêmica [cited 2013 Jul 23].Available from: http:// www.bvschagas.coc.fiocruz.br/php/trajetoria.php#polemical I made a pun here with Freud's book "Civilization and its Discontents".The original German title "Das Unbehagen in der Kultur" is rather translated into English as "The Uneasiness in Culture", but "Civilization and its Discontents" became such a popular title to the point of obfuscating the original one.Freud S. Das Unbehagen in der Kultur.Wien: Internationaler Psychoanalytischer Verlag; 1930.m Randall L. Curriculum vitae.Cambridge: Harvard University; s.d.[cited 2013 Jul 23].Available from: http://randall.physics.harvard.edu/CV.htmln The link refers to a consultation performed as of May 30 2013 and may be no longer valid and/or accurate when opened in subsequent moments: http://www.scopus.com.scopeesprx.elsevier.com/cto2/main.url?origin=AuthorProfi le&stateKey=CTOF_423753892