IAG- Pontifícia Universidade Católica do Rio de Janeiro

This article proposes a three-stage method, with the use of multiple rankings as a starting point in the first stage to help carry out bibliographical research in a more effective and systematic fashion. The relevance of this method is made evident by way of a pioneer empirical investigation on the use and importance of scientific journal rankings as perceived by Brazilian professors in the practice of bibliographical research in the field of Business Administration. The results suggest that, although there is an awareness of the importance of ranking as a criterion for bibliographical research, with a view to identifying the state of the art and relevant research gaps, ranking is in fact hardly used on a regular basis. It is believed that the proposed method can help overcome this problem and contribute to more effective literature reviews and research.


I I I INTRODUCTION NTRODUCTION NTRODUCTION NTRODUCTION
There is a consensus among scientific researchers that for scientific knowledge to progress it is necessary to conduct rigorous bibliographical research to identify the state of the knowledge, possible research gaps that may exist and opportunities for new contributions to the theme under study.
Scientific journals are the principal means of bibliographical research and publication of scientific research (Arenas, García, & Espasandín, 2001;Campos, 2003;Tahai & Meyer, 1999;Vilhena & Crestana, 2002), because they present the main results of investigations and constitute inputs for new ones (Romancini, 2004). Note that, in Brazil, in addition to scientific journals, especially in the field of Business Administration, academic conference papers (conference proceedings) have played an important role in divulging scientific research results.
Bibliographical research is an arduous and time-consuming task (Magarey, 2001;Podsakoff, Mackenzie, Bachrach, & Podsakoff, 2005;Tahai & Meyer, 1999). Although information technology, by providing powerful search tools, has indeed contributed to the identification of possible sources of consultation, the exponential growth of the amount of information available (Valiela & Martinetto, 2005) has made searches and decisions regarding the best sources to be adopted increasingly complicated.
Various criteria have been used to rank scientific journals, especially, the impact factor, which is calculated by dividing the total number of citations in any given year of articles from a specific scientific journal, published over the previous two years in a specific set of scientific journals, by the number of scientific articles published by the same scientific journal during the same period (a detailed explanation and example of the impact factor is given in Section Ways of Ranking). According to Campos (2003), this criterion is "currently one of the most often used to assess the quality of a scientific journal" (p. 1). In fact, various authors and organizations adopt the impact factor as a proxy of quality indicator (Baden-Fuller et al., 2000;Erasmus Research Institute of Management [ERIM], 2003;Financial Times, 2003;Garfield, 1999;Geary et al., 2004; Institute for Scientific Information [ISI], 2005;Mylonopoulos & Theoharakis, 2001;Rainer & Miller, 2005;Saha, Saint, & Christakis, 2003;Tahai & Meyer, 1999). Thus, according to various authors, the main purpose of ranking is to assess the quality of published scientific journals (Campos, 2003;Krzyzanowski & Ferreira, 1998;Tahai & Meyer, 1999;Vanti, 2002;Yamamoto et al., 1999). In Brazil, the Coordination for the Improvement of Higher Education Personnel [CAPES] (http://www.capes.gov.br) uses the impact factor to assess international scientific journals that are selected by the Institute for Scientific Information [ISI] (Campos, 2003;Vilhena & Crestana, 2002).
According to Vilhena and Crestana (2002), some scientific editors believe that the bibliometric indicators used to assess scientific impact do not gauge quality. However, they are considered useful as additional tools in the evaluation of scientific research (Tahai & Meyer, 1999).
The objective of this article is to propose a three-stage method, with the use of multiple rankings as a starting point in the first stage, to help carry out bibliographical research in a more effective and systematic fashion. Its relevance is illustrated by the results of a pioneer empirical investigation in Brazil on the use and importance of scientific journal rankings, as perceived by professors, in the practice of bibliographical research in the social sciences, precisely, Business Administration.
The article is divided into 5 sections. In the first, we present the investigation's theoretical references with respect to state of the art assessment, ways of classifying and ranking, scientific rankings and impact factor. In the second section, we describe our research methods, and in the third one we present the survey's main results and propose a bibliographical research method. These are discussed in the fourth section. Some final considerations are formulated in the last section.

State of the Art Assessment
State of the Art Assessment State of the Art Assessment State of the Art Assessment One of the main and necessary steps in any research methodology is to correctly assess the state of the knowledge in its field of investigation. According to Li and Cavusgil (1995) there are three basic approaches when conducting such an investigation: (1) the Delphi method, (2) meta-analysis and (3) content analysis.
The Delphi method, which was created as a forecasting method that makes use of the opinions of independent experts, can be adapted to state of the knowledge investigation. Due to time constraints and the availability of experts, the Delphi method is very rarely used in thesis and dissertation research processes. What happens often, at least as far as seminal works are concerned, is that the professors who direct their theses are consulted. Unfortunately, however, they do not benefit thus from the interactions at play in the Delphi method.
Meta-analysis is a statistic method that performs a combined analysis of the quantitative results of several previous empirical studies (Rosenthal & Dimatteo, 2001). The use of such results, given that they are statistically compatible (data from the different studies have the same characteristics), can address problems such as the lack of statistical power of each individual study (Magarey, 2001). It is a common method in evidence-based medicine research.
The origin of content analysis lies in the study of the meanings of textual communication content, by ways of selecting and categorizing meaningful parts of the text according to a defined theoretical framework. According to Krippendorff (2004), contemporary content analysis "… is an empirically grounded method, [that] … transcends traditional notions of symbols, contents and intents [and] … has been forced to develop a methodology of its own".
Another way to identify the state of the knowledge, especially in master and doctoral research, is to conduct bibliographical reviews. It has become increasingly difficult to conduct thorough reviews as there are more and more sources of information (journals, conferences, etc.). In order to cope with this large amount of information, researchers such as Arenas et al. (2001) have used rankings to define which journals to investigate. Due to the nature of their study, they use a closed set of journals and hence, bibliographical references. On the other hand, in master and doctoral research, the continuous building of theoretical references characterizes an ongoing process and, therefore, uses an open set of references.
Since the early 1990s, with the advent of the first World Wide Web browsers, many search mechanisms were made available. Huge amounts of scientific articles could be found by simply typing in some keywords. As noted in Section State of the Art Assessment, the use of keywords in these Web search mechanisms does not always lead to a thorough assessment of state-of-the-art knowledge in a particular field of research because there are many different keyword coding schemes.

Ways of Clas Ways of Clas Ways of Clas Ways of Classifying sifying sifying sifying
It is relevant to note that, for classifying articles, most authors use keywords. Libraries define book descriptors (subjects), for cross-referencing purposes, based on the thesaurus, but authors of an article are often free to use whatever keywords they choose, mainly because there is no universally agreed upon set of such keywords. This is especially true in the case of state-of-the-art research, where new concepts and terms are defined, and such keywords are frequently suggested and revised. The plethora of different keywords, some representing the same or very similar concepts, makes keyword-based bibliographical research very difficult for researchers, for example, when they need to apply such keywords in Web search mechanisms.  Arenas et al. (2001) draw attention to the importance of formalizing a clear investigation strategy based on rankings at the literature review stage. Content analysis of the articles chosen on the basis of keywords was adopted in the scope of their survey, with a view to defining the state of the art in strategic alliance network research. They propose a 6-step method: (1) determine, by using rankings, which journals to investigate; (2) review the articles published for the last five years on the selected journals; (3) assess the article's keywords; (4) build a data table; (5) determine research streams; and (6) do keyword analysis. It should be pointed out however that, according to Arenas et al. (2001), despite the fact that the use of statistics produces objective data, their method "has drawbacks, for in the last instance it is still the investigator who chooses the research streams and the keywords" (p. 12).

Scientific Ranking Scientific Ranking Scientific Ranking Scientific Ranking
Bibliometric indicators are used to index scientific journals. Ranking is one of the processes used to index and qualify those that are most recognized by the scientific community (Campos, 2003).
One of the types of ranking most cited by various authors is the one undertaken by the Institute for Scientific Information [ISI] (http://www.isinet.com) (Campos, 2003;Romancini, 2004;Vanti, 2002;Vilhena & Crestana, 2002). The ISI produces a bibliometric indicator of scientific journals based on the impact factor, but does not analyze the impact factor of all scientific journals. The ranking of scientific journals according to the impact factor has been published by the ISI's Journal of Citation Reports [JCR] (http://www.isinet.com/products/evaltools/jcr/) on an annual basis since 1975. The criteria used by the ISI to assess scientific journals are: (1) the journals' basic publication standards (including publication schedule, adherence to international editorial conventions, bibliographical information in English -title, keywords, abstract and citations); (2) the journal's editors; and (3) the international diversity of authors and editors (ISI, 2005).
In Brazil, the Qualis (Classification System of Periodicals, Annals and Journalshttp://www.qualis.capes.gov.br), created by the CAPES, ranks domestic and international scientific journals according to categories that are indicative of quality -A, B or C -and the breadth of their circulation -local, national or international. The combinations of these categories constitute nine levels that are indicative of the importance of the vehicle used, and, by inference, of the published work itself (Campos, 2003, p. 19;CAPES, 2005). One of the parameters used by the CAPES to assess scientific journals is the impact factor published by the JCR. Note that the CAPES uses ISI's impact factor only for ranking international journals.
Another important entity is the Scientific Library on Line [SciELO] (http://www.scielo.br), which began in 1997 as a partnership between the State of São Paulo's Foundation for the Support of Research [FAPESP] (http://www.fapesp.br) and the Latin American and Caribbean Health Sciences Information Center [BIREME] (http://www.bireme.br), and has been supported by the National Council for Scientific and Technological Development [CNPq] (http://www.cnpq.br) since 2002. SciELO's aim is to develop a methodology to prepare, store, disseminate and assess scientific production in an electronic format (SciELO, 2005). It also produces performance indicators by monitoring the access to scientific journals electronically and by analyzing the impact factor to decide whether to keep or not a publication in its database (Romancini, 2004).
Bibliometric indicators are also used to assess the productivity and quality of scientific research by measuring the number of researchers' publications and citations. These indicators are useful because they help researchers decide where to publish (Campos, 2003;Holsapple et al., 1994;Vanti, 2002).
It is important to emphasize that bibliometric studies of the scientific literature, according to Romancini (2004), "... are performed mainly in the exact and biological sciences" (p. 4). As social scientists need more space to present their arguments they prefer to publish their proposals in books (Romancini, 2004;Yamamoto et al., 1999). Romancini (2004) believes that scientific journals provide accessibility and visibility simultaneously, as "the criteria taken into account to judge the importance and quality of [a scientific journal] are linked to their effective use by a community of researchers -which assures the recognition of its merit" (p. 13). Thus, one seeks to analyze journals using indicators. Vilhena and Crestana (2002), considering that the scientific journal "is one of the most commonly used vehicles to communicate scientific research, [have observed] in recent years a concern, on the part of funding institutions, [with the ranking] of scientific journals and [assessment of] the number of citations" (p. 1).
Although it is impossible to measure the use of an article in a scientific journal, it is possible to verify the effect of scientific articles by measuring how often they are used and by identifying where they have been cited in other scientific articles (Vanti, 2002;Vilhena & Crestana, 2002). Thus, Baden-Fuller et al. (2000) argue that when an article is published in a scientific journal it is being certified; consequently, ranking is important and provides valuable information. They also believe that the publication of articles in scientific journals is influenced by the reputations of authors. Thus, wellknown authors who belong to prestigious educational institutions have more opportunities to be published. As we shall see below, ranking is often used by many authors and the impact factor is one of the most pragmatic ranking criteria.

Ways of Ranking Ways of Ranking Ways of Ranking Ways of Ranking
According to Peffers and Ya (2003) rankings seek to observe aspects "of the journals, such as the importance of the teaching staff, [...] frequency of use, quality, contribution, production preference and influence" (p. 3). According to these authors, who undertook a vast review of the literature produced by researchers who have sought to rank scientific journals in the information systems area over the past 20 years, rankings may use one of the three following kinds of measurement: 1) Citations -revealed preference study (Dubois & Reeb, 2000, 2001Peffers & Ya, 2003;Podsakoff et al., 2005;Rainer & Miller, 2005;Tahai & Meyer, 1999); 2) Perception of a representative group -study of the indication of preference of a group chosen at random (Arenas et al., 2001;Baden-Fuller et al., 2000;Dubois & Reeb, 2000, 2001Peffers & Ya, 2003); 3) Perception of an elite group -study of the indication of preference of a group of intentionally selected specialists (Geary et al., 2004;Peffers & Ya, 2003;Rainer & Miller, 2005;Tahai & Meyer, 1999).
For some authors (Geary et al., 2004;Tahai & Meyer, 1999), the studies that use citations as a form of measurement are more objective than surveys of perceptions, but are biased because they adopt a small number of scientific journals for collecting information. In their opinion, studies that use the perception of an elite group help to concentrate attention on journals that publish high-quality research but, "because they constitute a small and unrepresentative group, may fail to consider cutting edge research, in favor of research that is better known in the institutions they represent" (Geary et al., 2004, p. 65). Geary et al. (2004) and Baden-Fuller et al. (2000) argue that the best kind of analysis is based on data regarding the perception of a representative group of researchers. However, Geary et al. (2004) criticize the way data used in previous studies was gathered, in that the latter did not make a dynamic up-dating mechanism possible, given that the number of scientific journals assessed was determined prior to the research. In order to overcome this shortcoming, Geary et al. (2004) began to allow respondents to add other journals to their original list. These authors believe that each researcher has his favorite group of scientific journals, which includes those he has read or in which he has published or been cited. Other authors (Dubois & Reeb, 2000, 2001, however, believe that there is a strong correlation between objective and subjective ways of measuring the quality of scientific journals. Journals can be more valuable if researchers consider them to be of greater quality. Even though researchers may have their preferred journals, the most important finding of the research undertaken by Geary et al. (2004) was that 50% of citations were taken from 126 scientific journals out of a total  Linde (1998), 15% of scientific journals generally account for 50% of citations. This was also verified by Podsakoff et al. (2005) who found that the first 14 scientific journals ranked by them received 80% of citations. Since they had analyzed 28 journals covering a period of about 20 years this fully demonstrated the importance of ranking.
For Rainer and Miller (2005), although many authors (Baden-Fuller et al., 2000;Campos, 2003;Geary et al., 2004;Holsapple et al., 1994;Podsakoff et al., 2005;Vanti, 2002) have conducted studies ranking the quality of scientific journals over the past 15 years, with a view to helping researchers to identify the state of the art and strategy of research in themes addressed by academic courses (Holsapple et al., 1994), they differ in a number of ways, notably regarding the size and composition of the sample of respondents, number of articles, and the methods used to select and rank scientific journals. They also noted that each study drew up a ranking at a specific point in time. This led Holsapple et al. (1994) and Podsakoff et al. (2005) to criticize the fact that rankings did not take the scientific journal's age into account and that the studies undertaken covered a relatively short period of time.
To sum up, no matter what criteria are used to rank scientific journals -citations, perception of a representative group, perception of an elite group or other hybrid forms -according to our literature review, it is generally believed that rankings influence the assessment of the quality of scientific journals and that they are considered relevant by the scientific community.

Impact Factor Impact Factor Impact Factor Impact Factor
The impact factor deserves special attention due to its importance and because it is the indicator preferred by academia. It is the bibliometric indicator used by the ISI and the CAPES, and is calculated by dividing the total number of citations in any given year of articles from a specific scientific journal, published over the previous two years in a specific set of scientific journals (in this case, ISI's scientific journal database), by the number of scientific articles published by the same scientific journal during the same period (Campos, 2003;Garfield, 1999;Medeiros, 2003;Saha et al., 2003;Tahai & Meyer, 1999;Vilhena & Crestana, 2002), as shown in Table 1, regarding a hypothetical Journal Y.  Some authors have criticized the use of the impact factor to assess scientific journals (Campos, 2003;Coelho et al., 2003;Coura & Willcox, 2003;Uncles, 2004). However, only a few authors have undertaken empirical research to analyze whether or not the impact factor provides an accurate measurement of the quality of scientific journals. For example, Saha et al. (2003) examined the association of the perceived importance of general medical journals to its end users, as a proxy for quality, with the ISI impact factor of these journals. Campos (2003) highlights certain limitations in the use of the impact factor: "As it is a statistical measure, the impact factor can be distorted and its value can be misinterpreted" (p. 18). References vary considerably between different areas, and as the impact factor assesses citations of scientific journals over a two-year period, oscillations in their annual assessments may occur, especially in the case of journals that publish only a small number of articles annually. Campos (2003) and Linde (1998) believe that assessments should cover a longer period, for example five years, and that this calculation is also affected by the number of authors involved, given that there is a strong and positive correlation between the number of authors and the impact factor.
The selection of journals to be used can have a significant impact on the result of the assessment (Uncles, 2004), especially if the journals have a high rate of self-citation, and if there is a chance that each author may cite his or her own articles (Campos, 2003;Tahai & Meyer, 1999).
Until quite recently rankings did not receive the attention they do today. Garfield (1998) and Linde (1998) emphasize the dangers of using any kind of statistical information out of its appropriate context. Garfield (1998) provides examples of funding that was not granted because specific scientific journals had obtained a low impact factor, and Linde (1998) cites the use of the impact factor to guide the allocation of funding and assessments for faculty promotions.
Articles published in English are more likely to be cited than those written in other languages (Baden-Fuller et al., 2000;Campos, 2003;Garfield, 1998;Uncles, 2004), and articles published in local scientific journals have a lower impact factor than when they are published in international scientific journals (Garfield, 1998).
If used with the necessary awareness of its limitations, the impact factor could indeed be a significant and very useful indicator. Table 2 summarizes the main positive factors of the scientific rankings presented by the authors cited, including the impact factor.

Baden
The literature review was carried out to identify the existing methods in different fields of knowledge. The one that referenced more specifically to Business Administration and that was concerned with rankings was Arenas et al. (2001), thereby constituting a starting point for our proposed method. We built upon the Arenas et al. (2001) method concerned all the while with ensuring not only more systematic but also more productive bibliographical research, considering the increasing time and funding constraints.
An objective of the survey was to assess the relevance of having a method for carrying out systematic bibliographical research. Therefore, it was necessarily exploratory, aiming at the "discovery of intuitions" (Gil, 2002, p. 42).

Survey's Method Survey's Method Survey's Method Survey's Method
A triangulation strategy was adopted to obtain data from multiple sources with a view to producing more consistent conclusions (Yin, 1996). Data was collected by way of (1) a documental/telematic investigation and (2) a survey of perceptions captured by way of a predominantly structured questionnaire.
The Business Administration area was chosen because it is a multidisciplinary field of study, with administrators generally dealing with cultural, functional and technical matters, using knowledge from various fields, and which contains specialists with different backgrounds (Easterby-Smith, Thorpe, & Lowe, 1991) such as Economy, Sociology and Anthropology. Thus, instead of selecting professors

Factors Authors
Need to write in English for greater visibility. Baden-Fuller et al. (2000); Campos (2003); Garfield (1998); Magarey (2001) Error in the interpretation of the impact factor given that it is a statistical tool.
Influence of the authors' reputations. Baden-Fuller et al. (2000) Creation of anxiety in academic circles. Garfield (1998)  In order to carry out the aforementioned survey of perceptions, an on-line questionnaire was made available at the authors' website. This questionnaire included Likert format (with 1, never, and 5, always) questions, as well as open ones. These addressed the following issues: I1) the use of scientific journal rankings in a systematic way by scientific researchers to identify which articles to adopt in the preparation of scientific articles; I2) the potential benefit for scientific researchers from increasing their use of rankings to determine the scientific journals to be adopted as theoretical references for research; I3) the use of rankings by scientific researchers to determine where to publish scientific articles; I4) the comparison between factors that are either for or against the use of rankings; and I5) the use of rankings by scientific researchers to assess scientific journals.
Its content and functionality were preliminary tested. To this end, the questionnaire was sent by email to 24 doctoral students in Business Administration at one of the universities taking part in the research, asking them to evaluate it in terms of its reliability as a data capturing instrument. Twentythree replies were received with suggestions for possible improvements.
After refining the questionnaire, on the basis of the suggestions made regarding the scope of the test, a total of 236 e-mails were sent out to all doctoral-level professors on the staff of all Brazilian universities with stricto sensu PhD-level postgraduate courses in Business Administration, as certified by the CAPES in May 2005 (total of 12 courses).
Out of a universe of 236 professors, we received 76 fully-replied questionnaires from professors at all 12 postgraduate courses in Business Administration. We also received 31 partially-replied questionnaires that we decided to discard. Indeed, those which were fully-replied constituted an adequate sample in that they enabled us to perform a statistical generalization of results with a confidence level of 95%, for a confidence interval of ± 10%, according to the Rea and Parker (2000) formula for calculating the size of samples for small populations. The questionnaires were analyzed using the Kruskal-Wallis H-test to verify the existence of differences between groups of professors, defined according to their profile: university, level of education, number of years since obtaining their doctorate, number of articles published and number of theses and dissertations supervised. Content analysis was also carried out in the case of the answers to the open questions.
The respondents' profiles are presented in Tables 4 to 9. Note that the averages presented in Tables 5, 6, 7, 8 and 9 were calculated using the raw data instead of the frequency of each data range presented in these table's first column. In Table 4 the universities that took part in the research are listed, along with the percentages of respondents per university. Over two-thirds (68%) of the respondents had their doctorate, one-quarter had a post-doctorate and 7% did not inform their highest level of education.   Table 5 gives the number of years since professors obtained their PhD and the number of professors per period of time. Table 6 details the number of years that professors have been giving stricto sensu postgraduate courses. Table 7 shows the number of articles published, stratified by quantity of articles and numbers of professors during the respondents' academic careers.      On the basis of this study, we found that most doctoral level professors in Brazil use multiple means and sources of investigation for their bibliographical research, according to the averages presented in Table 10, from data collected using a 5-point Likert scale question. However, the use of university libraries and the consultation of published rankings are exceptions.  The respondents highlighted other kinds of bibliographical research that were not included among the options given in the questionnaire, notably, the following: (1) articles previously written by the author himself; (2) consulting other researchers; (3) technical bulletins; (4) classic books; (5) supervised dissertations.
Various reasons were given for using rankings, as can be seen in Table 11. The main one cited by respondents referred to the assessment of the quality of scientific journals. The respondents added other factors that favored the use of rankings, as follows: (1) helps to identify a diligent editorial body; (2) enables the credibility of the academic institution to be associated with the reputation of the researcher; (3) reveals the best quality scientific journals over time; and (4) encourages improvements in the quality of scientific journals. The least important aspect cited was assessing productivity. All the professors that took part in the survey could choose multiple answers to the questions related to Tables 11 and 12.  Creation of anxiety in academic circles 33% The age of the scientific journal is not analyzed 29% Error in the interpretation of the impact factor given that it is a statistical tool 26% Short term (< 2 years) and number of authors in the assessment of citations 18% Need to write in English for greater visibility 15% The replies in this item of the survey were homogeneous, with no significant differences in averages. The reason most often cited was the existence of self-citation, and the least, the need to write in English for greater visibility.
It is important to note, however, that hardly any respondents cited the rankings that they were using. A majority of those that that did so cited the CAPES' Qualis and only one mentioned the ISI. The results of the Kruskal-Wallis H-test, summarized in Table 13, show that there are differences in the averages of the answers (Tables 10 to 12) of the groups given in Tables 4 to 9, to a significance level of 95%, with the exception of the following two factors which did not present any differences: (1) approximate number of dissertations supervised (Table 9) and (2) number of articles published per professor (Table 7).
The results of the Kruskal-Wallis H-test (to compare the groups -Tables 4 to 9) summarized in Table 13, show that there were differences in the averages of the answers (Tables 10 to 12).
The null hypothesis for this test is that each group (Tables 4 to 9) is identical across the sets of answers -Bibliographical Research (Table 10), Factors in Favor of the Use of Rankings (Table 11) and Factors against the Use of Rankings (Table 12) -relative to the alternative hypothesis that the answers differ between groups.
Overall, the Kruskal-Wallis H statistical test showed that at least one of the answers in each set is different from at least one of the other answers (significant level of 95%), except for two groups which did not present any differences: (1) approximate number of dissertations supervised (Table 9) and (2) number of articles published per teacher (Table 7).
From the groups that did not confirm the null hypothesis (with answers different from the others) six questions were significant at 5%: (1) I verify the bibliographical references of seminal works; (2) I use university library services; (3) Determining where to research; (4) Need to write in English for greater visibility; (5) Influence of authors' reputations; and (6) Creation of anxiety in academic circles (Table  13).   Table 10 Review of Literature Table 11 In favor  Despite the fact that some answers did not confirm the null hypothesis, only two groups presented two answers that confirmed the null hypothesis: (1) Number of years since obtaining the doctorate (Table 5) and (2) Number of years as a teacher at the stricto sensu postgraduate level (Table 6). These results suggest that the number of years as a doctor and as a teacher in post graduate level impacts the perception of the use of rankings.
Thus, the great majority of respondents (74%) replied that the use of rankings should be encouraged. All respondents that disagreed with this viewpoint (26%) had taught at a stricto sensu postgraduate level for less than 16 years and half (50%) of the respondents had been teachers for under 10 years.

Proposal of a Bibliographical Research Method Proposal of a Bibliographical Research Method Proposal of a Bibliographical Research Method Proposal of a Bibliographical Research Method
In terms of helping to render literature review processes more systematic, we propose a three-stage bibliographical research method that makes use of rankings and draws on Arenas et al. (2001). The proposed method adds effectiveness and rigor to the literature review process because it defines how this process should start and evolve. There are research themes, especially those at a doctoral level, that address theoretical gaps as well as interdisciplinary issues. The use of a systematic method in this context can help researchers prepare a comprehensive literature review in less time than usual.
In comparison to that of Arenas et al. (2001), the proposed method borrows the idea of using rankings in its first stage. In the other two stages, it adds (a) authors' work search, (b) keyword search, and (c) two cycles for keywords and relevant reference feedback in order to guide the researchers to carry out specific steps that can help ensure a more thorough and efficient investigation. As shown in Figure 1, it includes three stages: (1) selection of initial sources; (2)  In the first stage, rankings are used to select the initial sources of information to be analyzed. The use of multiple rankings related to the research area, with the definition of a criterion of consolidation between them, is recommended. Some additional sources of information can be added during this stage, notwithstanding their position (or even citation) in the consolidated ranking.
The second stage involves selecting the first set of documents (articles, reports, books, etc.), based on the results of the bibliographical research of initial sources, the production of the main authors of the pertinent field of knowledge and the use of keywords in search mechanisms available on the Internet and in other sources. We would like to emphasize the importance of first establishing the period covered by the bibliographical research for example, the last five years. As is obvious, by covering a longer period of time, a more extensive research will be carried out. However, seminal works should always be considered. The number of initial sources and the a priori defined period of coverage can lead to a large set of documents because all the articles published in these sources and period should be considered. The first activity of this stage consists of assessing the titles of the articles, due to the large set of selected documents. It is assumed that most of the articles do not have misleading titles. If the articles are identified as important, their abstracts and keywords should be assessed; if not, the article should not be considered. If the article's importance cannot be determined by its title, its abstracts and keywords should be assessed as well. In the final activity of this stage, the The use of rankings increases the chances of a thorough bibliographical research. Due to the always present time constraints to conduct master and doctoral level investigations, the selection of an appropriate set of journals as a starting point of a bibliographical research is essential for this task. On the other hand, rankings of scientific journals are generally not used systematically by scientific researchers to identify scientific journals to be adopted in the preparation of scientific articles (issue I 1 ). The great majority of respondents resort to multiple ways of researching scientific articles.
It is worth pointing out that, although the answers of a great majority of professors that took part in the study suggested that they consulted published rankings, only a very small number actually cited a particular ranking and, of these, all but three indicated only one: the CAPES Qualis. Indeed, this led us to believe that the average of 2.6 (Table 10) resulting from the question I consult published rankings was in fact lower.
What is encouraging, however, is that the study revealed that a great majority (74%) would like to increase their use of rankings (issue I 2 ). The results indicated that doctoral-level professors in Brazil were aware of the fact that researchers would benefit from a wider use of rankings to determine which scientific journals should be included in bibliographical reviews. Therefore, it is relevant that the number of articles published on rankings of scientific journals has increased in recent years. Other forms of bibliographical research, specifically literature reviews, are perceived to be time-consuming, mainly because they involve consulting books, dissertations, theses, and other researchers that are not always available to most researchers. These findings could indicate the need and willingness for more widespread use of systematic bibliographical research methods, such as the one proposed in Section Survey's Method.
Although most averages amongst the set of variables that favor the use of rankings were significantly higher than in the set of variables that are opposed to the use of rankings (issue I 4 ), when the Kruskal-Wallis H-test was applied, we found that both sets had some differences. However, these were significant (5%) in only 4 questions out of a total of 19, suggesting that respondents have a homogeneous view of rankings (Table 13).
According to just over half (51%) of the professors that participated in the study, rankings were being used by scientific researchers as indicators of where to publish scientific articles (issue I 3 ). The assessment of the quality of a scientific journal (72%), determining where to research (44%), recognition of merit (43%) and the assessment of scientific research (40%), amongst other factors identified by Peffers and Ya (2003), were considered to be those that most contribute to the use of rankings for making a choice on where to publish.
Overall, the results of our research confirmed that rankings, when used by scientific researchers, were being adopted mainly to assess the quality of scientific journals (issue I 5 ). Indeed, we found that this was the most cited factor in the literature we reviewed (Baden-Fuller et al., 2000;Campos, 2003;CAPES, 2005;Dubois & Reeb, 2000, 2001ERIM, 2003;Financial Times, 2003;Geary et al., 2004;Holsapple et al., 1994;Krzyzanowski & Ferreira, 1998;Mylonopoulos & Theoharakis, 2001;Peffers & Ya, 2003;Podsakoff et al., 2005;Rainer & Miller, 2005;Saha et al., 2003;Tahai & Meyer, 1999;Vanti, 2002;Yamamoto et al., 1999), despite the criticisms regarding this practice on the part of some authors (Campos, 2003;Coelho et al., 2003;Coura & Willcox, 2003;Uncles, 2004). On the other hand, as observed by Saha et al. (2003), few authors have undertaken empirical research in order to analyze whether or not the impact factor measures accurately the quality of scientific journals. According to Campos (2003), oscillations in the number of journals published may have a negative effect on this assessment. Some other results are important, although they are not directly related to the survey's questions. First, we verified an average of 3.3 (Table 10) for the use of university libraries. This rather low average compared to the others is hardly surprising, given the growing use of Web search mechanisms such as Proquest, Ebsco, Google Scholar and Science Direct, and a consequent decline in more traditional forms of research. Secondly, we noted a high percentage of affirmative replies to the question regarding the influence of the authors' reputations, shown in Table 12. We believe that this percentage should be in fact lower, considering that high quality journals choose articles by blind review, i.e., without the prior knowledge of the article's author(s). On the other hand, it appears that most researchers do not share this perception.
We also verified that only 25 out of 76 (33%) doctoral level professors in Brazil think that the need to write in English constitutes a barrier (see Table 12). We expected that this question would receive a more negative assessment in Brazil, where English is not the official language, considering that articles published in other languages have fewer opportunities to be assessed internationally, as emphasized by Baden-Fuller et al. (2000), Campos (2003), Garfield (1998) and Uncles (2004).
Our study made it evident that the analysis of citations is generally believed to provide an objective measure of the qualities and impact of a journal. Time is a scarce resource for those in academia. Keeping up to date with the latest theoretical developments is extremely difficult, given the high number of publications. Rankings are considered useful because they focus on journals that contain the most recent research, and, therefore, on those that have a greater impact. At the same time, however, our investigation revealed that, to a certain extent, researchers are wary of using rankings to appraise a faculty's performance. Measuring the importance of the teaching staff, assessing performance and assessing productivity were the factors that obtained the lowest averages in the survey (Table 11). Assessing productivity was viewed as a quantitative criterion that hardly takes content into account.
Our study thus confirmed our belief that the use of rankings is indeed desirable in a bibliographical research process that underpins academic production. Some authors (Magarey, 2001;Rosenthal & Dimatteo, 2001) propose systematic research methods. However, they do not describe how to systematically perform bibliographical research.
In order to avoid being caught up in the vicious circle pointed out by some authors, we recommend the adoption of a systematic bibliographical research method that incorporates the use of rankings, not as a sole reference, but integrated into a broader non-deterministic process, that allows using various sources of important information. In other words, we are convinced that is important to use rankings, but not in an isolated fashion. Our proposed bibliographical research method was developed to address (a) the lack of systematic bibliographical research in the "traditional method" of theoretical reference building by master and doctoral students, (b) the huge amount of information sources one has to investigate to ensure some degree of thoroughness in bibliographical research and (c) the availability of search mechanisms, acknowledging the different keyword coding schemes.

C C C CONCLUSION ONCLUSION ONCLUSION ONCLUSION
In this article we proposed a method for carrying out systematic bibliographical reviews. To highlight its relevance, we also presented the results of a pioneering empirical survey regarding the perceptions of doctoral level professors of Business Administration courses at universities in Brazil on the use and importance of scientific journal rankings for bibliographical research, namely literature reviews. The results suggested that there is a great deal of awareness of the importance of ranking as a criterion for bibliographical research in identifying the state of the art and pertinent research gaps, as well as establishing solid theoretical foundations for field research.
At the same time, however, from the investigation's results it was evident that journal ranking is hardly used by doctoral level professors in Brazil on a truly regular basis. To help overcome this problem, we proposed the three-stage method for bibliographical research, the first stage of which includes the practice of scientific journal rankings as a starting point. To ensure effectiveness and rigor, we emphasized the use of multiple rankings that take into account both quantitative and qualitative measurement dimensions in conjunction with other sources of information. The cycle concept that we introduced in our proposed method stresses the need for constant feedback throughout the bibliographical research process. We believe such interactivity is important especially in Master's thesis and doctoral dissertation research that require continual assessment of the originality and innovativeness of their findings.
To conclude, we suggest extending this investigation to other countries, as well as to different fields of knowledge, and applying the proposed method, all the while making the necessary adjustments and seeking its continual improvement.