Academic rankings: an approach to rank portuguese universities

Rankings académicos: un abordaje para clasificar las universidades portuguesas

Rankings acadêmicos: uma abordagem ao ranking das universidades portuguesas

Academic rankings: an approach to rank portuguese universities

Rankings acadêmicos: uma abordagem ao ranking das universidades portuguesas

Rankings académicos: un abordaje para clasificar las universidades portuguesas

Pedro Bernardino^I; Rui Cunha Marques^II

^ICentre of Urban and Regional Systems, DECivil-IST; Technical University of Lisbon. E-mail: [e-mail]

^IICentre of Urban and Regional Systems, DECivil-IST, Technical University of Lisbon. E-mail: [e-mail]

ABSTRACT

The academic rankings are a controversial subject in higher education. However, despite all the criticism, academic rankings are here to stay and more and more different stakeholders use rankings to obtain information about the institutions' performance. The two most well-known rankings, The Times and the Shanghai Jiao Tong University rankings have different methodologies. The Times ranking is based on peer review, whereas the Shanghai ranking has only quantitative indicators and is mainly based on research outputs. In Germany, the CHE ranking uses a different methodology from the traditional rankings, allowing the users to choose criteria and weights. The Portuguese higher education institutions are performing below their European peers, and the Government believes that an academic ranking could improve both performance and competitiveness between institutions. The purpose of this paper is to analyse the advantages and problems of academic rankings and provide guidance to a new Portuguese ranking.

Keywords: Academic rankings. CHE. Higher education. Performance evaluation. THES.

RESUMO

Os rankings acadêmicos constituem um assunto controverso no ensino superior. No entanto, apesar de todas as críticas, os rankings parecem estar aqui para ficar e cada vez mais diferentes intervenientes os utilizam para obter informações sobre o desempenho das instituições de ensino. Os dois rankings mais conhecidos, o do The Times e o da Universidade JIAO Tong de Xangai, apresentam metodologias diferentes. No ranking apresentado pelo The Times a classificação é baseada em análise pelos pares, enquanto o de Xangai tem apenas indicadores quantitativos e baseia-se principalmente em resultados de investigação. Na Alemanha, o ranking CHE usa uma metodologia diferente dos rankings tradicionais, permitindo aos utilizadores escolher os critérios e os pesos. As instituições de ensino superior portuguesas apresentam um desempenho inferior aos seus pares europeus. O Governo acredita que um ranking acadêmico poderia melhorar tanto o desempenho como a competitividade entre as instituições. O objetivo deste trabalho é analisar as vantagens e os problemas dos rankings acadêmicos e fornecer orientações para um novo ranking português.

Palavras-chave:Rankings acadêmicos. CHE. Ensino superior. Avaliação de desempenho. Portugal. Xangai. THES.

RESUMEN

Los rankings académicos son un tema muy contradictorio en la enseñanza superior. Todavía, además de todas las críticas los rankings están para quedarse entre nosotros. Y cada vez más, diferentes stakeholders utilizan los rankings para obtener información sobre el desempeño de las instituciones. Dos de los rankings más conocidos, el The Times y el ranking de la universidad de Shangai Jiao Tong tienen métodos distintos. El The Times se basa en la opinión de expertos mientras el ranking de la universidad de Shangai presenta solamente indicadores cuantitativos y mayoritariamente basados en los resultados de actividades de investigación. En Alemania el ranking CHE usa un método distinto permitiendo al utilizador elegir los criterios y su importancia. Las instituciones de enseñanza superior portuguesas tienen un desempeño abajo de las europeas y el gobierno cree que un ranking académico podría contribuir para mejorar su desempeño y aumentar su competitividad. El objetivo de este artículo es analizar las ventajas e inconvenientes de los rankings académicos y contribuir para el nuevo ranking Portugués.

Palabras clave: Rankings académicos. CHE. Enseñanza superior. Evaluación del desempeño. Portugal. Shangai. THES.

Introduction

The use of rankings in the higher education sector does not generate consensus and leads to many discussions about the benefits and the costs of their use (MARGINSON; VAN DER WENDE, 2007). However, it is almost impossible to talk about the evaluation of higher education institutions, and consequently their comparison, without referring to academic rankings. Generally, institutions do not like to be ranked or compared with others, however, rankings are unavoidable. It is very interesting to observe that despite the criticism that academic rankings face, the main goal of several universities is to be at the top of an academic ranking. This happens because the world class rankings are very popular in this sector, give good reputation and free publicity to the institutions. A good example of this situation is the Victoria University of Wellington. When it went down several positions in the Asiaweek ranking, one of the top goals for the coming years of this university was to improve its position in the ranking (STELLA; WOODHOUSE, 2006). As rankings are gaining increasing attention and popularity, the stakeholders should seriously discuss two major related-issues on this subject "who should perform the higher education rankings?", and "is the information provided by academic rankings reliable?"

The Portuguese higher education sector is at this moment facing a set of new reforms, including the quality assurance system and the increase of the institutions' performance and competitiveness. In several international reviews and evaluations of the Portuguese higher education it was pointed out that the Portuguese institutions do not provide the minimum information about their performance to the stakeholders, and that they were not competitive enough. One of the measures taken by the Portuguese Government to oppose this situation was the creation of an academic ranking. Therefore, the main goal of this study is to analyse different ranking methodologies, and bring some conclusions and guidance for an approach to a Portuguese ranking. This research provides some contributions for the literature since ranking issues in higher education sector are usually associated with the countries which have the best universities of the world, like the USA or the UK and the example of Portugal might bring good lessons for other countries worldwide. This paper is organised as follows. After this brief introduction, section 2 discusses the costs and benefits of academic rankings. Section 3 analyses the most famous rankings, such as the Times Higher Education Supplement (THES) and the Shanghai rankings as well as the Centre for Higher Education Development (CHE) approach. Section 4 examines the importance of rankings in the Portuguese context and proposes a ranking for the Portuguese higher education institutions. Finally, section 5 presents the concluding remarks.

Costs and benefits of the academic rankings

Many critical aspects are pointed out to rankings by the universities. The main argument is that universities are different, so two different institutions with distinct goals and missions cannot be compared. Moreover, the universities are internally different, and it is very hard or even impossible to be measured as a whole (BOWDEN, 2000).

Academic rankings are frequently criticised due to the choice of performance indicators (MORRISON; MAGENNIS; CAREY, 1995). Very often the indicators of most academic rankings are based on what can be measured rather than on what is significant. In 2005 seven out of ten rankings did not include any indicator related to teaching quality because it is very hard to obtain an objective measure (MEREDITH, 2008). In contrast, the research activity has always a large influence on the rankings results. It is easily measured relatively to the teaching quality, for example, by the number of articles published in international peer review journals. Historically, the indicators that are chosen to make the ranking show, in most of the cases, a tendency to natural sciences.

One of the most complicated aspects of making an academic ranking is the danger that the results become a popularity contest rather than a serious discussion about which institutions are performing well. In general, this is a problem that occurs in rankings based on academic reputation and that use information from polls and surveys. As a consequence, this methodology leads to a distortion of the perception and to the halo-effect. For example, a US survey found the Princeton Law School ranked as seventh in the country, but Princeton did not have a law school (STELLA; WOODHOUSE, 2006).

The weights are also very controversial in higher education rankings. Some experts argue that the weights choice is sometimes subjective and arbitrary, lacking a theoretical basis. The major challenge is how to report results without assigning the weights (CLARK, 2002). The fact that the formula of computing a ranking generally changes every year is another negative aspect of the rankings (HARVEY, 2008). This problem is more often associated with the media rankings. If the formula is changed every year it means that the result, or the "quality" of the institution also changes significantly and it becomes very hard to interpret shifts in the performance of an institution. Similar conclusions can also be applied to the statistical data used or their validity. If the data adopted to perform the rankings are not uniform the results cannot be conclusive. Sometimes, the difference between the score of two institutions is statistically insignificant, but the methodology grades them in different positions (DILL; SOO, 2005). For example, the difference in quality between two institutions may be minimal and the gap between them might have several positions, which can be misleading.

In a seminal critical paper about rankings, Professor Lee Harvey (2008) states that the use of rankings in the higher education sector may lead to a focus of attention in some particular aspects of the institution's activity, or to some specific areas, and as a consequence decreases the institution's quality of teaching. Finally, one of the most criticised aspects about academic rankings is the unhealthy competition they produce between higher education institutions. The danger that the institution competes to improve its ranking position may lead to the decrease of cooperation with other institutions, with losses to students and to the higher education in general (DYKE, 2005).

Despite all the criticism and the negative aspects pointed to rankings, the academics still discuss their use and benefits. The truth is that all stakeholders use academic rankings and recognise that they can be very useful. Most institutions do not like to be ranked, but the reaction from the universities to rankings is mixed. Some support their use and others boycott them. Even if they are sometimes unhappy with the methodology used, the large majority of the institutions feel- compelled to contribute as the possibility of a good position in the ranking rewards them with a large amount of free advertising and may have relevant effects on the university's reputation.

The students are the first to use academic rankings. It is reasonable to expect that when a student chooses an institution he/she takes a decision based on many sources, and not only on academic rankings. A student's choice may depend on many different factors, such as the location, the sports facilities or the fees charged by the institution. But several studies indicate that one of the most common factors that lead to that choice is the university reputation (STELLA; WOODHOUSE, 2006). Since the students do not always have official data about the reputation, or the performance of the institutions, they use academic rankings as an answer to their questions.

Usually the employers' interests are similar to those of the students. In general the most important criteria for the recruitment choices are the personal qualities of the candidates, such as their communication skills or ability to teamwork. But the institution's prestige is frequently acknowledged and taken into account. A high position on the academic rankings and league tables is certainly prestigious to the institution, and the employers often use that information. However, governments and funding agencies rarely consider academic rankings. The quality assurance agencies normally are the source of the higher education data for the governments and in their decision of financing the institution more attention is generally given to the academic efforts. Nevertheless, some governments also use academic rankings. In the Philippines, for example, the government contacted the Asiaweek to obtain more information to take budget decisions (STELLA; WOODHOUSE, 2006).

The use of rankings in higher education is a very attractive and controversial subject, and the number of organisations that issue rankings is now considerable, as well as the number of experts on this theme. In 2004, in a meeting in Washington DC a group of experts founded the International Ranking Expert Group (IREG) with the purpose of advising the United Nations Educational Scientific and Cultural Organization – European Centre for Higher Education (UNESCO-CEPES) and the Institute for Higher Education Policy (IHEP) in this area. Two years later the IREG met again in Berlin and considered a set of principles of quality and good practice, called the Berlin Principles on Ranking of Higher Education Institutions.

The Berlin Principles introduced something new in the production of rankings. They were not focused on rankings problems, but rather on their benefits and included a set of recommendations about what should be done when an organisation produces a ranking. These recommendations were divided into four groups, respectively the purposes and goals of rankings, the design and weighting of the indicators, the collection and processing of data and finally the presentation of rankings. In short, the Berlin Principles assume that if the higher education rankings follow some basic recommendations and if they are correctly understood and interpreted they can produce many positive outcomes for the community (UNESCO-CEPES, 2006).

Later, the IREG on 18 April 2008 has decided to create the International Observatory on Academic Rankings and Excellence with the main purpose of establishing a collective representation of the ranking community, initiating research and training activities related to ranking and academic excellence and enhancing public awareness and understanding of academic work (HARVEY, 2008).

Although at the beginning academics were indifferent to rankings, especially to the media rankings, gradually they started to gain some respect from them and it is now acknowledged that they can bring some positive aspects to the higher education sector. The higher education rankings phenomenon appeared as a response to the increasing demand from the consumers to easily understand the information about the performance of higher education institutions (DICHEV, 2001). However, this may not be the main advantage that can come up from rankings. As generally recognised, the higher education is facing profound changes at this moment worldwide, but especially in Europe due to the Bologna Declaration. The global expansion of access to higher education and the emergence of an international market demand more from the higher education institutions, thus the rankings can bring something positive to the market competition among institutions.

The academic rankings can help to distinguish between different types of institutions and trigger a new market competition, within and across countries, for students and funding (HARVEY, 2008). The European Union, in the Lisbon Treaty, defines that governments need to invest more resources in research and development (R&D), 3% of the gross domestic product (GDP), and argues that university rankings can be useful to provide some of the rationale for allocations of funds. Therefore, it is better to give more money to successful institutions that guarantee a highly return on the investment.

As Marginson (2007) states, academic rankings have intensified the competition for the leading research, especially the youngest talents, all around the world. The high-performing researchers are being very well paid not only because they increase directly the incomes of the institutions, but also because increasing the research productivity improves the position of the institution in the most famous rankings, and consequently its prestige.

The most well-known rankings: the Shanghai and the THES rankings

As aforementioned, the higher education rankings play an important role in the quality assurance systems, and not always for the best reasons. Therefore, when we look at the rankings as a measure for the performance of higher education institutions we must consider the indicators and the weights that are used by those rankings. The most famous and respected higher education rankings are the Times Higher Education Supplement (THES) and the Shanghai ranking by Shanghai Jiao Tong University (SJTU).

THES (2007) is probably the most reputed media ranking in the world. The Times is also an extremely appreciated newspaper in the United Kingdom. This ranking is a composite indicator integrating peer review and opinion with quantitative data. The 2007 methodology is based on the fact that the experts' opinion is the most valid way to assess the standing top universities. The THES methodology encompasses two criteria based on peer review, being the most relevant the academic opinions that are worth 40% of the total score available (THES, 2007). The indicators used by the THES ranking methodology are given in Table 1.

The Shanghai ranking is compiled by the Shanghai Jiao Tong University's Institute of Higher Education, and its initial objective was to fill the qualitative gap between the Chinese universities and the world class universities. The Shanghai ranking is nowadays one of the most prestigious academic rankings in the world. The universities are ranked based on several indicators of academic and research performance. A weight is given to each indicator, the highest scoring institution is assigned a score of 100 and the other institutions are given a score as a percentage of the top score (SJTU, 2007). The indicators adopted by the Shanghai ranking are shown in Table 2.

It is interesting to notice that the two most well-known academic rankings in the world use distinct methodologies. The THES ranking gives an enormous importance to assessment based on the reputation of the universities by recognised researchers across the world, about 40%. Other indicator that is also obtained from survey by this ranking is the opinion of the employers. Therefore, half of the ranking weight is based on opinions. This evaluation methodology is highly criticised. Taylor and Braddock (2007) argue that opinions change according to the person's academic background, for example, researchers will a have sympathy for the universities of their areas. Conversely, the Shanghai ranking does not have any indicator measured by surveys. All the Shanghai indicators are quantitative measures rather than qualitative ones. With this methodology the ranking avoids errors associated with the qualitative indicators.

In the academic rankings the research outputs normally give an enormous contribution to the institutions final position. However, the THES ranking is an exception and the research outputs only contribute with 20% to the university final score, and are measured by the number of citations per full-time member of the staff. On the other hand, the Shanghai ranking is almost entirely based on the research performance of the universities, especially world class research. The Shanghai ranking weights the research outputs in 80%. The indicators rely on the number of Nobel Prizes, Field Medals, highly cited researchers and publications in the best journals in the world. However, the institution size is not considered in this evaluation and consequently the bigger institutions tend to perform better.

The quality of teaching is measured in the THES ranking by the staff to student ratio. Even though it is regarded as a good measure and very often used across the world, this indicator only measures the quality of teaching. In contrast, the Shanghai ranking measures the quality indirectly by the number of students winning Nobel Prizes and Field Medals. If a student wins a Nobel Prize it is not clear that is directly related to the quality of teaching. However, to measure the quality of teaching is very difficult. An important aspect considered by the THES ranking was the universities internationalisation, not only of their students but also of their staff.

In short, the two rankings follow very different methodologies. The Shanghai ranking is clearly based on research outputs, which is one of the essential functions of universities. In opposition, the THES ranking, which confers a great weight to peer review, includes more indicators in its methodology.

As noted earlier, rankings have both positive and negative aspects that advise consumers to look at them carefully. Nevertheless, this discussion is far from being over and the big question remains, "Are higher education rankings reliable?"

Looking at Table 3, we can observe the top 15 universities of the THES ranking and of the Shanghai ranking for the year 2007. These two renowned rankings use different criteria and different weights in their methodologies. Despite the obvious differences in the top 15 of both rankings, the first university is the same, Harvard University, and ten other universities repeat themselves in both top 15, which leads to some conclusions about their quality.

Nevertheless, analysing both rankings more thoroughly, we can see that the second and the third institutions of the Shanghai ranking do not appear in the top 15 of the THES ranking. This is probably the main discrepancy between both rankings. The Stanford University is only the 19^th, and the University of California Berkeley is the 22^nd in the THES ranking. Another institution that shows a significant difference between the two rankings is the Imperial College of London, number 5 in THES, but only 23 in the Shanghai ranking.

These differences can be explained by the criteria and weights used in the rankings methodology. The institutions with more foreign students and staff perform better in the THES ranking, and the smallest institutions tend to achieve worse results in the Shanghai ranking, because the outputs are measured in absolute terms and not per capita. As mentioned above, the choice of weights is very controversial, since the choice is itself a judgement of value and it differs according to the person that takes the decision. The institution that is performing the ranking needs to be very careful with the weights given to the criteria. The possibility that a statistical aspect dominates all the others is very real or, for example, several trivial dimensions prevail over more crucial ones.

The CHE ranking, a new approach

The German Centre for Higher Education Development is responsible for a new approach to the academic rankings, the CHE ranking (DAAD, 2007). It began by collecting data from all the higher education institutions in Germany and nowadays also includes Switzerland and Austria, whereas the Netherlands and Belgium are preparing to join and other countries are very interested, too. This ranking was created with the main objective of informing the students who want to enrol in higher education, or would like to change to another institution, and of providing information to all the stakeholders that want to know how the institutions are performing when compared to others. The CHE ranking is different from the other rankings because it does not have an overall point score. It justifies its choice arguing that there is no "best higher education institution" and "minimal differences produced by random fluctuations may be misinterpreted as real differences". Therefore, instead of the traditional rank positions, it was decided to offer a multidimensional ranking and always specifically about different fields of study. The final results are not presented in "league positions", but in league groups. At this point there are three different groups, respectively the top group, the middle group and the bottom group. This approach ensures that the top and the bottom groups deviate to a statistical significant extent from the arithmetic mean (DAAD, 2007). The indicators used in the CHE ranking methodology are displayed in Table 4.

In addition, CHE allows the students to choose the criteria they consider more important, and to decide the weights to each one of them. Therefore, it enables the stakeholders to compare the higher education institution according to their own interests. CHE does that through a web-enabled interactive database. The way in which the information is presented to consumers can be a helpful mechanism to decrease the ranking errors. That is why the CHE ranking has decided not to create league tables but to differentiate the results according to the criteria used. Like this, the universities can still be compared with each other but there is no rank position to avoid wrong conclusions.

Towards to a Portuguese ranking approach

The Portuguese higher education sector is going through a transition phase and it is facing deep changes in the organisation system. The Bologna Declaration, signed by the majority of the European countries in 1999, is one of the main reasons for this changing process. However, the main discussion issues do not end in the process of Bologna, and the institutions are facing many pressures from all the stakeholders. The students are paying high tuition fees, and as a result they are demanding more from the institutions. It is very reasonable to say that they are starting to behave more like clients rather than like students. The employers argue that they pay their taxes, and consequently demand more quality not only from the graduates, but also from the R&D activities without the support of the higher education institutions. On the other hand, the Portuguese government has been cutting public funds over the past years, which has led to an increase of competition for the resources available between institutions and has also demanded more accountability from them. Furthermore, the government has also felt the need of collecting information about the institutions' performance to know where to allocate better the public funds. The pressure of the international league tables has also increased the competition between higher education institutions, which feel compelled to perform better than their peers in a very competitive environment.

Therefore, when the Portuguese XVII constitutional government took over the country, the challenge it had to face was enormous, but it was also a unique opportunity to reform, restructure, and improve the entire higher education sector. It was in this context that the Portuguese government on November 21, 2005 requested an international evaluation of the Portuguese higher education. The aim was evaluating globally the Portuguese higher education system and its policies and also the current processes of accreditation, evaluation, and quality assurance. For that reason, the government aspired for an independent assessment not only of the higher education, but also of the organisation that evaluates the institutions, the Portuguese National Council for the Evaluation of Higher Education (CNAVES). These group of evaluations occurred independently from the national evaluations in progress. The international evaluation had the contribution of the Organisation for Economic Co-operation and Development (OECD), the European Association for Quality Assurance in Higher Education (ENQA), the European University Association (EUA), and finally of the European Association of Institutions in Higher Education (EURASHE). All these institutions have unique competences in the evaluation of higher education systems and institutions, which enables policy makers to launch new ideas to the Portuguese system with the main purpose of improving and modernising it. This group of studies made an evaluation of the Portuguese higher education system and this was crucial for the sector's future.

The OECD (2006) report about the tertiary education in Portugal reveals that the higher education sector is not performing according to the required levels, and that the poor performance of the Portuguese economy is directly related to the failure of the policies to increase the population education levels. In addition, the report also states that the relationship between the Portuguese industry and the higher education institutions is very weak. A large number of Portuguese companies still remain specialised in the production of labour-intensive and low value-added traditional goods but these markets are lost to new EU countries, Africa, or Asia. It is also clear that the productivity of the Portuguese workers is very low, half of the United States of America, and with high levels of labour force participation of the population with lower levels of education. The numbers are easily explained by the difficulty of the Portuguese companies to adapt themselves to new technologies, to innovate, and trade their products in other markets. Most experts defend that the only way for the Portuguese companies to increase their levels of technology and innovation is with higher levels of education. Portugal has an urgent need to invest more in human resources, and consequently in the higher education sector. The recommendations of the OECD (2006) report included a profound reform of the entire system, with the main purpose of increasing the autonomy, accountability, performance and competitiveness of the Portuguese higher education institutions.

The ENQA (2006) report about the Portuguese quality assurance system also defended profound transformations and reforms in the evaluation processes. The ENQA panel reported that the Portuguese system had not only a limited independence in the evaluation process, but also that it suffered from levels of inefficiency and inconsistency and the total lack of consequences or follow-up procedures. Therefore, the main conclusion was that Portugal urgently needs to create a new accreditation system based on the establishment of a strong and independent national agency, with the power of evaluation and accreditation.

The conclusions and the recommendations of the international evaluations of the Portuguese higher education were the starting point for the institutional reform of the higher education sector conducted by the Portuguese government. In November 2007 the government published a new law creating the new Evaluation and Accreditation Agency for Higher Education. At the time, the law caused some discussion within the Portuguese society, especially due to the loss of accreditation power of some professional associations. However, earlier in August the government had already published a law (PORTUGAL, 2007) defining the new evaluation criteria of the higher education institutions to be carried out by the agency. In that law there is a specific point, the article 22, which states that the results of the evaluation could be present in the form of rankings. Although academic rankings are always associated with some errors, they can provide useful information about the higher education institutions' performance. An academic ranking could have a positive effect on the Portuguese higher education sector, not only because it would provide more information to the stakeholders, but also because it could increase the competition between the institutions. Besides, the discussion and the controversy engendered about the academic rankings of universities could have a positive impact in the performance of the whole sector since, as it is known, the 'name and shame' policies often have a positive effect in the public institutions.

From the different approaches used by the THES (2007) and the Shanghai rankings both have strengths and weaknesses. However, they can be very useful to a new approach to rank the Portuguese higher education institutions, bearing in mind the Portuguese context. It seems that the Shanghai ranking is more reliable than the THES ranking, because it is not based on peer review, generally associated with some errors. It is important that the future Portuguese model avoids peer review, and the methodology should be rooted only in reliable indicators.

R&D activities are always crucial to measure or evaluate the universities' performance. However, they are not the only objective of an institution, and other indicators must contribute to the ranking's position weight. It is obvious that the Shanghai ranking attributes an excessive weight to the research. The indicators to evaluate the R&D activities in the Shanghai ranking are meant to evaluate the world class research, and are not the most appropriate to evaluate the Portuguese universities. Therefore, a Portuguese ranking must always include research indicators, but not Nobel Prizes or Field Medals. The R&D activities must be evaluated based on the publications in international peer review journals. However, for the system to be fairer a way to weight the different articles must be found, like for example the impact factors of the journals. In addition, the Shanghai ranking measures the research outputs in absolute terms and does not give enough attention to the per capita output. In the Portuguese approach, it is important to measure the research outputs per capita, otherwise it is not a fair ranking, because the larger institutions will always perform better.

A great advantage of the THES ranking compared with the Shanghai ranking is that it takes into account the quality of teaching, which is also an important aspect of the universities' mission. THES evaluates the quality of teaching using the ratio student/professor, because normally lower average class sizes provide better the relationships with the professors and the quality of teaching. However, it is not enough to have just one indicator to measure the quality of teaching, thus more indicators must be used to the Portuguese ranking methodology. Therefore, the students' drop out rates should also be included to evaluate the quality of teaching.

The Shanghai ranking has a great disadvantage when compared with THES. It does not consider the internationalisation of the institutions, respectively their students and staff, and does not include the "opinion" of the labour market. In the Portuguese higher education context both indicators are important, and must be included in the ranking methodology. In an environment of globalisation the Portuguese universities need to be able to attract the best students and staff not only from Portugal, but especially abroad. The high level of unemployment and the low education levels in Portugal are the top worries of the population, and the role of the higher education institutions to produce highly qualified graduates and increase the country's economy is more important than ever. Hence, the labour market must always be included in the final score of a university, but unlike the THES ranking, the indicator should not be based on surveys. The indicators chosen to evaluate the labour market for the graduates of a university are the employment rates and the average first salary.

Nowadays, the Portuguese higher education institutions are facing many financial problems, especially due to the decrease of public funds. The government expects that the universities diversify their source of incomes, by providing more services to the community, like consultancy works or training for the companies, and increase the R&D activities. In this context it is important to make sure that the institutions are achieving this objective. Therefore, the ranking methodology should have an additional indicator, the percentage of incomes from private sources. Table 5 describes the performance indicators proposed for the Portuguese higher education ranking.

The discussion of the creation of a ranking to the Portuguese higher education institutions does not end in the definition of indicators. There are three main issues that must be clarified.

The first one is related to the attribution of weights and the presentation of results. The definition of the weight is by itself a judgment of value that differs according to whom is looking at the institutions. Therefore, the CHE methodology is a good alternative to traditional rankings. The government could define the weights for the ranking indicators, but it should always allow the stakeholders to weight themselves the indicators and take their own conclusions about the institutions' performance. The presentation of the institution's position is also important, and the Portuguese government may opt for a "league position" or for different groups, like the CHE ranking. Because the Portuguese higher education sector is relatively small, to rank the institutions by different groups does not make much sense, and the presentation by "league position" could be more positive for the institutions' competitiveness. As the higher education institutions are very different from each other, it is recommended that the results should be presented by different areas of study, or even more specifically by department or course.

The second issue is related to the question: "Who should perform the Portuguese ranking?" The answer to this question is neither easy nor consensual. However, it is the Portuguese law that states that the institutions should be ranked, therefore the entity that should perform the ranking must be public and above any suspicion (BERNARDINO; MARQUES, 2009). At this moment, the Portuguese government has reformed the quality assurance system, and a new agency will have all the powers of evaluation and accreditation of the Portuguese higher education institutions. The new agency is one alternative to perform the ranking, but in our opinion there is a conflict of interests, so a different public entity should perform the ranking, in cooperation with the new agency. Our suggestion is that the National Statistics Institute (INE) would be the most adequate entity to perform the Portuguese ranking. The INE is nationally recognised for producing valid, reliable and quality statistical information about the performance of different sectors of the Portuguese society. The INE procedures are consistent with the international standards of quality, and the institute is acknowledged both for its transparency and independence.

The third issue concerns the way in which the data can be gathered for the performance indicators and therefore for computing the ranking. Indeed, some of the performance indicators, such as the employment rate or the average first salary might not be easy to handle. However, we believe that the university should keep closer to the career of their students and should maintain an updated database of the path their ex-students follow one or two years after they leave the university. Only this way can universities understand and adapt themselves to the needs of the market and of their students. The effort of collecting data and keeping the database is rewarding and quickly recovered.

Concluding remarks

The academic rankings are very controversial, partially because there is no consensus about what is the quality of a higher education institution. There is also no agreement about whether the quality of a school or programme can be quantified. Nonetheless, the rankings are here to stay and the lack of other public attractive methods to compare academic institutions will include them in the educational landscape for many years or maybe forever.

It will be very complicated to develop a universal ranking methodology able to provide reliable and international comparable performance data about the higher education institutions. However, if the ranking makers follow some basic recommendations both the information and the conclusions will be more reliable. The selection of the indicators and the assignment of weights are one of the most important aspects of the ranking methodology. They should have scientific basis, reliability and validity. The collection of the data must be coherent, public and transparent to avoid statistical errors. The weights assignment should appear from the conclusions of extensive analysis of the contribution of each indicator for the institution's performance. The higher education sector is always responding to the society needs, and the ranking systems should follow its changes. Therefore the evaluation procedures should be dynamic and able to change when the sector or society requires it, which includes the indicators and the weights.

Some specialists argue that rankings compare institutions that cannot be comparable, because they have different missions and contexts. Thus, every ranking system should state clearly the objective of the measurement, and identify precisely the audience addressed. This way we can improve some approaches to evaluate the institutions' performances, which sometimes are inadequate and, with some changes, transform them into useful tools for the students and other stakeholders.

An academic ranking would be important to boost competition among the Portuguese higher education institutions. It is essential that the indicators used by the Portuguese ranking reflect the requirements of the higher education institutions stakeholders and especially that they produce valid information about the institutions' performance in order to improve it continually.

Recebido em: 20/07/2009

Aceito para publicação em: 05/12/2009

• BERNARDINO, P.; MARQUES, R. Comparison of regulatory models of higher education in europe: the portuguese situation. Avaliação: revista da Avaliação da Educação Superior, Campinas, SP, v. 14, n. 1, p. 53-70, 2009.
• BOWDEN, R. Fantasy higher education: university and college league tables. Quality in Higher Education, London, v. 6, n. 1, p. 41-60, 2000.
• CLARK, M. Some guidelines for academic quality rankings. Higher Education in Europe, London, v. 27, n. 4, p. 443-459, 2002.
• DAAD. German Centre for Higher Education Development. CHE University Ranking 2009/10: your gateway to the university ranking. Berlin, 2007. Available at: <http://www.daad.de/deutschland/ hochschulen/hochschulranking/ 06543.en.html>. Accessed: 15 June 2008.
• DICHEV, I. News or noise?: estimating the noise in the U.S News University Rankings, Research in Higher Education, Dordrecht, Netherlands, v. 42, n. 3, p. 237-266, 2001.
• DILL, D.; SOO, M. Academic quality, league tables, and public policy: a cross-national analysis of university ranking systems. Higher Education, Dordrecht, Netherlands, v. 49, n. 4, p. 495-533, 2005.
• DYKE, V. Twenty years of university report cards. Higher Education in Europe, London, v. 30, n. 2, p. 103-125, 2005.
• ENQA. Quality assurance of higher education in Portugal. Helsinki, Finland, 2006.
• HARVEY, L. Rankings of Higher Education Institutions: A Critical Review. Quality in Higher Education, London, v. 14, n. 3, p. 187-208, 2008.
• MARGINSON, S. Global university rankings: implications in general and for Australia. Journal of Higher Education Policy and Management, London, v. 29, n. 2, p. 131-142, 2007.
• MARGINSON, S.; VAN DER WENDE, M. To rank or to be ranked: the impact of global rankings in higher education. Journal of Studies in International Education, Amsterdam, v. 11, n. 3/4, p. 306-329, 2007.
• MEREDITH, M. Why do universities compete in the ratings game?: an empirical analysis of the effects of the U.S News and World Report College Rankings. Research in Higher Education, Dordrecht, v. 45, n. 5, p. 443-461, 2008.
• MORRISON, H.; MAGENNIS, L.; CAREY, L. Performance indicators and league tables: a call for standards. Higher Education Quarterly, London, v. 49, n. 2, p. 128-145, 1995.
• OECD. Reviews of national policies for education: tertiary educational in Portugal. Paris, 2006.
• PORTUGAL. Law No. 38, of 16 August, 2007. Approves the legal framework for the evaluation of higher education. Legislation, Lisboa, PT, 2007. Available at: <http://www.mctes.pt/legislation/Law_38-2007.pdf>. Accessed: 22 February 2010.
• SJTU. Academic Ranking of World Universities 2007. Shanghai, 2007. Available at: <http://www.arwu.org/rank/2007/ranking2007.htm>. Accessed: 21 May 2008.
• STELLA, A.; WOODHOUSE, D. Ranking of higher education institutions. Melbourne: AUQA, 2006.
• TAYLOR, P.; BRADDOCK, R. International university ranking systems and the idea of university excellence. Journal of Higher Education Policy and Management, London, v. 29, n. 3, p. 245-260, 2007.
• THES. World University Rankings 200. London, 2007. Available at: <http://www.timeshighereducation.co.uk/ hybrid.asp?typeCode=142&pubCode=1&navcode=118>. Accessed: 23 May 2008.
• UNESCO-CEPES. Berlin principles on rankings of higher education institutions. Berlin, 2006.

Publication Dates

• Publication in this collection
  27 May 2010
• Date of issue
  Mar 2010

History

• Received
  20 July 2009
• Accepted
  05 Dec 2009