ENADE indicators for themanagement of university libraries

Indicadores do ENADE na gestão das bibliotecas universitárias

ABSTRACT

The functions of university libraries (UL) are to provide the bibliographic, documentary and informational infrastructure to support scientific activities, as well as to disseminate the knowledge produced by their academic entity to society. To achieve their mission, ULs need to be evaluated and, in Brazil, this evaluation is done by the National Student Performance Exam (ENADE - ExameNacional de Desempenho de Estudantes) through the Student Questionnaire (SQ). The present study evaluated the quality of the indicators used in the SQ for the evaluation of ULs. This quality was evaluated by considering the presence of the following essential properties for good indicators: stability, validity and importance of parameters. For the stability analysis, the SQ questions were repeated in the period from 2004 to 2014. Regarding the validity, a correlation analysis was performed between the results obtained in the SQ with another theoretically related indicator. For the analysis of the importance, a group of specialists was formed, composed of librarians with experience in the management of ULs and, through the Delphi method, indicators were evaluated as to their relevance. This work verified that the indicators used by ENADE are considered important, but they do not have stability and validity.

keywords:
Evaluation of Libraries; Construction of Indicators; Delphi method

RESUMO

As funções das bibliotecas universitárias (BUs) são de fornecer a infraestrutura bibliográfica, documentária e informacional para subsidiar as atividades científicas, além de divulgarem o conhecimento produzido por seu corpo acadêmico para a sociedade. Para alcançarem a sua missão as BUs precisam ser avaliadas e, no Brasil, essa avaliação é feita pelo Exame Nacional de Desempenho de Estudantes (ENADE) por meio do Questionário do Estudante (QE). O presente trabalho avaliou a qualidade dos indicadores utilizados no QE para a avaliação das BUs. Essa qualidade foi avaliada quanto à presença das seguintes propriedades essenciais para os bons indicadores: estabilidade, validade e importância dos parâmetros. Para a análise da estabilidade, foi analisada a repetição das perguntas do QE no período de 2004 a 2014. Quanto à validade, foi feita a análise de correlação entre os resultados obtidos no QE com outro indicador teoricamente relacionado. Já para a análise da importância, foi formado um grupo de especialistas, composto por bibliotecários com experiência na gestão de BUs e, por meio do método Delfos, foram avaliados os indicadores quanto a sua relevância. O trabalho constatou que os indicadores utilizados pelo ENADE são considerados importantes, porém não possuem estabilidade e validade.

Palavras-chave:
Avaliação de bibliotecas; Construção de Indicadores; Método Delphi

1 Introduction

Within the technical and administrative infrastructure that the university needs to achieve its objectives, there are libraries. These “aim to provide informational support, complementing the curricular activities of the courses, offering resources to facilitate scientific research” (DEUS, 2011, p.4). Directly linked to the mission of the university and based on the ‘teaching, research and extension’ tripod, libraries have the basic function of “providing the bibliographical, documentary and informational infrastructure to subsidize the scientific activities, as well as to disseminate the knowledge produced by the academic entity to the society” (DEUS, 2011, p. 5).

Given their importance, university libraries (ULs) are present in the evaluations that make up the National System for the Evaluation of Higher Education (SINAES - SistemaNacional de Avaliação da Educação Superior). Law Nº 10.861 of April 14, 2004 instituted this system and listed the dimensions that should be evaluated, among others, the physical infrastructure, the place where the libraries are located (BRASIL, 2004).

An integral part of SINAES, the National Student Performance Survey (ENADE) collects information on students' socioeconomic profile and their perception of the components of their training process, which includes the products and services offered by the ULs. The main objective of this survey is to provide information to assist directors and managers of Higher Education Institutions (HEIs) in decision-making.

The evaluations of SINAES and ENADE aim to verify the effectiveness of pedagogical and administrative actions, “since they contribute significantly to an internal reflection focused onimproving the quality of undergraduate education” (INEP,2013, p. 3). The importance of these evaluations for libraries is emphasized by Dziekaniak (2006), in a study which emphasizes that a university library can only achieve a high-quality standard in its products and services, besides providing adequate informational support, if it is effectively evaluated.

Per Almeida (2005), the numbers pointed out by evaluations are important, but they need to make sense and carry a meaning, which is guaranteed if the choices of what will be measured are properly made. In addition, “the construction of models for the evaluation of services provided by libraries is a dynamic process and must be constantly updated” (DUARTE; BARBOSA; COSTA,2013, p. 2).

It is in this context that the Central Library (CL) of the University of Brasilia (UnB), created in 1962 and with a collection of almost 1.5 million copies, is placed. In its 53-year history, the CL has made continuous efforts to maintain a diversified portfolio and modernize services. Its work aims not only to satisfy with quality, but also to comply with the vision of “being an academic library that is a reference in Brazil and Latin America following the standard of excellence in information and knowledge management.” (UnB, 2015).

This standard of excellence can only be achieved if the CL is aware of the information needed by its users. As stated by Almeida (2005, p. 13), “there must be a permanent attitude of inquiry and analysis, on the part of the librarian, regarding the real situation of the information unit.”

At this point, any evaluation becomes an instrument for the library to carry out the analysis of its actions and define the necessary improvements, enhancing its processes and developing best practices (DUARTE; BARBOSA; COSTA, 2013). However, for an assessment to reflect reality properly and to contribute to decision-making, it is important that indicators are correctly chosen and measured.

According to the ABMES (2017, p. 1), the ENADE indicators present limitations,highlighting the statement of the INEP/MEC coordinator: "these indicators are not comparable between their editions and require adjustment for the purposes of regulation, supervision and financing". In addition, specifically for the evaluation ofuniversity libraries, given the limited scope of the indicators used, the lack of university libraryspecialists is perceived in the definition of these indicators.

Thus, the present work aims to analyze the quality of the indicators used by ENADE for the evaluation of the products and services offered by university libraries (UL), focusing on the CL of UnB.

2 Theoretical reference

2.1 Evaluation of university libraries

The ULs play a role that goes far beyond simply keeping and organizing books, periodicals, and other documentary media. It is also not expected that these places must be just reading rooms or internet access locations. According to Gelfand (1969, p. 24), they must:

[...]feed the student's intellect, stimulate faculty research, and invite everyone under their roof to share their cultural food in full. In this context, the use of the library becomes a teaching method, occupying its place next to the expository class and the discussion groups.(GELFAND, 1969, p. 24).

Thus, they must actively attend to the teaching and research needs of the university community, while the librarian acts to guide students in the ways of research. The more efficient and effective the operation of its services is, the better the possibilities of supporting the educational programs and, consequently, the greater the contribution to the development of the HEI (DORIGON, 2006).

These units need to assume a strategic function within the institutions and need to be evaluated so that they can continuously improve and ensure the quality of the services provided (TUTIKIAN; SUÑÉ, 2011). Theirperformance is not only expressed by the number of books or loans made, above all it must:

[...] considering the functional and structural characteristics of the university library, the evaluation should take as reference its mission, goals, objectives, collection, personnel, physical area, services, accommodation and budget. Each of these variables should be evaluated on the basis of indicators that can measure whether the outcome of the action plan is compatible with institutional goals and proposals.(BEZERRA, 2010, p. 80).

This new form of operation changes the functioning of ULs, which “begin to adopt strategic planning, quality-oriented management models and evaluation as tools for control, improvement and decision-making” (LUBISCO, 2007, p. 46). The evaluation process is one of the essential elements to coordinate the execution of what is planned, and its main function “is to produce knowledge about the information unit, the organization in which it is located and its environment, to serve as a planning subsidy.” (ALMEIDA, 2005, p.11).

Equally important is the role of evaluation as a way of being accountable to the university and tosociety (PACIOS, 2011). Libraries, as well as other academic units, feel strongly pressured to demonstrate their contribution to the organizations to which they belong in quantitative form.(FAGAN, 2014).

Despite the importance of the theme, the use of models and various aspects related to the evaluation of ULs only began to be addressed in many countries after the 1990s.(BATISTA; SÁNCHEZ; RIVERO, 2012). One evaluation model that has stood out is the LibQUAL +. Developed by the Association of Research Libraries (ARL), the tool was created to measure the quality of service in academic libraries from the user's perspective and has been the predominant tool used in North America and the United Kingdom.(McCAFFREY, 2013).

Other forms of evaluation have emerged (TARANGO; OROZCO, 2009 andBEZERRA, 2010), each in its own way, and using different methods of evaluation, different criteria and indicators, in order to obtain a better understanding of the results. One of the last and more in-depth models arose precisely from the dissatisfaction with the indicators used by the Ministry of Education (MEC) for the evaluation of libraries (LUBISCO, 2007). The evaluation proposal was developed as a thesis, and its final product was presented at the Evaluation Seminar of the Brazilian University Library, where it gave rise to a new document based on the consolidation of the contributions of the working groups of the event.(LUBISCO; VIEIRA, 2009).

3 Evaluation by SINAES and ENADE

The State has also developed and applied its evaluation models, which has generated a legal obligation for ULs in Brazil. Since 2004, with the entry into force of Law 10.861, the National System for the Evaluation of Higher Education (SINAES) has been created, becoming the process that evaluates institutions, courses and students in Brazil (BRASIL, 2004).

One of the pillars of the SINAES is the National Examination of Student Performance (ENADE) which assesses undergraduate students' performances in relation to the program content through a test. In addition, each participant responds to the Student Questionnaire (SQ), with the purpose of being a tool that provides information to support management and contributes to the continuous improvement of the quality of education.(INEP, 2013).

Over the years, the SQ underwent several changes, the last one carried out in 2013 with the objective of adapting the instrument to current demands, in addition to improving its contribution as a quality indicator of Higher Education.(INEP, 2015). In the first part of the questionnaire, the student's socioeconomic profile is surveyed and, later, the SQ raises questions that address the formative processes throughout his or her academic life, among which are those that deal with the products and services of ULs.

4 Use of indicators for evaluation

It is not only through scientific necessity that the human being measures objects and events;on the contrary, to measure “is a routine activity of the human being and his interaction with nature.” (ROZADOS, 2005, p. 61). However, to perform any measurement, appropriate metrics and instruments are needed for each purpose. Among the various forms of measures, there are those whose objective is to describe and represent an event or phenomenon: these are indicators.(ROZADOS, 2005).

In the context of organization management, indicators are essential instruments in monitoring and evaluating activities, allowing goal achievements, progress achieved, quality improvements, and other aspects to be monitored. (BRASIL, 2010).

According to the type of measure they use, indicators can be classified as objective or subjective.(JANUZZI, 2005). While the former “refer to concrete occurrences or to empirical entities of social reality, constructed from available public statistics,” the latter “correspond to measures constructed from the evaluation of individuals or specialists regarding different aspects of reality, raised in public opinion polls or discussion groups” (JANUZZI, 2005, p.143). Thus, objective indicators usually end up being of a quantitative nature, while the subjective ones are qualitative (BRASIL, 2010).

In the set of subjective indicators, there are those that are used to measure the quality of a service according to the needs and satisfaction of the clients. Per Ferreira et al. (2014, p. 11), the quality indicators permit one to measure “the results of the work from the perspective of the users, allowing the perception of a given phenomenon or a condition in a simplified, understandable and comparable way”. In this category, the indicators used in the SQs of the ENADE are framed, since these instruments are based on the students' perception to measure, among other aspects, the quality of the products and services offered by the ULs.

There are many indicators available, so the process of choice should “seek the highest possible degree of adherence to some properties that characterize a good performance measure” (BRASIL, 2010, p.17). Januzzi (2002, p. 57) points out that “the selection of indicators is a delicate task, since there is no formal theory that can guide it with strict objectivity”. In addition, this selection should be guided by “a set of desirable properties and by the logic of the application that will define the typology of appropriate indicators” (JANUZZI, 2005, p.139).

Although the taxonomy related to these properties is still not well defined (MONTIBELLER FILHO, 2010), there are several characteristics that are pointed out in the literature as determinants of the quality of an indicator and can influence its use or not use, depending on the purpose established. Some of the most cited characteristics are presented in Chart 1.

Chart 1
Properties of the indicators

Comparability	“[...] indicators should be easily comparable with internal or external references and also historical events series” (BRASIL, 2009, p. 47). MontbiellerFilho (2010) considerscomparability an important feature since a set of indicators has the purpose of monitoring and evaluating processes, that is, something that occurs not only in a single moment, but over a period of time.
Reliability	“[...] indicators should come from reliable sources, using recognized and transparent methodologies for the archive, processing and dissemination” (BRASIL, 2012, p. 18).
Stability	“[...] ability to establish stable historical series that allow monitoring and comparisons” (BRASIL, 2010, p. 26). In order to be useful to management, it is necessary that the indicators “are normalized and that their historical production (temporality) always adheres to the same standard or form of measurement” (ROZADOS, 2005, p. 63). It is a desirable feature because it allows for inferring trends and evaluating the effects of implemented actions (JANUZZI, 2005).
Importanceorselectivity	“[...] provides information on the key strategic variables and priorities for actions, products or expected impacts” (BRASIL, 2009, p. 46). In addition, according to D'Este, Castro Martinez and MolasGallart (2009), indicators should be able to measure what is intended to be measured.
Validity	“[...] ability to represent, as closely as possible, the reality that is meant to be measured and modified. An indicator should be significant to what is being measured and maintain this significance over time “(BRASIL, 2010, p.25). It must be ensured, at least in theory, that there is “a reciprocal relationship between the indicating (concept) and the proposed indicators” (JANUZZI, 2002, p.57).

• Source: Authors.

As Fechine (2014, p. 53) points out, “despite attempts to demonstrate the important requirements for the ideal indicator, it is not so easy to find indicators that truly meet all these characteristics.” However, it is important to seek the use of indicators that have the greatest number of properties listed in Chart 1.

For the evaluation of libraries, ISO 11620 establishes 34 indicators, among them those related to the perception and satisfaction of users and similar to those proposed by ENADE (POMBAL;WILSON; AZEVEDO, 2009). In information units, satisfaction indicators would be related to aspects such as:

[...]opening schedules, number of places available for reading, research, consultation in computer terminals, document availability, loan services, interlibrary loan service, referral service, staff behavior, online catalog ergonomics, archive signaling or reference to a more global assessment. (ROZADOS, 2005, p.64).

Per Rozados (2005), these indicators are valuable tools and are still little used in management, but their systematic use is able to point out the strengths and weaknesses, besides allowing the comparison of the organization’s results over time and with other organizations, enabling practices such as benchmarking.

5 Material and methods

5.1 Analyzing the quality of ENADE indicators

The evaluation of the quality of ENADE indicators for the ULs considered three attributes: stability, validity and importance of the indicators.

One of the properties present in good indicators is stability, in other words, the ability to maintain the same form of measurement, allowing the establishment of a stable historical series and making it possible to monitor and compare their results over time. For the stability analysis, all the indicators used by ENADE in the evaluation of libraries between 2004 and 2014 were described and presented. After the description, the frequency of the indicators in the ENADE assessments was calculated from the beginning of their application in 2004.

Another important feature is validity. A good indicator should produce accurate data that can reflect the effects of interventions. For the analysis of this property, the results of ENADE on the updating of the CL's books and periodicals, measured from the students' perception, were used. This information has been worked out together with data on the investments made by the CL, represented by the acquisitions incorporated into the archive, whose measurements were obtained from the library management system. What is expected is that a positive perception of the students in relation to the updating of books and periodicals is directly related to the CL's investments in these areas, otherwise the indicator of updating the archive through the students' perception may not be valid for producing information that helps direct the resources earmarked for the acquisition of bibliographic materials.

The method used to evaluate the validity of the indicators was the correlation analysis. The sample took information from the Administration, Economics, Law and Psychology courses, since these were the only ones included in all the ENADE assessments and because they made available the data disaggregated bycourse.

Another central criterion for a good indicator is its importance. For this, it is necessary to provide information about the main strategic variables, priorities in the decision-making and about the products or expected impacts.

The importance of the indicators used by ENADE was verified by the Delphi technique in conjunction with the Mann-Whitney non-parametric statistical U-test and the Box Plot type graph. To that end, a group of experts judged and reached consensus on the importance of the indicators used by ENADE.

5.2 Delphi Method

The Delphi method or Delphi methodology “[...] aims to obtain the consensus of expert opinions on what is being investigated.” (VERGARA, 2005, p. 172).

The selection of the specialists was not random, because “the relevant level of professional qualification about the subject area to be studied is of extreme importance in order to obtain a consensus of specialized ideas.” (SCARPAROet al., 2012, p. 245). Therefore, to be part of the group there is a prerequisite - the experience of at least one year as a manager in university libraries. For example, occupying such a position, these professionals acquire a broader view of the functioning dynamics of the library, not restricting themselves to the activities inherent to the sector in which they work.

Librarians from the 20 best federated universities were invited to join the group, per the General Index of Courses (GIC) for the year 2013, the last one published by INEP (INEP, 2015). In addition, professionals from the two best private higher education institutions of the Federal District received invitations. These professionals were invited personally, through e-mail and through social networks, and 28 people expressed an interest in participating in the research.

After defining the group of experts, an electronic questionnaire containing the five indicators that appeared most frequently in the ENADE assessments was applied and the analysts were asked to score the degree of relevance of each indicator on a five-point ordinal scale. The indicators refer to issues related to the operating hours, frequency of use of the library by the students and the updating and availability of the archive. After indicating the degree of relevance, the specialists were asked to justify the reasons for their response in an open space, according to Chart 2.

Chart 2
Questionnaire for the first round of the Delphi method

In the Delphi method, the consensus of the specialists is obtained through the application of questionnaires, in successive rounds, allowing information exchange among the participants, but preserving the anonymity.(VERGARA, 2005). The exchange of information among the participants, known as “controlled feedback”, is the main success factor of the method.(MUNARETTOet al., 2013). According to theforementioned authors, this exchange of information generates reciprocal learning among all the respondents, spontaneously, because the identities of each one are preserved by anonymity.

In the first round held, 21 experts returned the questionnaire sent. After obtaining the data with the first round, the frequencies of the answers in the five attributes of relevance were calculated and their respective justifications summarized, objectively highlighting the favorable and unfavorable reasons for each indicator. In the second round, 14experts answered the same questions;however, they possessedthe knowledge of the answers given by the group in the previous phase, which allowed the participants to reflect on their answers, being able to maintain or change their opinion about the relevance of the indicators.

The number of rounds required to reach consensus in the Delphi technique depends on the complexity of the subject and expert knowledge. According to Vergara (2005), it would take at least two rounds, but, in general, it would not exceed five. However, the author herself points out that the use of the method by electronic channels tends to be fast, since the internet permits swift sending and receiving of questionnaires.

The definition of consensus among the experts counted on the application of the Mann-Whitney U-test for independent samples. The U-test compares the location center of two samples to detect differences between them.(PESTANA; GAGEIRO, 2005). Thus, the consensus is obtained at the moment that the test did not identify a statistically significant difference between the opinions of the specialists for all ENADE indicators throughout the rounds of application of the questionnaires. The consensus indicatorswere omitted from the questionnaire in subsequent rounds of the Delphi method application.

After the consensus, the U-test was applied again, no longer to compare the results of the same indicator in different rounds, but to compare the degree of relevance between the indicators used by ENADE.

6 Results and discussions

6.1 Analyzing the quality of ENADE indicators

Stability

The analysis of all the indicators used by ENADE for the evaluation of university libraries since the beginning of the exam in 2004 (Chart 3) indicated the lack of stability of the indicators used in the evaluation of libraries.

Chart 3
ENADE questions for libraries between 2004 and 2014

2004-2008	2009-2012	2013-2014
How frequently do you use the library of your institution?	How frequently do you normally use the library of your institution?	The library provides the bibliographic references that students needed.
How do you evaluate the library archive, with regard to updates, given the curricular needs of your course?	How do you evaluate the library archive, with regard to updates, given the curricular needs of your course?	The institution featured virtual library or has given access to available works in virtual archives.
How do you evaluate the scientific-academic periodicals available at the library, with regard to updates?	How do you evaluate the scientific-academic periodicals available at the library, with regard to updates?
Operating hours that satisfy your needs?	Operating hours that satisfy your needs?
Concerning books commonly used in the course, the number of copies available in the library attends all the students?	Among the times you needed to use the library archive, did you manage to do so?
What source(s) do you use the most to perform research activities forthe course subjects? (A) The library collection of my institution. (B) The library collection of another institution.	Are the general conditions of the physical facilities of classrooms, libraries and workplaces and study for the operation of the course appropriate?
Does the library of your institution offer book loan services?
How is the literature search service offered?
Infrastructure for reading and study

• Source: Authors

The first ENADE evaluation had nine indicators, reduced to six in the subsequent monitoring, and it had only two in the last evaluation, i.e. a 78% numerical reduction of indicators used.

In addition to the drastic reduction in the number of indicators, it is worth noting the modification of these indicators. The second and third periods (2009 - 2012 and 2013 - 2014) included the inclusion of two indicators that were not used in the previous period:

• Among the times you needed to use the library archive, did you manage to do so?

• Did the institution feature a virtual library or has it given access to available works in virtual archives?

Of the eleven indicators used by ENADE between 2004 and 2014, only five took place in two evaluation periods, these being those evaluated in the Delphi technique. They are:

• How often does the student use the library?

• How does the student evaluate the archive, with regard to updates?

• How does the student evaluate the journal archive, with regard to updates?

• Do the hours of operation satisfy the needs?

• Concerning the most used books in the course, does the number of copies available satisfy the demand?

The changes made in the questionnaires generated a problem of indicator instability. Questions used atthe beginning of the evaluation are no longer available nowadays, which impairs the analysis of the library's results over the evaluation years. Because they have been interrupted, it is not possible to have a clear view of the path traveled, and that hinders the perception of a tendency since, to evaluate this component, at least 10periods should be necessary.(CUNHA; AMARAL; DANTAS, 2015).

Similarly, the indicators currently used will not have a past situation to be compared. As pointed out by Rozados (2005), it is necessary that these indicators follow a pattern and that the historical production should follow the same standard and form of measurement for possible comparisons to be useful for management.

The absence of the stability criteria was also observed by Dufloth and Silva (2008) when analyzing Brazilian public security indicators published in the UNODC reports (United Nations Office on Drugs and Crime). In this study, the authors pointed out as fragility the fact that they could not verify long and comparable historical series. For D'Este, Martinez and Gallart (2009), this comparability over time must be a general condition of any good indicator.

Throughout the reviewed period, the world has undergone major changes and, consequently, this new context has modified the products and services of university libraries. Therefore, it is expected that the instruments of evaluation will follow and adapt to this evolution.

As described in Chart 3, the questionnaire underwent two changes, the last one in 2013, which had the objective of adjusting the instrument to the current demands (INEP, 2015). To that end, a group of experts was assembled to reformulate the SQ in order to “produce better quality inputs and greater quantity for the survey of the students' perceptions about the educational conditions of their formative processes.” (INEP, 2014, p. 1). However, the assembled experts, despite being mostly PhDs in areas such as Education and Statistics, apparently did not have a background in Library Science nor had experience in working with libraries. For this reason, the modifications may not have been appropriateregarding the evaluation of ULs, which has led to the quantitative and qualitative loss of indicators.

The instability of the indicators used by ENADE can be explained by the constant evolution of university libraries, mainly due to technological changes with the Internet. However, given the content of the changes in the ENADE indicators, this justification is not enough to explain the changes that have occurred in the ENADE indicators.

Validity

The results in Figure 1 suggested a lack of correlation between students' perceptions and the CL investments in updating the archive. The correlation degree for the collection of books and journals was very low, contrary to the expected results and not significant at the 5% level (Table 1).

Figure 1
Relationship between the students' perception about the update of books and periodicals and library investments in these aspects

Table 1
Correlation degree between the students' perception about the update of books and periodicals and library investments in these aspects

		Number of Books Acquired			Number of Journals Acquired
Students' perception about the update of the book archive	Person Correlation	-0.20	Students' perception about the update of the journal archive	Person Correlation	-0, 01
	Sig.	0.54		Sig.	0 98
	Samplesize	12		Samplesize	12

• Source: Authors

The AA (Book Archive Update) and AP (Journal Archive Update) indicators used by ENADE apparently do not meet the validity criteria, since it is expected that an indicator will representthe reality, as closely as possible, and should berelevant to what is being measured.(Brasil, 2010). It is understood that its use intends to measure how up-to-date the collections of books and periodicals were at the time. Consequently, a low satisfaction with the updating of the collections could be interpreted as an effect of a lack of investment in the acquisition of books and periodicals;however, the results showed that it is not possible to make such an association.

The lack of validity can lead to mistaken decision making, because it makes it difficult to correctly identify situations that need to be improved upon. In the case of libraries, the results could boost investments in collections that are not outdated in the proportion suggested by the user's perception.

In the context of sustainable development, a similar situation is exposed by Melo (2013), when referring to the choice of the Gross Domestic Product (GDP) as the only indicator of influence on economic dynamism. According to the author, the increase in GDP is not, for example, capable of reflecting the depreciation of the natural capital. Therefore, attention to the definition of valid indicators is one of the factors that avoids “the risk of defining inadequate indicators to reflect the desired situations.” (SANTOS FILHO, 2007, p. 1007).

Importance

The U-test did not identify a difference between the opinion of the experts of the first and second rounds of the Delphi method at the 5% level, therefore, obtaining the consensus among the participants regarding the relevance degree of all ENADE indicators occurred in the second round of the questionnaire applied. The median value of the answers, as well as the data dispersion for each analyzed indicator in the two rounds, is shown in Figure 2.

Figure 2
Expert perceptions for the indicators used by ENADE in the evaluation of university libraries in the first and second round of the Delphi method

The realization of only two rounds to achieve a consensus suggests a high conviction and knowledge of the experts on the subject matter and, consequently, little influence of the group's responses on the change inindividual opinions. In addition, the anonymity of the method enabled the participants to express their ideas without constraints of any nature. Apparently, the main change of opinion occurred for the ED indicator, highlighting the lower data dispersion around a lower median in the second phase of the research;however, even this change was not statistically significant at the 5% level.

The number of two rounds required to reach consensus was widely corroborated by the literature. According to Giovinazzo (2001), when using the Delphi method, two rounds are sufficient, since new steps would be uninteresting for the participants. Moreover, this number of rounds has been enough to reach consensus on most of the discussed issues, as can be seen in the results of studies by Araújoet al.(2015) andFechine (2014).

According to the opinion of the experts consulted, all ENADE indicators are relevant. No indicator reached a median value in its responses lower than the “relevant” attribute in the questionnaire, codified by the score 4.(Figure 2).

The results also indicated a tendency of the specialists interviewed to evaluate the indicators in a positive way, since the assigned scores are concentrated from the middle of the scale towards the high degreeof relevance. This behavior can be the result of an evaluation based on the importance of the service or product represented by the indicator, rather than the judgment of the capacity of this indicator to measure and generate the information for which it was designed.

A similar situation was found by Pereira (2013), where he also applied a questionnaire to managers of ULs investigating the relevance of 44 performance indicators. The obtained result indicated that 82% of the indicators were considered relevant for decision-making, receiving scores above the neutral value. However, a low percentage of use was observed, suggesting a discrepancy between the relevance attributed in the evaluation and the choices carried out in practice.

This tendency increases the need for a relative evaluation between indicators, not only absolute evaluation, since it is possible to perceive which of the relevant ones stand out for the formation of a parsimonious measurement group. Thus, by observing the results of the U-test, it is possible to verify that the specialists pointed to FB (Frequency of use of the library) and HF (Hours of operation) as indicators that are more important than AA (Book Archive Update) and AP (Journal Archive Update). The ED (available copies) indicator layin the middle ground, and it is not possible to affirm at the 5% level that this is less important than FB and HF, nor that it is more important than AA and AP (Chart 4).

Chart 4
Difference between the degreeof relevance of the indicators used by ENADE to evaluate university libraries

Groups
1	2
FB
HF
ED	ED
	AA
	AP

• Source: Authors

The reasons that led the experts to point out FB (Frequency of use of the library) and HF (Hours of operation) as the main indicators of ENADE were related to the relevance of the library in the performance of its own role as a support to teaching, research and extension. In other words, it is useless to worry about the collection of books and journals (AA and AP) if the availability of the library for the users is not adequate (FB and HF), according to the justifications presented openly in the questionnaire:

“The library must support students in teaching and research”

“Relevant item since the library fosters the academic bibliography and research. It needs to attract its users and demonstrate its products and services (mainly digital) that are unknown by most of its users and are of paramount importance in the scientific environment”

“We can consider that the presence of the user in the library is the currency of exchange for the service provided by this sector in the institution, the greater, the presence the more precious are the services offered by the library”

“The library needs to reach the greatest availability necessary for its users and needs to cover all the academic shifts”

“In addition to the library having a schedule compatible with academic activities, it is important to know if it meets the personal needs of students”

On the other hand, the indicators of group 2 were considered relevant although to a lesser degree if compared to the importance of FB and HF. Some justifications of the specialists that denote the importance of the updating of the bibliographical materials were:

“In the context of university libraries, the collection must meet the bibliographies of the disciplines, so the adequate collection is relevant to guarantee the quality of teaching and support to scientific research”

“It is important to get a collection evaluation from the students to obtain some feedback about the quality of the collection”

“Fundamental, because recent researches are published in the form of an article in newspapers”

“It is very relevant because it is scientific literature that promotes main researches, projects, completion of graduation works, thesis, dissertations and other academic documentary types. The journals, when updated, bring modern scientific information, peer- reviewed research results in relatively less time than other channels”

“Relevant to guide the acquisition of titles of the basic and complementary bibliography. It goes beyond being just a chart that guides the work of the evaluators, the important thing is to guarantee access to these books”

The UL has a general mission based on ensuring access to information and disseminating scientific knowledge in response to the demands of the academic community, which involves a broad range of products and services. When assessing an indicator considering its importance as an essential property, it is necessary to verify if it provides information about the main products, actions and strategic objectives.(BONNEFOY, 2005). In addition, it is necessary to consider the relevance of the production and use of the indicator.(COSTA; CONCEIÇÃO, 2012).

When asked about the importance of the ENADE indicators, it was clear that the experts considered the reliability aspect of the indicator in their judgment, suggesting that the measurement of FB (Frequency of use of the library) and HF (Hours of operation) through students' perceptions may be more reliable than the AA (Book Archive Update) and AP (Journal Archive Update) measurements by the same method. Thus, the problem of ENADE indicators does not seem to be in the use of measures of student perception; rather, the problem stems from not considering if the student has the appropriate knowledge to evaluate a certain issue.

7 Final considerations

University libraries play an important role within the framework provided by the university forits academic community. This importance can be perceived by the weight that this unit carries within the evaluations of the National System of Evaluation of Higher Education (SINAES), which is part of the National Student Performance Examination (ENADE).

Since 2004, the Student Questionnaires (SQ) applied by ENADE have sought to capture students' perceptions of ULs to contribute to the improvement of management. For this reason, it has been important to analyze the quality of the indicators used.

Per the perception of the group of experts who participated in the research, the indicators used by ENADE are considered important, but are not pointed out as the main ones when thinking about an ideal UL evaluation. Their forms of measurement did not prove effective in analyzing the results of the CL, and did not allow the formation of longhistorical series. Thus, they did not present the properties of stability and validity, considered essential characteristics for the quality of an indicator.

The methodology was also relevant because it was done with the participation of the main stakeholders: librarians. Considering that the participants hold or have already held positions aschiefs and directors of libraries, they are more likely to be able to propose and identify which indicators are relevant so that they can better base their decisions on the management of ULs. Another important point is that the group was formed by experts from universities in different states, both public and private. It is known that each organization has its own characteristics and is part of different contexts, which makes it questionable to use a single evaluation model. However, the unfeasibility of a government evaluation in a more individualized way is perceived. It is therefore important that the instrument used must be composed of indicators present in the reality of most existing libraries.

References

• ABMES. Associação Brasileira de Mantenedoras do Ensino Superior. Available at: https://abmes.org.br/noticias/detalhe/2068/mec-admite-limitacoes-nos-indicadores-de-qualidade-do-ensino-superior Accessedon: 24 Sept. 2018.
  » https://abmes.org.br/noticias/detalhe/2068/mec-admite-limitacoes-nos-indicadores-de-qualidade-do-ensino-superior
• ALMEIDA, Maria Christina Barbosa de. Planejamento de bibliotecas e serviços de informação. 2. ed. Brasília, DF: Briquet de Lemos, 2005. Availableat:https://docero.com.br/doc/cecxcn Accessedon: 26 Mar. 2020.
  » https://docero.com.br/doc/cecxcn
• ARAÚJO, Tamires Sousa et al. Problemas percebidos no exercício da docência em contabilidade. Revista Contabilidade & Finanças, v. 26, n. 67, p. 93-105, Jan./Apr. 2015. Availableat: http://www.revistas.usp.br/rcf/article/view/98101/96937Accessed on: 27 Oct. 2015.
  » http://www.revistas.usp.br/rcf/article/view/98101/96937
• BATISTA, Darlin Salgado; SÁNCHEZ, María Victoria Guzmán; RIVERO, YaidelynMacías. La investigaciónen Cuba sobre evaluación de bibliotecas. ACIMED, v. 23, n. 2, p. 160-174, 2012. Available at: http://scielo.sld.cu/scielo.php?script=sci_arttext&pid=S1024-94352012000200006 Accessedon: 26 Mar. 2020.
  » http://scielo.sld.cu/scielo.php?script=sci_arttext&pid=S1024-94352012000200006
• BEZERRA, Neiliane Alves. Avaliação da qualidade dos serviços das bibliotecas universitárias pelos usuários discentes. Dissertation (Mestrado profissional em Políticas Públicas e Gestão da Educação Superior). Universidade Federal do Ceará, Fortaleza, 2010. Availableat: http://www.repositorio.ufc.br/handle/riufc/694 Accessedon: 03 Dez. 2014.
  » http://www.repositorio.ufc.br/handle/riufc/694
• BONNEFOY, Cristóbal Juan; ARMIJO, Marianela. Indicadores de desempeñoenel sector público. Santiago do Chile: Instituto Latinoamericano y del Caribe de Planificación Económica y Social, 2005. (Manuales, 45). Available at: http://repositorio.cepal.org/bitstream/handle /11362/5611/S05900_es.pdf Accessedon: 27 Oct. 2015.
  » http://repositorio.cepal.org/bitstream/handle /11362/5611/S05900_es.pdf
• BRASIL. Lei n. 10.861, de 14 de abril de 2004. Institui o Sistema Nacional de Avaliação da Educação Superior - SINAES e dá outras providências. Diário Oficial [da] República Federativa do Brasil, Brasília, DF, 15 Apr. 2004. Available at: http://www.planalto.gov.br/ccivil_03/_ato2004-2006/2004/lei/l10.861.htm Accessedon: 26 Maio 2014.
  » http://www.planalto.gov.br/ccivil_03/_ato2004-2006/2004/lei/l10.861.htm
• BRASIL. Ministério do Planejamento, Orçamento e Gestão. Indicadores: orientações básicas aplicadas à gestão pública. Brasília, 2012. Availableat: http://www.planejamento. gov.br/servicos/central-de-conteudos/publicacoes/121003_orient_indic_triangular.pdfAccessedon: 04 Nov. 2015.
  » http://www.planejamento. gov.br/servicos/central-de-conteudos/publicacoes/121003_orient_indic_triangular.pdf
• BRASIL. Ministério do Planejamento, Orçamento e Gestão. Indicadores de programas: guia metodológico.Brasília, 2010. Available at: http://www.planejamento.gov.br/secretarias/upload/Arquivos/spi/publicacoes/100324_indicadores_programas-guia_metodologico.pdf.Accessed on: 23 Oct. 2015.
  » http://www.planejamento.gov.br/secretarias/upload/Arquivos/spi/publicacoes/100324_indicadores_programas-guia_metodologico.pdf.
• BRITO, Márcia Regina F. de. O SINAES e o Enade: da concepção à implantação. Avaliação, Campinas; Sorocaba, SP, v. 13, n. 3, p. 841-850, nov. 2008. Availableat: http://www.scielo.br/pdf/aval/v13n3/14.pdfAccessedon: 04 Nov. 2015.
  » http://www.scielo.br/pdf/aval/v13n3/14.pdf
• BRITO, Gisele Ferreira de; VERGUEIRO, Waldomiro de Castro Santos. Avaliação da qualidade da biblioteca acadêmica: a metodologia LibQUAL+® e suas perspectivas de aplicação no Brasil. BJIS, Marília, SP, v. 7, n. Especial, p. 26-46, 1º. Sem. 2013. Availableat: http://www2.marilia.Unesp.br/revistas/index.php/bjis/indexAccessedon: 01 Dec. 2015
  » http://www2.marilia.Unesp.br/revistas/index.php/bjis/index
• CARVALHO, Maria Margarida Melo de; DOMINGUEZ, Caroline Elizabeth. New challenges for the quality of librarian services delivery in Portugal.Library Management, v. 33, n. 4/5, p. 272-291, 2012.Available at: http://www.emeraldinsight.com/doi/abs/10.1108/01435121211242317Accessedon: 19 Out. 2015
  » http://www.emeraldinsight.com/doi/abs/10.1108/01435121211242317
• COSTA, Alexander Josef Sá Tobias da; CONCEIÇÃO, Rodrigo Silva da. Reflexões sobre a seleção de indicadores sociais e ambientais na política nacional de proteção e defesa civil em âmbito local. Geo UERJ, v. 2, n. 23, p. 413-436, 2012.Available at: http://www.e-publicacoes.uerj.br/index.php/geouerj/article/view/4755Accessedon: 14 Out. 2015.
  » http://www.e-publicacoes.uerj.br/index.php/geouerj/article/view/4755
• CUNHA, Murilo Bastos da; AMARAL, Sueli Angélica; DANTAS, Edmundo Brandão. Manual de estudos de usuário da informação. São Paulo: Atlas, 2015. Available at: https://rbbd.febab.org.br/rbbd/article/view/625Accessedon: 26 Mar. 2020.
  » https://rbbd.febab.org.br/rbbd/article/view/625
• D’ESTE, Pablo; CASTRO MARTÍNEZ, Elena; MOLAS GALLART, Jordi. Documento de base para un “manual de indicadores de vinculación de launiversidadconel entorno socioeconómico”: un marco para ladiscusión. Buenos Aires: OEI/AECID, 2009. Availableat: http://www.observatorio cts.org/files/Archivo%20Documental/Documentos%20de%20proyectos/indicadores_de_vinculacion.pdfAccessedon: 16 Oct. 2015.
  » http://www.observatorio cts.org/files/Archivo%20Documental/Documentos%20de%20proyectos/indicadores_de_vinculacion.pdf
• DEUS, Cátia Costa Rocha Daniel de. Evolução das bibliotecas universitárias e suas relações com as políticas educacionais no Brasil. In: CONGRESSO BRASILEIRO DE BIBLIOTECONOMIA, DOCUMENTAÇÃO E CIÊNCIA DA INFORMAÇÃO, 24, 2011, Maceió, Alagoas. Anais [...]Maceió: FEBAB, 2011. Available at: http://www.febab.org.br/congressos/index.php/cbbd/xxiv/paper/viewFile/418/537 Accessedon: 05 Nov. 2014.
  » http://www.febab.org.br/congressos/index.php/cbbd/xxiv/paper/viewFile/418/537
• DORIGON, Derli Sandra. Gestão de operações e qualidade dos serviços nas unidades de informação. 2006. 203 f. Dissertação (Mestrado em Ciência da Informação) - Centro de Ciências da Educação, Universidade Federal de Santa Catarina, Florianópolis, 2006. Available at: https://repositorio.ufsc.br/handle/123456789/89091 Accessed on: 07 Out. 2015.
  » https://repositorio.ufsc.br/handle/123456789/89091
• DUARTE, Maria Bernarda Teixeira; BARBOSA, Luiz Fernando Sampaio; COSTA, Helder Gomes. Seleção de critérios para a avaliação de serviços oferecidos por bibliotecas: uma revisão bibliográfica da base Scielo. In: CONGRESSO NACIONAL DE EXCELÊNCIA EM GESTÃO, 9., 2013. Anais [...] Rio de Janeiro: UFF, 2013. Available: http://www.excelenciaemgestao.org/Portals/2/documents/cneg9/anais/T13_0575_3310.pdfAccessedon: 10 Dez. 2014.
  » http://www.excelenciaemgestao.org/Portals/2/documents/cneg9/anais/T13_0575_3310.pdf
• DUFLOTH, Simone Cristina; SILVA, Armando Leonardo Linhares de Araújo Ferreira da. Indicadores de segurança pública para a gestão de políticas públicas: análise dos atributos de qualidade dos indicadores propostos nas perspectivas de organizações internacionais, do governo federal e de governos estaduais. In: ENCONTRO DE ADMINISTRAÇÃO PÚBLICA E GOVERNANÇA, 2008, Salvador. Anais [...] Salvador: ANPAD, 2008. Available at: http://www.anpad.org.br/diversos/trabalhos/EnAPG/enapg_2008/ 2008_ENAPG372.pdf Accessedon: 13 Out. 2015.
  » http://www.anpad.org.br/diversos/trabalhos/EnAPG/enapg_2008/ 2008_ENAPG372.pdf
• DZIEKANIAK, Cibele Vasconcelos. Participação do bibliotecário nas comissões de avaliação do Ministério da Educação (MEC) nos processos de reconhecimento de cursos de graduação. BIBLOS, v. 18, p. 169-178, 2006. Available at: http://www.seer.furg.br/biblos/article/view/90Accessed on: 10 Dez. 2014.
  » http://www.seer.furg.br/biblos/article/view/90
• FAGAN, Jody Condit. The dimensions of library service quality: a confirmatory factor analysis of the LibQUAL+ instrument. Library & Information Science Research, v. 36, p. 36-48, 2014. Available at: https://www.sciencedirect.com/science/article/abs/pii/S0740818814000073?via%3Dihub Accessedon: 26 Mar. 2020.
  » https://www.sciencedirect.com/science/article/abs/pii/S0740818814000073?via%3Dihub
• FECHINE, Roberta. Indicadores de sustentabilidade como instrumentos para avaliação de programas de coleta seletiva na cidade de Salvador-BA. 2014. 149 f. Dissertação (Mestrado em Meio Ambiente, Águas e Saneamento) - Universidade Federal da Bahia, Salvador, 2014. Available at: http://www.maasa.eng.ufba.br/dissertacoes/indicadores-de-sustentabilidade-como-instrumentos-para-avaliacao-de-programas-de-coleta Accessedon: 19 Set. 2015.
  » http://www.maasa.eng.ufba.br/dissertacoes/indicadores-de-sustentabilidade-como-instrumentos-para-avaliacao-de-programas-de-coleta
• FERREIRA, Beatriz Lagnier Gil; OLIVEIRA, Cintia Machado de; DAGOSTO, Marcio de Almeida. Avaliação da percepção do usuário sobre a qualidade do sistema de transporte público do Recreio dos Bandeirantes. In: CONGRESSO DE PESQUISA E ENSINO EM TRANSPORTES, 28., 2014, Curitiba. Anais[...]Curitiba: ANPET, 2014. Available at: http://www.anpet.org.br/xxviiianpet /anais/documents/AC266.pdf Accessed on: 27 Ago. 2015.
  » http://www.anpet.org.br/xxviiianpet /anais/documents/AC266.pdf
• GELFAND, Morris A. University libraries for developing countries. Paris: Unesco, 1969. 157 p. Available at: https://unesdoc.unesco.org/ark:/48223/pf0000000714 Accessedon: 26 Mar. 2020.
  » https://unesdoc.unesco.org/ark:/48223/pf0000000714
• GIOVINAZZO, Renata. Modelo de aplicação da metodologia Delphi pela Internet: vantagens e ressalvas. Administração on line, v. 2, n. 2, 2001.Available at: http://www.fecap.br/adm_online/art22/renata.htm Accessedon: 22 Jan 2015.
  » http://www.fecap.br/adm_online/art22/renata.htm
• INSTITUTO NACIONAL DE ESTUDOS E PESQUISAS EDUCACIONAIS - INEP. Igc_2013_09022015.xlsx.Brasília, 2015. Available at: http://download.inep.gov.br/educacao_superior /enade/igc/2013/igc_2013_09022015.xlsx Accessed on: 20 Apr. 2015.
  » http://download.inep.gov.br/educacao_superior /enade/igc/2013/igc_2013_09022015.xlsx
• INSTITUTO NACIONAL DE ESTUDOS E PESQUISAS EDUCACIONAIS - INEP. Nota técnica nº 70: utilização dos insumos do Questionário do Estudante aplicado em 2013. Brasília, 2014. Availableat: http://download.inep.gov.br/educacao_superior/enade/notas_tecnicas/2013/nota_tecnica_n_70_2014_utilizacao_insumos_questionario_estudante_2013.pdf Accessedon: 25 Jun. 2015.
  » http://download.inep.gov.br/educacao_superior/enade/notas_tecnicas/2013/nota_tecnica_n_70_2014_utilizacao_insumos_questionario_estudante_2013.pdf
• INSTITUTO NACIONAL DE ESTUDOS E PESQUISAS EDUCACIONAIS - INEP. Relatório da IES. Brasília: Universidade de Brasília. Brasília, 2013.(Enade 2012). Available at: http://enadeies.inep.gov.br/enadeResultado/.Accessed on: 11 Oct. 2014.
  » http://enadeies.inep.gov.br/enadeResultado
• JANUZZI, Paulo de Martino. Indicadores para diagnóstico, monitoramento e avaliação de programas sociais no Brasil. Revista do Serviço Público, v. 56, n. 2, p. 137-160, 2005. Availableat: http://seer.enap.gov.br/index.php/RSP/article/view/222 Accessedon: 09 Out. 2015.
  » http://seer.enap.gov.br/index.php/RSP/article/view/222
• JANUZZI, Paulo de Martino. Considerações sobre o uso, mau uso e abuso dos indicadores sociais na formulação e avaliação de políticas públicas municipais. Revista de Administração Pública, v. 36, n. 1, p. 51-72, 2002. Availableat: http://bibliotecadigital.fgv.br/ojs/index.php/rap/article/view/6427 Accessedon: 15 Out. 2015.
  » http://bibliotecadigital.fgv.br/ojs/index.php/rap/article/view/6427
• LUBISCO, Nídia Maria Lienert. La evaluaciónenla biblioteca universitariabrasileña: evolución y propuesta de mejora. 2007. 439 f. Tese (Doutorado). Universidad Carlos III de Madrid, Madrid, 2007.
• LUBISCO, Nídia Maria Lienert; VIEIRA, Sônia Chagas (Orgs.). Biblioteca universitária brasileira: instrumento para seu planejamento e gestão, visando à avaliação do seu desempenho. Documento final consolidado a partir das contribuições dos Grupos de Trabalho do Seminário Avaliação da Biblioteca Universitária Brasileira. Salvador: EDUFBA, 2009. Available at: https://repositorio.ufba.br/ri/bitstream/ufba/588/3/Biblioteca%20universitaria%20brasileira.pdf Accessed on: 22 Nov. 2014.
  » https://repositorio.ufba.br/ri/bitstream/ufba/588/3/Biblioteca%20universitaria%20brasileira.pdf
• MCCAFFREY, Ciara. LibQUAL in Ireland: performance assessment and service improvement in Irish university libraries. The Journal of Academic Librarianship, v. 39, p. 347-350, 2013. Available at: https://www.sciencedirect.com/science/article/abs/pii/S0099133312002042 Accessedon: 26 Mar. 2020.
  » https://www.sciencedirect.com/science/article/abs/pii/S0099133312002042
• MELO, Paulo Thiago Nunes Bezerra de. Indicadores da dimensão institucional do desenvolvimento sustentável e os objetivos da Rio +20. Desenvolvimento em questão, v. 11, n. 23, 2013. Availableat: https://www.revistas.unijui.edu.br/index.php/desenvolvimentoemquestao /article/view/308 Accessedon: 15 Out. 2015.
  » https://www.revistas.unijui.edu.br/index.php/desenvolvimentoemquestao /article/view/308
• MONTIBELLER FILHO, Gilberto. Indicadores e equidade social: propriedades dos indicadores de sustentabilidade e ausência do princípio de justiça social em estudos sobre mudança climática e CO2. RevistaInternacionalInterdisciplinarINTERthesis, v. 7, n. 1, p. 199-221, 2010. Available at: https://periodicos.ufsc.br/index.php/interthesis/article/view/1807-1384.2010v7n1p199 Accessed on: 10 Out. 2015.
  » https://periodicos.ufsc.br/index.php/interthesis/article/view/1807-1384.2010v7n1p199
• MUNARETTO, Lorimar Francisco; CORRÊA, Hamilton Luiz; CUNHA, Júlio Araújo Carneiro da. Um estudo sobre as características do método Delphi e de grupo focal como técnicas na obtenção de dados em pesquisas exploratórias. Revista de Administração da UFSM, v. 6, n. 1, p. 09-24, 2013. Availableat: http://cascavel.ufsm.br/revistas/ojs-2.2.2/index.php/reaufsm/article/view/6243 Accessedon: 16 Nov. 2014.
  » http://cascavel.ufsm.br/revistas/ojs-2.2.2/index.php/reaufsm/article/view/6243
• PACIOS, Ana Reyes. Evaluación y planificaciónenlas bibliotecas universitariasespañolas. In: Biblioteca universitária: elementos para o planejamento, avaliação e gestão. Salvador: EDUFBA, 2011. 263 p. Available at: https://repositorio.ufba.br/ri/bitstream/ri/5620/1/_Biblioteca.pdfAccessed on: 26 Mar. 2020.
  » https://repositorio.ufba.br/ri/bitstream/ri/5620/1/_Biblioteca.pdf
• PEREIRA, Carla Purcina de Campos. Avaliação de desempenho e tomada de decisão em bibliotecas universitárias. Encontros Biblio: Revista eletrônica de Biblioteconomia e Ciência da Informação, Florianópolis, v. 18, n. 38, Nov. 2013. Available at: https://periodicos.ufsc.br/index.php/eb/article/view/31454Accessedon: 29 Out. 2015.
  » https://periodicos.ufsc.br/index.php/eb/article/view/31454
• PESTANA, Maria Helena; GAGEIRO, João Nunes. Análise de dados para Ciências Sociais: a complementaridade do SPSS. 6. ed. Lisboa: Edições Silabo, 2014. Availableat: https://www.researchgate.net/publication/272817141_ANALISE_DE_DADOS_PARA_CIENCIAS_SOCIAIS_A_Complementaridade_do_SPSS_6_EDICAO_Revista_Atualizada_e_Aumentada_MARIA_HELENA_PESTANA_JOAO_NUNES_GAGEIRO Accessedon: 20 Mar. 2020.
  » https://www.researchgate.net/publication/272817141_ANALISE_DE_DADOS_PARA_CIENCIAS_SOCIAIS_A_Complementaridade_do_SPSS_6_EDICAO_Revista_Atualizada_e_Aumentada_MARIA_HELENA_PESTANA_JOAO_NUNES_GAGEIRO
• POMBAL, Bruno; WILSON, Tom; AZEVEDO, Ana. Gestão da eficiência em serviços de documentação e informação. 2009. 138 f. Dissertação (Mestrado) - Faculdade de Engenharia, Universidade do Porto, 2009. Availableat:http://hdl.handle.net/10216/19295Accessedon: 23 Ago. 2015.
  » http://hdl.handle.net/10216/19295
• ROZADOS, Helen Beatriz Frota. Uso de indicadores na gestão de recursos de informação. Revista Digital de Biblioteconomia e Ciência da Informação, v. 3, n. 1, p. 60-76, 2005. Available at: http://www.sbu.unicamp.br/seer/ojs/index.php/rbci/article/view/316Accessedon: 03 Ago. 2015.
  » http://www.sbu.unicamp.br/seer/ojs/index.php/rbci/article/view/316
• SANTOS FILHO, Serafim Barbosa. Perspectivas da avaliação na Política Nacional de Humanização em Saúde: aspectos conceituais e metodológicos. Ciência & Saúde Coletiva, v. 12, n. 4, p. 999-1010, 2007. Availableat: http://www.scielo.br/scielo.php?pid=S1413-81232007000400021&script=sci_arttextAccessedon: 15 Out. 2015.
  » http://www.scielo.br/scielo.php?pid=S1413-81232007000400021&script=sci_arttext
• SCARPARO, Ariane Fazzoloet al. Reflexões sobre o uso da técnica Delphi em pesquisas na enfermagem. Revista da Rede de Enfermagem, v. 13, n. 1, p. 242-251, 2012. Availableat: http://www.revistarene.ufc.br/revista/index.php/revista/article/download/36/31 Accessedon: 25 Set. 2015.
  » http://www.revistarene.ufc.br/revista/index.php/revista/article/download/36/31
• TARANGO, Javier; HERNANDEZ OROZCO, Guillermo. Evaluación de bibliotecas universitarias: un modelo de avance y desarrollo. Revista Interamericana de Bibliotecologia, v. 32, n. 2, Dez. 2009. Availableat: http://www.scielo.org.co/scielo.php?script=sci_arttext&pid=S012009762009000200005&lng=en&nrm=iso Accessedon: 09 Dez. 2014.
  » http://www.scielo.org.co/scielo.php?script=sci_arttext&pid=S012009762009000200005&lng=en&nrm=iso
• TUTIKIAN, Jane; SUÑÉ, Letícia Sampaio. Prefácio. In: Biblioteca universitária: elementos para o planejamento, avaliação e gestão. Salvador: EDUFBA, 2011. 263 p. Available at: https://repositorio.ufba.br/ri/bitstream/ri/5620/1/_Biblioteca.pdfAccessed on: 26 Mar. 2020.
  » https://repositorio.ufba.br/ri/bitstream/ri/5620/1/_Biblioteca.pdf
• UNIVERSIDADE DE BRASÍLIA. Biblioteca Central. Sobre a BCE. 2015. Available at: http://www.bce.unb.br/sobre-a-bce/.Accessedon: 04 Nov. 2015.
  » http://www.bce.unb.br/sobre-a-bce
• VERGARA, Sylvia Constant. Métodos de pesquisa em administração. São Paulo: Atlas, 2005. 287 p. Available at: https://www.passeidireto.com/arquivo/35102746/sylvia-constant-vergara-metodos-de-pesquisa-em-administracaoAccessed on: 26 Mar. 2020.
  » https://www.passeidireto.com/arquivo/35102746/sylvia-constant-vergara-metodos-de-pesquisa-em-administracao

Publication Dates

• Publication in this collection
  20 May 2020
• Date of issue
  Jan-Mar 2020

History

• Received
  29 Nov 2018
• Accepted
  04 Feb 2020