Efficiency assessment in university libraries

Eficiência em bibliotecas universitárias

Abstract

University libraries provide an important service to society, contributing to spread knowledge and cultivating new talents in the academic environment. The main objective of this article is to perform a review of the literature on efficiency assessments in the context of university libraries. The databases Web of Science and Scopus were adopted as reference to search for papers in the aforementioned context and identify the methodologies used and perspectives set out by different authors. A complementary search was also made on Google Scholar to obtain additional articles. In sum, 34 papers were found to compose the core of analyzed publications. As result, we observed that nearly 90% of the papers use Data Envelopment Analysis to assess the efficiency of university libraries and other methodologies with the same proposal were identified. Moreover, the variables used in these publications were analyzed, contributing to the mapping of main inputs and outputs that directly affect the services of university libraries. Furthermore, other characteristics were also considered, such as: temporal placement of publications and countries with the largest production of papers. Finally, based on the results of this study, further researches are suggested.

Keywords
Academic libraries; Evaluation methods; Bibliographical revision

Resumo

As bibliotecas universitárias prestam um importante serviço à sociedade, contribuindo para difundir o conhecimento e cultivar novos talentos dentro do ambiente acadêmico. Neste estudo, a revisão da literatura sobre avaliação de eficiência no contexto das bibliotecas universitárias é o principal objetivo. Foram adotadas como referência as bases de dados Web of Science e Scopus para a busca de trabalhos no contexto acima mencionado e para a identificação das metodologias utilizadas e perspectivas estabelecidas pelos diferentes autores. Realizou-se uma pesquisa complementar na base Google Scholar para a obtenção de outros artigos. Em suma, foram selecionados 34 trabalhos para compor o núcleo de publicações analisadas. Como resultado, observou-se que cerca de 90% dos artigos utilizam a Análise de Envoltória de Dados para avaliar a eficiência das bibliotecas universitárias, tendo sido identificadas outras metodologias com a mesma proposta. Além disso, analisaram-se as variáveis utilizadas nessas publicações, contribuindo para o mapeamento dos principais inputs e outputs que afetam diretamente os serviços prestados pelas bibliotecas universitárias. Ademais, foram consideradas outras características, tais como o arranjo temporal das publicações e os países com a maior produção de artigos. Por fim, foram sugeridas pesquisas adicionais com base nos resultados deste estudo.

Palavras-chave
Bibliotecas acadêmicas; Métodos de avaliação; Revisão bibliográfica

Introduction

From Farrell´s work (1957) on, appeared the first researches about the measurement of productive units’ performance, which until our days serve as the basis for many studies that propose models to assess the efficiency of those units. The concept of efficiency suggests the comparison between what was produced with a certain amount of a resource and the greatest amount that could have been produced considering the same amount of resources (Soares de Mello et al., 2005).

Such concept can be applied within the context of university libraries, which provide an important service to society, contributing to spread knowledge and cultivating new talents at the academic environment, specially in a so-called information, digitalization and intellectualization era (Kuang et al., 2010).

Factors such as stricter budgets (Shahwan; Kaba, 2013), pressure for more services and results using less resources (Shim; Kantor, 1998), and for fast technological changes that require more investments (Noh, 2011) lead the necessity of continuous efficiency assessments in those institutions. Shim (2003) adds that university libraries handle continuous pressures to justify the use of resources according to provided services and their impact to users.

It is also emphasized that university libraries are included in an environment of continuous changes, highlighting technological innovations that transform their services and products. In this context, collections are not limited to physical ones, but include more and more electronic resources that require new management models. Thus, digital libraries become important information resources to both students and researchers (Joo; Lee, 2011). In recent years, academic libraries expanded digital assets to supplement the services of traditional libraries and support research, teaching and learning (Joo; Lee, 2011).

Electronic resources fast growth and high costs challenge managers to keep the signature of those resources when planning library actions. Furthermore, traditional assessment processes are not consElectronic resources fast growth and high costs challenge managers to keep the signature of those resources when planning library actions. Furthermore, traditional assessment processes are not considered fully adequate to digital means (Hwang et al., 2012).idered fully adequate to digital means (Hwang et al., 2012).

Thus, many studies have given special attention to those institutions, employing quantitative methods to assess efficiency in this scenery and setting out prospective models that may include all variables inherent to services and products offered.

In this challenging context, we propose an “exploratory study using a systematic literature review” to identify the main characteristics of those studies and analyze publications with the objective of providing a set of information that assists researchers and other specialists in the theme to develop new frontiers to assess efficiency, and improve the performance of academic libraries. In this sense, our reference is the work of Emrouznejad et al. (2008). The study was also based on the paper presented by Oliveira and Campello (2016) concerning the delimitation of the research objective to elucidate the state of the art regarding the evaluation of efficiency in the context of university libraries. Another inspiration for this study was the work of Lima et al. (2007) that adopted a systematic research on scientific data bases as a research method, involving the formulation of a search strategy using keywords for retrieval of relevant information to the search subject.

Therefore, we intend to show the methodologies used to measure and suggest improvements of those institutions, based on the proposed methods, models and variables used, providing managers with information that can help in decision-making and indicate opportunities to apply and distribute the resources in an efficient way.

This paper is divided into four sections. Section one proposed a contextualization of the theme and presented the scope of work. Section two explains the methodological procedures used in this study. Section three presents the main characteristics observed in the bibliographical review, with special focus on variables found in the studies analyzed, and brief comments on the objectives of the main research. It is also noteworthy how these papers deal with the measurement of efficiency in the context of university libraries. Finally, section four deals with the conclusions.

Methodological procedures

This article suggests a bibliographical revision on efficiency assessment in university libraries. Therefore, we initially set out the scientific bases Web of Science (ISI) and Scopus as references to search papers that are the core of examined publications, so as to enable the understanding of used methodologies in this research area and perspectives adopted by different authors to address this theme.

We opted to define a set of terms used in different ways in both scientific databases, attempting to include the greatest possible number of articles that address the efficiency assessment in university libraries.

At this stage, we used the terms “evaluation”, “measuring”, “measurement” and “assessment” jointed or severed, to emphasize the assessment scope of research, being complemented by the terms “efficiency” and “performance”. Finally, the expressions “academic library”, “university library” and the isolated term “library” were considered, emphasizing the title of articles. Those three last terms were included both in the plural and in the singular, intending a greater comprehension of obtained results.

After searches made in those two databases with aforementioned parameters and terms and using the title and the abstract of the articles as analysis criterion, 25 papers considered relevant to the scope of this research were selected.

In the following stage, the references from the previously found papers were analyzed to include other researches with the same purpose that, for any reason, were not found in results of searches performed in ISI and Scopus. Thus, other five articles were incorporated to the core of publications that will be analyzed.

Google Scholar was also used to complement the research. For being a reference database that includes academic journals, it is similar to previously cited subscription databases. The combinations of keywords used emphasized the terms “efficiency”, “performance” and “libraries” in the title of the articles. The related articles link for each result was also analyzed. Thereby, four other works that directly handle the mentioned subject were selected and integrated to the core of examined publications. They are: Han and Hong (2002), Lili (2008), Jo et al. (2009), and Carvalho et al. (2011).

It must be emphasized that an analysis of references of each of those articles was also carried out. However, it was not possible to find other articles besides those already in the set of publications.

In the last stage, we proposed the individual analysis of each of the 34 selected documents from considered scientific databases to understand how the existing literature addresses the efficiency assessment in the specific context of university libraries, and thus enable a deeper understanding of the subject, providing a base for subsequent works that intend to assess those institutions and assisting in the development of new models to reach this objective.

For instance, at this stage we will observe the most used variables within this context, the most used methods and techniques to assess efficiency, the assessed units and data related to publications of those works.

Figure 1 sums up the process of definition of selected articles that study the efficiency in university libraries to analysis and shows different combinations of keywords that were used in the databases Scopus and ISI.

Results and Discussion

In this section, we present a survey of the main characteristics of the 34 articles that form the core of examined publications, highlighting some information related to the variables involved in the methodological process. Furthermore, we suggest a data analysis of this set of articles.

Figura 1
Overview of the path to obtain the core articles.

Characteristics of empirical data

The assessment of libraries is a recurrent subject in journals of different interest areas, especially those connected to Information Science, Management and Engineering.

In this section, we set out important data that enables a more comprehensive view of the set of papers, such as chronological placement and the regions where this subject’s discussion is more mature, with the purpose of providing information related to the characteristics of the 34 articles that form the core of analyzed publications, as observed in the studies by Lima et al. (2007) and Emrouznejad et al. (2008).

In Figure 2, we can observe the number of studies that have been published over time which are part of the subjects mentioned in this paper. This figure helps us notice the periods in which the works are concentrated, in addition to the years when the publications were more intense or non-existent.

Figura 2
Distribution of papers by publication year. .

Finally, we performed an analysis of the main origins of publications found on scientific databases and articles that are directly related to the subjects addressed by this study.

The intense participation of Asian countries in research related to efficiency assessment in university libraries is noticeable. Among them, Taiwan has the largest number of publications inserted in the context of evaluation of these institutions, a total of 9 published articles. This is the country of origin of the work proposed by Kao and Liu (2000), for instance. These authors had their work widely mentioned both in Scopus Database (61 citations) and in Web of Science database (46 citations) by 2015.

Another widely mentioned Taiwanese author is Chen (1997a, 1997b), who assesses the efficiency in libraries in the city of Taipei. We highlight that the works that assess libraries in Asia, or of Asian origin, represent almost 50% of works that are part of the sample.

Other countries where there is a significant amount of important studies in those subjects are the United States (5 papers), Brazil (4 papers) and South Korea (4 papers). Austria and China each have 3 published papers on this topic. In addition, two articles of Spanish origin were also found.

Researches have also been found in other countries, such as Belgium, Bulgaria, the United Arab Emirates and Poland, each with an article published in this context.

Variables analysis

To achieve the proposed objective, Chart 1 presents the inputs found among the articles selected for analysis, proposing a comparative view between the sets of variables used in the measurement of efficiency in different university libraries according to the established by each author.

Chart 1
Inputs used in the context of efficiency evaluation in university libraries.

INPUTS	Chen (1997a)	Chen (1997b)	Shim and Kantor (1998)	Shim and Kantor (1999)	Kao and Liu (2000)	Han and Hong (2002)	Saunders (2002)	Kao and Liu (2003)	Shim (2003)	Stancheva and Angelova (2004)	Kao and Lin (2004)	Osiewalski and Osiewalka (2004)	Reichman (2004)	Reichmann and Sommersguter-Reichmann (2006)	Simon de Blás, etal. (2007)	Lili (2008)	Jorge et al. (2009)	Jo et al. (2009)	Liu and Chuang (2009)	Reichmann and Sommersguter-Reichmann (2009)	Kuang et al. (2010)	Noh (2011)	Simon et al. (2011)	Carvalho et al. (2011)	Carvalho et al. (2012)	Noh (2012)	Hwang et al. (2012)	Carvalho et al. (2013)	Shahwan and kaba (2013)	Hsieh et al. (2014)	Stroobants and Bouckaert (2014)
Staff related variables	X	X	X	X		X			X	X		X	X	X	X		X	X		X	X	X	X	X	X			X	X		X
Book Collection			X	X		X			X			X	X	X	X		X	X		X	X	X		X	X			X	X
Area of library space	X	X				X				X		X			X		X	X			X		X	X	X			X
Acquisition of books andperiodicals (expenditure or quantity)	X	X	X	X					X			X									X										X
Total serials / Number of journals			X	X					X						X							X
Student employees			X	X					X						X
Patronage					X			X								X			X
Computers / Equipment cost /Technical Equipment										X		X										X								X
Monographs purchased			X	X					X
Total student enrollment			X	X					X
Full-time instructional faculty			X	X					X
Wages / Average salary / Manpower cost										X		X																		X
Budget																		X				X							X
Library operation costs / Expenditure																					X		X								X
Web DB (purchased or subscritions)																						X				X	X
Expenses on electronic
DB and Softwares /										X																	X
Purchase fee
Seating Capacities / Quantity of seats												X									X
Eletronic resources (e-journals / e-books)																						X				X
Normalized Costs							X
Printed Edition Expenses										X
Size Index											X
Staff with bachelor degree or higher.																					X
Staff with medium professional title and above																					X
Internally developed																						X
DB units
e-resources built (expenditure or quantity)																										X
Collection cost (Expenses that libraries spent on e-resources)																														X
Number of programs for using e-resources																										X
Number of devices for using e-resources																										X
Number of trainings for using e-resources																										X

• Source: Prepared by Authors (2017).

This table seeks to help in the identification of the most common indicators measured and used in this context.

Chart 1 enables the visualization of the diversity of input variables involved in the studies selected. Such diversity is related to each author’s adopted methodology and defined objectives.

Altogether, 29 inputs were arranged from the modeling of articles that propose the application of evaluation tools in university libraries. It is noteworthy that in three of the thirty-four articles we could not identify the variables used in the efficiency evaluation process. Thus, the number of articles listed in the table with the main inputs is smaller than the number of publications found on the scientific basis.

Moreover, we observed that the same set of inputs and outputs were repeated frequently. This fact is originated in similar studies, which have common authors, and consequently similar approaches. These articles have origin in other papers with the same concept or approach, such as those written by Shim and Kantor (1998) and Shim and Kantor (1999). It is possible to point out other examples as well: Kao and Liu (2000); Kao and Liu (2003); Kao and Lin (2004); Liu and Chuang (2009); Jorge et al. (2009); Carvalho et al. (2011); Carvalho et al. (2012); Carvalho et al. (2013).

To identify the inputs used in literature, specifically in studies related to checking efficiency in the context of university libraries, and contribute with subsequent studies that intend to develop assessment models in this area, we conducted an analysis of the variables mentioned in Chart 1. Subsequently, the same procedure was performed with the outputs.

The input related to “Library Staff” or employees of assessed libraries is present in most of the analyzed articles, showing their importance to measure efficiency in university libraries. It is known that the larger the number of employees and the better trained the team is, the greater the impact on providing a good service to users of those institutions and on the organization and maintenance of their structures.

Some peculiarities might deserve further attention when the input related to the library staff is used. Chen (1997a), Han and Hong (2002), Reichmann (2004) and Stroobants and Bouckaert (2014) are some of the authors who use the total number of employees as input. On the other hand, Shim and Kantor (1998), Shim and Kantor (1999) and Shim (2003) consider three categories of variables related to the staff: “Number of professional staff”, “Number of support staff” and “Number of full time equivalents of hourly student employees”. Simón de Blas et al. (2007) also considers “Number of student employees” separately from “Library staff”.

Stancheva and Angelova (2004) simultaneously use two inputs that may present some similarity to each other as they are related to the workforce. The authors consider the variables “Staff” and “Wages” in the same set of inputs, the first referring to the total number of employees and the second to the total of their salaries.

Finally, in the article proposed by Kuang et al. (2010) we observed that the authors divide the staff variable in three categories: “Full Time staff number”, “Number of staff with bachelor’s degree or higher” and “Number of staff with medium professional title and above”.

Another input that must be highlighted in this context is the variable related to collections of each library, in other words, the volumes available to users, which represent the collection of each assessed unit. Simón de Blas et al. (2007) divide the variables related to the collection into two categories: “Number of books belonging to the library” and “Number of journals”.

Similarly to the variable “library staff”, the input “library collections” is widely found in selected studies. As they represent a considerable cost to libraries, both on basis of acquisition, allocation and conservation of those materials, we understand that, to be considered efficient, an institution must optimize the number of items in the collection, avoiding excessive and unnecessary expenses.

Besides the physical collection of the library, some studies consider the electronic collection an important variable in the set of inputs. Noh (2011), for example, includes both the inputs “paper books possessed by the library” and “number of e-books and the number of e-journals” in the proposed modeling, which makes a lot of sense when the main objective is to evaluate digital libraries or the use of e-resources.

In addition to the inputs “Collection”, “Volume Held” and those related to volumes maintained by the libraries, it is possible to verify variables related to the “Collection Cost” (Hsieh et al., 2014) and expenditures for acquiring new books and materials as found in the papers proposed by Kuang et al. (2010), Stroobants and Bouckaert (2014). Shim and Kantor (1998), Shim and Kantor (1999), and Shim (2003) include the variable “monographs purchased, in volumes” in the analysis.

The insertion of variables related to the physical space offered to users, size of the library area and surface allows the evaluative methodology to consider the results achieved by the library in relation the available size of the installation, allowing a small advantage to the smaller libraries that perform good results despite their infrastructure limitations. Studies such as Chen (1997a), Chen (1997b), Han and Hong (2002), Jo et al. (2009), and Simon et al. (2011) use this kind of variable in the evaluation process. On the other hand, Osiewalski and Osiewalska (2004), and Kuang et al. (2010) use the input “Seating capacities” as a proxy for the size of the library.

Finally, Kao and Lin (2004) use a unique input called “Size Index” that considers the university size in terms of the students and faculties. The value of this index is set between 0 and 1.

As seen in Chart 1, there are many types of resources used by the authors. Other substantial inputs that can be found in this context are: “Student Enrollment” (Shim; Kantor, 1998; Shim; Kantor, 1999; Shim, 2003); “Fixtures, Computer and devices for using e-resources” (Osiewalski; Osiewalska, 2004; Noh, 2011, 2012); “Budget” (Jo et al., 2009; Noh, 2011; Shahwan; Kaba, 2013).

Also, it should be noted that the variables “Patronage” (Kao; Liu, 2000; Kao; Liu, 2003; Lili, 2008; Liu; Chuang, 2009) and “Normalized Costs” (Saunders, 2002) are indexes used by authors in specific situations as unique inputs. In the first case, “Patronage” represents the size of the university. This input is a weighted sum of the standardized scores of faculties, graduate students, undergraduate students, and extension students in the range of 0 to 1. As mentioned above, Kao and Lin (2004) use a similar variable in their paper. In the second case, “Normalized Costs” are obtained through the sum of normalized staff expenditures, cost of materials and binding costs.

Outputs, as well as inputs, were chosen according to the perspective and objectives set out in each paper. Chart 2 shows the diversity of the output variables involved in the selected studies.

Chart 2
Outputs used in the context of efficiency evaluation in university libraries.

OUTPUTS	Chen (1997a)	Chen (1197b)	Shim and Kantor (1998)	Shim and Kantor (1999)	Kao and Liu (2000)	Han and Hong (2002)	Saunders (2002)	Kao and Liu (2003)	Shim (2003)	Stancheva and Angelova (2004)	Kao and Lin (2004)	Osiewalski and Osiewalka (2004)	Reichman (2004)	Reichmann and Sommersguter-Reichmann (2006)	Simon de Blás, etal. (2007)	Lili (2008)	Jorge et al. (2009)	Jo et al. (2009)	Liu and Chuang (2009)	Reichmann and Sommersguter-Reichmann (2009)	Kuang et al. (2010)	Noh (2011)	Simon et al. (2011)	Carvalho et al. (2011)	Carvalho et al. (2012)	Noh (2012)	Hwang et al. (2012)	Carvalho et al. (2013)	Shahwan and kaba (2013)	Hsieh et al. (2014)	Stroobants and Bouckaert (2014)
Book Circulation / Loan Transactions	X	X	X	X		X	X		X	X		X	X	X			X	X		X	X	X	X	X	X			X	X		X
Reader Visits / Number of users	X	X				X				X							X	X				X		X	X			X
Total interlibrary lending	X	X	X	X					X						X						X		X
Collections / Collection cost					X			X		X	X	X				X	X		X				X	X	X			X	X	X
Registred Readers
Reference Transaction and Online search	X	X	X	X					X																		X
Total interlibrary borrowing			X	X					X						X						X		X
Opening hours							X						X	X							X		X								X
Participation in group presentations or instructions			X	X			X		X			X
Personnnel					X			X			X					X			X
Expenditures					X			X			X					X			X
Buildings (area and seats)					X			X			X					X			X
Services					X			X			X					X			X
Number of serial subscriptions													X	X						X			X
Book materials added													X	X						X									X
Consultations																	X							X	X			X
Number of Web DB sessions / Connections																						X				X	X
Number of documents downloaded																							X			X	X
Number of Reference Questions							X
Interlibrary Loan							X
Interlibrary borrows							X
Home Delivery loans															X
Number of books for lending and borrowing																		X
Full time bachelor and college students																					X
Master and doctor students																					X
Number of web site visits																						X
Number of seats																							X
Number of Web DB hits																										X
Number of e-journal sessions																										X
Number of e-book sessions																										X
Number of e-book loans																										X
Number of internally built e-resource use																										X
Number of other e-resource use																										X
Hours of e-resource device use																										X
Service hour (hours that libraries provide e-services)																														X
Service item																														X

• Source: Prepared by Authors (2017).

Initially, Chart 2 provides thirty-six outputs located in the articles selected. As in Chart 1, the variables could not be identified in three publications. Hence, they do not appear in Chart 2.

The outputs related to “Book Circulation”, such as ‘Total Circulation”, “Loaned Books” and “Borrowed Books”, are present in most of the articles analyzed. They reflect a connection between the efficiency of those assessed institutions and their main service: transferring knowledge to users.

Another form of book circulation used by the authors is the “Transactions among libraries”. Some of the authors prefer to use two different outputs to represent the “Total interlibrary lending” and the “Total interlibrary borrowing” (Shim; Kantor, 1998; Sauders, 2002; Shim, 2003; Simón de Blas et al., 2007; Kuang et al., 2010; Simon et al., 2011), while Chen (1997a, 1997b), for example, uses only the “Interlending Service” to refer to the external circulation of books.

The output for the number of “Registered Readers” also often appears among verified articles. Osiewalski and Osiewalska (2004) use this variable by means of controlling of “library cards holders”. In the model established by these authors, they proposed to increase the number of registered members, along with “Trained users” and “Loan Transaction”, reducing resources related to the structure of the 20 Polish libraries evaluated, such as “Library space”, “Salary” and “Seating capacity”. The authors should note that increasing the number of registered users and reducing the space and number of seats may affect the quality of service. Authors such as Jorge et al. (2009), Carvalho et al. (2011), Carvalho et al. (2012), Carvalho et al. (2013), and Shahwan and Kaba (2013) also use this variable.

The optimization of library operating times also seems to be a concern of some authors. Saunders (2002), Reichmann (2004), Reichmann and Sommersguter-Reichmann (2006), Simon et al. (2011), and Stroobants and Bouckaert (2014) include the output “Weekly Library Service Hours” in the evaluation of the libraries. This variable is considered by establishing a relation between the efficiency of the library and its availability for users to have access to the library services.

Upon analyzing the variables, we also verified the proposal by Kao and Liu (2000, 2003), Kao and Lin (2004), Lili (2008), Liu and Chuang (2009). In those studies, some variables usually used as outputs are treated as inputs, from the point of view of applied metrics in university libraries.

Some of those studies justify the use of variables “Collections”, “Personal” and “Expenditures” as outputs, claiming that the objective is studying the proper allocation of those resources to clients, being required that they be treated as outputs. According to the authors, the consumption of the mentioned resources by libraries reflects a quality service to the user, as opposed to the concept of efficiency. In this case, the purpose is to evaluate whether a library is providing sufficient collections, large and comfortable space and enough librarians for its patrons.

We observed that other variables, which can commonly be verified as inputs, are presented in some cases as outputs. For example, Reichmann (2004) and Reichmann and Sommersguter-Reichmann (2006) consider the variable “Serial Subscriptions” an output, which comprises the number of serials in the narrow sense as well as newspapers, annual reports and other journal-like series for which the library has a subscription in the respective year. In a normal situation, this variable would be equivalent to the input “Collection” or “other materials (such as journals) added”.

It should be noted that the article proposed by Simon et al. (2011), unlike other authors, suggests a three-stage service model, using basic inputs, intermediate outputs, and final outputs to examine the changes in productivity in the relations among the libraries. The variables “Collection”, “Serials”, “Seats” and “Service Hours” were treated as outputs in this paper, but they perform a dual function due to their intermediate variable characteristic. In the others articles they are mostly used as an input.

Table 1 summarizes the frequency of the inputs and outputs found within the analyzed articles. In this case, to condense the information and make it possible to construct a lean graph, we only considered the variables that appeared three or more times.

Table 1
Frequency of the variables found within the analyzed articles.

Variables		Occurrence		Frequency
Inputs
Staff related variables		22		71.0
Book Collection		17		54.8
Area of library space		13		41.9
Acquisition of books and periodicals		8		25.8
Total serials / Number of journals		5		16.1
Student employees		4		12.9
Patronage		4		12.9
Technical Equipment		4		12.9
Monographs purchased		3		9.7
Total student enrollment		3		9.7
Full-time instructional faculty		3		9.7
Wages		3		9.7
Budget		3		9.7
Library operation costs / Expenditure		3		9.7
Web DB		3		9.7
Outputs
Book Circulation / Loan Transactions		22		71.0
Reader Visits / Number of users		10		32.3
Total interlibrary lending		8		25.8
Collections / Collection cost		7		22.6
Registred Readers		7		22.6
Reference Transaction and On-line search		6		19.4
Total interlibrary borrowing		6		19.4
Opening hours		6		19.4
Participation in group presentations or instructions		5		16.1
Personnnel		5		16.1
Expenditures		5		16.1
Buildings (area and seats)		5		16.1
Services		5		16.1
Number of serial subscriptions		4		12.9
Book materials added		4		12.9
Consultations		4		12.9
Number of Web DB sessions / Connections		3		9.7
Number of documents downloaded		3		9.7

• Source: Prepared by Authors (2017).

Efficiency in University Library context

In this section, we will discuss some aspects of the articles listed in Chart 1 and 2, showing how some authors deal with the assessment of efficiency in university libraries.

Reading the selected articles allows us to deduce that data envelopment analysis is the most used tool for efficiency assessment in the context of university libraries. From the 34 selected articles, almost 90% used data envelopment analysis (DEA) to check the efficiency in these institutions. This is because it is a popular technique for evaluating efficiency and for the evaluation of scenarios with multiple inputs and multiple outputs, unlike other efficiency evaluation tools, besides being possible to apply it in many situations and environments.

Data envelopment analysis, is a widely known approach for evaluating the efficiency in decision-making units. This tool is based on a non-parametric approach, using linear programming problems to calculate the efficient frontier based on consumed resources and on the results obtained (Sengupta, 1999).

In general, the concept of efficiency in the selected works refers to the ability of the library to transform its resources in products and services.

There are two classical models in DEA, both widely used in the 34 selected articles. The Banker, Charnes and Cooper model (BCC) proposed in 1984, which assumes variable returns of scale (VRS), represents most of those publications (Saunders, 2002; Reichmann; Sommersguter-Reichmann, 2009; ). The Charnes, Cooper and Rhodes model (CCR), proposed in 1978, which assumes constant returns of scale (CRS), can be seen at the paper by Noh (2011). In this assessment, the author used the CCR model jointly with BCC model to compare the results. In this context, the use of the BCC model is more appropriate, given the characteristics of these institutions.

In addition to classic models, there are applications of DEA using advanced models, such as Fuzzy DEA (Kao; Liu, 2003; Liu; Chuang, 2009) and Assurance Region (Kantor, 1999; Kao; Shim, 2003; Shim; Lin, 2004).

Furthermore, considering the characteristics of an university library, it is possible to observe different ways to apply DEA, specially together with other assessment methods, for instance: Markovian Hypothesis (Jorge et al., 2009; Carvalho et al., 2013), Free Disposal Hull Approach (FDH) (Stroobants; Bouckaert, 2014), Stochastic frontier regression (Saunders, 2002), Malmquist index approach (Reichmann; Sommersguter-Reichmann, 2009; Simon et al., 2011) and Bootstrap method (Simon et al., 2011).

As previously mentioned, besides efficiency analysis, it was possible to verify the diversity of objectives, methods and tools used in these articles. Regarding the timeline, the literature review will contemplate publications between the years of 1997 and 2014.

Chen (1997a, 1997b) can be considered one of the first authors to start the evaluation of efficiency in university libraries. For this, he used data envelopment analysis to evaluate 23 university libraries. These studies demonstrated that DEA is a powerful assessment tool that mathematically estimates the efficiency score by means of a combination of inputs and outputs, helping these libraries reach better performances.

The efficiency evaluation in academic research libraries, measured with DEA, was the subject developed by Shim and Kantor (1998, 1999), Kantor and Saracevic (1999), and Shim (2003). In these studies, the application of the data envelopment analysis tool, besides providing the efficiency score of each library analyzed, indicates the best practices of the group, thereby revealing information on improving operations of some libraries in particular.

In their paper, Shim and Kantor (1999) propose the efficiency assessment of major academic research libraries based on data obtained from the Association Research Libraries (ARL). The patterns of efficiency scores of both individual libraries and libraries in peer groups (private vs. public) were observed. The paper considered the transition from traditional libraries to the use of electronic resources, justifying the use of DEA in the new environment of digital library.

In this context, digital libraries were the subject of a complex study by Kantor and Saracevic (1999), which evaluated 21 different services in 5 major academic research libraries using statistical methods associated with the DEA tool. The study stands out as a model for evaluation of digital libraries focusing on the costs of operations and the impact of the services produced. This paper also used data from the ARL, as well as seen in Saunders (2002), and Reichmann and Sommersguter-Reichmann (2009).

Reichmann (2004) and Reichmann and Sommersguter-Reichmann (2006) evaluated the efficiency in 118 university libraries from German- and English-speaking countries.

Reichmann (2004) observed that, when comparing the efficiency scores among libraries, there were no significant differences between the libraries of English-speaking countries and German-speaking countries, or between small and large libraries. Reichmann and Sommersguter-Reichmann (2006) considered university libraries in countries such as Australia, Austria, Canada, Germany, Switzerland and the United States. There are considerable differences in the efficiency scores of European and non-European academic libraries. The research results also indicate that the frontier of best practices is superior in non-European universities.

The study by Lili (2008) uses data envelopment analysis to measure the efficiency of 10 university libraries in the region of Changsha, China. This work investigated the sensitivity of DEA efficiency to various input-output variable combinations. The results suggest that the DEA approach may be useful to develop comparative analyzes.

Other Asian researches can be found in this context, for example Jo et al. (2009) that evaluates 26 Korean university libraries between 2005 and 2007. In this article, important variables such as “Library Size” and “Staff” were considered.

To the best of our knowledge, there are four articles in Brazilian scenario - Jorge et al. (2009) and Carvalho et al. (2011, 2012, 2013). They analyzed the performance of libraries affiliated to a federal university in Rio de Janeiro, using DEA and markovian analysis. The DEA model allowed the quantification of the efficiency of each library unit, identifying efficiency levels and ranking them accordingly to their operation plans, what helps the managers in their quest for library efficiency.

Among the more recent papers on this topic are those proposed by Hwang et al. (2012), Shahwan and Kaba (2013), Stroobants and Bouckaert (2014).

The efficient use of electronic resources in libraries was the subject of study conducted by Hwang et al. (2012), which aimed to empirically analyze a database acquired by a specific university. According to the results, they realized that information provided by DEA was more diverse and objective than past evaluating methods, allowing a calculation of the maximum value of efficiency of each database, and thus providing some comparisons and suggestions for improvement.

Shahwan and Kaba (2013) focused on the measurement of efficiency in 11 academic libraries in the United Arab Emirates, Qatar, Oman and the Kingdom of Saudi Arabia, member countries of the Gulf Cooperation Council, applying an output-oriented version of the DEA model along with a questionnaire to managers and directors of these libraries. The results support the identification of possible areas for improvement of inefficient libraries. This study is the first attempt to apply the DEA approach to evaluate the performance of academic libraries in this part of the world.

A combination of methods to measure the efficiency of local public libraries in Flanders was proposed in the study by Stroobants and Bouckaert (2014). Based on the results, the use of non-parametric frontier methods can be verified. FDH and DEA were presented as alternative techniques for benchmarking the performance of organizations in relative terms, mainly for benchmarks’ proposition for public services.

Finally, few articles have evaluated the efficiency or the productivity of these institutions using other tools (non-DEA techniques). Osiewalski and Osiewalska (2004) used Bayesian stochastic frontier approach to measure cost efficiency of public and academic libraries in Poland. Kuang et al. (2010) proposes a Partial Least Squares - Support Vector Machine model (PLS-SVM model) to evaluate the library efficiency of 14 colleges and universities. The authors state that the PLM-SVM model is capable of generating consistent and accurate results. Xiaomei (2011) suggests an application of AHP model in Tianjin university library. Noh (2012) proposed a performance measurement model from a brainstorming session with Delphi panelists and experts to evaluate electronic resources in academic libraries in Korea.

Conclusion

This article performs a bibliographical review on efficiency assessment in university libraries. A total of 34 papers were found in two scientific bases and through the search performed on Google Scholar. The search details and procedures were described above. From this review, the scope of most articles was analyzed, highlighting their objectives, characteristics of assessed libraries and some of the results found, showing the different points of view proposed.

The articles show the DEA technique is widely used in this context, being perfectly adequate to efficiency assessment of libraries, including university ones. However, four articles proposed an evaluation using non-DEA techniques, for instance, stochastic frontier, PLS-SVM model, AHP model, and a specific performance measurement model proposed by its authors.

The variables used by each author in the efficiency evaluation were organized in tables, highlighting the inputs “Library Staff” and “Book Collection” and the outputs “Circulation” / “Loan Transactions” and “Readers visits” as the most used. We believe that mapping those variables can help subsequent works, based on the main variables that directly impact the services of university libraries.

To complement the information inserted in those tables, we provided a frequency distribution relative to the inputs and outputs found. These may help visualize a wider range of variables and points of view regarding the efficiency assessment of university libraries.

Other characteristics of 34 selected articles were also analyzed, such as the temporal and regional placement of publication of the works.

For future works, we propose the use of a DEA modeling based on the main variables verified in the present study and a comparative analysis with other studies in the Brazilian scenario.

Contributions

• R.S. Tavares and G.M. Drumond contributed to the conception and design, analysis and interpretation of data and review. L. Angulo Meza and M.P. Méxas contributed to the design, review and approval of the final version of the article.

References

• Banker, R. D.; Charnes, A.; Cooper, W. W. Some models for estimating technical scale inefficiencies in Data Envelopment Analysis. Management Science, v. 30, n. 9, p. 1078-1092, 1984.
• Carvalho, F. A. A. et al Análise envoltória de dados na gestão do desempenho de bibliotecas universitárias: o caso de IFES no Rio de Janeiro. Revista Brasileira de Biblioteconomia e Documentação, v. 7, n. 1, p. 4-21, 2011.
• Carvalho, F. A. et al Análise de eficiência e desempenho no longo prazo: ilustração empírica de um modelo para avaliação de bibliotecas públicas. Investigación Bibliotecológica, v. 27, n. 60, p. 71-95, 2013.
• Carvalho, F. A. et al Library performance management in Rio de Janeiro, Brazil: Applying DEA to a sample of university libraries in 2006-2007. Library Management, v. 36, n. 4-5, p. 297-306, 2012.
• Charnes, A.; Cooper, W. W.; Rhodes, E. Measuring the efficiency of decision-making units. European Journal of Operational Research, v. 2, n. 6, p. 429-444, 1978.
• Chen, T. Y. A measurement of the resource utilization efficiency of universities libraries. International Journal Production Economics, v. 53, n. 1, p. 71-80, 1997a.
• Chen, T. Y. An evaluation of the relative performance of university libraries in Taipei. Asian Libraries, v. 6, n. 1-2, p. 39-50, 1997b.
• Emrouznejad, A. et al Evaluation of research in efficiency and productivity: A survey and analysis of the first 30 years of Scholarly Literature in DEA. Socio-Economic Planning Sciences, v. 42, n. 3, p. 151-157, 2008.
• Farrell, M. J. The measurement of productive efficiency. Journal of the Royal Statistical Society, Series A, Part 3, p. 253-290, 1957.
• Han, D. O.; Hong, B. Y. Efficiency of University libraries in Seoul by DEA. Journal of the Korean Society for Library and Information Science, v. 36, n. 3, p. 275-286, 2002.
• Hsieh, L. F. et al. Cost efficiency and service effectiveness for university e-libraries in Taiwan. The Electronic Library, v. 32, n. 3, p. 308-321, 2014.
• Hwang, M. J. et al. Using Data Envelopment Analysis to Evaluate Library Electronic Databases. Journal of Educational Media and Library Sciences, v. 49, n. 3, p. 343-367, 2012.
• Jo, S. et al An analysis of the relative efficiency for the National University Libraries using DEA Model. Journal of Korean Library and Information Science Society, v. 40, n. 1, p. 253-74, 2009.
• Joo, S.; Lee, J. Y. Measuring the usability of academic digital libraries. The Electronic Library, v. 29, n. 4, p. 523-37, 2011.
• Jorge, M. F. et al Gestão do Desempenho de Organizações Públicas: um estudo empírico sobre bibliotecas universitárias em uma IFES no Rio de Janeiro. In: Encontro da Associação Nacional de Pós-Graduação e Pesquisa em Administração, 33., 2009, São Paulo. Anais.. São Paulo: Enanpad, 2009.
• Kantor, P. B.; Saracevic, T. Quantitative study of the value of research libraries: A foundation for the evaluation of digital libraries. Proceedings of the ASIS Annual Meeting, v. 36, p. 407-419, 1999.
• Kao, C.; Lin, Y. C. Evaluation of the university libraries in Taiwan: Total measure versus ratio measure. Journal of the Operational Research Society, v. 55, p. 1256-1265, 2004.
• Kao, C.; Liu, S. T. Data envelopment analysis with missing data: An application to university libraries in Taiwan. Journal of the Operational Research Society, v. 51, p. 871-905, 2000.
• Kao, C.; Liu, S. T. A mathematical programming approach to fuzzy efficiency ranking. International Journal Production Economics, v. 86, n. 2, p. 145-154, 2003.
• Kuang, J. et al A new model for libraries efficiency evaluation. In: International Conference on Network and Finance Development. Proceedings.. Wuhan, China: NFD, 2010.
• Lili, Z. Measures university libraries efficiency of Changsha Region using DEA. In: International Conference on Logistics Engineering and Supply Chain, 2008. Conference Paper.. Changsha, China: WASET, 2008. p.160-163.
• Lima, R. A. et al Delimitação de uma área multidisciplinar para análise bibliométrica de produção científica: o caso da Bioprospecção. Transinformação, v. 19, n. 2, p. 153-168, 2007. http://dx.doi.org/10.1590/S0103-37862007000200006
  » http://dx.doi.org/10.1590/S0103-37862007000200006
• Liu, S. T.; Chuang, M. Fuzzy efficiency measures in fuzzy DEA/AR with application to university libraries. Expert Systems with Applications, v. 36, n. 2, p. 1105-1113, 2009.
• Noh, Y. Evaluation of the resource utilization efficiency of university libraries using DEA techniques and a proposal of alternative evaluation variables. Library Hi Tech, v. 29, n. 4, p. 697-724, 2011.
• Noh, Y. A study measuring the performance of electronic resources in academic libraries. Aslib Proceedings New Information Perspectives, v. 24, n. 2, p. 134-153, 2012.
• Oliveira, I. R.; Campello, B. S. Estado da arte sobre pesquisa escolar no Brasil. Transinformação, v. 28, n. 2, p. 181-194, 2016. http://dx.doi.org/10.1590/2318-08892016000200005
  » http://dx.doi.org/10.1590/2318-08892016000200005
• Osiewalski, J.; Osiewalska, A. Measuring cost efficiency of public and academic libraries in Poland – a methodological perspective and empirical experience (Keynote paper). In: IATUL Conferences, Krakow, 2004. Proceedings.. Krakow, Poland: Krakow University of Technology, 2004.
• Reichmann, G; Sommersguter-Reichmann, M. Efficiency measures and productivity indexes in the context of university library benchmarking. Applied Economics, v. 42, n. 3, p. 311-323, 2009.
• Reichmann, G. Measuring university library efficiency using data envelopment analysis. Libri, v. 54, n. 2, p. 136-146, 2004.
• Reichmann, G.; Sommersguter-Reichmann, M. University library benchmarking: an international comparison using DEA. International Journal Production Economics, v. 100, n. 1, p. 131-47, 2006.
• Saunders, E. S. Cost efficiency in ARL Academic Libraries. Libraries Research Publications, Paper 8, 2002.
• Sengupta, J. K. A dynamic efficiency model using data envelopment analysis. International Journal of Production Economics, v. 62, n. 3, p. 209-218, 1999.
• Shahwan, T. M.; Kaba, A. Efficiency analysis of GCC academic libraries: an application of Data Envelopment Analysis. Performance Measurement and Metrics, v. 14, n. 3, p. 197-210, 2013.
• Shim, W. Applying DEA Technique to library evaluation in academic research libraries. Library Trends, v. 51, n. 3, p. 312-332, 2003.
• Shim, W.; Kantor, P. B. A novel economic approach to the evaluation of academic research libraries. Proceedings of the ASIS Annual Meeting, v. 35, p. 400-410, 1998.
• Shim, W.; Kantor, P. B. Evaluation of digital libraries: A DEA approach. Proceedings of the ASIS Annual Meeting, v. 36, p. 605-615, 1999.
• Simon, J. et al Changes in productivity of Spanish university libraries. Omega, v. 39, n. 5, p. 578-588, 2011.
• Simón de Blas, C. et al Applicación de la técnica DEA en la medición de la eficiência de lãs bibliotecas de la Universidade Complutense de Madrid. Revista Española de Documentación Científica, v. 30, n. 1, p. 9-23, 2007.
• Soares de Mello, J. C. C. B. et al Curso de Análise Envoltória de Dados. In: Simpósio Brasileiro de Pesquisa Operacional, 37., 2005, Gramado, RS. Anais.. Gramado, RS: Sobrapo, 2005.
• Stancheva, N.; Angelova, V. Measuring the efficiency of university libraries using Data Envelopment Analysis. In: Conference on Professional Information Resources, 10., 2004, Prague. Proceedings.. Prague: Inforum, 2010.
• Stroobants, J.; Bouckaert, G. Benchmarking local public libraries using non-parametric frontier methods: A case study of Flanders. Library and Information Science Research, v. 36, p. 211-224, 2014.
• Xiaomei, H. Research on the performance evaluation of University Library in Tianjin. Energy Procedia, v. 5, p. 1148-1152, 2011..

Publication Dates

• Publication in this collection
  Jan-Apr 2018

History

• Received
  08 Nov 2016
• Reviewed
  24 Apr 2017
• Accepted
  14 May 2017