Data envelopment analysis for evaluating public hospitals in Brazilian state capitals

Abstracts

OBJETIVO: Aplicar a metodologia de Análise Envoltória de Dados na avaliação do desempenho de hospitais públicos em termos das internações em suas clínicas médicas. MÉTODOS: A eficiência dos hospitais foi medida a partir do desempenho de unidades de decisão nas variáveis estudadas para cada hospital, no ano 2000. Foram analisados dados referentes às internações em clínica médica dos hospitais SUS das capitais estaduais do Brasil e Distrito Federal (taxas de mortalidade e tempo médio de internação, valor médio da internação e perfil de doenças). A técnica de análise de correlação canônica foi introduzida na restrição do intervalo de variação das variáveis. O modelo Constant Returns to Scale foi utilizado para gerar escores que permitissem avaliar a eficiência das unidades. A partir dos escores obtidos, os municípios foram classificados de acordo com seu desempenho relativo nas variáveis analisadas. Procurou-se correlação entre os escores de classificação com variáveis exógenas: despesas com programas de saúde básica por habitante e índice de desenvolvimento humano das capitais. RESULTADOS: Nos hospitais estudados se destacaram as doenças do aparelho circulatório (23,6% das internações); e a taxa de mortalidade foi de 10,3% das internações. Das 27 capitais, quatro alcançaram 100% de eficiência (Palmas, Macapá, Teresina e Goiânia), sete ficaram entre 85% e 100%, dez entre 70% e 85% e dez com menos de 70%. CONCLUSÕES: A ferramenta utilizada mostrou ser aplicável para a avaliação de desempenho de hospitais públicos, revelando a grande variabilidade entre as capitais brasileiras, no que se refere às internações em clínica médica.

Sistema Único de Saúde; Avaliação de serviços de saúde; Serviços hospitalares; Eficiência organizacional; Sistemas de informação; Análise de dados

ORIGINAL ARTICLES

Data envelopment analysis for evaluating public hospitals in Brazilian state capitals

Antonio C Gonçalves^I; Cláudio P Noronha^I; Marcos PE Lins^II; Renan MVR Almeida^II

^ICoordenação de Indicadores Gerenciais. Secretaria Municipal da Saúde. Rio de Janeiro, RJ, Brasil

^IIPrograma de Engenharia da Produção. Coordenação dos Programas de Pós-graduação de Engenharia (COPPE). Universidade Federal do Rio de Janeiro. Rio de Janeiro, RJ, Brasil

^{Correspondence}

ABSTRACT

OBJECTIVE: To apply the Data Envelopment Analysis (DEA) methodology for evaluating the performance of public hospitals, in terms of clinical medical admissions.

METHODS: The efficiency of the hospitals was measured according to the performance of decision-making units in relation to the variables studied for each hospital, in the year 2000. Data relating to clinical medical admissions in hospitals within the public system in Brazilian state capitals and Federal District (mortality rate, mean length of stay, mean cost of stay and disease profile) were analyzed. The canonical correlation analysis technique was introduced to restrict the variation range of the variables used. The constant returns to scale model was used to generate scores that would enable assessment of the efficiency of the units. From the scores obtained, these cities were classified according to their relative performance in the variables analyzed. It was sought to correlate between the classification scores and the exogenous variables of the expenditure on primary care programs per inhabitant and the human development index for each state capital.

RESULTS: In the hospitals studied, circulatory diseases were the most prevalent (23.6% of admissions), and the mortality rate was 10.3% of admissions. Among the 27 state capitals, four reached 100% efficiency (Palmas, Macapá, Teresina and Goiânia), seven were between 85 and 100%, ten were between 70 and 85% and ten had efficiency of less than 70%.

CONCLUSIONS: The tool utilized was shown to be applicable for evaluating the performance of public hospitals. It revealed large variations among the Brazilian state capitals in relation to clinical medical admissions.

Keywords: National Health System (BR). Health services evaluation. Hospital services. Efficiency, organizational. Information systems. Data analysis.

INTRODUCTION

Data envelopment analysis (DEA), which was introduced by Charnes et al³ in 1978 and extended by Banker et al¹ (1984), provides a representation of the structure formed by decision-making units (DMUs), with inputs and outputs that are defined in such a way as to be able to assess the relative efficiency of these DMUs. This efficiency is defined from the observed performance of the DMUs in relation to the variables analyzed. It is an empirical measurement and not a theoretical or conceptual reference.^10,11 This means that its scores are a comparison measurement that is more appropriate than the more commonly used indicators (e.g. number of procedures per time period or mortality rates), which may be highly dependent on the specific characteristics of a population.

This method establishes a "common region" on the basis of the data (variables) of the DMUs, thereby creating an efficiency index that reflects the importance of each variable for each DMU. Thus, in the common region, units with behavioral patterns that are most optimized for these variables are sought. The maximum value for this index (in each DMU) is then assumed to be an "empirical maximum" efficiency, from which a relative classification of the units becomes possible.¹⁰ From this, the method also provides "excellent" values that the variables should attain, for the DMU to be able to move from "inefficient" to "efficient". DEA has recently been used in the health sector for establishing reference standards for hospitals, clinics or health services, particularly in developing countries.^{3,4,7,9,12,13} In Brazil, one of the rare studies using this methodology was carried out in 2001, to compare university and general hospitals in the municipality of Rio de Janeiro.¹¹

In December 2000, there were 913 hospitals available to the Brazilian national health system (Sistema Único de Saúde – SUS) in the country's state capitals (public, university and philanthropic hospitals and those available through access agreements). During that year, these hospitals were responsible for 742,833 admissions relating to clinical medicine. Methodologies that allow assessment of these hospitals' performance urgently need to be developed, both because of the scarcity of resources in the health sector and because users demand and have a right to a system with quality services.¹⁴

The objective of the present study was to apply DEA in studying the efficiency of the a hospital network, using the SUS hospitals in Brazilian state capitals as an example.

METHODS

The database was formed by admissions to SUS hospitals in the country's state capitals in 2000, and the data were obtained from the SUS hospital information system (Datasus).¹ The DEA was performed using the Frontier Analyst Professional software.² The canonical weights, canonical correlation, restriction intervals for the weights of the variables and the other statistical procedures were generated in the Statistica software.

To comparatively assess the efficiency of the SUS hospitals in the Brazilian state capitals, their admissions in the clinical medical category were analyzed. In addition to clinical medical admissions in the strict sense, admissions in other clinical sub-specialties were also included, such as: cardiology, endocrinology, clinical oncology, infectology and pneumology. The following variables were used:

Inputs: mortality rate (mortality) and mean length of stay in hospital (mean length of stay).

Outputs: percentages of admissions relating to the three chapters of the International Classification of Diseases (ICD) with the greatest mortality percentages, respectively: neoplasias; infectious and parasitic diseases (IPD) and diseases of the circulatory system (circulatory); and mean value paid through the Hospital Admission Authorization (mean HAA).

The DEA used the Constant Returns to Scale (CRS) model, in which efficiency was defined as the ratio of the weighted sum of inputs and outputs, and the objective of the method was to maximize this ratio for each DMU. A unit (capital) that obtained the maximum value for this maximization (1, by definition) was considered to be "efficient" and if not, it was said to be "inefficient" "inefficient" (^Annex).

Firstly, a canonical correlation analysis between the input and output variables was used to identify restriction intervals for the weights of these variables that were needed for DEA (^Annex).2,8,³ Next, the scores thus obtained were correlated (Pearson's coefficient) with the exogenous variables "per capita expense of primary healthcare programs" and "human development index (HDI) for the cities studied", for the year 2000.⁴

RESULTS

Diseases of the circulatory system were prominent, accounting for 23.6% of the admissions in the hospitals studied, with a range from 28.7% in Cuiabá to 8.9% in Macapá. The infectious and parasitic diseases group, which included AIDS and tuberculosis, corresponded to 9.9% of the admissions (maximum of 18.7% in Manaus and minimum of 5.9% in Brasília). The neoplasia group represented 7.5% (maximum of 19.3% in Belo Horizonte and minimum of 0.3% in Aracaju) (Table 1). These three groups totaled 41% of all the admissions within the system.

The mortality rate was 10.3% of the admissions (maximum of 17.6% in Natal and minimum of 4.1% in Macapá). The mean length of stay was 8.8 days (maximum of 12.6 in Rio de Janeiro and Florianópolis, and minimum of 4.7 in Palmas). The mean amount for admission reimbursements via HAAs was R$ 405.34 for all the admissions (maximum of R$ 542.23 in Campo Grande and minimum of R$ 207.90 in Macapá).

Table 2 summarizes the results from the canonical correlation analysis (canonical weights, canonical correlation coefficients and restriction intervals for the weights of the variables). Table 3 shows the classification of the state capitals according to the efficiency attained using DEA, the observed values and the estimated values for minimization of the inputs. Among the 27 state capitals, four achieved 100% efficiency (Palmas, Macapá, Teresina and Goiânia), seven were between 85% and 100%, ten were between 70% and 85% and ten presented less than 70%. Table 3 shows the estimated values for the inputs needed for each capital to achieve 100% efficiency. For example, Rio de Janeiro (66.5% efficiency) has observed values for mortality and mean length of stay of 16.0% and 12.6 days, respectively. In this case, for the city to achieve 100% efficiency, it would be necessary to reduce these rates to the levels of 7.6% and 8.9 days, respectively.

No linear correlation was found between the classification scores and the municipal HDI values (r=0.03; p>0.05), or between the classification scores and expenses per capita (r=0.03; p>0.05).

DISCUSSION

Contrary to other studies that utilized the DEA methodology in the field of health sector assessment in Brazil, the present study was restricted to one specific specialty (clinical medicine) and did not cover hospitals as a whole. It was thus sought to ensure that comparisons were made between entities with intrinsically greater homogeneity. For this, classical indicators were used, such as length of stay and mortality rate, and the admissions relating to the three chapters of the ICD with greatest weight in the system.

In the Brazilian public health system, admissions to hospitals in the system are paid for through HAAs. The amounts of these payments depend on the services provided, the technological backup and the materials used, excluding salaries and infrastructure expenditure. In defining the DEA model, the disease profile and the mean amount of the HAA payments were taken to be "fixed", since they represent real demands from affections that are prevalent among the population and the hospital resources at a given time.

Contrary to what is commonly done in developing causal models, in the present study the mortality variable was used as an input to the system, because of the differentiated characteristics of the methodology used. In DEA, the groups of variables called "inputs" and "outputs" are used to generate the factor that is the great differential of the method, i.e. the classification scores resulting from the minimization of the inputs or the maximization of the outputs. In the present study, the form that is considered most natural was used, i.e. minimizing the inputs "mortality rate" and "length of stay". Nonetheless, no methodological or interpretative difference would arise if these were used as outputs. Thus, if the inputs had been considered to be outputs, and vice versa, and the analysis had been undertaken such that outputs were maximized, the same hierarchical classification (the scores) would have been obtained, without introducing any alterations of logic in the results obtained.

The mathematical structure of DEA models often means that a DMU is considered to be efficient because zero weight is attributed to variables that are then disregarded in evaluating the unit. Defining restrictions from the canonical weights,⁸ as introduced in the present study not only allows the importance of the variables for DEA to be evaluated, but also minimizes the quantity of variables with zero weight. This is an important methodological step, because it avoids rejecting variables that may be relevant in the process of forming the efficiency scores. In the original concept for DEA models (in economics) only "desirable" outputs were considered, i.e. those for which maximization is of interest (for example, maximizing production while considering fixed supplies).⁶ In the present study, the percentages of admissions relating to the three ICD chapters of greatest weight and the mean amounts of HAA payments were considered to be outputs, and the inputs (to be minimized) were the mortality rate and mean length of stay. The mean amount of HAA payments was used as a "proxy" for the complexity of the procedures carried out, and this made it possible to reject the hypothesis that the results had been influenced by the differentiated levels of complexity of these procedures.

Some studies⁹ have used DEA to perform economic assessments on health care units. The present study, however, was not concerned with economic performance, which in any case depends on parameters that are difficult to measure in developing countries.¹³ Thus, the central idea in applying it was to classify the performance of the state capitals in relation to the mortality rate and the mean length of stay, from fixed values for the input variables. From this, the model described was applied, in which the aim was to minimize inputs, i.e. to answer the question of what proportional reduction in the inputs (mortality rate and mean length of stay) it was possible to achieve for a set of hospitals in one state capital while still maintaining the observed disease profile and the mean amounts of HAA reimbursements. The units (capitals) for which it was not possible to reduce the variables were considered to be efficient in comparison with the others, thus generating efficiency scores.

The canonical correlation indicated that there was greater dependence between the variables "mean length of stay" (-0.724) and "neoplasia" (-0.656), which had the highest canonical coefficients among the variables analyzed (Table 2). Thus, it is inferred that, among the population studied, this was the group of diseases with the greatest impact on the patients' length of stay. This corroborates the hypothesis that neoplasias generally require greater length of stay, particularly regarding surgical conditions, and moreover, it shows that the same HAA procedure requires a longer stay if associated with a neoplasia group.

Using the scores generated by DEA, it could be seen that 16 state capitals were operating at less than 75% relative efficiency. The four cities identified as "100% efficiency" (Palmas, Macapá, Teresina and Goiânia) were not among the states with greatest per capita gross domestic product (GDP) or in which the country's major technological and educational centers are located. This indicates that, for the municipalities studied, significant performance gains are still possible with the existing supplies. This observation is reinforced by the independence between the classification scores and the variables "per capita expense on primary healthcare programs" and "HDI of the capitals". For example, the city of Macapá has one of the worst HDI and per capita expenses among the set of municipalities studied, but was classified as an "efficient unit". The HDI combines schooling, income and longevity data and is widely used as a quality-of-life indicator.

The capitals identified as having the worst performance had the most complex characteristics, and they included cities with a tradition of training healthcare human resources and other cities that, similar to the ones with the best performance, were distant from the country's main technological and educational centers.

One of the most important features of the methodology presented is it compares efficiencies while taking real functional conditions into consideration. Moreover, one original characteristic of the present study is the definition of weight limits for the variables, without the need for a decision-maker to intervene, since the restriction intervals were obtained from characteristics of the classification variables themselves (the inputs and outputs). The estimates for the mortality rate and mean length of stay may help health administrators by being a comparative reference point for clinical medicine indicators.

On the other hand, the work to improve these indicators does not dispense with identifying the intrinsic features of the units studied or other evaluations. For example, qualitative satisfaction surveys on the population attended may serve as parameters for demarcating the results. It is unlikely that any single reason for the relative positions of the state capitals will be identified, but the tool presented is a powerful and simple method for ranking performance, thereby opening the doors to more particular studies. Thus, the approach presented in this study is important and independent, and it provides managers with relevant information for wide-ranging evaluations of the system.

REFERENCES

Received: 9/8/2005

Reviewed: 8/1/2006

Approved: 2/7/2007

Canonical correlation analysis and CRS model

I - Canonical correlation analysis (CCA)

CCA was developed by Hotelling in 1936. It studies linear relationships between two groups of variables (a and b), and its fundamental concern is to find the pair of linear combinations of a and b that has the maximum linear correlation.^3,11 From the scheme shown in the Table, the linear combination of the variables a and b is defined as:

The coefficients V_i,i=1,....,m and U_r,r=1,....,s must be such that the square of the correlation between z and w, r² (z,w), presents its maximum value. It is assumed that the variables of the two groups are linearly independent, i.e. the rank X_mxn=m and the rank Y_sxn=s.A_1nxn are the orthogonal projectors of w_nx1 and A_2nxn is the orthogonal projector of z_nx1, i.e. A₁ projects w in the subspace Z and vice versa. The vector w must be collinear with the orthogonal projection of z in W (the vector that makes a minimum angle with z).

This condition is expressed as:

A₂z = rw

In which r = cos(z,w) and A₂ is the orthogonal projection operator in W.

Likewise:

A₁ w = rz

From this, the following can be deduced:

A₁ A₂z = r² z and A₂A₁w = r²w

In which l₁ = r² = cos²(z,w)

Consequently, z and w are respectively eigenvectors of the operators A₁A₂ and A₂A₁ that are associated with the greatest eigenvalue l₁ which is equal to its cosine squared (its squared correlation).

After appropriate algebraic operations, and assuming that A₂ can be inverted, the canonical variables z and w can be written in the following form:

Likewise it can be deduced that:

The canonical variables are the eigenvectors of A₁A₂ (A₂A₁) , which are associated with the eigenvalues ranked in decreasing order. At each stage, a pair of variables associated with the greatest eigenvalue () is generated. The interest is in the canonical weights of the variables from the first stage (greatest correlation), which are used in the proportions:

The values of these weights indicate the importance of each variable in obtaining the maximum correlation between the combinations, and can thus be utilized to generate restriction intervals for the inputs and outputs in a DEA model. The matrixes A₁ and A₂ and the canonical weights are obtained by:

That is, V(m x 1) and U(s x 1) are deduced from each other by linear transformation, such that D(n x n) is a diagonal weighting matrix of the variables.

II - Constant Returns to Scale (CRS) model

In the case of a unit with a single input-output pair, the efficiency of the unit can be defined simply as the output/input ratio. In the case of several inputs and/or outputs, the efficiency is the ratio between the weighted sum of the outputs and the weighted sum of the inputs, and the following is a measurement of this efficiency:¹⁵

(additional restrictions for the weights, in accordance with the output and input levels of the j^th DMU, respectively).

U_r, V_i >= 0, r=1, .... s and i=1, .... m are the weights (multipliers) to be determined and e y_rj, x_ij >=0 are the outputs and inputs known from the j^th DMU. The limits are obtained a priori, by substituting the canonical weights of the inputs and outputs in the above proportions, and they generate a value for each DMU. Consequently, there is a set of n values for each variable, and the minimum and maximum for each set define the limits and importance of each variable in the DEA, without direct interference from a decision-maker.

The Figure illustrates the optimum input values that would turn an inefficient unit into an efficient one, according to this definition. In this particular case, points A, B, C and E correspond to inefficient units. Point D is an efficient unit, situated on the straight line that represents the efficient CRS frontier. The displacement of A to the efficient frontier (point P) implies the optimum input value that would make this unit efficient.

• 1. Banker RD, Charnes A, Cooper WW. Some models for estimating technical and scale inefficiencies in Data Envelopment Analysis. Management Science.1984;30(9):1078-92.
• 2. Bouroche JM, Saporta G. Análise de Dados. Rio de Janeiro: Zahar Editores; 1982.
• 3. Charnes A, Cooper WW, Rhodes E. Measuring the efficiency of decision-making units. Eur J Oper Res 1978;2(6):429-444.
• 4. Chilingerian JA. Data Envelopment Analysis. Boston: Kluwer Academic Publishers; 1996.
• 5. Dexter F, O'Neill L. Data Envelopment Analysis to determine by how much hospitals can increase elective inpatient surgical workload for each specialty. Anesth Analg. 2004; 99(5):1492-500.
• 6. Färe R, Grosskopf S. New directions: efficiency and productivity. Boston: Kulwer Academy Publishers; 2004.
• 7. Felder S, Schmitt H. Data Envelopment Analysis based bonus payments. Theory and application to inpatient care in the German state of Saxony-Anhalt. Eur J Health Econ.2004;5(4):357-63.
• 8. Friedman L, Sinuany-Stern Z. Scaling units via the canonical correlation analysis in the DEA context. European J Operational Res. 1997;100(3):62937.
• 9. Kirigia JM, Emrouznejad A, Sambo LG, Munguti N, Liambila W. Using Data Envelopment Analysis to measure the technical efficiency of public health centers in Kenya. J Med Syst. 2004;28(2):155-66.
• 10. Lins MPE, Meza LA. Análise Envoltória de Dados e Perspectivas de Integração no Ambiente de Apoio à Decisão. Rio de Janeiro: COPPE/UFRJ; 2000.
• 11. Marinho A. Estudo de eficiência em hospitais públicos e privados com a geração de rankings. Rev Adm Publica.1998;32(6):145-158.
• 12. Retzlaff-Roberts D, Chang CF, Rubin RM. Technical efficiency in the use of health care resources: a comparison of OECD countries. Health Policy 2004;69(1):55-72.
• 13. Valdmanis V, Kumanarayake L, Lertiendumrong J. Capacity in Thai public hospitals and the production of care for poor and nonpoor patients. Health Serv Res. 2004;39(6Pt 2):2117-34.
• 14. Viacava F, Almeida C, Caetano R Fausto M, Macinko J, Martins M, et al. Uma metodologia de avaliação do desempenho do sistema de saúde brasileiro. Cienc Saude Coletiva. 2004;9(3):711-724.

Annex

1

Database of the Brazilian national health system [homepage on the Internet]. Brasília; 2005. Available at:

http://tabnet.datasus.gov.br/ tabnet/tabnet.htm#AssistSaude [Accessed on March 3, 2005]

2

Banxia Frontier Analyst Professional. Glasgow: Banxia Holdings Limited; 1998.

3

Lins MPE, Gonçalves AC, Gomes EG, Silva ACM. Performance assessment of dental clinics through PC-oriented Data Envelopment Analysis. Accessibility and quality of health services. In: Proceedings of the 28th Meeting of the EURO Working Group Operational Research Applied to Health Services; 2004; Frankfurt, Germany. Frankfurt: Peter Lang Publishing Group; 2004. v. p. 95-109.

4

Fundação Nacional de Saúde. Consulta de pagamentos. Brasília; 2005. Available at:

http://www.fns.saude.gov.br/consultafundoafundo.asp [Accessed on March 3, 2005]

Publication Dates

• Publication in this collection
  29 Mar 2007
• Date of issue
  June 2007

History

• Accepted
  07 Dec 2007
• Received
  08 Sept 2005
• Reviewed
  01 Aug 2006