Composite indicator to evaluate quality of municipal management of primary health care

OBJECTIVE: To develop a composite indicator to evaluate the quality of municipal management of primary health care. METHODS: The evaluation model focuses on aspects of health system management. Fifty-fi ve performance indicators were used and classifi ed according to the criteria of relevance, effectiveness, effi cacy and effi ciency. The measures were aggregated through an additive data envelopment analysis model for measures of value, merit and quality. Data was utilized from 36 municipalities in Santa Catarina State (Southern Brazil), with populations between 10 thousand and 50 thousand residents in 2006. RESULTS: The results are presented as monotonic measures over the interval [0, 1] (score = 1: effi cient; other values: ineffi cient). Five municipalities had a score of 1 in the quality of management for actions promoting access, while eight municipalities received a score of 1 in the quality of management of actions for service provision; the other municipalities were classifi ed as ineffi cient (score < 1) for both dimensions. CONCLUSIONS: The quality of municipal management in primary health care can be evaluated with a composite indicator, constructed through linear programming techniques, which simultaneously considers the criteria of relevance, effectiveness, effi cacy and effi ciency and expresses them as measures of value, merit and quality. DESCRIPTORS: Quality Indicators Health Care. Primary Health Care. Health Management. Municipal Management.


INTRODUCTION
Evaluations have existed since early civilization. 6Their application in public programs increased with the Second World War, due to a need to control the spending of scarce national resources.In Brazil, the fi eld began to be developed in the 1980s. 19 is a challenge to transform the concept of quality evaluation into criteria, indicators and standards that assure validity. 10Sander 17 contributed considerably with evaluation studies applied to the quality of management.He utilized the historical retrospective of administration theory and his infl uence in education in Latin America to highlight the constructs of administration based on efficiency, effi cacy, effectiveness and relevance.These four constructs point to four criteria to evaluate and guide administrative performance.Its theoretical essence is intimately connected to the nature of each construct, which corresponds to the economic, institutional, political and cultural dimensions associated with the respective criteria.Scriven 18 described the concept of an object's quality as dimensions of value and merit: an object has quality when it has value and merit, be it a system, a process or a program.It has value when its resources are well applied to meet the needs of stakeholders; and it has merit when it performs well what it intends to do.An object can have merit and not have value when the manager does not meet the needs of the population of interest.Therefore, all objects without merit do not have value, since resources should not have been spent with effi cacy and effi ciency in order to meet the needs of interested parties.
The combination of the proposals by Sander 17 and Scriven 18 may explain the concept of quality, considering value and merit.These conditions are suffi cient for systems, processes, projects and programs to exhibit quality, and the criteria of effi ciency, effi cacy, effectiveness and relevance are necessary to exhibit quality. 7e of the challenges to evaluate management in health in a deterministic fashion, instead of probabilistic, is fi nding techniques that allow for simultaneous analysis of all the aspects involved.
Data envelopment analysis (DEA) is a widely used method in the study of productivity and technical effi ciency or organizations that utilize multiple inputs and generate multiple products.It allows for the identifi cation of improved practices through the empirical identifi cation of frontiers using linear programming.In recent years, there has been a substantial increase in international publications using DEA for health evaluations. 11,14,15,16,20In Brazil, articles report the use of DEA in economic studies of education and health. 4,8,9,13e quality of municipal management can be expressed by the ability of the manager to take actions that reduce the risk of disease and other harms and that make access universal and equitable for all municipal residents to the actions and services necessary for health promotion, prevention and rehabilitation.This study sought to develop a composite indicator to evaluate the quality of municipal management in primary health care.

METHODS
A methodological study was undertaken to develop an evaluation model focused on the management of the health system through use of DEA and indicators for effi ciency, effi cacy, effectiveness and relevance, consolidated into a composite indicator of quality.The model was tested in small municipalities (from 10 to 50 thousand residents) of Santa Catarina State (Southern Brazil) in the year 2006.
The evaluation matrix considered two dimensions: the management of actions promoting access (intersectoral activities; popular participation; human resources and infrastructure) and the type of actions (external; internal); and the management of actions for service provision (child; adolescent; adult; older adults) and the type of actions (promotion and prevention; diagnosis and treatment).This resulted in 12 sub-areas of analysis, for which indicators were selected that refl ected the criteria used for quality: relevance, effectiveness, effi cacy and effi ciency.The selection was performed based on a search of the literature, workshops to create consensus among specialists and technical experts from the Santa Catarina State Secretary of Health.Indicators and measures were not selected for the criteria of effi ciency in actions of service provision, since this is a fundamental preoccupation of management and not service provision.Indicators were selected for external actions of management for intersectoral activities and popular participation.Eight indicators for each type of action (internal and measured for the four criteria) were selected for the management of human resources and infrastructure (Table 1).
The model evaluated the relative quality of municipal management in three stages: in the fi rst, measures of relevance and effectiveness of management were used to generate a measure of value; in the second, measures of efficacy and efficiency were used to generate measure of merit; and in the third, the measures of value and merit were used to generate a measure of quality.A mathematical algorithm for linear programming was developed to evaluate the performance of the municipal manager compared to the performance of other managers through use of the function-impact performance of the most relevant factors, from the view of the manager.The mathematical algorithm produced variable relative measures in accordance with the manager evaluated.The algorithm used for the aggregation of the measures was applied with Lingo© software (Lindo Systems, Chicago, USA).
The resulting curves for excellent performance were defi ned by the best combinations of value and merit.The curves were denominated "frontiers of observed quality" and considered as excellent the quality observed in municipalities represented by points on the frontier and considered other municipalities as ineffi cient.The algorithm calculated the distance from each point to the frontier of observed quality, and associated a measure that was inversely proportional to the distance for each point, in order to obtain a monotonic and increasing measure for quality over the interval [0,1].The same principal was assumed in the aggregation of measures of relevance and effectiveness to generate the measure of value, as well as in the aggregation of measures of effi cacy and effi ciency to generate the measure of merit.Value rankings were produced to position the municipalities from the sample ("good" management for the 25% best positioned, "poor" for the 25% worst positioned and "normal" for other municipalities between positions from 25% and 75%).
An additive model was utilized for the analysis. 5In the algorithm developed, a municipality was designated as Mun o whose management was simultaneously evaluated for various criteria of performance (Cj,J=1,2,...,J) and the municipalities were associated with measures(Mj,J=1,2,...,J) that were monotonic and increasing over the interval [0,1].
Observed values of 0 ≤ m j ≤ 1 were considered for the measures (Mj,J=1,2,...,J).The management of Mun o can be evaluated in an absolute and relative manner with these values.In the fi rst case, the standards for excellent performance are recognized (mj*,J=1,2,...,J j), and management is considered effi cient when (mj*=mj* j); in other cases, management is considered ineffi cient.In the second case, excellent standards m j * do not exist or are not recognized, and the management of Mun o is evaluated relative to the management of municipalities similar to Mun n , (n=1,2,...,N) considering the combination of the measures (Mj,J=1,2,...,J).
In the mathematical models that utilize DEA to verify of the management of a municipality Mun o is effi cient of ineffi cient, it is assumes that the measures M1,...,Mk assume values m1,...mk such that: Always that: (1) Therefore, the problem in verifying if there exists a Mun n better than Mun o can be resolved by verifying of there exist numbers such that: To verify the existence of such z n , the linear programming problem is solved Sk ≥ 0, k = 1, 2,..., K e Zn ≥ 0, n = 0, 1, 2,..., N; Which maximizes S = When S* > 0, the manager of Mun o is ineffi cient, since s* k > 0 for some k, the observed data demonstrate the possibility that managers can increase the performance of the municipality in one of the criteria without harming performance in another criteria.On the other hand, when S* = 0, management can be considered excellent, since s* k = 0 for all k indicates that the managers cannot increase the performance of this organization in any of the criteria without harming performance in another criteria.The 55 indicators were aggregated by the algorithm developed (Figure ).
Table 1.Indicators and criteria according to the dimension and focus for evaluation of the quality of municipal management in primary health care.

RESULTS
The number of reports generated by applying the mathematical algorithm depended on the desired characteristics of the evaluation.Reports were generated for each aggregation of measures in each type of action, focus and dimension, in addition to partial reports for performance in relevance, effectiveness, effi cacy, effi ciency, value and merit.Table 2 presents measures of relevance, effectiveness, effi cacy, effi ciency, value and merit for each of the types of actions in each focus, as well as the quality measures of municipal management in primary health care, its dimensions and their respective focus, for each municipality in the sample.
The values were presented for each evaluation criteria (relevance, effectiveness, effi cacy and effi ciency) relative to all the other municipalities.The criteria of value resulted from aggregating the measures of effectiveness and relevance; the criteria of merit resulted from aggregating the measures of effi ciency and effi cacy.Value = 1 indicated that the municipality was in the observed frontier for that measure (effi ciency), and the smaller the value, the farther the municipality was from the ideal value of that measure.The quality of actions was expressed as the performance resulting from aggregating value and merit for each action taken -external action, internal action, promotion and prevention, diagnosis and treatment -for each of the four focuses of the two dimensions.Quality indicates the performance resulting from the aggregation of the four focuses of each dimension, allowing for measurement of management performance in actions promoting access and actions for the provision of health services.Aggregation of the latter two measures resulted in the measure of performance for the management of primary health care.
The measures (1.0), (0.0) and (0.5) indicated levels of quality for the municipal management of primary health care, according to the standard of quality adopted for study.The quality of a municipality's management was considered: (i) Good, when represented by the measure (1.0); (ii) Poor for the measure (0.0), and (iii) Normal for the measure (0.5).Measures with (*) indicate that a ranking was not produced for this management action, since it was not included in the analysis (Table 3).
The classifi cation was proposed as a summary alternative for the rankings.In the example presented, the municipality can have good management for intersectoral activities and infrastructure and normal

Ind n Ind n
Quality of management of primary health care (Q of G_ABS) Ind -performance indicator used by the model for aggregation of measures Q da G_SMS: Management Quality of the Municipal Health System: evaluates the quality of management of actions promoting access to health services to residents Q da G_PROV: Management Quality of Provisionment: evaluates the quality of management of actions that provide health services to residents management for popular participation and human resources in the dimension of management for actions promoting access to health services.It also had good management for children and poor management for adolescents, adults and older adults.In comparison with the 35 other municipalities evaluated, this municipality had a normal quality of municipal management in primary health care.Reports were generated for each municipality analyzed.
Excellent values (1.000) were obtained by five municipalities in the quality of management for actions promoting access (Q_SMS) and by eight municipalities in the management of action for provisioning services (Q_PROV); one municipality obtained a value (1.000) for the quality of management in primary health care (Q_GABS) and tem (28%) obtained values above 0.900.
The lowest value obtained was for human resources (0.219 in municipality 31).The lowest mean for values was observed for popular participation and the highest for children (Table 4).

DISCUSSION
This is one of few studies that focuses on the evaluation of the performance and provision of primary health care.
The proposed model uses a mathematical algorithm as an alternative to construct a composite indicator that allows for identifi cation of potential areas for improvement in the overall performance of municipal management in primary health care (Sint_G_ABS).The evaluation model and its results point to large differences in the quality of services performed in the health sector.4).This suggests that management in health in these municipalities occurs through the prioritization of some types of actions in detriment to others, and this prioritization varies within the municipalities analyzed.
Evaluation studies of management effi ciency in health care through the services of general surgery, ophthalmology and orthopedic trauma surgery in 22 Valencian hospitals 3 (Eastern Spain) utilized a non-parametric DEA approach and discriminant analyses to show the effectiveness of the DEA model to classify health services as effi cient or ineffi cient.The study adopted a scale of 0 to 1, with a value of 1 considered effi cient and a value less than 1 as ineffi cient.The same procedure was adopted in the model presented here and resulted in fi ve municipalities with an excellent value (1) in the management of actions promoting access, eight in the management of actions of service provision and only one municipality when considering the two dimensions simultaneously.The result suggests that actions of service provisioning continue to be prioritized in Table 3. Levels of quality of municipal management in primary health care according to the criteria, the action, the focus and the dimension evaluated.Santa Catarina State, Southern Brazil, 2006.The same can be observed in the opposite direction, except for municipality 20 (effi cient); all municipalities with a composite indicator of 1 in actions of service provisioning have measures below the third quartile of management actions for access.This indicates that the manager makes decisions between prioritizing actions the municipalities analyzed, which does not necessarily signify a management that guarantees improved services.All other occurrences where management actions promoting access had a composite indicator of 1, the value generated for management actions of service provisioning are below the third quartile.Another possibility of the model developed by this study is the identifi cation of ineffi cient areas relative to other municipalities, similar to other studies. 9The distance between the values for each municipality relative to the reference point for excellence allows for verifi cation of where and how much the municipality could improve its situation relative to other similar municipalities.For the 36 municipalities, the distances to the frontier were 8,472 (0.235/municipality) for actions promoting access, 3,382 (0.094/municipality) for actions of service provision and 5.92 (0.164/municipality) for the management of primary care.Again the analysis identifi ed prioritization of actions of service provision by municipalities.

Quality of Management Actions for
The use of the model resulted in 22.5% of composite indicators that measured the quality of items evaluated with values above 0.900, a level similar to the evaluation of technical effi ciency of 89 health centers in Ghana. 1 The DEA approach was utilized in the evaluation of productive effi ciency in Brazilian hospitals 4,12 and of public spending in health. 8This indicates the potential of this approach in health evaluations, but also demonstrates that greater importance has been given to the identifi cation of technical ineffi ciency.
The results indicated one municipality with an excellent quality of management in primary health care; 27.8% of municipalities presented values > 0.900 in performance and none presented performance <0.678.The item with the best average performance was "children" with an average of 0.870, traditionally a priority in all health systems, and the worst performance was for popular participation with an average of 0.553, which appears to be less prioritized in small municipalities.
Two other types of studies could be conducted: one including factors not controlled by municipal managers (social, economic and environmental factors), which impact the results of primary health care and affect municipal management; and the other can utilize more complex DEA models (invariant models, with two phases), which allow for more robust comparison between the management of primary health care in municipalities with different characteristics.
The study is supported by other international applications and publications in the fi eld.The study allowed for evaluation of management quality in primary health care in municipalities with small populations, through use of DEA approach.The evaluation used multiple performance indicators also utilized by the Ministry of Health and aggregated them by dimensions, types of actions and focus of activities in the primary health care of municipalities.The results of this evaluation process were grouped according to multiple criteria of performance that refl ect the capacity of the municipal health manager to allocate resources to meet the needs of health promotion, prevention and rehabilitation in their municipalities.
basic sanitation Qualifi cation of providers for working in the Family Health Strategy 4 Obeyance of Constitutional Amendment Nº 29 Turnover of providers in their functions Popular Participation (Evaluation Focus) Infrastructure (Evaluation Focus) 1 Participation of society in establishment of priorities for primary care Quality of working conditions 2 Participation of social entities Access to health services 3 Participation of client representatives in the Municipal Health Council Suffi ciency of infrastructure to meet the needs of the municipal health secretary 4 Participation of client representative in seminars and meetings for community control Productivity of the system To be Continued

*
indicators were not identifi ed for the criteria or focus Q_ Quality

Table 1
Model proposed for the aggregation of measures to evaluate the quality of municipal management in primary health care. Figure.

Table 2 .
Quality measures of municipal management in primary health care according to the criteria, to the action, to the focus and to the dimension evaluated.Santa Catarina State, Southern Brazil, 2006.

Table 4 .
Composite indicators for each type of action, of dimension and of management of primary health care.Santa Catarina State, Southern Brazil, 2006.Management of Primary Health Care, X: mean of the values that guarantee access and/or service provisioning, and the situation results where only one municipality has a value of one for the composite quality indicator.