SciELO - Scientific Electronic Library Online

 
vol.25 issue2National Health Promotion Policy the dilemmas of autonomyReflections on the role of health economics units regarding national health care systems author indexsubject indexarticles search
Home Pagealphabetic serial listing  

Services on Demand

Journal

Article

Indicators

Related links

Share


Saúde e Sociedade

Print version ISSN 0104-1290On-line version ISSN 1984-0470

Saude soc. vol.25 no.2 São Paulo Apr./June 2016

http://dx.doi.org/10.1590/S0104-12902016144614 

Articles

Performance evaluation of health systems and management in the Brazilian public administration

Leonardo Carnut1 

Paulo Capel Narvai2 

1Universidade de Pernambuco. Recife, PE, Brazil. E-mail: leonardo.carnut@upe.br

2Universidade de São Paulo. Faculdade de Saúde Pública. São Paulo, SP, Brazil E-mail: pcnarvai@usp.br

Abstract

The logic of the new public administration considers health services as non-state activities, and therefore executable by private entities. In this scenario, the control function reaches the proscenium in the discussion about the consolidation of the Regulatory State. Thus, this study used the narrative review technique, described by Rothers, to create the web of an initial critique regarding the content that the administrative control of the State has been outlining in the health sector. The focus was on the direct way of contracting results, expressed in the performance evaluation of health systems. As a counterargument, we turned our attention to the complexity of the field, both from the point of view of the intense disagreement regarding key terms, as well as the methodological problems in the study of health systems. Furthermore, we have comments about the acting perspective that goes beyond the control function and the need for the scientific community to refocus on the topic.

Keywords: Health System; Health Evaluation; State; Public Health; Unified Health System

Introduction

The transformations of the contemporary State have led, in the West, to the hegemony of the so-called Regulatory State, whose influence has spread throughout the world, in the historical context of globalization. With the notable exception of Cuba, we have a prevalence of this type of State, shaped by the capitalist production relationships and consolidated in the historical period of deep crisis (first decades of the 21st century) from the neoliberal State model used by the United States of America and Europe, with effects even in China - which, under a one-party political regime, has slowly, gradually, and steadily been opening up to private business segments, so they are able to undertake and produce under State control. In Brazil, this State feature (in addition to its effects on the economic area) resonates in important ways also on the processes of defining and implementing social policies, including public health (Cherchiglia; Dallari, 1999).

The assumption, in this kind of Regulatory State, is that public institutions are essentially responsible for making decisions about health policies and programs and implementing the relevant actions only to areas of expertise that cannot be delegated, such as the areas of epidemiological surveillance and health. The general instrument created by the State to regulate the actions developed by the private sector, with or without a lucrative purpose, is the regulatory agency. Regarding public health, the Brazilian Unified Health System (SUS) coexists with two regulatory agencies, the National Supplementary Health Agency (ANS) and the Anvisa (National Health Surveillance Agency), and with 27 state departments, and about 5,600 offices and municipal health departments, which operate in the health sector as regulators, since, performing part of the services under direct management, maintain and expand - in a rapid pace - partnerships of various formats with social organizations and private companies, seeking to provide health actions and services, and thus assume only a regulatory role in the health institutions of the country.

In this context, the role of the planning process (including the control and evaluation functions) comes back into focus, both in Brazil and abroad, especially regarding the exercise of State activities in general, and health activities in particular. Admitting the thesis, victorious in most Western countries, that the State must not provide direct assistance to the health of its population, being responsible instead for funding it and establishing rules and conditions for it to be executed by private parties, are the result of the strategic importance of defining resources to perform these actions, their allocation, those that are able to manage it, how the workers of this sector can participate in the management of work processes, since they are main agents, and also how to define what should - and should not - be done in health, how to regulate public control of these resources, and other critical aspects of the health systems, within democratic regimes like Brazil. The country has, by the way, specific legislation (Law 8.142/1990) on the control that society should exert on public health policies, through conferences and health councils, among others.

In scenarios similar to Brazil's, where the State health sector is responsible for the execution of certain activities and services, while coexisting with actions taken by private parties with (supplementary health) and without (supplementary health) intent for profit, the need for the State to control investments and the efficient use of these resources is highlighted. The exercise of this activity (control function) by the State implies, among other aspects, the assessment of the performance of all the organizations that participate in the health system, in addition to the state institutions. In Brazil, despite the consensus among analysts regarding the chronic underfunding of the system, the assertion that there is no lack of resources, but a mismanagement in using these resources is recurrent (Nishijima; Biasoto-Júnior, 2013). Public opinion, social movements, political and trade union leaders have all been pressuring for the "government" to "evaluate the performance" of the health system and "take the necessary steps to improve it", demanding also "FIFA-standard hospitals" (an analogy to the quality of football stadiums imposed to the Brazilian government by the International Football Federation, during Brazil's Football World Cup in 2014).

In response to these expectations, a new management paradigm of public resources emerged in 1990 (Behn, 1998), in which the "results" appeared as a way to solve the "problem" of a complex public, bureaucratic and procedural administration (Garces; Silveira, 2002): it is the new public management, public administration, public management administration or managerial paradigm. Although there are some questions regarding the power of the managerial paradigm to ensure all is provided (Secchi, 2009), the new public administration has been growing roots in Brazil and abroad. With this consolidation, the importance of performance evaluation increases as an indispensable instrument for the exercise of the function control by State institutions.

The identification of the five typical functions of any level administrator in any organization is attributed to Jules Henri Fayol, considered the founder of the Classical Theory of Management, including: predict and plan; organize; command; coordinate; and control, which is defined as the permanent verification of compliance with the established standards and rules (Fayol, 1990). This article focuses on the control function, in the specific context of health system performance, since the health sector has been following the general trend in management administration, to use management indicators, performance indicators and measurement programs of quality (Andrade, 2012; Brasil, 2011) (although some studies have considered the polysemy of these terms and methodological obstacles to capture them) (Samico et al., 2010). We seek, in this organizational scenario, to check the respective performance evaluation model for the conditions that allows it to capture the complexity of these services, since evaluation is essential to the control function, though evaluating the production of screws is quite different than evaluating the performance of service systems, such as health and education, restricting this to only two examples regarding the exercise of social rights.

Thus, when it comes to complex objects such as the measurement of the performance of health systems (Roemer, 1991; Lobato; Giovanella, 2012), the problem occurs not only in the paradigmatic-conceptual dimension, but also in the methodological base (Hoffman, 2012). The very understanding of what is a "health system", for this evaluative purpose, requires a specific methodological approach that should consider the compatibility of the indicators with what is understood as the system in this context (Brouselle, 2011), hoping that its scope includes, for consistency, the health sector itself. However, the performance incentives based strategies have increased. An example is the ranking system implemented in England, since the year of 2000, in which hospitals were assigned stars, with the results public and establishing sanctions or benefits as the establishment fulfill goals (Conill, 2012). Thus, the complexity of the health assessment process is reduced to the performance, and the performance is reduced to a mere ranking of the organizations.

In this article, we present and discuss the results of a narrative review, as conceptualized by (Rother, 2007), from a critical perspective, starting with the international literature regarding the evaluation of health system performance, given the strategic nature such evaluations incorporate in the management of health policies and programs of the contemporary State, and also the growing interest in the subject in Brazil, as evidenced by the creation of the initiative SUS Performance Index (IDSUS) in early 2012, criticized by some analysts (Viacava, 2012).

Control function according to the bureaucratic and managerial paradigms in the Brazilian public sector

To govern means to exercise the function of controlling the actions and operations performed by the State or private sector, that are of public interest. The degrees such controls may take on vary from one organization to another, and according to ownership and the organization's mission. In the case of the Brazilian State, three models of public administration are well known: the patrimonial, the bureaucratic, and the managerial. Patrimonialism was hegemonic in Brazil until the Revolution of 1930 and the bureaucratic administration predominated until the advent of the Managerial State Reform in 1995, which sought to introduce the managerial paradigm.

The clear separation between State and market is, for analysts such as (Bresser-Pereira, 1996), essential to capitalism. Democracy can only exist when civil society, formed by citizens, differs from State, at the same time controlling it. Therefore, due to the demands for new roles for these economic agents and the impregnation of science as an organizer of social life, it has become necessary to develop a type of management that not only establishes a clear distinction between public and private, but also generates the separation between the political and the public administrator. There is no unanimity, however, regarding the thesis of the State-market separation. For (Mascaro, 2013), in capitalism there is no way to separate State and market, since the interests of one shape the other; its political form can only be understood as a derivation of the commodity form. Therefore,

Understanding the state can only occur based on the critique of the capitalist political economy necessarily backed by social totality. Not on the ideology of the common good, or order, or the praise given to, but within the holdings of dominations and capital reproduction crisis [...]. In this sense, one should understand the State not as a neutral apparatus available to the bourgeoisie, so that it can exercise power. We must understand it within the dynamics of capitalist relationships, the reason it is the structural base of the State [...] [which is] a necessary derivative of capitalist reproduction; these relationships desire their constitution or their formation (Mascaro, 2013, p. 89).

In the history of the public sector organization, the rational-legal paradigm is the intellectual heritage of three thinkers: Woodrow Wilson, Frederick Taylor, and Max Weber. In fact, these three thinkers have built the conceptual basis on which most States, until that point, organized their administrations. (Wilson, 1955) defended that the administration should be separate from politics. For him, after State policy makers settled their decisions, its practice should be delegated to those well versed in the "management science", who in turn would perform the task in the most efficient way possible. This would be possible because, according to (Taylor, 1995), among the various methods and tools used in each element of each case, there is always a method and a tool that are more agile and better than all the others. (Weber, 1988), in turn, claimed that bureaucracy was the most efficient organizational mechanism; thus, bureaucracy would be ideal to put into practice the scientific principles referred by Taylor (Behn, 1988).

The establishment of the bureaucratic paradigm in the organization of State quickly gained acceptance, very motivated by a socio-historical context, which pointed to the need of fighting patrimonialism arising from the formation of the Brazilian State. As stated by Gilberto (Freyre, 1998), the origin of the Brazilian Patrimonial State is located in the connection between the "family unit" and the "production unit", which got mixed up in the Plantation House, and that was consolidated over the centuries. In the 1930 post-Revolution period ("Era Vargas"), to rationalize the management was a way of reaffirming the bureaucratic values of the public administration (Bresser-Pereira, 1996), aiming to uproot any manifestation of patrimonialism strongly entrenched in the Brazilian social fabric.

However, during the consolidation of the "bureaucratic model" within the Brazilian government, many significant changes took place, in particular those related to the questioning of the efficiency of public actions when faced with the advance of capitalist societies, the deepening of relationship conflicts between State-market and the new resignification of the National States role in protecting the life of its citizens (Bresser-Pereira, 1996; Fiori, 1992). So in the mid-1970s, criticism regarding the size of State, the adoption of Keynesian7 mechanisms of economic regulation and the State's small capacity for agile and efficient response regarding new social demands grew in scale. Pari passu regarding criticism to the State's role, the oil crisis in 1974, and the globalization of the economy after the fall of the Berlin Wall, thus ending the communist utopia (Hobsbawm, 1995), further highlighted the problems of an Interventionist State. Nevertheless, as a solution to the structural crisis of capital, State retreated, aiming for an economic recovery, as listed in most neoliberal political projects adopted in several countries (Laurell, 2002).

With the fall of the Berlin Wall as a world event, and then immediately thereafter the end of the Soviet Union (a typical Interventionist State), the public machine adapted quickly, in many countries, to the molds of the Regulatory State. The same happened in Brazil, with the redefinition of the rules that would govern public administration. For (Bresser-Pereira, 1996), these new rules stand "in the proposed new public management as a response to the great 1980s State crisis and the globalization of the economy" (p.1). For the author, the two aforementioned phenomena imposed worldwide redefined functions of the State and its bureaucracy, with the new, main mission now focused on results. Despite the different meanings raised by the new public administration, (Bresser-Pereira, 1996) argues that the need for this model stemmed not only from the differentiation of the structures and from the increasing complexity of problems to be faced by the State, but also from the need to legitimize the bureaucracy to meet the demands of the citizens. (Behn, 1998) points out that the new public management is a new concept of public administration that "consists of several interrelated components", seeking to

provide high quality services to be valued by the citizens; increase the autonomy of public managers, especially the control of the central agency; measure and reward organizations and individuals based on the achievement of the required performance goals; make available human and technological resources that managers require to perform their tasks well; and, recognizing the virtues of competition, keep an open attitude about which public purposes should be taken over by the private sector, and not by the public sector (p.39, emphasis added).

It is, in short, a restructuring of the public organizations, allowing for administrative flexibility and accountability (Garces; Silveira, 2002). For this proposal, the "contractualization of results" is one of the most used tools, becoming therefore an increasingly built-in mechanism used by public policies. The results become the potential target of public organizations, which now focus on improving their performance, trying to adequately equate autonomy (management) and control requirements (results) (Pacheco, 2009).

During the implementation of the management paradigm, the contractualization of results took on different meanings in the public administrations, which (Kettl, 1997) identified as two types of contractualization. On the one hand, we have the make managers manage, adopted by countries that created incentives to influence behavior and, on the other hand, let managers manage, expressing the view that there are numerous barriers to be removed, rules, procedures and rigid structures that stand in the way of the public administrator. In the first case, the contractualization of results represents a new form of control and is accompanied by the establishment of positive and negative sanctions; the country that went deeper into this road was New Zealand, with the first generation of reformers. In the second case, the agreement on result is seen as a coordination, adjustment and learning organizational tool. Experimentation, and not control, is the answer to improve performance. This has been the hallmark of the reforms, which occurred in Australia and Sweden.

Clearly identified with the first proposal of contractualization, the Brazilian public administration has been developing a management model, whose achievement of results is accompanied by mechanisms of gain/loss of incentives. Even knowing that the greatest restriction on the Brazilian public administration is the rigidity of administrative procedures and a budget execution that is legally difficult (the second proposal would apply), the bet of the Brazilian mangers rests on changes to the servers' work; i.e. the acquisition of an entrepreneurial behavior by public managers, inserted in an classic model of an organizational environment that is traditionally bureaucratic (Garces; Silveira, 2002).

Therefore, the overall objective of the administrative reform conducted in the Brazilian context was to foster the transition from a bureaucratic public administration to a managerial public administration. The characteristics of the managerial administration can be listed as: (1) decentralization of the political point of view, transferring resources and responsibilities to the regional and local political levels; (2) administrative decentralization, with the delegation of authority for public administrators transformed into increasingly autonomous managers; (3) organizations with few hierarchical levels, rather than pyramidal; (4) the assumption of limited trust, though not total distrust; (5) an administration focused on serving the citizen, rather than self-reported; and (6) control by results, a posteriori, instead of strict control, step by step, of the administrative processes (Bresser-Pereira, 1996).

Due to the strong emphasis on budget and financial execution based on the obtained results (Garces; Silveira, 2002), the managerial paradigm recommends special attention to planning and strategic management. The strategy, understood as a form of contemporary survival in a market increasingly competitive (Touraine, 2000), invades the public space and generates public competition from within. The strategy becomes the cornerstone for achieving results and often forces the management, in the name of performance, to make use of selectivity and focus (Garces; Silveira, 2002). In this perspective, the introduction of "programs" and a management process increasingly driven by "programmatic packages" enter into the current environment of public administration, progressively creating new values and entrepreneurial attitudes, characteristic of an administration driven by results (Garces; Silveira, 2002), which causes some authors (Smith, 2002; Matias-Pereira, 2008) to baptize the managerial administration as a process of "managing by results" in commodified way. The specificity of the result as the maximum product of this management philosophy, places the reduction of the administrative scope on the results, representative of the entire chain of intangibles, intrinsic to the organizational processes, such as quality. At this juncture, as stated by (Busanelo, 2011), the organizational goals boil down to a typical control system that involves four basic steps: "(i) establish performance parameters; (ii) measure performance; (iii) compare the performance parameters and determine deviations; and (iv) take corrective action (in most cases, the behavior of individuals)" (p. 2, emphasis added).

Meritocratic logic and the control function: results, performance and awards

The entity's performance that we seek to assess usually enjoys certain consensus. However, from a scientific point of view, the term has different meanings and forms of application which differ, basically, from it, or from what is being evaluated or even the emphasis on the process that is under evaluation (Irwin, 2010).

The word "performance" has several meanings, depending on the author and the scientific basis from which it stems. In a broad sense, we can say that performance is a set of behavioral characteristics or capabilities and the efficiency of an individual8, an organization (Corrêa, Horneaux Júnior, 2005) or a group of human beings (Reifschneider, 2008), animals9 or other living beings (Ling, 1977), machinery or equipments (Porto; Creppe, 2002), products10, systems11, enterprises (Lourenzani, Quieroz, Souza Filho, 2008) or processes12, especially when compared to goals, requirements or expectations previously defined. In the strictest sense, (Bergamini and Beraldo, 1988) define performance as an action, an operation, or qualified behavior stemming from an expectation. To (Siqueira, 2002), performance is a lag to be measured between an expectation created and a certain behavior. (Misoczky and Vieira, 2001) identify some meanings for the term performance, present in literature dealing with organizations: a) as a "dramatic" and "cultural" performance in the interaction between managers and other members of the organization during the process of building the meaning of an organizational identity; b) as a result of quantifiable activities through the use of measures, such as net income or return on investment, for example, being synonymous with efficiency; c) as the part of the objectives that are formally achieved. As it turns out, there is consensus that the performance is linked to an expectation of behavioral compliance with that which has already been previously established.

Performance is linked to the logic of overcoming expectations that attribute a functional merit to that which performed beyond what was expected. Conversely, performance can be understood as a gradation of results conceptually related to the search for an award for merit obtained and, therefore, is part of the logic of meritocracy. However, the managerial paradigm, performance was associated only to productivity, to the amount of work (Barbosa, 1996). Every organization needs to be evaluated by a capable system, using feedback processes, of reviewing strategies and the work methods. Thus, it recycles, oxygenates and is able to survive in turbulent and mutable environments (Souza, 2002). Good performance, under the meritocratic logic, is encouraged by awards. The focus approach of a process that evaluates the performance of a particular entity is the ratio between contribution and consideration. We find that, depending on the times, the type and the way in which the assessment was conducted, the contribution-reward relationship changes (Neto; Gomes, 2003).

Performance can be understood in several ways, depending on the theoretical basis on which the organizational review rests. In the achieving goals model (most used by analysts and by technicians in organizations) is linked to the functionalist, rational, and organization concept, which was and remains the dominant perspective in the organizational theory. According to this approach, an organization exists to fulfill specific objectives and the evaluation of its performance is therefore geared to assess the extent of the organization's ability to achieve its objectives (Sicotte; Champagne; Contandriopoulos, 1998).

A second model, commonly used, is the internal process model, which establishes that an organization with high performance is one that works without bumps, according to the established rules, without extensive tensions. This model values stability and control. It also measures the organizational performance by the level of internal production processes.

When organizations follow the open systems model, they give great importance to the relationships between the organization and their environment. The acquisition and the maintenance of an appropriate level of resources is then a major organizational challenge. This acquisition of resources model is, for many managers, the operational definition of the organization's object, suggesting that success rests with the acquisition of resources, growth and adaptation. Even with new, more advanced ideas, such as the tracer condition, which considers not only the procedures, but also the results, the outcomes and the therapeutic effects (Tanaka, Espírito Santo, 2008), this model argues that the ability of an organization to obtain resources needed for its proper operation and its survival in the environment are the most important criteria for evaluating their performance (Sicotte; Champagne; Contandriopoulos, 1998).

In the human relations model (which is based on an organic or natural view of organizations), emphasis is given to the activities necessary for the maintenance of a satisfactory climate of collaboration within the organization and in meeting the needs of the people who work there. This suggests that an organization with high performance is one that can function as a healthy working environment. The stability, consensus, motivation, and work environment are core values.

A fifth model that conforms the idea of organizational performance is the strategic constituencies model, according to which an organization with high performance is one that can satisfy the internal and external objectives. This model is based on a political or strategic view that organizations are political arenas, in which the actors interact according to their own strategic interests. Here, the emphasis is on negotiation and compromise.

The social legitimacy model, which works with an ecological perspective of the organizations operations, considers that an organization is efficient when it maintains and survives, combining the processes and results with social values, norms and goals. Reputation, prestige and image are therefore indicators of performance (Lobato; Giovanella, 2012).

These evaluation models of organizational performance are based on several conceptual performance representations. Other authors have chosen to forego defining performance (at the conceptual level) and proposed the so-called "methodological" models for evaluating performance. The fault-driven model estimates that an organization is considered to be of high performance if it does not make mistakes or if no signs of inefficiency appear. Instead of defining what performance could be, it identifies and evaluates moments of poor performance. The comparative high performance model is another methodological model, according to which an organization is evaluated in comparison to other similar organizations. Generally, the performance criterion is then chosen on the basis of data available for the various organizations being compared. Lastly, the most popular methodological model of performance assessment of health care organizations is the normative model of the rational action system. Inspired by action theories in sociology (Weber, Parsons, Simon and others), (Donabedian, 1996) proposed that the quality of care or, more generally, performance, can be assessed using standards of results, and also, of process and of structure, according to the care related to certain regional network in construction.

Control function and difficulties in assessing the performance of health systems as a complex object

In the industry, measuring performance is possible for there is a concrete product derived from the work process. The evaluation of system performance is made basically in two circumstances: when the production of a given organization is guided by the product or when there is a production process of product, isolated from other processes. In this type of arrangement, causality is known, performance can be defined by indicators, the products are uniform and the environment is stable (Champagne, 2003; Klazinga, 2010).

However, considering only the health service as the focus being measured, performance assessment requires taking into account various attributes. The nature of healthcare work, by itself, confers characteristics that do not interact with the notion of control, characteristic of industrial processes. Health work is essentially: a) collective; b) compartmentalized; c) interdependent; and d) marked by the uncertainty of demand (Ribeiro; Pires; Blank; 2004). Since service is a work process (Meirelles, 2006), health services can be considered as a health work in process.

According to the characteristics listed by (Meirelles, 2006), services in general have some characteristics that are: (1) intangibility: it cannot be touched, that is, it has no material base; (2) unstockability: cannot be stocked, i.e., does not have a material base that can be stored, waiting of its consumption; (3) irreversibility: the service once started cannot be undone -, it might even be stopped in the middle of its execution, but cannot be taken back, it cannot go back; (4) it is dependent on human resources: there is no way to provide a service without people to implement it; (5) is dependent on information: due to the provider-user interface, information is critical to understand the service that you are paying for; (6) indistinguishability: the process and the product are indistinguishable, i.e., the product is consumed in the act of execution.

Health services, in addition to all these characteristics (simply because they are services) still have four more (directly linked to the fact of being health related): (1) are not amenable to standardization: even with the use of protocols that guide its actions, health services are virtually incapable of being identically replicated (Spiller et al., 2009; Nogueira, 1997); (2) are highly dependent on interpersonal relationships: bond, listening, patience, tolerance and other attributes of human psychology in the treatment of the other; (3) rely on highly qualified human resources: all healthcare professionals require at least a high school diploma and even two years (minimum) of training in the specific area of health (Mendes, 2011); (4) presents with organizational fragmentation: a patient often depends on the provision of various services in various organizations to solve their health problem, since not all organizations have sufficient resources to solve all the problems of a single user (Nusbaumer, 1984).

For contemporary authors, health services can be classified as: a) end services, related to well-being and quality of life of consumers (Marshall, 1988); b) support services for personal needs (Walker, 1985); c) government services, due to the fact that the State promotes these services, with no added value (Conill, 2006); d) pure type service, i.e., one that is unique and exclusive - the result of the work process is work itself, without necessarily having a resultant product (Meirelles, 2006), it being external to the process.

Thus, compared to industrial conditions and all aspects of the specificity of health as a service, the complexity of the issue when it comes to health systems becomes evident. They deal with organizations that have obligations focused on users with the production of multiple "products" have a strong focus on process orientation and their production is often confused with the other co-producers of health. Still, the products of health systems are intertwined with unknown causalities. There are difficulties in defining quality standards in performance indicators and the challenges increase with the escalation in services and environment dynamics (Klazinga, 2010).

Moreover, as stated by (Conill, 2006), when seeking to comparatively analyze health systems, the focus tends to fall on the "services and care systems". However, reducing health systems to health services and care systems is one of the largest problems of the area. The former are much more comprehensive and refer to health in a broad sense, which is the result of a complex interaction of a number of factors and actions of different social systems. Health systems include a set of interventions that target specific, social or health problems; they cover the full range of interventions, preventive services to palliative services, through the diagnostic and curative services. It composes the major public health functions (surveillance, protection and health promotion, disease prevention, assessment of the health service systems, development of public health skills) (Levesque; Bergeron, 2003), but health systems do not have direct responsibilities or governance, on all the social, economic, cultural and demographic conditions that affect the ability of people to live long and well.

Thus, understanding the performance of health systems studies has received many negative reviews for its shortcomings. Many performance systems suffer from incompleteness, however it is inevitable that some aspects of the service be omitted when measuring performance. The implication is that some measured aspects gain priority and non-measured are neglected, i.e. the old adage that "what gets measured gets managed" (Exworthy, 2010). There are at least three main problems related to the measurement of health system performance: first, consistency over time is important in measures of performance, consistency in vision, regulation and use of market incentives. Second, the voltages are between the self-improvement, self-regulation and external pressures. Where better balance is found between these forms of regulation, there will be room for other considerations, such as the role of managers, peer reviews and external regulators. Third, there is a concern that the use of targets tend to focus on the areas that are measured, while those that do not, are likely to be neglected, so that what gets measured gets managed, but questions remain about what it is not measured (Smith, 2002). Thus, to evaluate health system performance, one requires a clear conceptual model, in which what is considered performance of the health system is explicit, that have databases that provide the data necessary for the construction of indicators available and a policy and management system that constantly use the information produced in the evaluations, in the decision making (Klazinga, 2010).

Studies on health systems have a wide range of designs and methods. The Methodological Reader, published by the Alliance for Health Policy and Systems Research in 2012, classifies research designs in this area in two main research strategies: fixed designs that are established prior to data collection, and flexible designs that evolve during the study process. Fixed strategies usually use more positivist approaches in study design, data are generally quantitative, and researchers mainly seek to measure the impact of a phenomenon in highly specific and controlled conditions. Experimental design and modeling are coupled with general statistical analysis. Among the common techniques, we have data collecting through surveys, structured and semi-structured interviews and opinion polls. Flexible designs, on the other hand, are more interpretive in nature and deal primarily with qualitative data. Global projects include case study, grounded theory, ethnography, life histories and phenomenological methods. Interviews, focus groups, various forms of observation and documented reviews are common data sources. The data are analyzed interactively by interpretative processes (Hoffman et al., 2012).

For (Hoffmann et al., 2012), challenges in measuring health system performance are linked to: a) Generalization: research findings often depend on the particular context in which the studies were conducted, making it difficult to generalize them. In addition, methods for achieving external validity and promoting generalization are subject to scrutiny of different research traditions with different methodological guidelines; b) Comparability: the small number of national health systems around the world (just over 200), is a problem for the use of certain empirical methods such as cross-sectional analysis, in addition, health systems are rarely reformed in great scale (and always constantly experimenting with small changes); c) Applicability and transfer: a health systems research is highly context-specific, this means that studies in one jurisdiction may not be applicable or ipsis litteris transferable to another jurisdiction; d) Standardization: as seen, there is a lack of conceptual agreement on key terms, a variety of theoretical frameworks and paradigms and disagreement as to how different methods dialogue and the circumstances in which they may be helpful in answering different types of questions; e) Prioritization: the community of health systems researchers also lacks a broad process of consensus on priorities that are dynamically identified and the type of research that should be conducted and financed. Given the specific context of health systems research, this challenge is compounded by the possibility of limited transfer of prioritization from one jurisdiction to another and, therefore, requires supranational ability to prioritize; f) Community diversity: in health systems research, community is not really a community. As a research field that is devoted to strengthening health systems and understanding the context in which they work (particularly advanced in the debate on the theoretical aspects and development of particular methods), health systems researchers come from different places, were trained in different disciplines, bring different traditions, speak different languages, prefer different methods and focus on different issues.

Health system performance beyond the control function: to evaluate with whom and for whom?

Building a theoretical framework of health systems performance is not a simple activity or a neutral academic exercise, but should capture both administrative as political notions. Therefore, the construction of many performance frameworks based on development processes that include key groups of stakeholders (Klazinga, 2010).

Measuring performance, understood as an input for evaluation management, should aim to provide a timely decision process in time, with reliability and completeness of information, according to the goals of different actors (Tanaka; Melo, 2008). In Health Services Management Reviews (HSMR), it is important that all those who participate or who will be affected by the actions triggered by the making and practice of the decisions, become participants and therefore interested parties in evaluating results. The assessment has to converge to the needs of the evaluating actors, i.e. not only those who are responsible for the decision-making, but also those who will be responsible for implementing the actions arising from the decisions. Since this constitutes a chain of events, if one does not perform the task that is their due, the action is not implemented. Therefore, it is crucial that actors accept so their results are legitimate and used by managers and technicians involved in the action (Tanaka; Tamaki, 2012).

For purposes of characterizing the actors involved in the performance assessment processes of health systems, (Andrade, 2012) states that, in this sense, there are at least five scenarios in which the actors interested in the decision-making are: 1) the citizen- user; 2) family; 3) society; 4) technical and management; 5) managers. For (Brouselle et al., 2011), the actors involved in the evaluation process can be individuals or organized groups of agents (organizations, pressure groups, trade unions etc.). The actors are characterized by: 1) their values, their beliefs, their knowledge; 2) their projects, their intentions; 3) the resources they possess or control; 4) by their willingness for action. The actors interact in a permanent game of cooperation and competition to improve their positions, to have or control the critical characteristics of the action system (money, power, influence, commitments due to social norms). The practices of the actors (managers, doctors, care staff, among others) are simultaneously influenced by the system's structures and compose it procedurally.

The practices of the actors organize the processes by which the system resources are mobilized and used to produce the goods and services needed to achieve the desired goals (organizational, personal, group). The actors are therefore interdependent (Brouselle et al., 2011), and one could say that this interdependence is the basis of the dynamics of health systems. The dynamics of health systems can be characterized by functions and relationships established between their components. These result in policies, actions and services, contribute to results - negative or positive - and determine the performance of health systems (Lobato; Giovanella, 2012).

The dynamics of health systems can be understood as an organized action system. Like any organized action system, the system of health services is situated in a particular context at a given time. Its structure is formed by the interaction of a particular physical structure (buildings, architecture, technical levels, budgets), organizational structure (governance) and a specific symbolic structure (representations, values, collective norms). It defines a social space in which four major groups of stakeholders interact (professionals, managers, commercial world and the political world), in a permanent game of competition and cooperation, guided at the same time by the system's purposes, in order to achieve or control resources. The very system of health services, as an organized action system is made up of several action subsystems organized and interdependent, each with a certain degree of autonomy (Hoffman et al., 2012), especially in Brazil, where regional subsystems are ordered legally by the policy of regionalization and are empirically determined by the link between political decisions and institutional level actions (Lima et al., 2012).

Recognizing the existence of these political relationships and incorporating them to the study of systems, is to identify who the key players are in the decision-making process or in the making of particular system guideline; what they think, what their projects are, which resources they hold, which strategies they use (Lobato; Giovanella, 2012). In some systems, the strong party-political influences with high turnover, discontinuity and management technical disqualification are classic barriers of dynamic systems to the health systems performance (Conill, 2012), therefore, we are once again justified in the need to look at these actors and their intrinsic interests to the performance evaluation of these systems. For each component or function of health systems, there is a set of social relationships that interfere with their dynamics. There is no denying its presence in the conduction of the systems, hence the need for studies to incorporate them as elements inherent to the emergence, development and why not, for the performance of health systems (Lobato; Giovanella, 2012). Therefore, in health systems studies, incorporating the perspectives of social actors in the evaluation process is of paramount importance. The appreciation of the living experience of these social actors, conceiving them as the "other" - subject and protagonist of a program or service, together with the epistemological stance of research guided by the inter-subjectivity of the subject-researcher relationship contemplate, doubly, the ethical demands of reflecting the otherness in the services and government actions (Uchimura; Bosi, 2002).

Final remarks

Although recurrent among managers and analysts of the Brazilian public health system, the theme of health system performance assessment, due to its complexity and strong political component, is sparsely covered in the international scientific literature, as highlighted by the review performed in this essay. There is a clear choice on the part of scholars of narrowly broad themes, few articles discuss the performance theme using comprehensive and totalizing designs. Most of the research deals with partial aspects, selective and focused, directing the analysis to dimensions that could be classified as secondary, considering the importance of this type of study for public policies of all countries. Such restrictions on these approaches are the result of the inherent complexity of the theme, although one should not underestimate the role of the political dimension (and, therefore, ideological) present in these choices.

In this essay, focusing on the administrative function (not exclusively) of control, we observed a relative utility, for this purpose, of the focused and restricted studies on the insufficient theoretical and methodological development of this field of study regarding the more inclusive and totalizing approaches. We have, therefore, an important gap in the scientific knowledge on the subject, with significant impact on the assessment practices, in a damaging way. This finding is relevant, since the consolidation of the Regulatory State requires this theoretical development, essential in providing it with the tools and resources necessary to fulfill its role. Advancing these knowledge interests leaders, managers, union leaders and social movements, among other social groups, so they can have a scientific basis for evaluating health system performance, thus supporting their actions. Lastly, the predominance of selective approaches, combined with the insufficiency or absence of comprehensive studies, as identified in this review, indicates the need to reorient the production of knowledge regarding health system performance - not just in Brazil.

Referências

ANDRADE, L. O. M. Inteligência de governança para apoio à tomada de decisão. Ciência & Saúde Coletiva, Rio de Janeiro, v. 17, n. 4, p. 829-832, 2012. [ Links ]

BARBOSA, L. Meritocracia à brasileira: o que é desempenho no Brasil? Revista do Serviço Público, Brasília, DF, v. 120, n. 3, p. 58-102, 1996. [ Links ]

BEHN, R. D. O novo paradigma da gestão pública e a busca pela accountability democrática. Revista do Serviço Público, Brasília, DF, v. 49, n. 4, p. 5-43, 1998. [ Links ]

BERGAMINI, C. W.; BERALDO, D. G. R. Avaliação de desempenho humano na empresa. 4. ed. São Paulo: Atlas, 1988. [ Links ]

BRASIL. Portaria n. 1.654 de julho de 2011. Institui, no âmbito do Sistema Único de Saúde, o Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica (PMAQ-AB) e o Incentivo Financeiro do PMAQ-AB, denominado Componente de Qualidade do Piso de Atenção Básica Variável - PAB Variável. Diário Oficial da União, Brasília, DF, 19 jul. 2011. Seção 1, p. 37. [ Links ]

BRESSER-PEREIRA, L. C. Da administração pública burocrática à gerencial. Revista do Serviço Público, Brasília, v. 47, n. 1, p. 1-28, 1996. [ Links ]

BROUSELLE, A. et al. (Org.) Avaliação: conceitos e métodos. Rio de Janeiro: Fiocruz, 2011. [ Links ]

BUSANELO, E. C. Um panorama dos estudos sobre avaliação de desempenho logístico Indicadores e sistemas de mensuração. In: XXXV ENANPAD 2011. Rio de Janeiro: ANPAD, 2011. v. 1. [ Links ]

CHAMPAGNE, F. Defining a model of hospital performance assessment for European hospitals. Barcelone: WHO regional office for Europe, 2003. [ Links ]

CHERCHIGLIA, M. L.; DALLARI, S. G. A reforma do Estado e o setor público de saúde: governança e eficiência. Revista de Administração Pública, Rio de Janeiro, v. 33, n. 5, p. 65-84, 1999. [ Links ]

CONILL, E. M. Sistemas comparados de saúde. In: CAMPOS, G. W. S. et al. Tratado de saúde coletiva. São Paulo: Hucitec; Rio de Janeiro: Fiocruz, 2006. p. 563-613. [ Links ]

CONILL, E. M. Sobre os impasses dos usos da avaliação para a gestão: não é preciso inventar, nem basta simplificar. Ciência & Saúde Coletiva, Rio de Janeiro, v. 17, n. 4, p. 834-836, 2012. [ Links ]

CORRÊA, H. L.; HOURNEUAX JÚNIOR, F. A Evolução dos métodos de mensuração e avaliação de desempenho. In: Anais do XXIX ENANPAD. Brasília: ANPAD, 2005. v. 1. p. 1-15. [ Links ]

DONABEDIAN, A. The effectiveness of quality assurance. International Journal for Quality in Health Care, London, v. 8, n. 4, p. 401-407, 1996. [ Links ]

EXWORTHY, M. The performance paradigm in the English NHS: potential, pitfalls, and prospects. Eurohealth, London; v. 16, n. 3, p. 16-19, 2010. [ Links ]

FAYOL, H. Administração industrial e geral. 10. ed. São Paulo: Atlas, 1990. [ Links ]

FIORI, J. L. Para repensar o papel do Estado sem ser um neoliberal. Revista de Economia Política, São Paulo, v. 12, n. 45, p. 76-89, 1992. [ Links ]

FREYRE, G. Casa-Grande & Senzala. 34. ed. Rio de Janeiro: Record, 1998. [ Links ]

GARCES, A.; SILVEIRA, J. P. Gestão pública orientada por resultados no Brasil. Revista do Serviço Público, Brasília, DF, v. 53, n. 4, p. 53-77, 2002. [ Links ]

HOBSBAWM, E. O século: vista aérea: olhar panorâmico: In: ______. Era dos extremos: o breve século XX: 1914-1991. São Paulo: Companhia das Letras, p. 11-26, 1995. [ Links ]

HOFFMAN, S. J. et al. Background paper on conceptual issues related to health systems research to inform a WHO global strategy on health systems research. Hamilton: WHO, 2012. [ Links ]

IRWIN, R. Managing performance: an introduction. Eurohealth, London, v. 16, n. 3, p. 15-16, 2010. [ Links ]

KETTL, D. F. The global revolution in public management: driving themes, missing links. Journal of Policy Analysis and Management, Storrs, v. 16, n. 3, p. 446-62, 1997. [ Links ]

KLAZINGA. N. Health system performance management: quality for better or for worse. Eurohealth, London, v. 16, n. 3, p. 26-28, 2010. [ Links ]

LAURELL, A. Avançando em direção ao passado: a política social no neoliberalismo. In: ______. Estado e políticas sociais no neoliberalismo. 3. ed. São Paulo: Cortez, 2002. p. 151-178. [ Links ]

LEVESQUE, J. F., BERGERON, P. De l'individuel au collectif: une vision décloisonnée de la santé publique et des soins. Ruptures - Revue transdisciplinaire en santé, Montreaux, v. 9, n. 2, p. 73-89, 2003. [ Links ]

LIMA, L. D. et al. Regionalização da Saúde no Brasil. In: GIOVANELLA, L. (Org.). Políticas e Sistema de Saúde no Brasil. Rio de Janeiro: Fiocruz; p. 823-852, 2012. [ Links ]

LING, G. N. The physical state of water and ions in living cells and a new theory of the energization of biological work performance by ATP. Molecular and Cellular Biochemistry, Amsterdam, v. 15, n. 3, p. 159-172, 1977. [ Links ]

LOBATO, L. V. C.; GIOVANELLA, L. Sistemas de saúde: origens, componentes e dinâmica. In: GIOVANELLA, L. et al. (Org). Políticas e Sistema de Saúde no Brasil. 2. ed. Rio de Janeiro: Fiocruz; Centro Brasileiro de Estudos de Saúde, 2012. p. 89-120. [ Links ]

LOURENZANI, W. L.; QUEIROZ, T. R.; SOUZA FILHO, H. M. Scorecard Sistêmico: modelo de gestão para empreendimentos rurais familiares. Organizações Rurais & Agroindustriais (UFLA), Lavras, v. 10, n. 1, p. 1-15, 2008. [ Links ]

MARSHALL, J. N. Services and uneven development. Oxford: Oxford University Press, 1988. [ Links ]

MASCARO, A. L. Estado e forma política. São Paulo: Boitempo, 2013. [ Links ]

MATIAS-PEREIRA, J. Administração pública comparada: uma avaliação das reformas administrativas do Brasil, EUA e União Européia. Revista de Administração Pública, Rio de Janeiro, v. 42, n. 1, p. 61-82, 2008. [ Links ]

MEIRELLES, D. S. O conceito de serviço. Revista de Economia Política, São Paulo, v. 26, n. 1, p. 119-136, 2006. [ Links ]

MENDES, E. V. As redes de atenção à saúde. Brasília: Organização Pan-Americana da Saúde, 2011. [ Links ]

MISOCZKY, M. C.; VIEIRA, M. M. F. Desempenho e qualidade no campo das organizações públicas: uma reflexão sobre significados. Revista de Administração Pública, Rio de Janeiro, v. 35, n. 5, p. 163-77, 2001. [ Links ]

NETO, A. S.; GOMES, R. M. Reflexões sobre a avaliação de desempenho: uma breve análise do sistema tradicional e das novas propostas. Revista Eletrônica de Ciência Administrativa (RECADM), Campo Largo, v. 1, n. 1, p. 1-24, 2003. [ Links ]

NISHIJIMA, M.; BIASOTO-JUNIOR G. Análise de eficiência técnica em saúde entre 1999 e 2006. Planejamento e Políticas Públicas, Brasília, DF, v. 40, p. 45-65, 2013. [ Links ]

NOGUEIRA, R. P. O Trabalho em Serviço de Saúde. In: SANTANA, J.P. et al. Desenvolvimento Gerencial de Unidades Básicas do Sistema Único de Saúde (SUS). Brasília, DF: Programa de Desenvolvimento de Recursos Humanos, 1997. p. 183-186. [ Links ]

NUSBAUMER, J. Les services: nouvelle donne de l'economie. Paris: Economica, 1984. [ Links ]

PACHECO, R. S. Mensuração do desempenho no setor público: os termos do debate. Cadernos de Gestão Pública e Cidadania, São Paulo, v. 14, n. 55, p. 149-161, 2009. [ Links ]

PORTO, L. G. C.; CREPPE, R. C. Modelo matemático para análise de desempenho de motores elétricos em máquinas de processamento de arroz. In: Anais do IV Encontro de Energia no Meio Rural, 2002. [ Links ]

REIFSCHNEIDER, M. B. Considerações sobre avaliação de desempenho. Ensaio: avaliação e políticas públicas em educação, Rio de Janeiro, v. 16, n. 58, p. 47-58, 2008. [ Links ]

RIBEIRO, E. M.; PIRES, D.; BLANK, V. L. G. A teorização sobre processo de trabalho em saúde como instrumental para análise do trabalho no Programa Saúde da Família. Cadernos de Saúde Pública, Rio de Janeiro, v. 20, n. 2, p. 438-446, 2004. [ Links ]

ROEMER, M. National Health Systems of the World. (Vol.1: The Countries). Oxford: Oxford University Press, 1991. [ Links ]

ROTHER, E. T. Editorial: Revisão sistemática X revisão narrativa. Acta Paulista de Enfermagem, São Paulo, v. 20, n. 2, p. v-vi, 2007. [ Links ]

SAMICO, I. et al (Org.). Avaliação em Saúde: Bases Conceituais e Operacionais. Rio de Janeiro. IMIP MedBook. 2010. [ Links ]

SECCHI, L. Modelos organizacionais e reformas da administração pública. Revista de Administração Pública, Rio de Janeiro, v. 43, n. 2, p. 347-369, 2009. [ Links ]

SICOTTE, C.; CHAMPAGNE, F.; CONTANDRIOPOULOS, A. C. La performance organisationnelle des organismes publics de santé. Ruptures - Revue Transdisciplinaire en Santé, Montréal, 1998; v. 6, n. 1, p. 34-46, 1998. [ Links ]

SIQUEIRA, W. Avaliação de desempenho: como romper amarras e superar modelos ultrapassados. Rio de Janeiro: Reichmann & Affonso, 2002. [ Links ]

SMITH, P. C. Performance management in British health care: will it deliver? Health Affairs, Philadelphia, v. 21, n. 3, p. 103-15, 2002. [ Links ]

SOUZA, V. L. Gestão do desempenho: julgamento ou diálogo. Rio de Janeiro: FGV, 2002. [ Links ]

SPILLER, E. S. et al. Fundamentos da gestão dos serviços em saúde. In: ______. Gestão dos serviços em saúde. Rio de Janeiro: FGV, 2009. p. 19-44. [ Links ]

TANAKA, O. Y.; ESPIRITO SANTO, A. C. G. Avaliação da qualidade da atenção básica utilizando a doença respiratória da infância como traçador, em um distrito sanitário do município de São Paulo. Revista Brasileira de Saúde Materno Infantil, Recife, v. 8, n. 3, p. 325-332, 2008. [ Links ]

TANAKA, O. Y.; MELO, C. Avaliação de serviços e programas de saúde para a tomada de decisão. In: ROCHA, A. M.; CESAR, C. L. M. (Ed.). Saúde pública: bases conceituais. São Paulo: Atheneu, 2008. p. 119-131. [ Links ]

TANAKA, O. Y.; TAMAKI, E. M. O papel da avaliação para a tomada de decisão na gestão de serviços de saúde. Ciência & Saúde Coletiva, Rio de Janeiro, v. 17, n. 4, p. 821-828, 2012. [ Links ]

TAYLOR, F. W. Princípios de Administração Científica. São Paulo: Atlas S.A., 1995. [ Links ]

TOURAINE, A. Igualdad y diversidad: Las nuevas tareas de la democracia. México, DF: Fondo de Cultura Económica, 2000. [ Links ]

UCHIMURA, K. Y.; BOSI, M. L. M. Qualidade e subjetividade na avaliação de programas e serviços em saúde. Cadernos de Saúde Pública, Rio de Janeiro, v. 18, n. 6, p. 1561-1569, 2002. [ Links ]

VIACAVA, Francisco et al. Avaliação de Desempenho de Sistemas de Saúde: um modelo de análise. Ciência & Saúde Coletiva, Rio de Janeiro , v. 17, n. 4, abr. 2012. [ Links ]

WALKER, R. Is there a service economy? Science and Society, New York, v. 49, n. 1, p. 42-83, 1985. [ Links ]

WEBER, M. Economia e sociedade. Brasília, DF: EdUNB, 1988, v. 1. [ Links ]

WILSON, W. O estudo da administração. Rio de Janeiro: FGV, 1955. [ Links ]

Received: January 07, 2015; Revised: November 30, 2015; Accepted: December 10, 2015

Leonardo Carnut Rua Gumercindo Cavalcanti, s/n. São Cristóvão, Arcoverde, PE, Brazil. CEP 56512-600.

Authors' contribution Carnut wrote the critical review of the article's content and Narvai was responsible for the critical reading and correction of the contents presented by the first author.

Creative Commons License Este é um artigo publicado em acesso aberto sob uma licença Creative Commons