Evaluation of public policies in Brazil and the United States: a research analysis in the last 10 years

Evaluación de las políticas públicas en Brasil y Estados Unidos: análisis de las investigaciones en los últimos 10 años

Abstract

This study compares research on evaluation of public policies in Brazil and the United States. To examine the extent to which evaluation research has been established in these countries, this article uses the methods of bibliometric and social network analysis to compare recent use of evaluation research in these countries. The results show that Brazil and the United States have focused on the same areas of research, especially the areas of health, education and social welfare. It was also observed that researchers from both countries use research methods, although American researchers make more use of quanti-quali methods, and that research networks on policy evaluation in the two country have similar characteristics, although the average size and density of networks in Brazil is less than that in the United States. On the basis of the evidence, it is concluded that evaluation research in Brazil evolved during the period investigated and is becoming consolidated as a field of study and practice.

Keywords:
policy evaluation; public policy; public management; evaluation research; comparative studies

Resumen

Este estudio hace una comparación entre la investigación sobre la evaluación de políticas públicas en Brasil y en Estados Unidos. Con el objetivo de examinar el grado en que la investigación en evaluación se establece en los ya citados países, este artículo utiliza los métodos bibliométricos y de análisis de red social para comparar las publicaciones recientes de la investigación en evaluación en ambos países. Los resultados muestran que Brasil y Estados Unidos han investigado y publicado en las mismas áreas, en especial en las áreas de salud, educación y el bienestar social. También se observó que los investigadores de los dos países utilizan métodos de investigación similares, aunque los estadounidenses hacen un mayor uso de métodos cuanti-cuali, y que sus redes de investigación tienen características similares, aunque las redes americanas son de un tamaño y una mayor densidad maiores. A partir de las evidencias encontradas, se concluye que la evaluación de la investigación en Brasil ha evolucionado durante el período de investigación y está en el proceso de consolidación como un campo de estudio y práctica.

Palabras clave:
evaluación de la política; política pública; gestión pública; investigación en evaluación; estudios comparativos.

Resumo

Este estudo compara a pesquisa sobre avaliação de políticas públicas no Brasil e nos Estados Unidos. No intuito de examinar o grau em que a pesquisa em avaliação está estabelecida nesses países, este artigo usa os métodos bibliométrico e de análise de rede social para comparar as publicações recentes da pesquisa em avaliação em ambos os países. Os resultados mostram que o Brasil e os Estados Unidos têm pesquisado e publicado nas mesmas áreas, em especial, nas áreas de saúde, de educação e de bem-estar social. Observou-se também que os pesquisadores dos dois países utilizam métodos de investigação similares, embora os americanos façam maior uso de métodos quanti-quali e suas redes de pesquisa apresentem características semelhantes, apesar de as redes americanas serem de tamanho e densidade maiores. A partir das evidências encontradas, conclui-se que a pesquisa em avaliação no Brasil evoluiu durante o período investigado e que está em vias de consolidação como um campo de estudo e prática.

Palavras-chave:
avaliação de política; política pública; gestão pública; pesquisa em avaliação; estudos comparativos.

1. Introduction

In the United States the analysis of public policies and programs is as old the practice of public policy-making (Munger, 2000). The Federalist Papers might be viewed as the most prominent example of intensive analysis of prospective public policy. However, the use of "expert analysis" to evaluate the effectiveness of public policies and programs is a recent development (Munger, 2000). According to DeLeon, (2006:39), a scientific approach to the study of public policy can be traced back to the mid-twentieth century:

By most accounts, the academic discipline generally referred to as the study of public policy grew out of the approach called the policy sciences... The policy sciences approach has been primarily credited to the work of Harold D. Lasswell, writing in the late 1940s and early 1950s, most prominently articulated in his essay, "The policy orientation," which was the opening chapter to Lasswell and Daniel Lerner's The Policy Sciences...The policy sciences orientation was explicitly focused on the rigorous application of the sciences...to issues affecting governance and government.

The recent demand for thorough analysis of American public policy and programmatic responses grew out of the massive mobilization of public resources during the Second World War. The practicality of applying scientific methods to systematic analysis of large quantities of data followed upon the emergence of electronic data computation during the post-war period. This coincided with an escalating focus among students of social, political, and economic phenomena on the rigorous application of the scientific method, particularly in the form of quantitative analysis, to the study of societal problems. The Johnson presidency's War on Poverty provided impetus for the continued growth of policy analysis and the establishment of evaluation research as a distinct field within the social sciences (DeLeon, 2006; Munger, 2000; Rossi, Lipsey and Freeman, 2004).

Since the 1960s policy analysis and evaluation research have had a strong presence in education, research, and political and public administration practice in the United States. Policy analysis and evaluation research have been consolidated as academic and practice fields taught in graduate level courses in a variety of social science disciplines. Research teams that carry out policy analysis and evaluation research projects are employed by universities, private nonprofit and for-profit organizations and perform work for legislative bodies, organs of public administration, philanthropic entities, and advocacy groups. A variety of professional associations recognize and support the practice of policy analysis and evaluation research. As a result, an underlying assumption of this article is that policy analysis and evaluation research are well-established features of the public policy and administration landscape in the United States (DeLeon, 2006; Lynn Jr., 1999; Munger, 2000; Rossi, Lipsey and Freeman, 2004).

By contrast, this field in Brazil is of more recent vintage. With the establishment of a new constitution in 1988 and the many social policy initiatives of the Luiz Inácio da Silva (Lula) administration (2003-2010) (Montero, 2005; Sugiyama, 2013; Vaitsman, Ribeiro and Lobato, (2013), interest in and a sense of the need for using the methods of social science in assessing the implementation and impact of public policy and programming has grown in Brazil. Although policy analysis and evaluation research are playing a growing role in Brazilian public policy and administration (Capobiango et al., 2011) a basic proposition of this article is that they are not yet used as extensively in Brazil as in the United States.

The study reported here builds on the works of Capobiango and partners (2011) and Vaitsman, Ribeiro and Lobato (2013), who have described the emerging practice of public policy analysis in Brazil. To this end, we utilize the methods of bibliometric analysis for comparing public policy analysis and evaluation research practice in the United States and Brazil. Specifically, we compare articles published on the subject in the academic journals of the two countries over the last 10 years. More specifically, in light of the emphasis on the development of social policy in Brazil since the establishment of the 1988 constitution and, more particularly, during the period beginning in 2003, we expect there to be a bias in this direction in the Brazilian policy analysis and evaluation research literature during the 10 years period covered by the analysis.

The importance of this field of research for improving practice in countries like Brazil that are engaged in strengthening their institutional arrangements and the relative lack of comparative research on the practices of public administration involving Southern hemisphere countries justify this effort to map the characteristics of the field of evaluation research in the two countries.

The variables used in the comparison include: objectives; areas of research; whether articles are empirical, theoretical, or theoretical/empirical; whether data are quantitative or qualitative; whether the sources of data consulted are primary, secondary, or primary/secondary; and the institutional affiliation of authors.

In addition to this introduction, the paper is organized into the following sections: literature review; methods; results and discussion, and final considerations.

2. Literature review

Evaluation research - as public policy analysis or program assessment - is a set of instruments for change that not only demonstrate problems or propose solutions to the problems identified, but point to appropriate changes that may lead to policy and program improvements (Bechelaine and Ckagnazaroff, 2014). The evaluation of policies, programs and government plans is not regarded as an end in itself, but rather as an important tool for improving the efficiency of public spending, quality management and social control of the effectiveness of actions of governments (Ceneviva and Farah, 2012). According to Ramos and Schabbach (2012), there is no "best" definition of evaluation research. All definitions, however, refer to the systematic and objective examination of a completed or ongoing policy, program or project in terms of its performance, implementation and results, in order to determine its efficiency, effectiveness, impact, sustainability and relevance (Costa and Castanhar, 2005).

Evaluation involves value judgments on the policy implemented and has the objective of providing information that can improve public decision-making. It requires the definition of the criteria to be adopted and the set of policy or program attributes or characteristics to be evaluated. Commonly used criteria are efficiency, effectiveness, effectiveness/impact, coverage, technical and scientific quality, user satisfaction and fairness. In addition to the definition of evaluation criteria, the analyst must consider the extent of the policy or program. Does it include a service or system? Is its service area local, municipal, regional or national? The target population of the policy or program also must be clearly delineated (Costa and Castanhar, 2005).

To evaluate is to form a judgment about where the object being evaluated is to be placed on a continuum: more/less; a lot/a little; near/far; good/bad; adequate/inadequate; achieved/unachieved (Neirotti, 2012). In every evaluation there is a referent (a model, a situation, an expected or desired condition of what is being evaluated) and a referred object (the object to be evaluated). The analyst's challenge is to make a meaningful assessment of the gap between the reality and the intention and to provide evidence that supports understanding the gap. Evaluation is distinctive because it deals with applied knowledge concerning public policy decisions made in public spheres (Neirotti, 2012). To this end, evaluation research is important to the world of public action because it is integrally tied to policies, programs, and practices that affect the everyday lives of people (Mark and Henry, 2004).

Analysis of public policies and programs can be seen as involving four aspects of the their design and operation: 1) identification of the need for governmental intervention; 2) consideration of the intervention theory and operational components; 3) assessment of the implementation and operation of the policy or program; and, 4) measurement of the impact of the intervention. To accomplish each of these objectives the analyst draws upon the methods of social science research and concepts from the domains of economics, management, sociology, and political sciences (Chambers and Wedel, 2009; Kisby, 2007; Michael, 2006; Rossi and Wright, 1991; Weiss and Birckmayer, 2006).

Ideally, identification of the need for policies and programs, including specification of social problems and identification of governmental response gaps or inadequacies in responding to them, should be one of the primary ways in which policy and program analysts assist governmental leaders. Nonetheless, evaluation research professionals are usually not involved in the policy process prior to the introduction of a policy or programmatic response to a social problem. Rather, after they are engaged to assess the performance of policy and program interventions they must, by necessity, perform retrospective assessment of the need for them (Altschuld and Kumar, 2010; Cook, Leviton and Shadish, 2000; Rossi, Lipsey and Freeman, 2004; Rossi and Wright, 1991).

When analysts consider the theories behind policy and programmatic interventions and the operating characteristics to which they lead, they contribute to public policy and public administration in more familiar ways. They assist political and administrative leaders in determining whether the policies and programs deliver intended public products and services to those identified as in need of government assistance. As policy and program evaluators translate their understandings of policy and program operations into outcome and cost data, they play the roles of most interest to the policy and administration communities - and to interest groups and citizens seeking to hold government accountable in terms of effectiveness and efficiency. Evaluators of public policies and programs seek to provide political and administrative leaders with data and analyses needed to confirm that they have made good choices in the application of public resources. They also provide information that can be helpful in improving or abandoning policy and programmatic intervention choices (Altschuld and Kumar, 2010; Cook, Leviton and Shadish, 2000; Costa and Castanhar, 2005; Jannuzzi, 2005; Ramos and Schabbach, 2012; Rossi, Lipsey and Freeman, 2004; Rossi and Wright, 1991).

According to Appleton-Dyer and partners (2012), evaluation research can be classified according to the way that it is used to develop or support public policies and programs. It can be used: (1) instrumentally, when it is used to give direction to policies and practices; (2) politically or symbolically, when it justifies preexisting preferences and actions; and, (3) conceptually, when it is used to provide new generalizations, ideas, or concepts that are useful for making sense of the policy scene. Evaluation studies are rarely used as the direct basis for decisions. Symbolic use of evaluations provides legitimation to justify what decision makers want to do anyway. In this case, evaluation provides support for policies on the basis of intuition, professional experience, self-interest, and organizational interest. Evaluation researchers offer conceptual support for public policy and program building when they provide analyses of evidence from evaluations of policy and programming in other settings to policy makers and managers who are seeking to respond to a problem in their policy or program environments (Appleton-Dyer et al., 2012; Johnson et al., 2009; Weiss, 2005).

Evaluation research also plays an important role in terms of accountability. The evidence produced through analyses of public policies and programs can be viewed as supporting internal accountability - that is, demonstrating the productiveness of the intervention in question within a policy sector or an organization. Evaluation evidence also supports external accountability - demonstrating to the broader social or political environment that the policy or program is effective in dealing with a targeted social problem. Thus, evaluation research can support greater political transparency to assure that social planning and innovation align with social and political interests (Neirotti, 2012).

According to Cotta (2001), evaluations can be classified according to: their timing - before, during or after implementation of policies or program; the evaluator's position relative to the evaluated policy or program - internal, external or semi-independent; and the nature of the evaluated object - context, input, process and results.

While analysis of public policies and programs in the United States has been established as an academic and professional area of practice for nearly a century, it is in the process of institutionalization in Brazil. Arretche (2003) and others observe that the study of public policy has been established as a sub-discipline of political science in Brazil. Disagreeing with this assessment, Frey (2000) argues that Brazilian policy studies began only recently and have not, as yet, found an institutionalized academic or professional home. In their assessments of the public policy field, Melo (1999) and Faria (2005) have argued that knowledge about evaluation of public policies in Brazil is very limited. According to Bechelaine and Ckagnazaroff (2014), despite the growing interest in evaluating government programs in Brazil, there is a national shortage of theoretical studies to verify the use of such assessments as correction tools for improvement of public services.

Among the main reasons for the growth in evaluation research in Brazil, especially since the 1990s are: the fiscal crisis that reduced the spending capacity of governments and increased the pressure for greater efficiency; the deepening of democracy, which brought new social actors and new demands on governments; the search for social programs that mitigate social differences; and, pressure from international organizations to promote and improve social programs (Ramos and Schabbach, 2012).

In Latin America beyond Brazil, the emergence of evaluation research took place in the late 1980s and early 1990s. The emergence of the systematic analysis of policies and programs was stimulated by multilateral organizations, which required evaluations as a condition for financing projects in the region (Faria, 2005). According to Ramos and Schabbach (2012) in the last decade major international institutions including the United Nations, World Bank, Organization for Economic Co-Operation and Development (OECD), and Inter-American Development Bank (IDB) have developed methodologies for evaluation of public policies.

Pressure resulting from frequent economic crises since the 1970s as well as problems with the operation of welfare states have put pressure on governments to respond to questions about the effectiveness of their actions to impact social reality through public policies (Trevisan and Van Bellen, 2008). As a result, evaluation research has become a major tool in the pursuit of improving the efficiency of public spending and social control over the actions of governments (Ramos and Schabbach, 2012).

In distinct contrast to Brazil, evaluation research in the United States has been an important part of policy study for over half a century, in terms of theoretical advancement, methodological development and professionalization (Albaek, 1998). The first post-war surge in public policy and program evaluation took place in the United States between the 1960s and 1980s. The use of policy analysis and program evaluation grew with the social policy experimentation of the Great Society and War on Poverty in the 1960s (Caracelli, 2000). This growth in applied social research paralleled the emergence of electronic data processing and the ability to manipulate large volumes of data. The initial surge was characterized by a distinctly top-down approach. This was gradually reversed with the emergence of New Public Management in the 1990s, which emphasized a bottom-up approach and the measurement of results (Faria, 2005).

Since the 1960s, evaluation research as a profession has experienced dramatic growth, development, and institutionalization in the United States (Johnson et al., 2009). A product of three predecessor organizations, the American Evaluation Association (AEA) was founded in 1986 (Gargani, 2011; Shulha and Cousins, 1997). In 2013, the AEA reported more than 7,500 members. The conceptual development of evaluation has been equally impressive. This development has included debates on evaluation theory and practice, discussions concerning strategies and methodology, deliberations on evaluation competencies, and examinations of the many contributions evaluation can make to the global community (LaVelle and Donaldson, 2007).

At the beginning of the twenty-first century, evaluation research has assumed an established role in investigating the extraordinarily complex and difficult problems facing society (Caracelli, 2000). In response to the complex political, social and economic context within which they work, evaluation practitioners embrace a variety of paradigms, perspectives, and values. Policy analysts and program evaluators conduct evaluation research for many purposes, assume different roles, and use diverse practices (Caracelli, 2000).

3. Research method

Bibliometric research was used to investigate the questions of interest to this study. Bibliometric methods are used to explain and interpret the behavior of a given field of knowledge in order to observe events, objects, agents, products and contexts whose elements can be counted, measured and quantified (Borgman and Furner, 2002). They are also used to describe the number of publications, authors, type of work, institutional ties of researchers, and other relevant information published with respect to a given field of knowledge during a specific time period. Bibliometrics is a quantitative and statistical technique for measuring the production of knowledge in the field analyzed (Araújo, 2006).

For the purposes of this study, research was carried out using six databases, two from Brazil - Scielo Brazil and Spell - and four international, - the Wiley Online Library, Sage Publications, Oxford Journals, and Science Direct, including Elsevier. We have tried to cover all major international databases that index the leading journals in United States. In Brazil, Scielo Brazil and Spell were chosen because they are the largest open databases in Brazil, and access to them is free. The study was conducted in these databases in order to increase the reliability of data relating to Brazil, since these two, between them, index all or nearly all Brazilian academic journals publishing works on public administration, and the evaluation of public policies. The use of these two databases resulted in the identification of a comparable number of studies in the Brazilian and international databases, contributing to the robustness of the analysis of some of the variables of interest.

The search of these databases covered 10 years of published articles, 2005 to 2014. Search terms used include descriptors of evaluations, assessments and analyses concerning public policies, programs, and projects. In the Wiley Online Library, Sage Publication, Science Direct, and Spell the search was conducted on titles, abstracts and keywords. In Scielo Brazil, and Oxford Journals it is not possible to search for keywords and therefore the search was conducted only on titles and abstracts. This does not appear to have caused a methodological problem because commonly key words also appear in the abstracts. In addition, it should be noted that the Spell database indexes two types of articles, those presented in scientific events and those published in scientific journals. In this study only articles published in scientific journals were included in order to maintain comparability with the other five databases used.

The search in Scielo Brazil resulted in 173 articles. Of these, 94 met the search criteria. In the Spell database 49 articles were found, with 25 meeting the search criteria. In the Wiley Online Library search, 884 articles were found, but only 106 met the search criteria. In Science Direct, 1,947 papers were found, but only 72 met the search criteria. In the Oxford database we found 875 articles, with only 20 meeting the research criteria. Finally in the Sage Publications database, 73 articles were found, and of these 21 met the criteria. The primary reason for eliminating articles from the sample was because they pertained to work in countries other than Brazil and the United States and, thus, fell outside the scope of the study. The total result of the search has included duplicate articles, but these papers were not considered in the analysis of the selected papers. The final sample was made up of a total of 320 papers, 100 from Brazil and 220 from the US.

As previously mentioned, data of interest with respect to the articles analyzed included: journal title; year of publication; research objectives; areas of research; whether the article was empirical, theoretical or theoretical/empirical in nature; nature of the data (quantitative or qualitative); nature of data sources (primary, secondary, or primary and secondary); and institutional affiliation of authors. The analysis of social collaboration networks of researchers used the following structural measures of the networks: density, average degree, betweenness centrality and closeness centrality.

Quantitative data analysis, descriptive and cross tabulation analysis, was carried out using SPSS software, version 18. Content analysis was used in the identification of the objectives established for the respective articles in order to determine not only the areas of research involved but also the topics of interest within each of these areas.

Given the volume of data involved, NVivo software was used to support the content analysis process. The classification into research areas followed that of the articles themselves. Articles having no clear focus on a specific area of research or having a dual focus, for example, health and criminal justice, were classified in one of these categories, according to the prominent subject. Some theoretical articles, which dealt with method, teaching, theory, or policy in general, were not classified as a research area. Articles dealing with agricultural policy were categorized under the heading of economic development. For classification in terms of the nature of the data sources used it was determined that theoretical articles would be classified as data source "not applicable," as can be seen in the tables presenting study results. Finally, the free software Pajek, version 3.14 was used for analysis of research networks in terms of the elaboration of the network of research collaboration and the calculation of network structural measures (size, density and centrality).

Initially, then, quantitative analysis was carried out in order to separately describe the principal variables of interest for each country. Following this, comparative analysis was undertaken of the quantitative data for the United States and Brazil. Then content analysis was conducted to identify the nature of the articles published and what researchers were looking at in the respective countries. Finally network analysis was carried out to identify relations between researchers and institutions. In this analysis, intraorganizational researcher relationships were not considered.

The main limitation of the study is a possible selection bias resulting from the databases used for identifying the scientific articles that made up our sample. While every attempt was, made to include the databases that include journals that publish articles in the area of interest, it is possible that some relevant publications were missed.

4. Results and discussion

In this section we present and discuss the results of the study undertaken. In table 1 we report the number of articles published during each year of the period of interest, 2005-2014, in order to determine the behavior of the field over time.

Table 1
Publications on evaluation research (2005-14)

• Source: Research data (2015).

As can be observed in table 1, publication on evaluation research varied over time in both countries. In Brazil, only 2 papers meeting the research criteria were found for 2006, while in 2011 and 2012 there were 15 each year and 6 in 2014. Thus, no pattern or trend could be identified with respect to publication in the field in Brazil during the period of interest. With respect to the United States, it is noted that there was less variation in numbers during this period than in Brazil. Overall, during the period of interest, the number of publications meeting study criteria indexed in US databases consulted are more than double the number in Brazil, although this proportion varied greatly from year to year.

As previously noted, some Brazilian researchers have stated that field of evaluation research is still in its infancy in Brazil (Faria, 2005; Frey, 2000; Melo, 1999). Given the number of articles published from 2005-2014, however, it appears that Arretche (2003) is correct in arguing that evaluation research has now taken root in the country. In support of this argument is the fact that the number of publications encountered on the subject in Brazil represents nearly half of those found for the United States in four of the most important American databases for the same period. Based on the evidence of this study, then, it cannot be said that there is little research on the subject in Brazil. Of the 320 papers published on evaluation research identified in this study, nearly a third (31.25%) were published in Brazil. Given the research path of the respective countries (Albaek, 1998; Caracelli, 2000; Faria, 2005; Johnson et al., 2009) the expectation was that this difference would be much larger; and these findings point to advances in evaluation research in Brazil.

Table 2 provides an analysis of the most researched areas in Brazil and the United States, respectively.

Table 2
Principal areas of evaluation research

• Source: Research data (2015).

It is observed that for Brazil, the areas of health (27%), education (25%), and evaluation research (9%) together make up 61% of total publications. For the United States, health (44.1%), education (9.5%), and evaluation research (8.2%) are responsible for 61.8% percent of total publications. In other words, in both countries, the same three areas receive the greatest attention, differing only regarding percentages of the individual area. What might explain this? That the research concerns are the same? Influence of US research on Brazilian research? Influence of international institutions contributing to the increase of research and the orientation of analysis toward the areas that concentrate the most resources (Faria, 2005)? Pressures for the efficiency of public expenditures, transparency and accountability (Ramos and Schabbach, 2012)?

The size and large budgets of these areas also might explain these results. Under Brazil's new constitution, the so-called social areas such as health, education, and welfare have gained importance (Montero, 2005; Sugiyama, 2013; Vaitsman, Ribeiro and Lobato, 2013), and investments have been growing over the past two decades. Thus, the areas of policy analysis and evaluation research are assuming a growing role in Brazil's social public policy (Capobiango et al., 2011). Any or all of these questions or justifications may be relevant. However, further study and analysis will be necessary if these questions are to be answered with any precision.

The analysis of the data sources used by researchers, presented in table 3, reveals that in both Brazil and the United States secondary data sources are more prevalent. The most significant difference is with respect to the use of both primary and secondary data in the same study, which is considerably greater in the United States. American researchers make more use of multiple data sources than Brazilian researchers, permitting the additional observation that they also make greater use of mixed methods, both quantitative and qualitative, for data analysis. The category "not applicable" indicates that articles with this classification are theoretical rather than empirical in nature. The respective percentages are not inconsiderable. They represent 26 percent of the work published in Brazil and 17.3 percent in the United States, providing evidence that substantial effort is being dedicated to theoretical production in both countries.

In addition, as can be seen in table 3, there is considerable use of qualitative data in both countries.

Brazilian and American research present significant differences in terms of the quantity of papers and nature of the data at the 0.05 statistical significant level in the Chi-square tests. The greatest difference observed is with respect to the qualitative approach. In Brazil, taking into consideration the fact that theoretical papers are included in this category, the qualitative approach dominates, with 58 percent. For the United States, this approach was observed in only 43.2 percent of the papers analyzed. The situation is inverted when the quantitative approach in analyzed, being used in 38.2 percent of the US papers analyzed, but in only 30 percent of the Brazilian papers. Another distinct difference is with respect to mixed methods. Studies involving both qualitative and quantitative research were used in 18.6 percent of the American studies and only 12 percent of the Brazilian studies. The evidences of our study, therefore, points to that the field is characterized by a concern for measuring efficiency, efficacy, impact and satisfaction (Costa; Castanhar, José, 2005) and by judgments of quantitative aspects such as more/less, much/little, good/bad, adequate/inadequate, achieved/not achieved (Neirotti, 2012).

Table 3
Relationship between nature and source of research data

• Source: Research data (2015).

It can be noted in table 3 that secondary data predominates in the quantitative studies, which is according to expectations. Only in the American studies occur instances of the collection of both primary and secondary data in quantitative studies. In fact, the use of both primary and secondary data by the American researchers is greater for all categories. Brazilian researchers appear to make more use of secondary sources in qualitative studies.

For both countries there was greater use of qualitative studies, followed by quantitative studies and mixed methods studies. Given the complexity of the field, more quali-quanti studies might be opportune for mitigating biases arising from the use of a single methodological approach, as Donaldson and Grant-Vallone (2002) have pointed out.

With respect to the vehicles that publish the scientific production of the field of evaluation research, we found 164 journals in the six data bases: 51 in Brazil and 112 in the United States. In the American data base the main journals are the Journal of Policy Analysis and Management, Evaluation and Program Planning, New Directions for Evaluation, Journal of School Health, Policy Studies Journal, and Nicotine and Tobacco Research. Together they published 22.6 percent of the papers analyzed in our study. In the case of Brazil, the main journals are Ciência e Saúde Coletiva (Science and Collective Health), Revista de Administração Pública (Review of Public Administration), e Ensaio: Avaliação e Política Pública de Educação (Essays: Education Policy and Evaluation), responding for 22.5 percent of the articles found in the Brazilian data base. It is observed that in Brazil there appears to be less concentration of works on evaluation research in the three principal journals identified than in the United States.

Also we have noted that in the United States there are specialized journals in the field of evaluation research, such as Journal of Policy Analysis and Management, Evaluation and Program Planning and New Directions for Evaluation while in Brazil cases of specialized journals such as Essays: evaluation and Public Policy Education are rare.

Analysis of the number of authors per article found that 58 percent of the Brazilian articles were written by one or two authors, while this was the case for 47.3 percent of the American articles. In Brazil, 95 percent of the articles were written by four or fewer authors and in the United States, 80.5 percent. Our data suggest that in the United States, in this field of study, it is more usual to involve a greater number of researchers. We found papers with as many as 12 authors in the United States, while in Brazil the greatest number of authors observed was six. There appear to be two ways of interpreting this evidence: i) that Brazilian journals do not accept or discourage articles written by more than six authors, or; ii) that Brazilian researchers may be less involved in research networks than American researchers. However, an analysis of guidelines in most Brazilian journals showed that these journals do not limit the number of authors in their publications, favoring the second interpretation.

Analysis with regard to the second point mentioned above, presented in Figures 1 and Figures 2, respectively, reveals that the National School of Public Health is the Brazilian research institution with the greatest number of relationships with other institutions in Brazil with respect to evaluation research. In the case of the United States, the University of North Texas has the greatest number of researchers collaborating with researchers from other institutions.

Figure 1
Networks of researchers - Brazil

Figure 2
Networks of researchers - US

The main structural measures of the Brazilian and American collaboration networks in evaluation research are presented in table 4.

Table 4
Structural measures of collaborative networks of researchers studying evaluation research

• Source: Research data (2015).

Network density is 37 percent for the Brazilians and 60 percent for the Americans. The average degree measurement represents the average number of effective connections between the possible nodes of a network and is considered a better measurement than density to evaluate the structural cohesion between networks that have different sizes (Nooy, Mrvar and Batagelj, 2011). The average degree of Brazilian network is lower than the American network. This result was expected, given the fact that articles were collected in more databases in the US, and more scientific organizations exist in that county than in Brazil. In addition, the lower measurement of density and average degree in Brazil may be explanabale by the comparatively smaller time for development of collaborative networks in Brazil. In the US, the field of evaluation research began to emerge at least 30 years earlier than in Brazil (Albaek, 1998; Caracelli, 2000). On the basis of the data available, it is not possible to determine whether in the United States there are more incentives for development of collaborative research networks, such as financing for projects involving more than one research institution, which could be another possible explanation for the results observed.

Two metrics of table 4 are worthy of further note: that of betweenness centrality and that of closeness centrality. According to Prell (2012) the first metric presents the most influential actor of the network in terms of potential for network control, while the second focuses on actor independence. Table 4 shows the five most important institutions in the respective networks from these perspectives. This analysis draws attention to the leadership role of the Brazilian National School of Public Health in policy evaluation, even though its metric is lower in this regard than that of the Centers for Disease Control and Prevention - CDC, BC (0076) and CC (0195) and the California Department of Public Health, BC (0061) and CC (0219).

Also of interest is the evidence that in both countries the institutional ties of researchers on policy analysis and evaluation are not just with universities. They also have institutional ties to public organizations providing services to the citizenry, such as local government and departments of health of the federal government and local governments, which suggests that these latter organizations may be playing an active role in the scientific research taking place with respect to policy evaluation.

5. Final considerations

The results found in this study allow us to state that the research evaluation area is more consolidated in the US because there are twice as many journals that publish in the field (112 against 51 in Brazil) and a considerably greater number of publications-nearly 70% of the publications found in the period analyzed (2005-14). In addition, in the United States, unlike Brazil, in addition to specialized journals there are professional associations dedicated to evaluation research, made up of researchers, as is the case of The American Evaluation Association (AEA) (Gargani, 2011; Shulha and Cousins, 1997), another indication of greater consolidation of the field in US.

Another piece of evidence in the regard is the fact that the American research networks are larger and denser, which can be at least partly explained by the longevity of this field of study in the USA, which emerged at least 30 years earlier than in Brazil: 1960 and 1990, respectively. It can be thus be said that there is a significant difference in the generations of researchers that have dedicated themselves to the field.

The findings of this study thus reaffirm our proposition that evaluation research is not yet as established a research field in Brazil as in the United States. Nonetheless, these findings provide evidence of advances in evaluation research in Brazil, surpassing the initial expectation of the authors of this work that the differences in the development of the area in the respective countries would be more significant.

It is relevant to note in this regard that, in both countries, researchers have ties not only to universities and research institutions but with public organizations that provide services to citizens, as well. The existence of research networks involving universities, research institutes, and organizations of various levels of government support the characterization of evaluation research as a fields of applied research (Neirotti, 2012), in both countries.

The fact that the four main areas in which evaluation research is conducted, that is, health, education, policy analysis, and social welfare, converge in the two countries, albeit with different percentages, is worthy of note and raises some questions: Are the research interests the same in the United States and Brazil? Do these results represent pressure from international organizations to focus on areas that involve greater expenditure of public resources or which are government priorities? More studies are needed to respond adequately to such questions.

Some directions suggested for future studies on the subject include: (i) the inclusion of other scientific databases; (ii) analysis of the relationships among organizational settings of research groups, research area, methodological approaches and theoretical categories; (iii) identification of the theories used in the respective studies and analysis of the relationship between theory use and specific areas of evaluation research.

In light of the argument made by Vaitsman, Ribeiro and Lobato (2013) that Brazil lags behind the United States in the conceptualization and methodological development of policy analysis, in general, and based on the findings of this study, we also offer recommendations for Brazilian researchers in the field of evaluation research: (i) given the complexity of the field, it is recommended that Brazilian researchers make more use of multiple sources of data collection; and (ii) that Brazilian researchers make greater use of mixed approaches involving both qualitative and quantitative methods and techniques for data collection and analysis. Finally, it is suggested that to investigate the factors that influence the formation and consolidation of research networks in the field and the relationship of this to overall consolidation of the field may contribute to further advances in evaluation research in Brazil.

References

• ALBAEK, Erik. Knowledge, interests and the many meanings of evaluation: a developmental perspective. Scandinavian Journal of Social Welfare, n. 7, p. 94-98, 1998.
• ALTSCHULD, James W.; KUMAR, David Devraj. Needs assessment. Thousand Oaks, CA: Sage Publications, 2010.
• APPLETON-DYER, Sarah et al. Understanding evaluation influence within public sector partnerships: a conceptual model. American Journal of Evaluationv. 33, n. 4, p. 532-546, 2012.
• ARAÚJO, Carlos A. Bibliometria: evolução histórica e questões atuais. Em Questão, v. 12, n. 1, p. 11-32, 2006.
• ARRETCHE, Marta. Dossiê agenda de pesquisa em políticas públicas. Revista Brasileira de Ciências Sociais, v. 18, n. 51, p. 4, 2003.
• BECHELAINE, Cinthia H. De O.; CKAGNAZAROFF, Ivan B. Por que as avaliações vão para a gaveta? Uma revisão teórica dos fatores relacionados ao uso dos resultados das avaliações de políticas públicas. Contabilidade, Gestão e Governança, v. 17, n. 2, p. 78-93, 2014.
• BORGMAN, Christine L.; FURNER, Jonathan. Scholarly communication and bibliometrics. Annual Review of Information Science and Technology, v. 36, n. 1, p. 2-72, 2002. Available at: <Available at: http://onlinelibrary.wiley.com/doi/10.1002/aris.1440360102/abstract >. Accessed on: 25 feb. 2015.
  » http://onlinelibrary.wiley.com/doi/10.1002/aris.1440360102/abstract
• CAPOBIANGO, Ronan Pereira et al. Análise das redes de cooperação científica através do estudo das coautorias dos artigos publicados em eventos da Anpad sobre avaliação de políticas públicas. Rev. Adm. Pública, v. 45, n. 6, p. 1869-1890, 2011. Available at: <Available at: http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0034-76122011000600012&lang=pt >. Acessed on: 25 feb. 2015.
  » http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0034-76122011000600012&lang=pt
• CARACELLI, Valerie J. Evaluation use at the threshold of the twenty-first century. New Directions for Evaluation, v. 2000, n. 88, p. 99-111, 2000.
• CENEVIVA, Ricardo; FARAH, Marta F. S. Avaliação, informação e responsabilização no setor público. Revista de Administração de Empresas, v. 46, n. 4, p. 993-1016, 2012.
• CHAMBERS, Donald E.; WEDEL, Kenneth R. Social policy and social programs: a method for the practical public policy analyst. Fifth ed. Boston: Allyn and Bacon, 2009.
• COSTA, Frederico L.; CASTANHAR, José C. Avaliação de programas públicos: desafios conceituais e metodológicos. Rev. Adm. Pública, v. 37, n. 22, p. 969-992, 2005.
• COTTA, Tereza C. Avaliação educacional e políticas públicas: a experiência do Sistema Nacional de Avaliação da Educação Básica (Saeb). Revista do Serviço Público, v. 52, n. 4, p. 89-111, 2001.
• DELEON, Peter. The historical roots of the field. In: MORAN, Michael; REIN, Martin; GOODIN, Robert E. (Org.). The Oxford handbook of public policy. London: Oxford University Press, 2006. p. 1000.
• DONALDSON, Stewart I.; GRANT-VALLONE, Elisa J. Understanding self-report bias in organizational behavior research. Journal of Business and Psychology, v. 17, n. 2, p. 245-260, 2002.
• FARIA, Carlos Aurélio Pimenta de. A política da avaliação de políticas públicas. RBCS, v. 20, n. 59, p. 97-169, 2005.
• FREY, Klaus. Políticas públicas: um debate conceitual e reflexões. Planejamento e Políticas Públicas, n. 21, p. 211-259, jun. 2000.
• GARGANI, John. More than 25 years of the american journal of evaluation: the recollections of past editors in their own words. American Journal of Evaluation, v. 32, n. 3, p. 428-447, 2011.
• JANNUZZI, Paulo De Martino. Indicadores para diagnóstico, monitoramento e avaliação de programas sociais no Brasil. Revista do Serviço Público, v. 56, n. 2, p. 137-160, 2005.
• JOHNSON, Kelli et al. Research on evaluation use: a review of the empirical literature from 1986 to 2005. American Journal of Evaluationv. 30, n. 3, p. 377-410, 2009.
• KISBY, Ben. Analyzing Policy Networks. p. 71-90, 2007. Available at: <Available at: http://dx.doi.org/10.1080/01442870601121502 >. Accessed on: 25 feb. 2015.
  » http://dx.doi.org/10.1080/01442870601121502
• LAVELLE, John M.; DONALDSON, Stewart I. The state of preparing evaluators. New Directions for Evaluation, n. 114, p. 107-114, 2007.
• LYNN Jr., Lauence E. A place at the table: policy analysis, its the future of practice and postpositive critics. Journal of Policy Analysis and Management, v. 18, n. 3, p. 411-425, 1999. Avaliable at: < Avaliable at: http://onlinelibrary.wiley.com/doi/10.1002/(SICI)1520-6688(199922)18:3%3C411::AID-PAM5%3E3.0.CO;2-G/full >. Accessed on: 20 feb. 2015.
  » http://onlinelibrary.wiley.com/doi/10.1002/(SICI)1520-6688(199922)18:3%3C411::AID-PAM5%3E3.0.CO;2-G/full
• MARK, Melvin M.; HENRY, Gary T. The mechanisms and outcomes of evaluation influenceEvaluation, v. 30, n. 1, p. 35-57, 2004.
• MELO, Marcos A. Estado, governo e políticas públicas. O que ler na ciência social brasileira. São Paulo/Brasília: Sumaré; Anpocs; Capes, 1999. p. 55-100.
• MICHAEL, Ewen J. Public policy: the competitive framework. South Melbourne, Australia: Oxford University Press, 2006.
• MONTERO, Alfred P. Brazilian politics: reforming a democratic state in a changing world. Cambridge, UK: Polity Press, 2005.
• MUNGER, Michael C. Analyzing policy: choices, conflicts, and practices (new institutionalism in american politics). {S.l.}: W.W. Norton, 2000. Avaliable at: < Avaliable at: https://books.google.com.br/books?id=IYd5QgAACAAJ >. Accessed on: 20 feb. 2015.
  » https://books.google.com.br/books?id=IYd5QgAACAAJ
• NEIROTTI, Nerio. Evaluation in Latin America: paradigms and practices. In: KUSHNER, Saville; ROTONDO, Emma (Org.). Evaluation voices from Latin America. San Francisco: Jossey-Bass, 2012. p. 128.
• NOOY, Wouter De; MRVAR, Andrej; BATAGELJ, Vladmir. Exploratory social network analysis with pajek. New York: Cambridge University Press, 2011.
• PRELL, Christina. Social network analysis: history, theory and methodology. London: Sage, 2012.
• RAMOS, Marília P.; SCHABBACH, Letícia M. O estado da arte da avaliação de políticas públicas: conceituação e exemplos de avaliação no Brasil. Rev. Adm. Pública, v. 46, n. 5, p. 1271-1294, 2012.
• ROSSI, Peter H.; LIPSEY, Mark W.; FREEMAN, Howard E. Evaluation: a systematic approach. 7. ed. Thousand Oaks, CA: Sage Publications , 2004.
• ROSSI, Peter H.; WRIGHT, James D. Evaluation research: an assessment. In: MILLER, Delbert C. (Org.). Handbook of research design and social measurement. 5. ed. Newbury Park, CA: Sage Publications, 1991. p. 87-97.
• SHADISH, William R.; COOK, Thomas D.; LEVITON, Laura C. Foundations of program evaluation Theory of practice. Newbury Park, CA: Sage Publications , 1991.
• SHULHA, Lyn M.; COUSINS, J. Bradley. Evaluation use: theory, research, and practice since 1986. American Journal of Evaluation, v. 18, n. 3, p. 195-208, 1997.
• SUGIYAMA, Natasha B. Diffusion of good government: social sector reforms in Brazil. {S.l.}: University of Notre Dame Press, 2013.
• TREVISAN, Andrei P.; VAN BELLEN, Hans M. Avaliação de políticas públicas: uma revisão teórica de um campo em construção. Rev. Adm. Pública, Rio de Janeiro, v. 42, n. 3, p. 529-550, maio/jun. 2008.
• VAITSMAN, Jeni; RIBEIRO, Jose M.; LOBATO, LenauraPolicy analysis in Brazil. Chicago: University of Chicago Press, 2013. Avaliable at: < Avaliable at: http://www.policypress.co.uk/display.asp?k=9781447306849 >. Accessed on: 15 feb. 2015.
  » http://www.policypress.co.uk/display.asp?k=9781447306849
• WEISS, Carol Hirschon. An alternate route to policy influence: how evaluations affect D.A.R.E. American Journal of Evaluation, v. 26, n. 1, p. 12-30, 2005.
• WEISS, Carol H.; BIRCKMAYER, Johanna. Social experimentation in public policy. In: MORAN, Michael; REIN, Martin; GOODIN, Robert E. (Org.). The Oxford handbook of public policy. Oxford, UK: Oxford University Press, 2006. p. 806-830.

Publication Dates

• Publication in this collection
  Nov-Dec 2016

History

• Received
  06 Nov 2015
• Accepted
  13 July 2016