Factors that influence human resources for health policy formulation: a multiple case study in Brazil and Portugal

Abstracts

This study aims to analyze whether the process by which policies for human resources for health that aim to improve the geographic distribution of physicians have been informed by scientific evidence in Brazil and Portugal. This was a multiple case study on a decision-making process for human resources for health in Brazil and Portugal. The respective case studies were based on Brazil’s More Doctors Program (Programa Mais Médicos - PMM) and Portugal’s strategy of hiring foreign physicians through bilateral agreements, to work in the country’s National Health Service (SNS). We interviewed 27 key actors in the policy-making process on the following topics: factors that influenced the policy decisions, actors that were expected to win or lose from the policy, and the scientific evidence and available data used in the policy-making, among others. The most evident factors appearing in the interviews as having influenced the PMM were: institutions; external factors (Presidential elections); group interests (e.g. physicians’ professional associations), governments (Brazil and Cuba), international organizations, and civil society; and ideas (scientific evidence). The most frequently cited factors in Portugal were: institutions and interests of government (from Portugal and the countries involved in the bilateral agreements), civil society, and groups (physicians’ professional associations). Contrary to the case study in Brazil, where the evidence was reported to having played an important role in the policy decisions, in Portugal, scientific evidence was not identified as contributing to the specific policy process.

Keywords:
Health Manpower; Public Health Policy; Policy Making; Medically Underserved Area; Health Services Accessibility

O estudo objetiva analisar o processo em que as políticas de recursos humanos em saúde (RHS), que visam melhorar a distribuição geográfica dos médicos, são (ou não) informadas por evidência científica no Brasil e em Portugal. Foi realizado um estudo de caso-múltiplo sobre o processo de decisão das políticas de RHS no Brasil e em Portugal. Para compor os estudos de caso, as políticas escolhidas foram o Programa Mais Médicos (PMM) e a estratégia de contratação de médicos estrangeiros por acordos bilaterais para o trabalho no Serviço Nacional de Saúde (SNS) português. Foram entrevistados 27 atores-chave no processo de formulação das políticas em análise nos seguintes tópicos: fatores que influenciaram a formulação, atores que eram esperados ganhar ou perder, evidências científicas e os dados disponíveis utilizados para a formulação, entre outros. Os fatores mais evidentes identificados nas entrevistas como sendo influenciadores do PMM foram: Instituições; Fatores Externos (eleições presidenciais); Interesses de grupos (por exemplo, associações de profissionais médicos), governos (brasileiro e cubano), organização internacional e sociedade civil; e Ideias (evidência científica). Os fatores mais listados em Portugal foram: Instituições e Interesses dos governos (português e envolvidos nos acordos bilaterais), sociedade civil e grupos (associações de profissionais médicos). Ao contrário do que se verificou no estudo de caso do Brasil, em que reconhecidamente a evidência teve um papel importante na formulação da política em análise, em Portugal a evidência científica não foi identificada como contributo para a formulação da intervenção em estudo.

Palavras-chave:
Recursos Humanos em Saúde; Políticas Públicas de Saúde; Formulação de Políticas; Área Carente de Assistência Médica; Acesso aos Serviços de Saúde

El estudio tiene por objetivo analizar el proceso en el que las políticas de recursos humanos en salud (RHS), que tienen como fin la mejora de la distribución geográfica de los médicos, son (o no) informadas por evidencias científicas en Brasil y en Portugal. Se trata de un estudio de caso-múltiple sobre el proceso de decisión de las políticas de RHS en Brasil y en Portugal. Para configurar los estudios de caso, las políticas elegidas fueron el Programa Más Médicos (PMM) y la estrategia de contratación de médicos extranjeros mediante acuerdos bilaterales para el trabajo en el Servicio Nacional de Salud (SNS) portugués. Se entrevistaron a 27 actores-clave en el proceso de formulación de las políticas en el análisis en los siguientes asuntos: factores que influenciaron la formulación, actores que se esperaba ganar o perder, evidencias científicas y datos disponibles utilizados para la formulación, entre otros. Los factores más evidentes, identificados en las entrevistas como de influencia en el PMM, fueron: instituciones; factores externos (elecciones presidenciales); intereses de grupos (por ejemplo, asociaciones de profesionales médicos), gobiernos (brasileño y cubano), organización internacional y sociedad civil; e ideas (evidencia científica). Los factores más registrados en Portugal fueron: instituciones e intereses de los gobiernos (como el portugués y los involucrados en los acuerdos bilaterales), sociedad civil y grupos (asociaciones de profesionales médicos). Al contrario de lo que se verificó en el estudio de caso de Brasil, donde se reconoció que la evidencia tuvo un papel importante en la formulación de la política en análisis, en Portugal la evidencia científica no fue identificada como una contribución para la formulación de la intervención en estudio.

Palabras-clave:
Recursos Humanos en Salud; Políticas Públicas de Salud; Formulación de Políticas; Área sin Atención Médica; Accesibilidad a los Servicios de Salud

Introduction

Health policy-making informed by the best available scientific evidence helps policy-makers make more assertive decisions ¹. It is characterized by transparent assessment of the relevant research, aimed at guaranteeing that others can determine which evidence was used appropriately ¹. The potential advantages are: better structuring of policy-making rather than sole determination by the main actors ^2,3; helping the public at large and decision-makers understand a problem and choose the most appropriate policy alternatives; achieving greater transparency in the use of public funds; and reducing or avoiding the risk of the policy’s failure ³. Meanwhile, failure to use evidence can lead to underutilization of effective interventions, incorrect use of adequate interventions, and excessive use of unproven interventions ^4,5. This scenario can create serious consequences for users of services, such as jeopardizing access to adequate healthcare, increasing health inequalities ¹, and inefficient use of limited resources for healthcare provision ⁶.

Despite progress in the last 20 years in the use of evidence, not all the lessons learned in policies related to clinical practice can necessarily be applied to other areas. Some review studies have examined the factors that hinder (vs. facilitate) the use of scientific evidence for informing policy decisions in legislative and administrative areas in the health sector ^7,8. Few studies have examined the use of scientific evidence in the policy-making process in a context of competing influences ⁹. The use of evidence in policy-making could be improved if researchers knew the competing influences in the political process, formed partnerships with the decision-makers, questioned the incentives from research institutions, and engaged in the public debate on the research problems ⁹.

This study aims to analyze whether the policy decisions for human resources for health that aim to improve the geographic distribution of physicians have been informed by scientific evidence in Brazil and Portugal. More specifically, the study aims to identify and analyze the factors that influence policy-making for human resources for health that aim to improve the geographic distribution of physicians, the efforts to use evidence in these policy decisions, and strategies used to improve human resources for health policies, informed by scientific evidence.

Method

This is a multiple case study on the policy-making process for human resources for health in Brazil and Portugal, developed in three phases: (I) and (II) analysis of the political contexts for human resources for health in the two countries ^10,11 and (III) the focus of this article, analysis of the policy-making process and the factors that influenced it, from the perspective of the respective actors.

The policies were selected because they have the same objective of reducing the asymmetrical distribution of physicians in areas with unmet needs for health services (rural, remote, and/or poor areas) and because they are implemented mainly in primary healthcare (PHC) (Table 1).

Table 1
Description of policy strategies that were analyzed for the case studies in Brazil and Portugal.

Strategies analyzed (country)	Description of strategies
I. More Doctors Program (Brazil)	Launched in July 2013 through Executive Order 621 and regulated by Law 12,871 of October 22, 2013, which established the program as a state policy. The purpose is to train physicians and deploy them in communities and areas with a shortage or lack of physicians. To achieve its objectives, the program was structured in three main lines of action: - Upgrading of the primary care structure: improvement in the infrastructure of the Healthcare Network (RAS), with a focus on primary care units; - Emergency provision, entitled “More Doctors for Brazil Project” (PMMB), working with emergency provision of physicians in vulnerable areas. The physicians are recruited individually, from Brazil and other countries (with more physicians per thousand inhabitants than Brazil), and through a bilateral agreement with the Cuban government 34. Participants in the program receive a monthly stipend and relocation allowance, higher for participants deployed to the remotest areas; and - Medical training: changes in undergraduate medical education and specialist training 35.
II. Hiring of foreign doctors through bilateral agreements (Portugal)	Formulated to respond to the shortage of family physicians in Portugal, especially in needy areas. The first bilateral agreement began in 2008 and was signed by the governments of Portugal and Uruguay, hiring 15 physicians to work in the National Institute for Emergency Medical Care (INEM). In 2009, an agreement was signed with Cuba, renewed in 2012 and 2014, including some 200 physicians, specifically to work in health centers in the Algarve, Alentejo, and Lisbon/Tagus Valley (LVT). In 2011, bilateral agreements were signed with Colombia (120 physicians) and Costa Rica (12 physicians). All the foreign doctors hired in this arrangement began the selection process in their own countries. The Porto School of Medicine (FMP) conducted written exams with the applicants. Before coming to Portugal, the selected physicians took a course in the Portuguese language. After validation of their diplomas by the FMP and an exam in Portuguese, they registered with the Portuguese National Medical Board. A brief training course was organized, covering the National Health Service (SNS), and there was a two-week period for adaptation to the service.

Data collection and sampling

From November 2015 to April 2016, face-to-face semi-structured interviews were conducted by the first author with key actors in the policy-making process for the target policies. A total of 27 actors were interviewed out of 35 requests that were sent (“snowball approach”), after a maximum of three attempts at contacts, so 8 actors were not interviewed (6 Brazilians and 2 Portuguese). The interviewed actors belonged to one of the three groups in Table 2.

Table 2
Groups of actors interviewed in Brazil and Portugal.

Groups of actors interviewed	Brazil	Portugal
Policy-makers (PM) at the central level and Ministry of Health staff involved in the policies	4 actors (3 PM and 1 staff)	8 actors (7 PM and 1 staff)
Researchers cited by policy-makers or referenced in the relevant documents	5	3
Interest groups: other actors, from international organizations (IO), medical associations (MA), councils, e.g., Federal Board of Medicine, National Council of State Health Secretaries (CS)	5 actors (2 IO, 1 MA, 2 CS)	2 MA

A semi-structured questionnaire was used with the following topics: the policy’s history, factors that influenced the policy-making, actors who were expected to gain or lose with the implementation, scientific evidence and available data used for the policy-making, and mechanisms promoted by the Ministry of Health to facilitate the use of and access to research results. Interviews focused on the interviewees’ personal perceptions and experiences in the policy process and do not express the views of the institution or the position occupied by the actor.

Analysis of the interviews

The interviews were recorded and transcribed. The analysis of the collected data used the 3Is (Interests, Ideas, and Institutions) conceptual framework, including External Factors (3Is+E) (Figure 1).

Figure 1
Conceptual framework: 3Is (Interests, Ideas, and Institutions) + External Factors (3Is+E).

The framework is based on three factors that the political science literature uses to explain the public policy development process ^12,13. The principle is that the policy’s development and choices can be influenced by the actors’ interests and ideas and the institutions’ configuration ^12,13 (Table 3).

Table 3
Factors that influenced the public policies.

Factors	Comments
Institutions	Institutions refer to the formal and informal rules, norms, precedents, and organizational factors that structure the policy behavior 12. The Institutions factor consists of various sub-factors, including the policy legacy, or the attributes of previous policies that can shape the policy process through the creation of incentives and giving some actors access to more or fewer resources in comparison to others; and the government structure and characteristics of the policy-making process (e.g., response time to a problem and level of approval required for a policy in a given country) that can structure policy behavior 13,36.
Interests	Interests relate to the various stakeholders’ agendas - social groups, public employees, elected officials, researchers, and policy advocates 12,13. The Interests factor (various stakeholders’ agendas) refers to the desire to influence the policy-making process to achieve their ends, in order to push or prevent the development of policy choices 12,13. This factor expresses the political actors’ perceptions concerning the winners and losers from the outcome of a given policy, and the extent of their gain or loss 12,13.
Ideas	Ideas can influence how actors define the problem and how they perceive the different policy options as effective, feasible, and acceptable. This factor includes research and other types of information and the values of the policy-makers, policy advisors, and other stakeholders 12,13.
External Factors	The framework also considers External Factors, which can provide opportunities to introduce changes in a policy process 12,13.

The thematic data analysis occurred deductively through predetermined categories and subcategories proposed in the conceptual framework and inductively with subcategories that emerged from the reading of the transcribed interviews.

Ethical aspects

The study was approved by the Institutional Review Board of the Institute of Social Medicine of the State University of Rio de Janeiro, Brazil (CAAE:49733215.4.0000.5260) and the Ethics Council of the Institute of Hygiene and Tropical Medicine of the New University of Lisbon, Portugal.

Results and discussion

The results are organized in subsections with the research objectives in order to allow comparisons.

Factors that influenced the policy decisions for the strategies in Brazil and Portugal

The factors that appeared most often in the interviews as having influenced the More Doctors Program (PMM, in Portuguese) were (Ia) Institutions, with the subcategories referring to government structure and policy legacy; (Ib) External Factors (Presidential elections); (Ic) group Interests (mayors and medical associations), governments (Brazil and Cuba), international organization (Pan American Health Organization - PAHO) and civil society; and Ideas (scientific evidence). The most frequently cited factors in the interviews in Portugal were (IIa) Institutions, policy legacies, and government structure and (IIb) Interests of governments (the Portuguese government and those of countries involved in the bilateral agreements), civil society and groups (medical associations). See Table 4 for examples of quotes by the interviewees.

Table 4
Examples of quotes concerning factors and sub-factors in the policies.

Strategy	Factor	Sub-factors	Citation
PMM	Institutions	Government structure - bureaucracy and centralization of the process	“The Ministry da Health is tough, it’s a heavy bureaucracy, it’s a difficult process, it’s a country with a lot of rules, with a lot of difficulties, with a lot of situations. It’s obvious that in Brazil and in the Ministry, a centralized process in a country this size is not an easy operation” (BR_PM1). “In the Ministry of Health in recent years there has been a misguided re-centralization of health activities. The Ministry moved from a normative and policy-making role to a more operational role, and with highly vertical action in the local and state territories. I think this is critical, I this contradicts the framework of the Brazilian Unified National Health System (SUS)...” (BR_CS1).
	Institutions	Policy legacies - constitutional principle and previous programs	“And at the same time we also reclaimed the Constitutional principle, where Article 200 of the [1988] Federal Constitution states that it’s the prerogative of the SUS to organize human resources training, so it provided the basis for the need, not only for programs, for example, to attract [health] professionals to a given location, but also to think of training, to think of continuing education, to think of various dimensions” (BR_PM2). “We have the understanding, for example, that it would not have been feasible to implement the PMM so quickly, or that in 9 months there would already be 14,000 physicians participating in the program, if it were not for the experience we already had with PROVAB. So this experience in the Ministry of Health to issue a national call for physicians, to allocate physicians in localities, to make the payments to these physicians, to supervise these physicians, and to monitor the development of the activities” (BR_PM2).
	External Factors	Presidential elections	“...based on all the existing evidences, it was apparent that the main problem was the government’s need in the pre-election stage to have a star program, a program known in the health area (...) It was very attuned with the election calendar, which makes sense, since a lot of things happen for political reasons, and this case had a strong political relationship” (BR_IO).
	Interests	Actors that gained - governments (Cuba and Brazil), mayors of municipalities served by the PMM, civil society, and PAHO	“...the President [Dilma Rousseff] wanted to guarantee the program, there was the population demanding more access [to healthcare] in the demonstrations, so there was this window of opportunity, the mayors were demanding, the President was supporting, and Brazil had the funds” (BR_Staff). “Referring to the demonstrations, ‘...that accelerated, made the President turn to us and say, that proposal you were developing, is it ready, or isn’t it ready?’. It’s ready. So let’s launch it” (BR_PM3). “One actor that gained a lot was PAHO, which got 50 million dollars and a lot of relevance with the [Brazilian] government” (BR_IO).
	Interests	Potential loser with the program - the medical profession	“I think nobody lost, I don’t see it. I noticed that the Brazilian Association of Physicians and the Federal Board of Medicine initially mounted strong opposition (...) but later they realized that these doctors were forbidden from working outside of this program, so they weren’t going to be, and never would have been a source of competition (...) I think the program proved that there was something to gain for everybody” (BR_IO).
	Ideas	Scientific evidence - use in the policy process	“Since 2011 when I’ve been in the Ministry, we use a lot of evidence to make decisions. For example, for us to discuss the PNAB, the President herself asked for scientific articles. She turned and said, look, I want some articles that show how the PNAB worked, is it really family health? What’s the impact? What is the best way to invest money in primary care?...” (BR_PM3). “...in mid-May 2013, the President convened a meeting with the medical associations, to hear these organizations about the problem the mayors were raising insistently, and to hear these organizations about the alternatives they saw (...) So they created channels to discuss the issue. The issue was the difficulty in acting as channels with proposals, because it was not about opening new medical schools, because it was not about brining doctors from other countries...” (BR_PM2). “One would say, there’s a shortage of doctors, the other would say there are doctors, but they’re poorly distributed. That actually both things are true, but one emphasized one thing and the other emphasized the other (...) But it all fell on deaf ears, it wasn’t a discussion. The Administration had already made the firm decision to bring doctors from Cuba, much earlier, at least a year before (...) And [medical associations] all over the world take the position against importing physicians from other countries” (BR_IO1).
	Ideas	Scientific evidence - mechanisms to promote the use of scientific evidence	“These data served as backing, they had already been used before independently, I mean this feeling, this evidence already existed. So the policy was backed by this evidence that already existed, to be implemented (...) The studies were financed by the Ministry of Health and by the Observatory, in our case, and via PAHO” (BR_P4). “So the mechanisms varied. When we want something more quickly and the collaborator can present interim results, precisely to shed light and produce evidence for the policy, we usually fund directly. Because if we use a bidding process, the research may start today and it’s two or three years before you have any findings. And then the guy is going to write the article, and he doesn’t release [the findings] before the article. By the time the article is published, you’re no longer in office. When we want to do [the policy] to feed the decision-making process, we fund directly” (BR_PM3).
	Ideas	Scientific evidence - reasons for use	“The second report was very interesting because it was used both by the CFM and the medical associations to prove that there was no shortage of physicians in Brazil and to say, ‘Look how unevenly physicians are distributed, how there’s a shortage of physicians.’ So if you look at the medical demographics, they serve as the basis for the position by the medical associations, against More Doctors, and it’s part of the PowerPoints and publicity materials, saying that the medical demographics prove, even give evidence, that there’s a shortage of physicians and that they’re needed. (...) So aspects that may be negative for one or the other political purpose are selected from the production and the scientific evidence” (BR_P1). “I think he promotes, like every policy-maker promotes. In principle the policy-maker does not decide on the basis of scientific evidence, he decides as a function of political pressure or the political demand that’s out there. Later he fine-tunes the response based on the scientific evidence, but this issue is triggered more by the perception, sometimes rather diffuse, of a certain reality, which is adjusted later by the evidence” (BR_P2).
Bilateral agreements	Institutions	Policy legacies - past experience with the bilateral agreement	“According to estimates in 2008, it would take us approximately 7 to 10 years to have enough physicians for every Portuguese citizen to have a family doctor. This was especially difficult in some regions of the country where [physicians] didn’t want to go. And because there was already some international experience with this. Because in 2006, when I needed physicians to work only in medical emergency departments (...) I couldn’t get them in Portugal. Because in Portugal, in the physicians’ market, the supply and demand are turned around. (...) So there was an opportunity to sign an agreement with Uruguay to bring doctors to work in medical emergency departments” (PT_PM3).
	Institutions	Government structure - centralization of the decision-making process	“...It was just me and two secretaries. So when I left there was nobody to deal with this. So I had to do everything, I hired the Camões Institute to teach Portuguese history. I hired Portuguese teachers… the entire policy was designed in the Ministry and thus with my work I answered directly to the Minister, so it was a lot easier, right?” (PT_PM3).
	Interests	Interests - political need and population’s need	“It was political, because any government administration wants to say it gave doctors to everybody. And there had to be alternatives, as I said, there were only two [either increasing the list of users per physicians or hiring foreign doctors] (...) because there weren’t enough physicians” (PT_PM3). “In 2008 we were faced with this, we had done what was strategically adequate to solve the shortage of physicians, but we had a shortage of physicians because the strategic measures were going to take a long time to take effect” (PT_PM2). “Portugal’s objective was always the same. What they gave the country had to be consistent with what the countries wanted or needed” (PT_F3).
	Interests	Certain resistance to the strategy - medical profession	“I couldn’t identify anyone that might lose (...) There’s always a little professional pressure, but I think everyone gained from this (...) And always based on an overall number, which is that Portugal has enough physicians by international comparison. Maybe it does. I’m not going to question that. Portugal may have [doctors] but there aren’t any in Odemira. I launched an admissions process, and nobody applied... I think my obligation as a policy-maker is to find a doctor that wants to go to Odemira” (PT_PM2).
	Ideas	Scientific evidence - did not contribute to the policy process	“I don’t know what scientific evidence would be better than the absence of physicians interested in a place in a procedure that everybody knows? What could I do differently?” (PT_PM2). “From there, for us to base these policies on a sociological study, on a scientific study, I doubt that anybody will ever do one, and even generally speaking I think that policy-makers rarely base their decisions on scientific studies. I hate to say it...” (PT_EM1).
	Ideas	Scientific evidence - mechanisms to promote the use of scientific evidence (in shaping the policy agenda)	“I commissioned the study, by a geographer who was very interested in spatial planning, and was thus very familiar with the country’s spatial situation in terms of where the health services are, where the health centers are, the population densities and the shortages of physicians, she worked extensively with those data and was writing a lot about it at the time, and she was particularly well versed on the subject. And it was a way to obtain a study quickly…” (PT_ PM4).
	Ideas	Scientific evidence - reasons for use	“Policy-makers often understand scientific evidence from an instrumental perspective, that is, scientific evidence interests them to the extent that it corroborates a decision one has already made, and not exactly because it changes a decision just because it’s scientific evidence. Besides, many of these decisions are based on common sense and don’t require a lot of scientific evidence” (PT_EM1).

• PAHO: Pan American Health Organization; PMM: More Doctors Program; PROVAB: Program for Valorization of Primary Care Professionals.

Ia - Institutional factors in the PMM (Brazil)

In relation to government structure, policy-makers reported that the internal difficulties in the Ministry of Health, bureaucratic issues, and centralization of the policy process and program deployment in a country the size of Brazil all hindered the policy process (BR_PM1). According to the policy-makers, the Ministry of Health played the role of designing, coordinating, and implementing the PMM and leading the changes in the physicians’ training process, together with the Ministry of Education and the Office of the Chief of Staff. The process also involved the Brazilian Ministry of Foreign Relations, Ministry of Defense, Internal Revenue Service, Federal Police, Ministry of Labor and Employment, and Bank of Brazil (BR_Staff/BR_PM1). The Ministry of Health’s role in providing the guidelines and policy decisions was challenged by interviewees in the interest group, according to whom the Ministry of Health, with the PMM, began to adopt an operational role, contradicting the organization of the Brazilian Unified National Health System (SUS) and indicating a possible re-centralization of healthcare activities (BR_CS1).

According to the policy-makers, policy legacies gave impetus to and shaped the policy process and also mitigated the difficulties. The process drew on a principle from Article 200 of Brazil’s 1988 Federal Constitution, according to which the SUS has the mandate to organize human resources for health training. This mandate provided the basis for the additional interventions in the training and continuing education programs, namely expanding the areas of policy intervention used to act on the issue of distribution of physicians in the country (BR_PM2).

The policy-makers and interest group actors also emphasized the importance of programs that preceded the PMM and aimed to correct the uneven distribution of physicians. The Program for Territorial Decentralization of the SUS (PISUS) and Program for Territorial Decentralization of Healthcare Work (PITS) helped demonstrate that isolated interventions tend not to spawn long-term sustainability, and that multiple interventions in different policies are needed to achieve the objectives and sustainability. The experience with these programs was short-lived, involving a limited number of health professionals, and they failed to change crucial issues in the medical training process (BR_CS1/BR_PM1/BR_PM2).

More recent programs strengthened the Ministry of Health’s institutional capacity, which proved essential for developing the conditions to design and implement the PMM. These previous experiences allowed greater agility and scale for the implementation of the PMM (BR_PM2/BR_Staff).

Ib - External Factors to the PMM (Brazil)

As for External Factors, nearly all of the interest group actors mentioned the Presidential election as a factor that influenced the PMM, citing the need for a quick policy and program that could be identified as the Administration’s brand in the health area.

Ic - The Interests factor in the PMM (Brazil)

According to the three groups of interviewees, the actors that gained the most with the program were: governments (Cuba and Brazil), mayors of the municipalities served by the PMM; civil society, and PAHO. The Brazilian medical profession was identified as a potential loser.

According to the policy-makers, the initial milestone of the PMM took place when a new Minister of Health took office: in his first few months in office in 2011, he organized a national seminar in Brasília on the shortage, provision, and retention of health professionals in remote and more vulnerable areas. In addition, the Multi-annual Plan and the National Primary Health Care Policy (PNAB), central pillars in the organization of the SUS, cited a shortage of physicians as one of the difficulties for upgrading and expanding primary healthcare, showing that this issue would be a priority on the health policy agenda at the time.

According to the policy-makers, the PMM was a political decision by the President of Brazil, who in February 2012 mandated the Ministry of Health to design a program capable of increasing the supply and quality of physicians in PHC. The issue was discussed for a year and five months, during which time information was gathered for organizing a diagnosis, proposing interventions, and identifying priority places for action. The possibility of cooperation with Cuba began to materialize in April 2013 (BR_Staff).

Also according to policy-makers, in early 2013 the Program for Valorization of Primary Care Professionals (PROVAB) filled 3,500 vacancies, but the demand from local governments was for 13,000 physicians. At the time, one of the factors cited in nearly all of the interviews (as one of the most important at the time), the National Coalition of Mayors (FNP), launched a campaign called “Where’s the Doctor?”. The FNP appreciated and drew on previous initiatives such as the Fund for Financing Students in Higher Education (FIES) and PROVAB, while emphasizing that more action was needed. To advocate for a solution to the problem, the mayors relied on non-systematic or unstructured evidence (BR_PM1).

Most of the interviewees cited the protests by Brazilian civil society in June and July 2013, spearheaded by the “20-Cents March” (against a fare hike in municipal buses), as catalysts for the policy ¹⁴. Health was a key issue on the national agenda, and the demands included quality improvement and expansion of access to services (BR_PM3). One of the actors in the interest group also identified this time as a window of opportunity for human resources for health policy in Brazil, which was subsequently confirmed (BR_CS1).

The Cuban government and PAHO were also identified as winners in the program’s policy-making. According to the interest group actors, the program was initially conceived as a political agreement between the government of Cuba and the government of Brazil with participation by PAHO, with specific characteristics of economic cooperation (BR_IO). There was an increase in PAHO’s relevance in the organization’s interaction with the Brazilian government (BR_IO).

Finally, interviewees showed some reluctance in identifying the actors that stood to lose from the program’s policy. The policy-makers stated that up to a certain point the medical associations, especially the National Federation of Physicians, Brazilian Medical Association, and Federal Board of Medicine (CFM), supported initiatives like PROVAB. Still, although the policy-makers and some interest group actors did not see the program as harmful to Brazilian physicians’ employability or income (referring to the emergency component of hiring foreign physicians), as perceived by these actors, the medical associations as a whole proved resistant and fearful that they might lose from the implementation of the PMM (BR_PM1). One interest group actor also identified an unexpected final gain for Brazilian physicians, who wanted the PMM physicians to cover unserved areas of the country (BR_IO).

Lobbying by medical associations against the PMM led to a change. The policy’s initial version included the “second cycle in medicine”, which consisted of extending the course of medicine for two more years to include an internship in PHC. In the face of the resistance, this intervention was changed.

IIa - Institutional factors in the bilateral agreements (Portugal)

As an institutional factor (policy legacies), the past experience with the bilateral agreement with the Uruguayan government for provision of emergency medical services was cited as a factor that influenced subsequent policy decisions. The possibility of this agreement was identified by the exchange of experiences between health policy-makers from both governments in an Ibero-American meeting, and the policy-making process began in 2005 and 2006, while the policy was implemented in 2008. Portugal’s part of the agreement was to train physicians and nurses in liver transplantation during a year at the Curry Cabral Hospital in Lisbon and to provide support by the Portuguese team in the first liver transplants performed in Uruguay. Based on this experience, in 2008 the Portuguese government commissioned a plan for hiring foreign physicians, to be designed by the person responsible for executing the bilateral agreement with the government of Uruguay (PT_PM3/PT_PM4).

Another facilitating factor cited by a policy-maker (from government) was the fact that the process was developed in the Ministry of Health and concentrated in a single policy-maker with direct access to the Minister (PT_PM3).

IIb - The Interest factor in the bilateral agreements (Portugal)

According to the interviewees, the actors that gained the most with the bilateral agreements were the governments and civil society. A potential “loser” that was identified was the medical profession, represented by the medical associations.

In relation to the interest factor, the policy process was identified by policy-makers as indispensable as a political need and the population’s need (PT_PM3). As for the population’s need, there was a large share of the Portuguese population without designated family physicians, which hindered the development of a reform program in PHC. Various isolated and ad hoc strategies were employed to deal with this problem, such as the implementation of a 25% quota for places in medical residency for family physicians. According to the policy-makers interviewed in this study, these strategies failed to produce an adequate short-term response, which required an intervention that would have obtained a “more immediate” response than training physicians (PT_PM2). But according to the interested actors, the bilateral agreements did not involve a significant number of physicians and were implemented to solve isolated problems in regions like the Alentejo and Algarve (PT_EM1).

According to the policy-makers, the decision to hire physicians from Latin America was due to the impossibility of hiring physicians from European countries, which was Portugal’s original choice due to the diploma’s automatic validation. However, physicians’ salaries in some of these countries are at least as high as in Portugal, while the cost of living is lower (e.g., Poland), and other European countries with shortages of physicians, like England, Finland, Sweden, Norway, and Germany, offer more attractive conditions (PT_PM2).

All the bilateral agreements included a counterpart role for the government of Portugal, except for the agreement with Costa Rica, where the negotiation was done with the Medical Council, which did not require it. Cuba received financial compensation specifically as an amount paid to the Cuban government for physicians with experience in the Community of Portuguese-Speaking Countries (CPLP) to provide healthcare in Portugal for three years. In the case of Colombia, Portugal’s part was an agreement between universities by which students would conduct research projects and Master’s and PhD theses between Portugal and Colombia, in Portugal.

According to the policy-makers, the Order of Physicians showed some resistance to the strategy, although Portuguese physicians were identified as actors that did not stand to lose from the policy, since the bilateral agreements provided that the foreign doctors would work in the country for three years, after which they would return to their own countries. The argument employed in the political opposition to the government was that Portugal was not short on doctors, but that the doctors were poorly distributed. As indicated by one of the policy-makers, this rhetoric had prevented effective political action for years (PT_PM2).

Ideas factor

The problem of geographic distribution of physicians is not exclusive to Brazil and Portugal and has been the subject of hot debate by academia for at least ten years. Although a series of possible strategies have been identified with greater or lesser strength of evidence ^15,16,17, there is no single model that can be implemented by any country and guarantee a solution to the problem. It is thus necessary to understand the problem’s causes, the characteristics and needs of the country’s system and health professionals, in order to make the policy choice and the necessary adjustments in the selected set of strategies.

The use of scientific evidence in policy-making is a complex process, not always systematic or transparent, and influenced by numerous factors, since decision-making is not an exclusively technical process, but is also inherently social ^1,12. The complexity of this process, specifically that of policy-making for human resources for health, has increased in recent years due to the number of actors involved and the growth in scientific evidence, both in volume and complexity. As revealed by the interviews, research is not the only factor that influences policy decisions, and we thus need to understand the policy-making process as a whole in order to understand the research needs and the moment in which it is possible to use the results.

When analyzing efforts to enhance the use of evidence in designing such strategies, we chose to examine the use of scientific evidence in the process and the moment in which it was used ^18,19, its purpose ^20,21, the approaches ²², and the strategies used to inform policy-making with scientific evidence.

Ideas factor in the PMM (Brazil)

According to the interviewees, the scientific evidence from research results and other data from the CFM and Ministry of Health were used in the PMM to both inform and oppose the policy ^{23,24,25,26,27}. As for the moment in which the evidence was used, the policy-makers reported that the use of domestic scientific evidence stemmed from the identification of the problem and its causes, from the needs for adjustments to strategies chosen for implementation, and more recently from the assessment of the PMM. In addition, sources used to identify possible solutions included the international literature, systematic international reviews, and other studies of strategies used in different countries ^15,16,17. The interviewees also identified the exchange of experiences with policy-makers from other countries during meetings of the World Health Organization (WHO), on strategies used to correct the uneven distribution of physicians. According to one researcher, no other policy had generated as much investment as the PMM to assess such a program (BR_P2).

According to policy-makers and researchers, PMM spawned a debate between medical associations and the Ministry of Health. According to the policy-makers, the medical associations used research results to oppose the PMM during the policy design and initial implementation stages ²⁷. They also used the argument that the lack of incentives for physicians to work in these underserved areas was the main problem, not the shortage of physicians (BR_P4).

According to researchers and policy-makers, one of the approaches that promoted the use of scientific evidence was the effort to collect evidence by users that turned to research to extract information and thus make decisions. In other words, such efforts are appropriate in situations in which the potential users have identified a knowledge gap and seek a timely solution ²². According to the policy-makers, researchers, and some interest group actors, another approach involved efforts at exchange, which occurs when producers or providers of research develop significant partnerships with a group of users who help them identify and answer relevant questions ²². The Secretariat of Labor Management and Health Education (SGTES) of the Brazilian Ministry of Health has a relationship with a set of researchers from the Network of Observatories on Human Resources in Health in Brazil (ObservaRH), who have produced a set of studies commissioned and/or financed by the Ministry of Health, and that served to inform the program’s policy-making (BR_P4).

According to policy-makers, a mechanism for streamlining the use of research results in the Ministry of Health was direct financing with the presentation of interim results by the researcher. According to the policy-maker, the traditional funding mechanism used by the Ministry of Health (call for projects) requires two to three years to produce results, plus the time for writing and publishing the article (BR_PM3).

Concerning the purpose for using the scientific evidence, the main discussion in the field focuses on three types of use of research: instrumental, conceptual, and symbolic ^8,20. In the specific case of the PMM, there is no consensus among the stakeholders concerning the purpose for the use of evidence generated by research, varying from conceptual use, which involves a more generic and indirect use of the evidence, instrumental use, with specific and direct application of the results, and symbolic use, in which the research is used to validate and support decisions that have already been made, for other reasons ^20,28.

Some interest group actors and researchers questioned whether the evidence had been used adequately in the case of the PMM. For example, one researcher mentioned that the report on medical demographics ²⁷ had been used both by medical associations contrary to the PMM, to contend that there was no shortage of physicians in Brazil, and by policy-makers, to support the claim that there was a shortage of physicians. Interviewees from the group of researchers and stakeholders in the PMM asked whether the abnormal and disproportional volume of investments in research, compared to other components of the health system, was really meant to validate the PMM (BR_P1).

Finally, two determinants were identified that facilitate the use of evidence, namely the acknowledgment of the importance of using evidence in policy-making by the stakeholders, including the policy-makers, and the characteristic of the policy-making process in Brazil, which includes forums for discussions.

The majority of the interviewees acknowledged the importance of: (1) use of research evidence in all phases of the policy process in Brazil and (2) an organizational culture for the use of such evidence. One of the policy-makers stated that evidence from both domestic and international research is used to shed light on policy debates, a position taken by the Office of the President, Office of the Chief of Staff, and the Ministry of Health (BR_PM3). However, some researchers interviewed in this study admitted that evidence is not the only factor that influences policy-making, and that it may not be the “priority” among competing factors. Meanwhile, evidence is used to enhance the response to a problem that was initially influenced by other factors such as political pressures and demands (BR_P2).

According to the policy-makers, there were various forums for discussion of the possible solutions to the problem of physician distribution in Brazil, such as the National Health Council (CNS). The discussions in these forums on the PMM lasted approximately four months, before the program’s implementation in July 2013 and during the process leading up to the Executive Order (BR_PM2).

According to the interest groups, these dialogues took place quickly and when the possible interventions were already being presented, rather than merely during the policy design. This stage involved technical discussion with divergent information among the actors. Based on data and studies, members of the government administration pointed to a lack of physicians in the target municipalities (counties), while the medical associations also drew on commissioned studies to contend that the problem was not the number of physicians but their distribution, and that the problem thus called for improving the career plan system for physicians (BR_IO1).

In a policy decision, even among specialists, there may be diverging opinions on the meaning, interpretation, and perception of sufficiency of evidence for making the decision, or even for its use to defend different policy solutions to a problem. In this scenario, some stakeholder groups, like specialists, can wield power and influence the type of evidence that will be used, as well as the purpose and moment of its use ²⁹. According to one researcher, the use of scientific evidence by renowned Brazilian institutes can lend a kind of power to policy-makers, a sort of power perhaps created by the “issue of scientificity”, based on the researcher’s impartiality. Thus, the empowerment of citizens and other actors in the use of scientific evidence to advocate for a strategy and participation in the policy debate can help reduce the incorrect use of such evidence (BR_P3).

Ideas factor in the bilateral agreements (Portugal)

Scientific evidence did not contribute to the policy decisions on the strategy for hiring foreign physicians in Portugal. According to the policy-makers, the policy decisions for this strategy drew on an exchange of experiences with policy-makers from other countries (PT_PM2).

Scientific evidence played a role in introducing the problem on the government agenda, helping identify the problem and thus raising the need to design a policy for the strategy through a descriptive study ³⁰ and a prospective quantitative study on the need for physicians ³¹. These studies were commissioned directly to the researchers by the Ministry of Health ³¹ and Ministry of Education ³⁰. Other types of evidence like technical reports with information taken from databases prepared by technical groups from the Ministry of Health were also used (PT_PM4).

The approach to promote the use of scientific evidence in shaping the agenda (inclusion of the physician distribution issue) was the effort to collect evidence by the user, the same mechanism used in Brazil. Direct financing was identified in Portugal as the way to streamline the process of using research results in the Ministry of Health (PT_ PM4).

The bilateral agreements in Portugal sparked a debate, and the medical associations were opposed to them, citing research results during the implementation phase ³². The Portuguese medical associations used the same argument as their Brazilian colleagues, namely that the lack of incentives for physicians to work in underserved areas was the main problem, and not a shortage of physicians (PT_EM1).

The majority of the interviewees acknowledged the potential for use of research in the initial phase of the policy decision-making process (shaping the policy agenda). In addition, some stakeholders questioned whether scientific evidence could really be used adequately, or only to support a decision already made (symbolic).

Thus, the low priority assigned by the main source of financial support (the national research funding agency) to research in health services or human resources for health ³³ and thus the limited number of studies in the area, besides the lack of recognition by the actors concerning the importance of using evidence in other phases of the policy, may have been reflected in the use of research centered on the identification of the problem for designing the bilateral agreements in Portugal.

The study presents some limitations that are inherent to the case study method, like the impossibility of extrapolating the findings to another context (external validity). To ensure the findings’ credibility (internal validity), the methodological steps were followed rigorously: a tested interview questionnaire and triangulation of data sources used different data collection procedures for analysis of the context ^10,11. Finally, the inferences were based on scientific evidence prior to the current study.

Final remarks

The factors most frequently identified in the PMM as influencing the policy process and competing with the ideas (scientific evidence) were the institutions (government structure and policy legacy), external factors (Presidential elections), and various stakeholders’ interests. In the bilateral agreements in Portugal, the interviewees reported that the process had been influenced predominantly by interests and institutions (centralization of the process), with a limited role for evidence in the problem’s inclusion on the policy agenda.

Contrary to the findings in the case study in Brazil, where evidence played an important role in the policy, in Portugal the scientific evidence produced by research was not identified as having contributed to the human resources policy. The study identified different determinants in the restriction of the role of evidence in the policy process for hiring foreign physicians in the bilateral agreements and the use of evidence in the policy process for the PMM.

On the one hand, the characteristic of the policy-making process in the country - the policy process in Portugal was concentrated in a few actors; and in the case of Brazil, the process involved a variety of actors, including the Ministry of Health, the Office of the Chief of Staff, and the Ministry of Education, and also forums for discussions inherent to the characteristic of the policy process in the country. On the other hand, the organizational culture for the use of evidence - in Brazil, evidence was included in the policy process with some instruments, especially the research agenda and with strategies like the ObservaRH. No institutional mechanisms were identified in Portugal that promoted the use of evidence in this human resources for health policy process.

In both countries, the study identified the need for interventions and policies that contemplate all areas of the health labor market and include measures that affect other sectors in addition to health. The potential reinforcement of the use of scientific evidence in policy-making could be achieved with investment in forums for discussion and platforms for interaction between researchers, policy-makers, and other users of research results and an increase in support for research through financing and mechanisms for the establishment of political priorities.

Future studies are important for evaluating the strategies implemented in these two countries, to analyze the drain of health professionals through emigration and the flow from the public to the private healthcare sector or to other sectors, besides dual employment, to produce a more accurate diagnosis of the distribution of physicians between levels of care, to explore the organizational culture for use of evidence by analyzing beliefs and attitudes, to analyze the work by knowledge-brokering organizations, using other policies, and to produce more studies on the factors that lead physicians to migrate or to work in underserved regions.

Acknowledgments

The authors wish to thank Prof. John N. Lavis for his valuable comments on the multiple case study method, the interviewees for their valuable time and information, and Pedro Leite Alves for his technical assistance. Thanks also to Brazilian National Research Council (PhD scholarship 01988/2012-7) and Rio de Janeiro State Research Foundation (E26/201359/2014) for the funding.

Referências

• ¹
  Oxman AD, Lavis JN, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP) 1: what is evidence-informed policymaking? Health Res Policy Syst 2009; 7 Suppl 1:S1.
• ²
  Strydom WF, Funke N, Nienaber S, Nortje K, Steyn M. Evidence-based policymaking: a review. S Afr J Sci 2010; 106:1-8.
• ³
  Campbell S, Benita S, Coates E, Davies P, Penn G. Analysis for policy: evidence-based policy in practice. http://preval.org/files/pu256_160407.pdf (acessado em 30/Dez/2016).
  » Analysis for policy: evidence-based policy in practice. http://preval.org/files/pu256_160407.pdf
• ⁴
  Ward V, House A, Hamer S. Developing a framework for transferring knowledge into action: a thematic analysis of the literature. J Health Serv Res Policy 2009; 14:156-64.
• ⁵
  Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, et al. Lost in knowledge translation: time for a map? J Contin Educ Health Prof 2006; 26:13-24.
• ⁶
  World Health Organization. World report on knowledge for better health: strengthening health systems. Geneva: World Health Organization; 2004.
• ⁷
  Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res 2014; 14:2.
• ⁸
  Innvaer S, Vist G, Trommald M, Oxman A. Health policy-makers' perceptions of their use of evidence: a systematic review. J Health Serv Res Policy 2002; 7:239-44.
• ⁹
  Waddell C, Lavis JN, Abelson J, Lomas J, Shepherd CA, Bird-Gayson T, et al. Research use in children's mental health policy in Canada: maintaining vigilance amid ambiguity. Soc Sci Med 2005; 61:1649-57.
• ¹⁰
  Oliveira APC, Gabriel M, Dal Poz MR, Dussault G. Desafios para assegurar a disponibilidade e acessibilidade à assistência médica no Sistema Único de Saúde. Ciênc Saúde Coletiva 2017; 22:1165-80.
• ¹¹
  Oliveira APC, Dussault G, Craveiro I. Challenges and strategies to improve the availability and geographic accessibility of physicians in Portugal. Hum Resour Health 2017; 15:24.
• ¹²
  Lavis JN, Rottingen JA, Bosch-Capblanch X, Atun R, El-Jardali F, Gilson L, et al. Guidance for evidence-informed policies about health systems: linking guidance development to policy development. PLoS Med 2012; 9:e1001186.
• ¹³
  Pomey M-P, Morgan S, Church J, Forest P-G, Lavis JN, McIntosh T, et al. Do provincial drug benefit initiatives create an effective policy lab? The evidence from Canada. J Health Polit Policy Law 2010; 35:705-42.
• ¹⁴
  Oliveira JPA, Sanchez MN, Santos LMP. O Programa Mais Médicos: provimento de médicos em municípios brasileiros prioritários entre 2013 e 2014. Ciênc Saúde Coletiva 2016; 21:2719-27.
• ¹⁵
  Chopra M, Munro S, Lavis JN, Vist G, Bennett S. Effects of policy options for human resources for health: an analysis of systematic reviews. Lancet 2008; 371:668-74.
• ¹⁶
  World Health Organization. Transforming and scaling up health professionals' education and training: World Health Organization guidelines 2013. Geneva: World Health Organization; 2013.
• ¹⁷
  World Health Organization. Increasing access to health workers in remote and rural areas through improved retention. Geneva: World Health Organization; 2010.
• ¹⁸
  Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst 2003; 1:2.
• ¹⁹
  Lavis JN. Research, public policymaking, and knowledge-translation processes: Canadian efforts to build bridges. J Contin Educ Health Prof 2006; 26:37-45.
• ²⁰
  Amara N, Ouimet M, Landry R. New evidence on instrumental, conceptual, and symbolic utilization of university research in government agencies. Sci Commun 2004; 26:75-106.
• ²¹
  Trostle J, Bronfman M, Langer A. How do researchers influence decision-makers? Case studies of Mexican policies. Health Policy Plan 1999; 14:103-14.
• ²²
  Lavis JN, Lomas J, Hamid M, Sewankambo NK. Assessing country-level efforts to link research to action. Bull World Health Organ 2006; 84:620-8.
• ²³
  Carvalho CL, Petta HL, Araújo JF, Girardi Junior JB, Oliveira VA. Avaliação nacional da demanda de médicos especialistas percebida pelos gestores de saúde. Belo Horizonte: Núcleo de Estudos em Saúde Coletiva, Faculdade de Medicina, Universidade Federal de Minas Gerais; 2009.
• ²⁴
  Carvalho CL, Farah JM, Araújo JF, Wan der Maas L, Campos LAB. Construção do índice de escassez de profissionais de saúde para apoio à Política Nacional de Promoção da Segurança Assistencial em Saúde. Belo Horizonte: Núcleo de Estudos em Saúde Coletiva, Faculdade de Medicina, Universidade Federal de Minas Gerais; 2010.
• ²⁵
  Observatório de Recursos Humanos em Saúde/Estação de Trabalho do IMS/UERJ. Sistema de Indicadores das Graduações em Saúde - SIGRAS. http://www.obsnetims.org.br/sigras/home.php (acessado em 30/Dez/2016).
  » http://www.obsnetims.org.br/sigras/home.php
• ²⁶
  Seixas PH, Corrêa A, Moraes J. Migramed - Migração Médica no Brasil: tendências e motivações. In: Pierantoni CR, Dal Poz MR, França T, organizadores. O trabalho em saúde: abordagens quantitativas e qualitativas. Rio de Janeiro: Centro de Estudos, Pesquisa e Desenvolvimento Tecnológico em Saúde Coletiva, Instituto de Medicina Social, Universidade do Estado do Rio de Janeiro/ObservaRH; 2011. p. 133-50.
• ²⁷
  Scheffer M, Biancarelli A, Cassenote A. Demografia médica no Brasil. v. 2: dados gerais e descrições de desigualdades. São Paulo: Conselho Regional de Medicina do Estado de São Paulo/Conselho Federal de Medicina; 2013.
• ²⁸
  Haynes AS, Gillespie JA, Derrick GE, Hall WD, Redman S, Chapman S, et al. Galvanizers, guides, champions, and shields: the many ways that policymakers use public health researchers. Milbank Q 2011; 89:564-98.
• ²⁹
  Greyson DL, Cunningham C, Morgan S. Information behaviour of Canadian pharmaceutical policy makers. Health Info Libr J 2012; 29:16-27.
• ³⁰
  Grupo de Missão. Plano estratégico para a formação nas áreas da saúde. Lisboa: Grupo de Missão; 2001.
• ³¹
  Santana P, Couceiro L, Alves I, Nogueira H, Costa C, Santos R. Caracterização demográfica dos profissionais de saúde no serviço nacional de saúde português. v. V: sumário executivo. Lisboa: Secretaria-Geral, Ministério da Saúde; 2007.
• ³²
  Santana P, Peixoto H, Loureiro A, Costa C, Nunes C, Duarte N. Estudo de evolução prospectiva de médicos no sistema nacional de saúde: relatório final. Coimbra: Ordem dos Médicos; 2013.
• ³³
  Dussault G. Plano nacional de saúde 2012-2016: roteiro de intervenção em recursos humanos em saúde (RHS). Lisboa: Direção-Geral da Saúde; 2014.
• ³⁴
  Ministério da Saúde. Programa Mais Médicos - dois anos: mais saúde para os brasileiros. Brasília: Ministério da Saúde; 2015.
• ³⁵
  Oliveira FP, Vanni T, Pinto HA, Santos JTR, Figueiredo AM, Araújo SQ, et al. Mais Médicos: um programa brasileiro em uma perspectiva internacional. Interface (Botucatu, Online) 2015; 19:623-34.
• ³⁶
  Pierson P. When effect becomes cause: policy feedback and political change. World Polit 1993; 45:595-628.

Publication Dates

• Publication in this collection
  2018

History

• Received
  30 Dec 2016
• Reviewed
  30 June 2017
• Accepted
  17 July 2017