evaluability of the Program to Value Primary Healthcare Professionals ( PrOVaB ) : management challenges

The objective of this article is to present the results of a study on the evaluability of the Program to Value Primary Healthcare Professionals (PROVAB in the Portuguese) that was created by the Brazilian Ministry of Health in 2011. The Program is part of the Manage Healthcare Work and Educationstrategy which seeks to invest in a number of measures aimed at improving and valuing the work carried out by primary healthcare teams. The research, which used qualitative methods, was carried out between February and November 2013 and involved five stages: (a) analysis of documents; (b) identification of potential users; (c) strategic analysis; (d) modelling of the intervention; (e) sharing of lessons learned. Data collection took place in three iterative phases: document analysis, key informant interviews and a workshop. The activities of the program were grouped into three areas: functional and working conditions, teaching/learning and management. The results showed that the program can be evaluated, since it was possible to specify its feasibility by means of a logical model. The potential and priority areas were mapped for future evaluations, whose central focus is to address the problem of unequal distribution of Brazilian health professionals.


introduction
Uneven distribution and the difficulty getting healthcare professionals to establish themselves in parts of Brazil, especially those in basic care, are important challenges that must be overcome to improve access, and the quality of healthcare in this country.This problem is not new, and distortions in the geographic distribution of healthcare professionals have been recorded since the 1970s.At that time, about 80% of all physicians were located in the Southeast and South, 17% in the Northeast and just 1% in the North 1 .Some 40 years later, even considering the increase in urbanization and higher wages offered, the distribution of physicians is still uneven.The ratio of physicians per thousand in habitants was 2.0 in 2012, below that of other Latin American countries such as Uruguay and Argentina, at 3.7 and 3.2 respectively.States with the highest ratio of physicians per thousand inhabitants are in the Southeast, where Rio de Janeiro has 3.62 and São Paulo 2.64, the highest in the country.The situation in the North and Northeast is still far from the country average.Pará has 0.84 and Maranhão 0.71 physicians per thousand inhabitants.While there is no specific standard for this indicator, the Ministry of Health (MH) uses the UK ratio of 2.7 physicians per thousand inhabitants, as theirs is also a universal public health system focused on basic care 2,3 .
The shortage of medical professionals in the more vulnerable cities and towns in Brazil was discussed in the study by Girardi et al. 4 , which found that the more isolated and remote parts of the country are more vulnerable to the shortage of medical care, in particular basic care.A survey conducted in 2008 in Aracaju, Belo Horizonte, Florianópolis and Vitória showed that the main reasons professionals do not stick to basic healthcare, in particular family health teams (FHT), were the poor working conditions, the requirement for full time dedication, and limited opportunities for professional growth and development.The authors explained that this situation leads to high turnover, compromising the continuity of care and thus the quality of the services provided 5,6 .
Providing healthcare professionals and getting them to stay in certain services and locations is one of the bottlenecks in healthcare management and education.This requires an incentive policy articulated with the training and education strategies to fulfill the constitutional requirements that underlie the Unified Healthcare System (Sistema Único de Saúde -SUS) 7 .
Maciel Filho 1 , in his doctoral thesis, reports some of the initiatives to address this situation, highlighting PITS, the Program to Bring Healthcare to the Interior, PROMED, the Program to Encourage Curricular Changes in Medical School, and Pro-Saúde, the National Program to Reorient Professional Healthcare Training.All of these proposals have emerged since 2001.They have different formats and suggest diverse types of intervention.The common goal is to promote changes in education and reduce the shortage of healthcare professionals by improving their geographic distribution in the various cities across Brazil's regions 8 .
Advancing in this debate to invest in addressing the issue of attracting and retaining physicians, nurses and dentists, in particular in the more vulnerable and remote locations, in 2011 the MH created PROVAB, the Program to Value Basic Healthcare Professionals.This strategy is part of the effort to Manage Healthcare Work and Education by investing in a set of measures that seek to qualify and value the work performed by basic healthcare teams, offering them adequate physical and financial working conditions, and access to on-the-job training [8][9][10] .
The purpose of this article is to present the results of the PROVAB Evaluability Assessment (EA) to check if the intervention is well designed, submitting it to a systematic assessment to provide subsidies to the decision on whether or not to implement the policy.

Method
This is an Evaluability Assessment (EA), also known as a pre-evaluation study to analyze PROVAB.This study is made up of a number of steps to maximize the use, potential and opportunity for assessment 11,12 .
This qualitative survey was performed between the months of February and November 2013, comprised of five phases or "moments", as per the approach developed by Thurston and Ramaliu 11 : (a) document analysis to explain the program goals and targets; (b) finding and listing the stakeholders; (c) strategic analysis; (d) intervention models; (e) publication of lessons learned.

Data gathering and sources
Document analyses used a variety of sources such as government directives, bid documents and published minutes regarding PROVAB between June 2011 and December 2012.We also checked the PROVAB Information and Management System (SIGPROVAB/SGTES/MS), and the SUS Arouca/Open University Platform (UnA-SUS/SGTES/MS).These sources allowed us to describe the program and its goals, targetaudience and targets, as well as actions to operate it, the resources required and the desired effect.This will help introduce new elements to explain the phenomenon under study.
This study is characterized as being a source for gathering data considered suitable to subsidize maps describing institutional policies to reveal and systematize converging and diverging points, as well as insights about the program practices.This type of study can be undertaken when a fact or phenomenon is actually happening, or after the fact 13 .Document analysis also enabled mapping potential users of the study -the so-called stakeholders, who were interviewed using a semi-structured questionnaire with questions pertaining to their profile, awareness of program resources, goals and actions developed.We also collected political-organizational context data, including barriers and possibilities for the intervention itself, and the use of the assessment results.The questionnaire was pre-tested and interviews conducted in August and September 2013.These were transcribed and analyzed using descriptive statistics and content analyses of interview responses 14 .Prior to collecting this data participants were asked to sign a Free and Informed Consent Form (FICF), and the secrecy and confidentiality of the data was explained to them.

PrOVaB Strategic analysis and logic Model
The purpose of the strategic analysis was to identify the relevance of the program or, in other words, the fit between PROVAB's goals and the problem situation identified as being relevant.This is an important step, as normally those involved in formulating and implementing the program have different ideas about its goals.The strategic analysis took into consideration the following guiding questions: 1.Is the problem selected pertinent?2. Are the goals suitable to address the problem situation?3. Are the partnerships created to solve the problem pertinent, bearing in mind their location and role?
The program was modeled on the proposition of Champagne et al. 15 for preparing Logical Models (LM), using data taken from document analyses and interviews.The LM shows that the intervention operation is internally rational.In other words, the interaction between the resources required, the activities to be performed and the expected effects in a favorable scenario, thus enabling a map of the program in operation and if it functions as expected.It will also reveal the more fragile aspects that may require further evaluation.
A workshop was organized to validate the strategic analysis and logical model with the stakeholders.During the workshop, we discussed and approved the relevance of PROVAB's goals and the plausibility of the relationships established in the LM.

results and Discussion
Concern with providing healthcare professionals and having them settle in parts of the country is nothing new.PROVAB is just one more experience focused on addressing this need, providing expanded access to quality healthcare to the entire population.
This program is managed by the Federal Government, which also funds scholarships for specialized studies in family health.Cities are in charge of hiring healthcare professionals and for compensation and living stipends, as per Interministerial Decree 2,087/2011.Oversight Institutions coordinate and supervise the medical professionals, while nurses and dentists are supervised by the monitoring area of the corresponding specialization courses 8 .
Eligible cities and towns can register on the PROVAB website.Eligibility is limited to extremely poor and hard to reach cities and towns, according to the criteria defined in the Directive, which are those that find it harder to attract and retain healthcare professionals, especially physicians 9,10 .
Chart 1 describes the stakeholders (potential users [of this study]), listing the role each one plays in the program, their potential interest in the assessment and their likely use of the findings.We identified the following potential users: The PROVAB National Coordinating Committee, the Committee to Implement and Monitor PROVAB, which plays a role as an advisory board, the Coordinators/Supervisors of the Oversight Institutions, independent consultants and city/ local managers.

PrOVaB Statistical analysis
In the first step of the strategic analysis, we listed the problems that lead to creating PROVAB to make sure it is pertinent.These are: professionals ill prepared to handle the new responsibilities of a service model centered on healthcare, difficulty consolidating the reform of the medical school curriculum, limited motivation for physicians to work in basic care, difficulty retaining physicians in the more remote and vulnerable areas, and a shortage of resources to pay for healthcare professionals in these locations.According to Ceccim and Pinto 16 , these problems have significantly compromised city and state management of healthcare systems for at least a decade.
In 2011, during the negotiations to create PROVAB, the Department of Labor and Education in Health Management (SGETS/MH) and the partner institutions decided that the biggest problem was the limited retention of physicians in the more remote and vulnerable areas.This problem was then selected as the platform for PROVAB's current strategic goals.This is a relevant problem, given its importance and feasibility in light of the problems SUS has faced in recruiting, providing and retaining healthcare professionals.Campos and Malik 17 considered limited retention of physicians to be a critical factor of success for implementing and expandchart 1. Grid with the potential users of the 2012 PROVAB Assessment.The assessment will gather reflections regarding how the working process of the professionals hired by PROVAB is organized, finding any issues that require adjustments, if any

role in using findings
The committee may use its findings to guide PROVAB management actions

Monitor PROVAB and abide by the recommendations
Help make any course corrections required, monitoring the educational activities proposed by PROVAB Implement the recommendations in the working process Improve the working process of basic care teams, and the care provided to SUS users ing Family Health teams in Brazil.In that study, the authors find that limited retention or high turnover of generalist physicians is due to three types of problems -the profile of the physicians enrolled in the family health program, how they are hired and the working conditions.
The next step was to check how well the likely causes of the problem (limited investment in training, inadequate working conditions, distance to healthcare units, inter-professional relationship problems and limited availability of supplies) matched the strategic goals of the intervention, formulated to address the problem situation, or in other words, to improve working conditions, promote access to on-the-job training and integrate teaching/service/community and on-the-job training.Even bearing in mind that there are different visions of this theme, the program goals are suitable, given what causes the problem and the ability to take action on these causes.According to Bezerra et al. 18 , one often sees healthcare projects/programs/policies display goals and guidelines that are not fully clear.That is not the case here.
In the third and last step of the strategic analysis, we found that the partnerships created to address the problem are also pertinent.Partnerships were adjusted taking into account the possibility of creating favorable situations for achieving the program's goals.With the stakeholders identified, and the role of each player in the program (Chart 1) known, it is clear that the partnerships established are consistent with the type of problem the program has been designed to address.
Figure 1, entitled strategic analysis round, summarizes all of the steps listed above.The goal of the person in charge of the strategic analysis round has to do with the program's institutional mission.PROVAB's mission has been defined as "allowing workers to feel valued and satisfied about their work in basic care".

intervention Modeling and logic analysis
Modeling the intervention and its logical analysis enables describing the paths the program has taken, illustrating its operating theory and describing the interaction between the resources required, the activities expected and the anticipated effects in a favorable scenario.
The set of activities expected by the program has been grouped along three dimensions: 1) Functional and operating conditions -those activities whose goal is to provide working con-ditions, housing and transportation, as well as compensation; 2) Teaching-learning, meaning those activities related to the supply or professional training and supervision courses, both face-to-face and distance; 3) PROVAB Management itself, with strategies to provide institutional support for the actions that will actually implement the program.It is worth pointing out that the logical model is not static, and must be reviewed during the course of the intervention to capture new aspects that would require adjustments or course corrections.
During the validation workshop, potential users analyzed the logical model based on the following criteria: clear description of the program goals; definition of plausible targets; actions/activities to make the program operational written in a consistent and objective way; well defined lists of the necessary structure/resources to undertake the actions; consistency with the anticipated results and a definition of the target-audience.The reviewed and agreed LM is shown in Figure 2.

Operational and Working conditions
To reach PROVAB's goals in terms of providing healthcare professionals, one must provide suitable work facilities in terms of infrastructure, equipment, medicine and other working materials.
PROVAB's model includes a team of supervisors and communication equipment to better monitor team working conditions, in an attempt to overcome large workloads, the shortage of medicines and materials and especially to provide backup in the form of other types of attention required to provide full healthcare.In fact, working conditions such that professionals may perform their tasks, which include a location that is clean and comfortable, suitable materials available and physical and environmental security are all important factors when attempting to provide healthcare professionals, in particular at the locations that are PROVAB targets.
Ney and Rodrigues 19 found high turnover among physicians to be associated with professional dissatisfaction, inadequate working conditions, overworked professionals, distorted compensation and problems with, or the absence of a position, career and wage plan.Regarding the working environment, this same study showed problems with the healthcare units, often operating in adapted facilities such as schools or community associations, and problems with basic utilities such as water, light and telephone ser-vices, all of which made it harder for the professionals to perform their tasks.The lack of computers and internet access also made it harder for teams to communicate with coordinators.
The equivalent compensation provided to family health teams, and the 10% score bonus in the medical residence entrance exams for those interested in such programs were mechanisms created to entice physicians to sign on with PROVAB.However, the study by Campos and Malick 17 shows that wages have not ensured retention, as these locations are normally far away and have no economic development or sociocultural attractions.
The nature of the employment contract has also led to dissatisfaction and insecurity on the part of many of the physicians and other basic healthcare professionals, as it is normally temporary.A study by Mendonça (2010) showed that a civil service exam and a career plan specific for healthcare are essential for retaining these professionals.In the case of PROVAB, contractual instability is aggravated by the cities and towns involved, which claim not to know the published rules and their responsibility in the agreements signed with the professionals.

teaching-learning
PROVAB also assumes consolidated integration of education-service-community and on-the-job training.Constant face-to-face and distance continuing education activities are included, through the SUS Open University.Educational activities have been included in PROV-AB to foster the acquisition of knowledge and the development of skills and mindsets, guided by the quality of the healthcare provided to the population 8,9 .
Continued education activities will be decided and agreed between PROVAB professionals and their supervisors, linked to Teaching Hospitals or Institutions, City or State Departments of Health, or Medical Residence Programs in Family and Community medicine.Supervision, which

Problems identified (Problem Situation)
-Professionals who are ill prepared to deal with the new responsibilities required by the care model that is centered on healthcare; -Difficulty consolidating any reform of the medical school curriculum; -Physicians have little motivation to work in basic care; -Limited ability to retain med in remote and more vulnerable areas; -Scarce financial resources to pay healthcare professionals, especially physicians, in the more vulnerable and remote cities and towns

Problem selected
Limited ability to retain med in remote and more vulnerable areas

Goal of the person in charge
Allow workers to feel valued and satisfied with working in basic care.

intervention strategic goals
-Qualificar e valorizar o trabalho realizado pelas equipes de atenção básica; -Induzir melhoria das condições de trabalho, tanto físicas quanto financeiras; -Promover acesso à formação em serviço; -Integrar ensino-serviço-comunidade e a educação pelo trabalho according to the parameters set by PROVAB, will be eligible for a 1% a month discount on their FIES (student) loan.Supervisory activities include an initial diagnostic of the territory and of the team's working processes, in relation to the professional who is being supervised.This will result in the joint development of a continued education program.Contracted professionals will be asked to undertake specific intervention projects that are focused on the specific territory 3,[8][9][10] .A survey by Girardi et al. 4 found that the main factor to attract physicians is the possibility of continued education as a strategy to reinforce the working process and invest in care practices.This scenario assumes the professionals and teams will have credibility with patients and other care providers, which depends on cultural changes and on the training of these professionals.

Final Goals
Concerned with ensuring the key skills and competences required to achieve the goals defined for reorganizing this level of healthcare, the MH offers continued education elements such as a specialization course focused on basic care through the SUS UNA System (Open University).However, in 2012 this course was not required for physicians, but focused primarily on the nurses and dentists selected for the PROVAB program 8 .Professionals also have a tele-health hub and a virtual community on the PROVAB Management Information website (SIG-PROV-AB), where professionals can join discussions forums to exchange experiences and ask questions of supervisors and professionals assigned to other locations.
According to Campos et al. 7 , a combination of regulatory, educational and financial strategies is required, such as expanded tele-health, permanent education and universal medical residence to enable a more sustainable movement towards providing healthcare professionals.In his Ph.D. thesis, Capozzolo 20 analyzed the working conditions of family health teams, finding that high physician turnover is partly due to the insecurity generated by the lack of professional training as general practitioners.Therefore, interactive virtual environments that combine different players may facilitate a more participative dialog that can impact the working process and healthcare practices, focusing on a model that is centered on healthcare 5 .

PrOVaB Management
PROVAB's organizational design includes a number of management activities in order to achieve the proposed goals, in particular sup-port for cities and states in implementing the program's activities, for the institutions that supervise the teaching-learning process, and for the implementation of the communication and monitoring system.It is worth mentioning that integration cities into the process of managing healthcare, while desirable, requires specific competences on the part of the cities, such as the ability to manage working teams, along with the required physical and technological infrastructure, inputs and strategies.PROVAB must include an arrangement that facilitates oversight and helps cities take on this pedagogical role in an integrated way, supporting the university within the SUS perspective 20,21 .
The role of the State as a partner in an inter-federative management policy has not been well explored.However, as getting professionals to settle in these cities and towns is a complex issue, the different spheres of management must partner and work together to expand the debate and increase the sustainability of this strategy.Although COSEMS and CONASEMS are part of the management committee, integrated support and policies across all areas and regions require an agenda of inter-institutional cooperation, which will broadly contribute to addressing the issue or providing and retaining healthcare professionals in all areas of the country.
Finally, regarding the PROVAB monitoring system, we found no grid with the structure and process indicators, which is a fundamental planning and management tool, subsidizing decisions made by managers at different levels (services, cities, states), as well as professional practices.

Political-organizational context: PrOVaB potential and challenges
According to Leviton et al. 22 , evaluability assessments are particularly recommended to identify the need for knowledge and information on the part of stakeholders, and as a means to explain the political and organizational context of the intervention, recommending a focus and assessment methods.According to the authors, in this step, one extracts the "social reality" of a given program, how it is viewed by those operating it and other players.
Based on this perspective, we interviewed fourteen key players representing their committees, including independent consultants, supervisors and PROVAB coordinating and progress committees.About 80% of the interviewees were directly involved in activities related to managing the program, especially tracking and monitoring professional activities.
The interviewees believe PROVAB was implemented to address the problem of a shortage of physicians, and the difficulty getting them to settle in remote and needy areas, in particular to provide basic healthcare.This is demonstrated in what the interviewees say about the problems leading up to the implementation of the PROV-AB program.
Difficulty retaining healthcare professionals, a shortage of professionals [physicians], training as a valuation strategy and closer proximity to basic care (Interviewee 3).
The shortage of basic care professionals, especially physicians, medical training that focuses on local specialists, and a broader teaching service (Interviewee 8).
Investments in professional training were mentioned by interviewees as one of the program's strengths, in particular the UNA-SUS distance learning courses focused on basic care, and the PROVAB virtual community, both spaces for professional education under the SUS umbrella.Something else interviewees mentioned as a professional advantage was the bonus for the Medical Residence entrance exams.
The strategy of offering physicians a bonus for the medical residence exam has attracted a number of physicians (Interviewee 2).
Another facilitating factor was "political will", which positioned the program as a priority for the MH agenda.The multi-institutional configuration of the program was also seen as essential for mobilization, which strengthened and legitimized the program.As mentioned in interviewee reports.
The political decision to create the program and the availability of financial resources (Interviewee 6).
Ministry of Health players assigned to the states improve the ability to listen to the local reality ... Internet and e-mail lists with external people and web-conferencing ... a permanent agenda of monthly meetings in Brasilia with state representatives of the Ministry of Health (Interviewee 9).
Among the barriers mentioned are the resistance of medical entities in adhering to the bonus granted to PROVAB professionals interested in entering residence programs, difficulty making the problem operational due to logistics issues, and changes in contracts with weak [employment] bonds.It is also important to mention that the players are aware that the resistance of professional entities has an unfavorable impact, leading to tension and conflict in attempts to destabilize the program.Most interviewees argued that to disqualify PROVAB with no positive proposition created negative interference.Another critical element is the political pressure that contributes to the need to reformulate answers within a given management period, making it harder to reflect on the decisions made.
Conservative medical education structures, resistance to implementing the 10% bonus for medical residence entrance exams [...] disagreements over the model of access to medical residence [...] weak contracts, the relationship between the city and the professionals, unstable employment bonds (Interviewee 5).
The stipend program provides a weak professional or employment bond, management skills at the city level are few or lacking, and the development of the program concept (Interviewee 4).
Communication problems between the spheres of management used as a tool for negotiating between multiple interests, and little clarity regarding the activities of each player involved in the program were also mentioned during the interviews.This situation is detrimental to good operations and understanding of the PROVAB guidelines.We also heard reports of city managers who are unfamiliar with the rules of the program, making it harder to disclose it and to get professionals to join.
Truncated data, rule changes during the course of the process, resistance by doctors, misinformation on the part of managers, reinforced by truncated information (Interviewee 2).
Too many players making communication difficulty, manager turnover at the Ministry level (Interviewee 3).
Despite the difficulties to implement the program, most (50%) of the interviewees believe PROVAB is reaching its goals, especially as regards providing and retaining healthcare professionals, in particular physicians, who are qualified to provide basic care, as explained by the interviewees.
Providing physicians with experience in basic healthcare, increased access to healthcare services for the population, especially in the more needy areas (Interviewee 14).
Train experts in a SUS-focused perspective (Interviewee 5).
The reports above show that providing/retaining professionals, in particular physicians, is a complex task, especially given the diversity of this nation's realities.PROVAB appears to be a valid initiative to address a job market, focus-ing on strengthening the family health teams.However, one should not ignore that the main attraction, according to those we interviewed, is the bonus in the medical residence entrance exams.Investing in training healthcare workers was mentioned as a positive, even though it must advance in its pedagogical approach, adding education and healthcare management 23,24 .

Final considerations
The Evaluability Assessment (EA) showed that PROVAB has clearly defined goals and targets as a response by the government to address a complex problem related to the difficulty providing and retaining professionals.However, it also revealed that implementation is diffuse and there is no consensus among stakeholders regarding its "political urgency", creating barriers to consolidating the program in cities and towns, similar problems having been faced by the "More Physicians" (Mais Médicos) program.
This intervention can be evaluated, as we were able to characterize the elements of which it is comprised, providing subsidies to understand the processes implied in making it operational.Strategies whose challenge is to overcome the shortage and uneven distribution of professionals in this country include diverse types of intervention -those that are by nature educational, regulatory or managerial, as well as direct and indirect financial incentives.Therefore, putting together a range of coordinated actions seem to be the path to strengthen the program.When considering lessons learned from the PROVAB evaluability assessment, it is valid to highlight a number of issues to improve the intervention, which may be prioritized in subsequent assessments.In addition to more consistent assignments to the different players, it is important to encourage a broader scope of action of the supervisor, and better communication flows between players to facilitate access to strategic information that will guide the decisions made by the various levels of the program.This is a very opportune moment for a further assessment, considering that the program is in the implementation phase and the results of the Evaluability Assessment described herein.Even considering some of the limitations of its applicability, the intention was to submit a pre-assessment of PROVAB, signaling the possibilities for improved implementation of the strategy, contributing to improving and enhancing the program in the scope of work management.

collaborations
CM Oliveira, MM Cruz, RMC Torres, SEC Kanso, ACGV Reis, AL Gonçalves, SC Carvalho, A Lima and V Grabois helped design the study concept and literature review, and were involved in defining the methodological design, discuss the results and draft and critically review the text.RMC Torres, SEC Kanso and SC Carvalho also helped with data processing and analyses.

Figure 1 .
Figure 1.Round of strategic analyses of the Program to Value Basic Care Professionals (PROVAB), 2012.