Evaluation of the Brazilian Family Grant Program: a quasi-experimental study in the State of Rio de Janeiro

A avaliação do Programa Bolsa Família: um estudo quasi-experimental no Estado do Rio de Janeiro

PSYCHOLOGICAL ASSSESSMENT

Evaluation of the Brazilian Family Grant Program: a quasi-experimental study in the State of Rio de Janeiro

A avaliação do Programa Bolsa Família: um estudo quasi-experimental no Estado do Rio de Janeiro

Luciana Mourão^I; Maria Cristina Ferreira^I; Anderson Macedo de Jesus^II

^IUniversidade Salgado de Oliveira, Niterói, Brasil

^IIPrograma das Nações Unidas para o Desenvolvimento, Brasília, Brasil

^{Endereço para correspondência}

ABSTRACT

This study aimed to evaluate a program of income distribution in Brazil (Family Grant Program, FGP) using a quasi experiment in shantytowns in the metropolitan region of Rio de Janeiro. Questionnaires, which contained questions regarding the three main axes of the program, were personally administered to 281 program participants and 249 non-participants (control group) in either their homes or in public spaces of the shantytowns. The results of several statistical tests (t-test, correlation and Mann-Whitney-Wilcoxon test) identified significant differences between the groups of beneficiaries and non-beneficiaries, which signal a positive influence of the FGP, including increases in the number of daily meals and family income. However, some conditionalities of the program are not being fulfilled.

Keywords: Social psychology, evaluation, transfer income, quality of life.

RESUMO

Este estudo teve como objetivo avaliar um programa de distribuição de renda no Brasil (programa de bolsa família, FGP) usando um quasi-experimento em favelas na região metropolitana do Rio de Janeiro. Questionários, que continha perguntas sobre os três eixos principais do programa, foram administradas pessoalmente a 281 programa participantes e 249 não-participantes (grupo controle) em qualquer de suas casas ou em circulam normalmente espaços públicos em favelas. Os resultados de vários testes estatísticos (teste t, correlação e teste de Wilcoxon-Mann-Whitney) identificaram diferenças significativas entre os grupos de beneficiários e não beneficiários, que sinalizam uma influência positiva da FGP, incluindo aumentos no número de refeições diárias e renda familiar. No entanto, algumas condicionalidades do programa não estão a ser cumpridas.

Palavras-chave: Psicologia social, avaliação, transferência de renda, qualidade de vida.

With regard to social programmes throughout the world and especially in developing countries, there has been an increase in actions and social politics based on the demand that is generated by the deepening of inequalities. In Brazil, however, there is no tradition of social programme evaluation (Jannuzzi, 2005; Rocha, Albuquerque, Coelho, Dias, & Marcelino, 2009). Such a culture is being initiated in the context of developing and strengthening public actions with two main goals: to offer resources to improve the efficiency and efficacy of the administration of these programmes and to provide answers to society regarding the social effectiveness of implemented politics within a process of accountability to society. However, although the importance of evaluating social programmes is nearly universally acknowledged, many challenges remain in countries such as Brazil, which began only a decade ago to systematically evaluate social programmes. Major improvement is still necessary in terms of research design and methods as well as the systematisation of procedures that allow for a more accurate reading of social politics and programme effectiveness.

In the early 1980s, Oskamp (1981) emphasised the need to adopt a scientific concept of research on programme evaluation because the application of research outcomes, decision theory and management system had previously received more attention than other areas, such as research design and validity. More than two decades later, methodological rigor in conducting programme evaluations remains a central issue. Several authors (e.g., Mourão & Laros, 2009; Shadish, Cook, & Campbell, 2002; Wholey, Hatry, & Newcomer, 2004) have expressed concerns regarding the rigor of the design, accurate use of measures and discussions of the validity of research in the evaluation of social programmes.

Based on these considerations, this current research aimed to investigate the degree to which the Family Grant Programme ([FGP] Programa Bolsa Família) has contributed to improving the quality of life of beneficiary families. There is currently no consensus regarding the concept of quality of life, although there is agreement that this notion is a multidimensional concept involving a state of well-being that encompasses health, environmental and social realms. Seidl and Zannon (2004) identified two dimensions regarding the conceptualisation of the term: quality of life as a generic term and quality of life as it relates to health. In this study, the first perspective will be considered; according to the authors, this perspective reflects a broader meaning that is influenced by sociological studies.

Based on this broader perspective, the World Health Organization Quality of Life Group (The WHOQOL Group, 1995) defines quality of life as "an individual's perception of their position in life in the context of the culture and value systems in which they live and in relation to their goals, expectations, standards and concerns" (p. 1405).

However, Minayo, Hartz and Buss (2000) argued that quality of life should not be defined solely through scientific or technical criteria but rather should be defined based on political scope to enable the elaboration of intervention agendas, social debates and basic consensus. In accordance with such concerns, the Quality of Life instrument that was developed by the WHOQOL Group addresses not only physical and psychological dimensions but also social and environmental relationships (with items that range from work capacity to safety and transport).

However, research on this phenomenon has been conducted nearly exclusively within the health field; thus, it is necessary to extend this study to other contexts, as this current project endeavours to complete. Hence, quality will be investigated through two pillars: the conceptual dimensions of quality of life (physical, psychological, social and environmental) and a comparison among quality-of-life indicators, which are extracted from the objectives of the FGP, among beneficiaries and non-beneficiaries.

This programme was chosen because of two main reasons: (a) the FGP is directly related to education (the participation of families with children and adolescents is conditional on their school attendance) and (b) this programme is the largest programme of direct income transference in the country and thus is extremely important to the poor population in Rio de Janeiro, which includes 758.251 children between 6 and 15 years old (http://www.mds.gov.br/bolsafamilia/noticias, accessed in 15/03/2009).

The metropolitan region of Rio de Janeiro was chosen because it has a large number of inhabitants (more than 11 million people, representing 73% of all inhabitants in the state), and the social inequality among the citizens is evident in the cities in this region. These cities contain slums with up to one million inhabitants and a massive presence of drug trafficking, including the use of heavy armament by drug traffickers, the employment of children and adolescents in front-line trafficking, the lack of public policies related to social inclusion and income generation for youngsters and the absence of safety policies motivated by silent rulers (Toledo, 1998).

According to the Brazilian Institute of Geography and Statistics (IBGE) census that was conducted in 2010, 22% of the population in the city of Rio de Janeiro lives in substandard clusters; the capital contains the largest number of people living in slums (nearly 1.4 million). If these statistics are extended to the metropolitan region, then more than 1.7 million people live in slums, which correspond to 14,4% of the population in the metropolitan region (IBGE, 2011).

The profile of the average slum dweller as determined by the IBGE (2011) shows a lower average age. In the slums, 28.3% of the total population ranges from 0 to 14 years old, whereas in the urban areas, this proportion is 21.5%. The average density of dwellers is higher in the slum residences than in regular urban areas of the city. In addition, the slums contain a higher concentration of people who self-identify as black or brown (68.4% versus 46,7% in regular urban areas). The 2010 IBGE census shows that this population has a lower percentage of people who have access to basic sanitation, running water and electricity (2011).

Hence, from the data presented, it was observed the urgent demand of these populations for the attention of the state, society and the scientific community. For this reason, the current research defined the dwellers of slums in the metropolitan region of Rio de Janeiro as the participants in this study to better understand their experienced reality and evaluate the outcome of the FGP for these people.

Evaluation of Social Programmes

Today, there is a worldwide renewed interest in the evaluation of social programmes. Research on programme evaluation has substantially expanded and has been applied to several governmental and private programmes. Both in Brazil and on an international scale, evaluative research allows for the provision of subsidies that enable resources to be employed with maximum efficiency. In addition, this type of research ensures a high level of programme effectiveness. In this sense, evaluation has been considered a powerful instrument for improving the performance of social programmes and a learning tool that allows for the redesign and improvement of such programmes in new cycles of social politics.

The general definition of evaluation differs from the more specific definition of programme evaluation. According to the definition provided by Scriven (1967), "to evaluate is to judge the value and merit of something" (p. 37). For Worthen, Sanders and Fitzpatrick (2004), "evaluation is the identification, clarification and application of defensible criteria to determine the value, the quality, the efficacy or the relevance of the evaluated object in relation to these criteria" (p. 35). According to Almeida (2006) evaluation is a social-political process; it is a shared and collaborative process, a process that constructs a reality.

Thus, evaluation comprises a transformative process that involves validating existing programmes, determining the need for adjustments and corrections and identifying the need for new programmes. In addition, evaluation is indispensable for the provision of transparency in public actions. A complete definition of evaluation is provided by Aguilar and Ander-Egg (1994):

Evaluation is a form of applied, systematic, planned and directed social research. It is intended to identify, to obtain and to provide, in a valid and trustful way, sufficient and relevant data and information to support the judgement of the merit and value of one programme's different components or of a set of specific conducted or to be conducted activities. Thus, it has the goal of producing concrete effects and results, proving the extension and the degree of the achievements, functioning as the basis for making rational and informed decisions concerning courses of action, as well as for solving problems and promoting knowledge and understanding about the factors associated with the outcomes' success or failure. (p. 31-32)

Considering the programme evaluation typologies, this research is a summative evaluation because of its focus on the outcomes of the FGP (Scriven, 1967); an external evaluation because the researchers who are involved have no relationship to the programme (Cohen & Franco, 1993); and an ex post evaluation because it measures the outcomes for people who are already beneficiaries of the programme (Campbell & Stanley, 1979). However, it is not a classic ex post evaluation because the beneficiaries continue to participate in the programme even after the research had been conducted. In addition, the research can be viewed as a formal evaluation with systematic and pre-defined research procedures (Worthen et al., 2004). Finally, this study focuses on the dimensions of efficacy and effectiveness in the analysis of the results achieved by the programme (Draibe, 2001).

The Family Grant Programme

The FGP, which was created by the federal government through law number 10.836 (Lei Nº 10.836, 2004), is a programme of direct income transference with conditionalities that benefits 12 million families in poverty and extreme poverty conditions, defined as a maximum income of R$ 140,00 (one hundred and forty reais) per person. The programme integrates Fome Zero (Zero Hunger), which has the goal of ensuring the human right to adequate food, promoting food and nutritional security and contributing to the achievement of citizenship by members of the population who are most vulnerable to hunger (Ministério do Desenvolvimento Social e Combate à Fome [MDS], 2010).

The amount that is received by families can vary from R$ 22 to R$ 200, depending on the number of children who are 18 years old or younger. The programme has three main axes: income transference, conditionalities and complementary programmes. According to MDS (2010), the income transference aims to promote immediate relief from poverty; the conditionalities reinforce access to basic social rights in the fields of education, health and social assistance; and the complementary programmes support family development to enable beneficiaries to overcome their vulnerabilities.

The criterion for participation in the FGP is based on family income and on the distribution of subsidies throughout the country. The programme chooses families based on information posted by the municipality administration in the Single Register for Social Programmes (MDS, 2010). The choice of registered people is made through an automated process, but the registration does not imply the immediate admission of families into the programme.

The management of the programme is decentralised and shared among Brazilian nation, the states, the federal district and the municipalities. The three federal entities collaborate to improve, expand and supervise the execution of the programme. The list of benefited people is public and can be accessed by any citizen (MDS, 2010). The following conditionalities ensure the right to receive subsidies from the FGP: (a) vaccinations must be monitored for growing and development children younger than seven years old; (b) women from 14 to 44 years old should also be monitored, particularly pregnant or nursing women who should seek prenatal care, for their health and that of their babies; (c) every child and adolescent from six to 15 years old should be registered in school and attend at least 85% of their classes per month; (d) students aged 16 and 17 should attend at least 75% of their classes; and (e) children and adolescents up to 15 years old who are at risk for or who were moved for child labour purposes by the Programa de Erradicação do Trabalho Infantil ([Peti] Programme for Eradication of Child Labor) should participate in the Serviços de Convivência e Fortalecimento de Vínculos ([SCFV] Living Services and Strengthening Linkages) within Peti and attend at least 85% of the established monthly hours.

The purpose of the conditionalities is that beneficiary families must assume responsibilities to expand their access to basic social rights. In this sense, a conditionality does not have a punitive character; rather, it aims to expand the opportunities for beneficiaries to exert their citizenship. Hence, families that encounter difficulties in meeting these conditionalities should seek orientation from the municipal manager of the FGP and consult the Social Assistance Reference Centres, the Specialised Social Assistance Reference Centres, or the social assistance team in the municipality. Thus, the government can further assist these families in overcoming their difficulties (MDS, 2010).

However, according to the programme rules, if all of the opportunities to reverse the noncompliance of the conditionalities fail, then a family's subsidies from the FGP may be blocked, suspended, or even cancelled.

Method

Participants

The current research was conducted using a survey of 530 low-income slum dwellers from the metropolitan region of Rio de Janeiro, in the cities of Niterói, Magé, São Gonçalo and Rio de Janeiro. The criteria to be included in the research were as follows: those in the beneficiary group needed to be participants in the FGP for at least one year and the control group participants needed to have a family income per person of up to one hundred and twenty reais, which was the maximum amount allowed for participation in the FGP at the time of data collection. The participants in both groups signed the consent form agreeing to participate in the study. The following individuals were excluded from the study: those with temporary or suspended participation in the FGP and those who failed to answer at least one-third of the survey questions.

The participants in the study were divided into two groups: 281 FGP participants; and 249 non-programme participants with family incomes that qualified them for programme participation (control group). The sample included workers who completed or did not complete an elementary level of education (42.4%) and those who completed or did not complete a middle level of education (57.6%). More women (80.5%) were in the study and the average age of the participants was 37 years old (SD=10 years) and the average income was 83.70 reais per person (SD=53.90 reais).

Instrument

The data collection instrument contained a total of 25 questions related to the three main axes of FGP: income transference, conditionalities and complementary programmes. The questionnaires for both the beneficiaries and non-beneficiaries had the same questions, except for questions specific to each condition (for example, how were you admitted to the programme, or why are you not an FGP beneficiary?).

The survey questions utilised different scales: nominal (for example, courses in which you enrolled after your admission to the programme), ordinal (How would you describe your family's current quality of life: very good, good, regular, bad or very bad?) and ratios (Including all of the people who live in your home, how much money do you receive monthly?). The questionnaire had undergone evaluation by six judges (experts in the evaluation of social programmes) who were responsible for the validity of its content. The changes that were suggested by the judges were processed and a semantic validation of the instrument including eight beneficiaries and seven non-beneficiaries was conducted. This validation indicated the need for further adjustments in the language adopted in the questionnaire. After these validation stages, the instrument was considered ready to be applied to the chosen population.

Procedures

The data were collected in the homes of the participants (87%) and in public spaces in the slums where people typically circulate (13%). The questionnaires were administered by five researchers who had an undergraduate level of education and had been trained in the research procedures. The questions were presented orally to the research participants and their answers were recorded.

To minimise the beneficiaries' overestimation of the positive aspects of the benefits that they received based on the fear that negatives answers could prevent them from receiving benefits, the researchers conducting the study did not inform the participants that the survey pertained to the FGP. The research was described as "research on life conditions of community dwellers in Rio de Janeiro". Because there were no questions concerning the evaluation of the programme (and only one question, among many others, asking whether the person received any benefits from the government and, if so, which benefits), this strategy could have eliminated or attenuated the aforementioned overestimation.

The data were processed using the SPSS software (version 15.0) by conducting descriptive analyses and inferential tests (t-test, correlations and Mann-Whitney-Wilcoxon test) to compare the results between the two groups: the beneficiaries and non-beneficiaries (control group).

Results

In the first stage of the analysis, the presence of missing values was identified and univariate and bivariate distributions were examined by graphical (branch and leaf diagram, box plot) and numerical (measures of symmetry) methods to assess normality, the homogeneity of variance and the presence of univariate outliers. In the analysis of multivariate outliers, the parameters of the Mahalanobis distance were used. Four multivariate cases were found and excluded from the analyses. The numerical criterion that was adopted to evaluate the need for changes in the variables was anchored in symmetry. There was no need for changes in the variables.

Regarding income, the data analysis indicated a lower family income among those who participated in the FGP (an average income of R$ 301.41; SD = 200.29) compared with the individuals in the same slums who did not participate in the programme (an average income of R$ 451.27; SD = 197.82). This difference between the average incomes was found to be significant by the t-test for independent samples (t ₍₅₂₈₎ = 8.63, p < .001); thus, the results indicated that the programme actually did serve people with greater need.

This result reflects the difficulty of research designs in the field of social programmes whose purpose is to serve the needs of those with specific characteristics (in the case of the FGP, low income). Thus, the control group (even those living in the same region) would not be expected to exhibit characteristics that are completely equivalent to the beneficiary group, as the purpose of the FGP is to serve those with the greatest needs. Nevertheless, despite this expectation, this phenomenon may represent a limitation of sampling inconsistency.

However, even if the income of the beneficiaries was lower than that of the non-beneficiaries, the income distribution of the studied population showed that the programme has high unmet demand, as 79.8% of the non-beneficiaries met the conditions for programme participation (per capita income of up to R$ 140.00). Among the beneficiaries, however, 9.6% of the individuals whose per capita income at the time of the study was higher than R$140.00 failed to fulfil the conditions of the programme.

Concerning to the possession of household appliances (refrigerator, radio, stove and TV) or basic conditions, such as bathrooms, light, sewer and running water, the non-parametric Mann-Whitney-Wilcoxon test (MWW test) showed identical distributions (with no statistically significant differences) between the beneficiaries and non-beneficiaries, as shown in Table 1.

The lack of difference between the beneficiary and the control groups with regard to the possession of domestic appliances and basic housing conditions can be considered a positive result with regard to the sample composition of the beneficiary and control groups because this finding indicates that the income differences between the two groups do not generate significant differences.

The Pearson's correlations identified positive and significant relationships between the years of schooling and family income. The correlation coefficient between the variables was .21 (p < .01), which, according to the classification of Miles and Shevlin (2001), can be classified as low in magnitude (from .10 to .30). However, given the target research population (slum dwellers in the metropolitan region of Rio de Janeiro), the relative lack of variation in years of schooling and income suggests that with greater variability, the magnitude of the correlation between income and education could be higher.

Nevertheless, with regard to the conditionalities of the FGP benefits for children and adolescents from 6 to 17 years old to remain in the programme, the results showed no significant differences between the average number of children and adolescents in this age range who attend school among the beneficiary and non-beneficiary families. However, the t-test results yielded a different outcome for children between four and five years of age (t₍₅₂₈₎= 2.20; p < .03). The average number of children out of school among the beneficiaries was .43 per family (SD = .73), whereas this average was .29 among the non-beneficiaries (SD = .66). These results show that children from the beneficiary families were not likely to attend school before six years of age, which corresponds to basic compulsory education. However, because the standard deviation was larger than the average, this result identifies a large discrepancy in the data distribution that indicates the existence of subgroups among both the beneficiaries and control group participants. Thus, lack of schooling may be a phenomenon that is localised in small groups among the residents in Rio's slums.

Another important indicator in the current study was the discrepancy in the number of beds per family (i.e., the number of mattresses that each family had divided by the number of members in each residence; for the purpose of calculation, double mattresses were counted as two mattresses). Among the FGP beneficiaries, there was a bed deficit in 35% of the families, whereas among the non-beneficiaries, this deficit was in 20% of the households. There was a significant difference in the average number of beds per resident (t₍₅₀₇₎ = 3.01; p < .01) among the beneficiaries (M = .9; SD = .23) and the non-beneficiaries (M = 1.0; SD= .19). This distinction between the deficit of beds in the beneficiary and non-beneficiary groups demonstrates the asymmetry between the surveyed groups and indicates a positive outcome for the FGP because the results showed that the programme assisted the people with the greatest need.

The number of meals made per day by each family was also lower among the beneficiaries (even after considering the monthly amount that they already received), with an average of 2.7 meals per day for the beneficiary families (SD = .8) and 3.0 meals per day for the non-beneficiary families (SD = .9); the t-test indicated that this difference was statistically significant (t₍₅₂₁₎ = 2.33; p < .02). The difference in the number of meals per family prior to receiving benefits is estimated to be even greater; in fact, when asked what had changed in their family lives after they began to participate in the programme, 42% of the beneficiaries provided answers related to the improvement in meals.

The MWW test was used to assess the conditionalities in the health area and to show different distributions among the beneficiaries and non-beneficiaries in relation to vaccination, prenatal care and men's visits to doctors. The results were more favourable for the beneficiaries in the case of updated immunisation and more favourable for the non-beneficiaries in the cases of seeking prenatal care and medical visits by men over 40 years old, as shown in Table 2.

The data analysis showed that in the case of annual gynaecological visits, breastfeeding and medical visits to monitor children's growth in health centres, there were no significant differences between the beneficiaries and non-beneficiaries. Notably, prenatal consultations and medical visits by men over 40 years old were found to be more frequent among the non-beneficiaries. This result reflects the differences between the two studied groups and suggests that the FGP must review the strategies for monitoring the conditionalities related to health because beneficiaries continue to present data that are indicative of poor health compared with non-beneficiaries.

Regarding to complementary programmes and the encouragement of FGP recipients to obtain civil documentation, there is a significant positive difference in relation to birth certificate requests and participation in professional and computer courses during the last three years. These three outcomes were more favourable for beneficiaries than for non-beneficiaries. However, the beneficiary and non-beneficiary groups exhibited no significant differences in terms of civil identification documentation for adults, courses for the Education of Youth and Adults (EJA) and participation in literacy courses, as shown in Table 3.

Finally, with regard to the perception of current and future quality of life (10 years from now), the beneficiaries were more likely to be more optimistic than the non-beneficiaries. However, the perceptions of both groups were identical with respect to their outlook on their quality of life in the next five years, as shown in Table 4.

The graph in Figure 1 shows the differences in the perceptions of current and future quality of life between the beneficiaries and non-beneficiaries. The results indicated that 29.1% of the beneficiaries currently considered their lives to be good or excellent, whereas this percentage among the non-beneficiaries was 19.8%. Similarly, 3.2% of the beneficiaries believed that their lives would be worse/much worse in 10 years, whereas this percentage among the non-beneficiaries was 8%.

Discussion

This study proposed to investigate quality of life based on the FGP indicators and considered the following variables and their conceptual dimensions: (a) physical – the number of daily meals and health aspects (medical appointments and vaccinations); (b) psychological – perceptions of current and future quality of life; (c) social – school attendance, courses and civil identification; and (d) environmental – basic conditions, such as bathroom, light, sewage, running water and the proportion of beds per person.

The results of the current study confirmed that the criterion of family income was observed in the choice of families that would benefit from the FGP because the average family income of the non-beneficiaries in this study was 50% higher than that of the beneficiaries (discounting the benefit value), even among those living in the same region. This result is an important indicator of the fairness criterion of FGP. Moreover, the results corroborate the findings from other authors who also observed a relationship between low income and the receipt of benefits (Costa, Salvato, & Diniz, 2010; Dias & Silva, 2010; Hall, 2006; Tavares, 2010). Soares, Ribas and Osório (2010) concluded that the FGP is well focused because 66% of the income from the programme is allocated to 10% of the poorest families. However, based on the 2004 PNAD, these authors estimated that the FGP would have a high error of inclusion (49%) but that its income distribution programmes would be lower than those in Mexico and Chile. Moreover, the IBGE estimates (2008) show that the average monthly income of the participating families is less than half of the minimum wage.

This study also highlighted the importance of the FGP in increasing the income of beneficiary families and consequently reducing poverty in the country. This result is confirmed by several other studies that evaluate the programme, such as the research of Dias and Silva (2010), who concluded that although the benefit amount does not substantially satisfy the real-life physical conditions of beneficiaries, the programme does translate into a significant income increase. The contribution of the FGP to the reduction of poverty has also been confirmed by data from the National Survey by Household Sample (PNAD) and from the Institute of Economic Applied Research (IPEA). The 4^th National Monitoring Report of the Millennium Development Goals indicated the decrease of extreme poverty from 12% in 2003 to 4.8% in 2008. According to Soares et al. (2010), the FGP was responsible for 21% of the reduction of the Gini index from 1995 to 2004. In the current study, the increased income provided by FGP represents was an average of more than half of the total income of the beneficiary families.

The Pearson's correlations were indicative of positive and significant relationships between the years of schooling and the family income of beneficiaries and non-beneficiaries; these relationships justify the conditionality imposed by the FGP regarding the frequency of school attendance for the children of beneficiaries, as education is a predictor of income for the studied population. This relationship between the programme and the increase in years of schooling has been confirmed by other researchers (Cacciamali, Tatei, & Batista, 2010; Tavares, 2010). However, Estrella and Ribeiro (2008) questioned this conditionality by arguing that basic education is already universal and that the government should focus on the quality of the educational and health services that are offered rather than their existence. Moreover, Monteiro, Ferreira and Teixeira (2009) argued that the dispersion surrounding the fulfilment of conditionalities emphasises the existence of limitations of the quality of information in registration, especially its validity.

The results of the current study also indicate that the number of meals of beneficiary families remains lower than the number of meals of non-beneficiary families that live in the same slums; this finding indicates the importance of these benefits in reducing hunger for participating families. This result is corroborated by Duarte, Sampaio and Sampaio (2009), who inferred that 88% of the benefit amount is used to purchase food and then concluded that the conditional income transference of the FGP exerts a positive influence on the food consumption of these families. Soares et al. (2010) noted that the FGP is an important mechanism of providing poverty relief for extremely poor families and has important impact on the malnutrition of children.

However, in contrast with other studies, the research by Saldiva, Silva and Saldiva (2010), who evaluated the health and nutrition of children under five years old in beneficiary families in a semi-arid municipality in Brazil, found a deficit in height and weight. The authors did not find statistically significant differences between the beneficiaries and non-beneficiaries in terms of the nutritional statuses of children. In both groups, the consumption of fruits and vegetables was similarly low. In addition, the authors found that the children benefiting from the FGP were at three-times higher risk for consuming junk food than the non-beneficiary children. This result suggests that it is not sufficient for families to have the necessary income to adequately feed their children; it is also necessary to invest in food education to ensure balanced and healthy nutrition for these families.

Regarding the conditionalities in health, the current study observed more effective use of health services only when comparing the vaccination history of beneficiaries and non-beneficiaries. In some cases (such as seeking prenatal care and medical appointments for men over 40 years old), the indicators are more favourable for the non-beneficiaries than for the beneficiaries. Therefore, the only conditionality that is imposed by the programme that has been effective is the monitoring of children's vaccines, possibly because this condition is more easily controlled by the programme's health agents. This result is corroborated by research conducted by Monteiro et al. (2009). The authors emphasise the problems in the CadÚnico (Registry for Social Programmes of the Federal Government) and in the management of "health" and "education" conditionalities. According to the authors of that study, these factors expose the weaknesses in the municipal social public management and compromise the programme's efficiency. Estrella and Ribeiro (2008) also confirmed the low level of efficiency in terms of controlling the conditionalities, especially those in the health area. According to the authors, the index of decentralised management (IGD) has registration problems that "make the structural aspects of FGP not so trustful and reduces the possibility of measuring in adequate ways the municipalities capacity to ensure the fulfilment of imposed conditional demands, especially those related to health" (p. 637).

With regard to complementary programmes, the current study observed a positive and significant difference in the participation of the beneficiaries in professional and computer courses during the last three years when compared with the non-beneficiaries. These analyses show that beneficiaries have taken advantage of increased opportunities to overcome poverty or extreme poverty conditions because of their more frequent participation in professional and computer courses compared with non-beneficiaries. The goal of these complementary programmes is to provide opportunities to overcome inequalities, promote social inclusion (MDS, 2010) and thus reduce the social vulnerability of participating families. These goals are important because, as Kerstenetzky (2009) claims, a frequent objection to the programme is that it would be assistentialist (i.e., it could increase poor people's dependence rather than encouraging responsibility and autonomy). Figueiró (2010) also discussed the dilemma between assistentialism and emancipation and considers that one of the problems of the FGP is the danger of failing to fulfil its original purposes and becoming merely an income transference policy. According to this author, the success of the FGP in achieving its proposed goals is directly related to the existence of an active and participatory civil society in the general context and in the regions reached by the programme. However, the results of the current study indicate that the programme has not been configured to be merely assistentialist, as demonstrated by the greater participation of the beneficiaries than of non-beneficiaries in professional programmes and the more favourable perceptions of current and future quality of life (10 years from now) among the beneficiaries.

Finally, it is important to discuss the question of the programme's sustainability. Kerstenetzky (2009) emphasised that the FGP cannot be viewed as a social policy that is oriented towards relieving poverty. According to the author, if the FGP is perceived as a policy that benefits only part of the population in an unsustainable manner, then the programme would be likely to encounter financial restrictions that would limit its effectiveness. Thus, Kerstenetzky (2009) proposed that the FGP should be understood as an integrative and developmental policy to leverage ample support. This researcher argues that by ensuring access to universal services, such as education and health and including those who are typically excluded, the FGP can leverage political and financial support to consistently contribute to the reduction of poverty and inequalities.

Conclusion

Evaluating social programmes requires an evaluation of the effects of intervention beyond the quantification of the people who benefit from such programmes, the benefits that are distributed, and the scope of the interventions. Thus, this study aimed to verify the influence of the FGP on the quality of life of the beneficiary families in the slums of the metropolitan region of Rio de Janeiro in comparison with a control group living in the same slums.

The study did not focus on the economic aspects that would allow for an evaluation of the programme's efficiency. This study focused on the indicators of the quality of life and well-being of the beneficiaries, such as the number of daily meals, the bed deficit per resident, the use of basic services (including education and health), and the perceptions of current and future quality of life.

The results pertaining to the FGP's efficacy, which is understood as the relationship between the process characteristics and outcomes, were positive regarding the participation of the beneficiaries in complementary programmes and in the increase in the number of children with birth certificates. However, the efficacy could not be demonstrated with regard to increases in education or the use of health services (except for vaccination).

The findings pertaining to the programme's effectiveness, which is defined by the relationship between the goals and objectives and the programme's effects, indicated a positive effect in the target population, with changes in the social conditions of the populations participating in the programme: an increase in the number of daily meals, a reduction in the bed deficit per residence, an increase in purchasing power, increased positive perceptions of current and future quality of life and greater participation in professional courses that contribute to overcoming social exclusion.

The results suggest that the programme's managers must devote special attention to conditionalities, especially those related to health. Moreover, the study indicates that, though small in percentage, there are still cases of participants whose income is higher than that of non-participants; such cases indicate the need for better care in selection for programme admission and/or frequent analysis of the conditions for leaving the programme. In addition, the low participation in professional courses is also deserving of attention. Although the average participation of the beneficiaries was found to be significantly higher than that of the non-beneficiaries, the figures are still low in absolute terms, especially considering the low qualifications of the target population.

One of the limitations of the current study is the income difference between the beneficiaries and the non-beneficiaries, which may reflect limitations in the research design and may complicate some of the explanations that are offered for the differences observed. In addition, a limitation is the tendency of the beneficiaries to offer more positive assessments when they suspected that the study pertained to their benefits. Although this research was not presented to the participants as being related to the FGP, some of the surveyed individuals may have altered their responses based on the phenomenon of social desirability.

One of the contributions of the current study is its inclusion of residents of slums in the metropolitan region of Rio de Janeiro, a population that is rarely studied in relation to the FGP, as most studies related to the programme focus on the Northeast region. Moreover, the use of a group of non-beneficiaries as a natural control group was important in verifying the programme's effectiveness. The control group included residents in the slums and presented conditions for participation in FGP, which allows for more accurate conclusions regarding the real influence of this programme. However, future research on the programme should adopt designs that allow for comparisons of the results before and after the intervention, using both beneficiaries and a natural control group.

Recebido: 03/06/2011

1ª revisão: 03/01/2012

Aceite final: 06/01/2012

• Aguilar, M. J., & Ander-Egg, E. (1994). Avaliação de serviços e programas sociais [Evaluation of services and social programs]. Petrópolis, RJ: Vozes.
• Almeida, V. P. (2006). Avaliação de programas sociais: De mensuração de resultados para uma abordagem construtivista [Evaluation of social programs: Measuring results for a constructivist approach]. Pesquisas e Práticas Psicossociais, 1(2), 1-13.
• Cacciamali, M. C., Tatei, F., & Batista, N. F. (2010). Impactos do Programa Bolsa Família federal sobre o trabalho infantil e a frequência escolar [Impacts of federal Family Grant in child labor and school attendance]. Revista de Economia Contemporânea, 14(2), 269-301.
• Campbell, D. T., & Stanley, J. C. (1979). Delineamentos experimentais e quase-experimentais de pesquisa [Experimental and quasi-experimental research designs]. São Paulo, SP: Editora da Universidade de São Paulo.
• Cohen, E., & Franco, R. (1993). Avaliação de projetos sociais [Evaluation of social projects] (7. ed.). Petrópolis, RJ: Vozes.
• Costa, A. A. B., Salvato, M. A., & Diniz, S. C. (2010). Análise do programa de transferência Bolsa Família para o período 2004-2006: Impactos sobre pobreza, desigualdade e focalização [Analysis of income transference program Bolsa Família in the 2004-2006 period: Impacts on poverty, inequalities, and focus]. Retrieved December 12, 2010, from http://www.ipc-undp.org/mds.do
• Dias, M. N. A., & Silva, M. R. F. (2010). O Programa Bolsa Família no município de Bacabal-MA: Avaliação de implementação com o foco nas condicionalidades [Family Grant in Bacabal-MA: Evaluation of its implementation with the focus on conditionalities]. Retrieved October 27, 2010, from http://www.ipc-undp.org/mds.do
• Draibe, S. M. (2001). Avaliação de implementação: Esboço de uma metodologia de trabalho em políticas públicas [Implementation evaluation: Outline for a working methodology in public policies]. In M. C. R. N. Barreira & M. C. B. Carvalho (Eds.), Tendências e perspectivas na avaliação de políticas e programas sociais [Tendencias ans perspectives in the evaluation of social policies and programs]. São Paulo, SP: Pontifícia Universidade Católica de São Paulo, Instituto de Estudos Especiais.
• Duarte, G. B., Sampaio, B., & Sampaio, Y. (2009). Programa Bolsa Família: Impacto das transferências sobre os gastos com alimentos em famílias rurais [Family Grant: Impact of transfers on spending on food in rural households]. Revista de Economia e Sociologia Rural, 47(4), 903-918.
• Estrella, J., & Ribeiro, L. M. (2008). Qualidade da gestão das condicionalidades do Programa Bolsa Família: Uma discussão sobre o índice de gestão descentralizada [Quality in managing conditionalities in Family Grant: A discussion on the index of descentralized management]. Revista de Administração Pública, 42(3), 625-641.
• Figueiró, A. L. (2010). Entre o assistencialismo e a emancipação: Uma análise da relação entre estado e sociedade civil, a partir das experiências do Programa Bolsa Família no entorno do Distrito Federal [Between assistentialism and emancipation: An analysis of the relationship between the state and civil society from the experiences of Family Grant in the Federal District]. Revista Sociedade e Estado, 25(1), 145-146.
• Hall, A. (2006). From Fome Zero to Bolsa Família: Social policies and poverty alleviation under Lula. Journal Latin America Studies, 38, 689-709.
• Instituto Brasileiro de Geografia e Estatística. (2008). Aspectos complementares de educação e acesso a transferências de renda de programas sociais [Complementary aspects of education and access to income transfers from social programs]. Retrieved May 12, 2010, from http://www.ibge.gov.br/home/presidencia/noticias/noticia_impressao.php?id_noticia=562
• Instituto Brasileiro de Geografia e Estatística. (2011). Censo 2010: 11,4 milhões de brasileiros (6,0%) vivem em aglomerados subnormais [2010 Census: 11,4 million Brazilians (6,0%) live in subnormal clusters]. Retrieved December 30, 2011, from http://www.ibge.gov.br/home/estatistica/populacao/censo2010/aglomerados_subnormais/default_aglomerados_subnormais.shtm
• Jannuzzi, P. M. (2005). Indicadores para diagnóstico, monitoramento e avaliação de programas sociais no Brasil [Indicators for diagnosis, monitoring and evaluation of social programs in Brazil]. Revista do Serviço Público, 56(2), 137-160.
• Kerstenetzky, C. L. (2009). Redistribuição e desenvolvimento? A economia política do Programa Bolsa Família [Redistribution and development? The political economy of Family Grant]. Dado, 52(1), 53-83.
• Lei Nº 10.836. (2004, 09 jan.). Cria o Programa Bolsa Família [It creates Family Grant]. Brasília, DF: Presidência da República.
• Miles, J. N. V., & Shevlin, M. E. (2001). Applying regression and correlation: A guide for students and researchers London: Sage.
• Minayo, M. C., Hartz, Z. M., & Buss, P. M. (2000). Quality of life and health: A necessary debate. Ciência & Saúde Coletiva, 5(1), 7-18.
• Ministério do Desenvolvimento Social. (2010). Manual de orientação do BPC [Manual guidance of BPC]. Retrieved August 4, 2011, from http://www.mds.gov.br/relcrys/bpc/manual_1.htm
• Monteiro, D. A. A., Ferreira, M. A. M., & Teixeira, K. M. D. (2009). Determinantes da gestão do Programa Bolsa Família: Análise do índice de gestão descentralizada em Minas Gerais [Management determinants of the Family Grant: Analysis of the index of decentralized management in Minas Gerais]. Saúde Social, 18(2), 214-226.
• Mourão, L., & Laros, J. A. (2008). Avaliação de programas sociais: Comparando estratégias de análise de dados [Evaluation of social programs: Comparing strategies for data analysis]. Psicologia: Teoria e Pesquisa, 24, 545-558.
• Oskamp, S. (1981). Applied social psychology. Washington, DC: Prentice-Hall.
• Rocha, F. E. C., Albuquerque, F. J. B., Coelho, J. A. P. M., Dias, M. R., & Marcelino, M. Q. S. (2009). Avaliação do Programa Nacional de Fortalecimento da Agricultura Familiar: A intenção de pagamento do crédito [Evaluation of the National Program for Strengthening Family Farming: The intent to pay credit]. Psicologia: Reflexão e Crítica, 22(1), 44-52.
• Saldiva, S. R. D. M., Silva, L. F. F., & Saldiva, P. H. N. (2010). Avaliação antropométrica e consumo alimentar em crianças menores de cinco anos residentes em um município da região do semiárido nordestino com cobertura parcial do Programa Bolsa Família [Anthropometric evaluation and food intake in children under five years-old living in a municipality in semiarid northeast region, with partial coverage of the Family Grant]. Revista de Nutrição, 23(2), 221-229.
• Scriven, M. (1967). The methodology of evaluation. In R. W. Tyler, R. M. Gagné, & M. Scriven (Eds.), Perspectives of curriculum evaluation (pp. 39-83). Chicago, IL: Rand McNally.
• Seidl, E. M. F., & Zannon, C. M. L. C. (2004). Qualidade de vida e saúde: Aspectos conceituais e metodológicos. [Quality of life and health: Conceptual and methodological aspects]. Cadernos de Saúde Pública, 20(2), 580-588.
• Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference Boston, MA: Houghton Mifflin Company.
• Soares, F. V., Ribas, R. P., & Osório, R. G. (2010). Evaluating the impact of Brazil's Bolsa Família: Cash transfer programs in comparative perspective. Latin American Research Review, 45(2), 173-190.
• Tavares, P. A. (2010). Efeito do Programa Bolsa Família sobre a oferta de trabalho das mães [Effect of the Family Grant on mothers' labor supply]. Retrieved December 12, 2010, from http://www.ipc-undp.org/mds.do
• Toledo, J. R. (1998, 26 set.). Os miseráveis são 25.000.000 [Miserable people are 25.000.000]. Folha de São Paulo, pp. 1-3.
• Wholey, J. S., Hatry, H. P., & Newcomer, K. E. (2004). Handbook of practical program evaluation (2^nd ed.). San Francisco, CA: Jossey-Bass.
• Worthen, B. R., Sanders, J. R., & Fitzpatrick, J. L. (2004). Avaliação de programas: Concepções e práticas São Paulo, SP: Gente.
• The WHOQOL Group. (1995). The World Health Organization quality of life assessment (WHOQOL): Position paper from the World Health Organization. Social Science & Medicine, 41, 1403-1410.

Publication Dates

• Publication in this collection
  23 Jan 2013
• Date of issue
  2012

History

• Received
  03 June 2011
• Accepted
  06 Jan 2012
• Reviewed
  03 Jan 2012