SciELO - Scientific Electronic Library Online

vol.22 issue8Rational use of medicines, pharmaceuticalization and uses of methylphenidateCoordination between pharmaceutical services for integrated pharmacotherapy: the case of Catalonia author indexsubject indexarticles search
Home Pagealphabetic serial listing  

Services on Demand



  • text new page (beta)
  • English (pdf)
  • Article in xml format
  • How to cite this article
  • SciELO Analytics
  • Curriculum ScienTI
  • Automatic translation


Related links


Ciência & Saúde Coletiva

Print version ISSN 1413-8123On-line version ISSN 1678-4561

Ciênc. saúde coletiva vol.22 no.8 Rio de Janeiro Aug. 2017 


Sustainability of ARV provision in developing countries: challenging a framework based on program history

Sustentabilidade da provisão de medicamentos ARV em países em desenvolvimento: modelo de avaliação com base na história do programa

Thiago Botelho Azeredo1 

Maria Auxiliadora Oliveira2 

Cláudia Du Bocage Santos-Pinto3 

Elaine Silva Miranda4 

Claudia Garcia Serpa Osorio-de-Castro2 

1Observatório de Vigilância e Uso de Medicamentos, Faculdade de Farmácia, UFRJ. Av. Carlos Chagas Filho 373, Cidade Universitária. 21941-170 Rio de Janeiro RJ Brasil.

2Departamento de Política de Medicamentos e Assistência Farmacêutica, Escola Nacional de Saúde Pública, Fiocruz. Rio de Janeiro RJ Brasil.

3Centro de Ciências Biológicas e da Saúde, UFMS. Campo Grande MS Brasil.

4Departamento de Farmácia e Administração Farmacêutica, Faculdade de Farmácia, UFF. Niterói RJ Brasil.


The provision of ARVs is central to HIV/AIDS programs, because of its impact on the course of the disease and on quality of life. Although first-line treatments costs have declined, treatment-associated expenses are steeper each year. Sustainability is therefore an important variable for the success of treatment programs. A conceptual framework on sustainability of ARV provision was developed, followed by data collection instruments. The pilot study was undertaken in Brazil. Bolivia, Peru and Mozambique, were visited. Key informants were identified and interviewed. Investigation of sustainability related to ARV provision involved implementation and routinization events of provision schemes. Evidence of greater sustainability potential was observed in Peru, where provision is implemented and routinized by the National HIV/AIDS program and expenditures met by the government. In Mozambique, provision is dependent on donations and external aid, but the country displays a great effort to incorporate ARV provision and care in routine healthcare activities. Bolivia, in addition to external dependence on financing and management of drug supply, presents problems regarding implementation and routinization. The conceptual framework was useful in recognizing events that influence sustainable ARV provision in these countries.

Key words: Program sustainability; HAART; HIV; Aids


A provisão de medicamentos ARV é central para programas de HIV/Aids, devido a seu impacto no curso da doença e na qualidade de vida. Embora os custos de tratamentos de primeira linha tenham diminuído, os gastos dos programas com os tratamentos tem aumentado a cada ano. A sustentabilidade torna-se fator fundamental para o sucesso dos programas. Um modelo conceitual para avaliação da sustentabilidade da provisão de ARV e instrumentos de coleta de dados foram desenvolvidos. Um estudo piloto foi realizado no Brasil e a pesquisa de campo cobriu Bolívia, Moçambique e Peru. Informantes-chaves foram identificados e entrevistados. Eventos críticos de implementação e rotinização foram investigados na história dos programas. Foi observado maior potencial para sustentabilidade no Peru, onde a provisão está implementada e rotinizada e os gastos são cobertos pelo governo nacional. Em Moçambique, o financiamento da provisão é quase totalmente dependente de ajuda internacional, mas há grandes esforços voltados à expansão da cobertura e rotinização do cuidado a PVH. Na Bolívia, além da dependência externa para o financiamento há problemas de implementação e gerenciamento da provisão. O modelo avaliativo mostrou-se útil na identificação de fatores que influenciam a capacidade para sustentabilidade dos programas nesses países.

Palavras-Chave: Sustentabilidade de programas; HAART; HIV; Aids


The fight against AIDS was set out as a global commitment by the United Nations (UN) Millennium Declaration and in the landmark UN Declaration of Commitment on HIV/AIDS (2001) which recognizes “that access to medication in the context of pandemics such as HIV/AIDS is one of the fundamental elements to achieve…” the realization of the right to health1,2. In 2006, UN Members move towards Universal Access to HIV prevention, treatment and care by 2010. By the end of 2011, around eight million (54%) eligible people living with HIV (PLHIV) were receiving antiretroviral therapy (ART) in low and middle-income countries (LMIC). As a consequence, AIDS-related illnesses and mortality dropped by more than 50%. The 2011 UN High Level Meeting reaffirms previous commitments and establishes, inter alia, the target of treating 15 million PLHIV by 20153.

Despite major advances and successes achieved in the last ten years, problems related with access to antiretroviral medicines (ARV) still persist and tend to increase in many developing countries: the maintenance and expansion of long-term financing; reducing early mortality by promptly starting ART and other medications against co-morbidities; implementation of novel 2010 WHO guidelines, which are more comprehensive and include costlier medicines in early treatment stages; improving the management of long-term treatment, which includes access to viral load tests needed for better clinical assessment and progression to second and third-line regimens4.

Although ARV prices have been decreasing over time5-7, population treatment-associated expenses are steeper each year, especially in face of new and costly medicines incorporated in treatment guidelines, most of which present very limited competition or none at all8. According to Hoen et al.8 the “policy space to produce or import generic versions (…) is shrinking” in developing countries, due to patent regimen pressures.

Provision of antiretroviral medicines includes procurement, supply chains, and operational and information systems, in order to ensure high quality services. Upscaling AIDS care and prevention is a complex and dynamic process that depends upon countries’ capacity to adopt new interventions and assess interactions among the interventions’ key components as well as between programs and health systems9.

This transition from emergency AIDS response to long-term response of large treatment cohorts requires health systems strengthening9 and regular access to ART5. Sustainability of ART provision has been pointed out as a major concern9-12.

Schell et al.13 point out that a minority of sustainability studies makes attempts at conceptualizing sustainability and at developing assessment tools accordingly. Stirman et al.14 argue that the absence of a working definition of sustainability and of guidance by an explicit theoretical model is an important limitation to the body of research. Scheirer15 suggests that the use of conceptual frameworks16, as well as in-depth assessments on program implementation, could help overcome limitations in sustainability studies.

The objective of this study was to challenge a conceptual framework applied to assess ARV provision sustainability in Peru, Bolivia and Mozambique.


Sustainability and related concepts

Sustainability may be interpreted in various ways, but is closely related to financial and organizational aspects, both of which depend on the commitment or political will of governments17. They involve the ability of a system to acquire the financial resources needed to fund programs, provide products, organize services and manage all non-financial resources18,19.

From an organizational point of view, Pluye et al.20 propose three aspects of sustainability that integrate the program with the organizational context and the political environment. The first aspect is implementation. According to Mazmanian and Sabatier21 implementation is the carrying out of a basic policy decision, usually incorporated in a statute but which can also take the form of important executive orders or court decisions. Denis and Champagne22 define implementation as the extension of the operational capacity of an intervention or the transfer of an intervention at an operational level; while for Love23 implementation is defined as activities focused on the carrying out of programs.

The second is routinization, which refers to the process that leads to the establishment of routines within the activities required by the program to achieve its objectives. Routines are operational procedures integrated into the organization, which reflect shared know-how or memory, exhibit adaptation to the context, convey values and other cultural artifacts, and adhere to rules such as plans and procedural manuals20.

The third aspect is standardization, which describes the existence of formal, legal, normative or institutional standards that guide the intervention. Standardization of established routines helps to simplify and disseminate them while at the same time introducing institutional standards such as guidelines, legislation and policies. Standardization strengthens institutional identity and anchors program sustainability20.

For the purposes of this study, sustainability was defined as an attribute of an intervention, program or policy that emerges from the implementation process by means of the routinization and standardization of a set of durable activities and resources aimed at program-related objectives.

In this perspective, sustainability can be assessed through the identification of critical events, which have built the program history, i.e. elements in temporal sequences constituting particular patterns, which can be explored to analyze underlying processes or mechanisms24,25. Conceptual events have been described as being associated to greater sustainability, some of which are thought to pertain specifically to: implementation (investment of adequate resources; compatibility of the activities with those of the organization); routinization (resource stabilization; risk-taking); or, both implementation and routinization (incentives; adaptation of activities; objectives fit; transparent communication; sharing cultural artifacts)24.

Framework development

The first step for developing the framework was the definition of the following assumptions:

  • Regularity of treatment provision is an important factor in the success of treatment programs26.

  • Central coordination is essential to the functioning of a sustainable AIDS program27.

  • Issues related to sustainability should be considered during both planning and implementation processes, which are concomitant and continuous17,20.

  • Program implementation and routinization may be assessed by means of identification and analysis of events throughout the program history24.

The building of the framework started with a simple list of research issues to be addressed in investigating achievement of ARV provision programs objectives and of program sustainability in order to determine feasibility in the investigation of sustainability.

In order to integrate theoretical categories (conceptual events) and operational elements, these issues were organized in four separate dimensions: Program Resources; Activities; Reinforcement Strategies; and, Context and Organizational Culture. The resulting framework helped structure data collection and analysis.

Field study and analysis

A pilot study, conducted in Brazil, helped to secure concept formulation (sustainability of ARV provision) within data collection instruments (content validity), and to identify small inconsistencies in the forms as well as to organize the data collection process. The framework was submitted to challenge through semi-structured interviews with program coordinators and other key informants in three countries: Mozambique, Peru and Bolivia. Key informants were selected according to their knowledge of program history, estimated by the level and length of their involvement in the program over time.

All interviewees were contacted personally through E-mail or telephone before country visits. A brief explanation of the nature and purposes of the study was given. If the intended interviewee manifested interest, an executive summary and project information were sent. After preliminary consent, further contact was made to set up appointments. These were followed up once the research team reached the country. Interviews began with a thorough reading of the written consent form in order to eliminate doubts or misunderstandings in regard to data collection, analysis and use of information. Specific authorization was asked for recordings.

Interviews guided identification of publicly available documents such as country bills and legislation on HIV/AIDS; program standards and norms; treatment protocols; planning and evaluation reports and newspaper clippings were collected. These materials were provided by interviewees themselves.

Recordings were transcribed and objective data were entered into a worksheet. Two researchers reviewed each interview and corresponding documentation, and wrote a detailed case description for each country, according to the dimensions: Program Resources; Activities; Reinforcement Strategies; and, Context and Organizational Culture. The entire group revised case descriptions; documentation and transcriptions were double-checked every time any ambiguity or inconsistency was detected, making the corresponding modifications. Then authors examined the case descriptions and sorted out the information (critical events in program history) according to the theoretical categories (conceptual events). Savaya et al.28 organized their comparative case study analysis on the sustainability of social programs in a similar fashion.

Ethics Statement

Our ethical procedures were carried out after consultation with our IRB and adhered to Ethics in Research Legislation in effect at the time of the study29. Key informant interviews referred only to the public history of programs. No personal data was collected. Participation was strictly voluntary and all interviewees gave written informed consent. Furthermore, there was no identification of respondents’ position or name in data collection forms. Confidentiality and anonymity was assured.



Chart 1 presents the framework as organized as Program Resources, Activities, Reinforcement Strategies and Cultural and Organizational Issues. In each of these dimensions, conceptual events that correspond to program operational elements are listed. These, in turn, are detailed in the topics that were investigated.

Chart 1 Framework for investigating the sustainability of ARV provision. 

Theoretical event* by dimensions Program operational elements Investigated topics
IE – Adequate investment of resources Trained staff; facilities; financial resources. •Provide an overview of resources on which the ARV provision has counted on since its beginning: site, infrastructure, volumes and sources of funds
•Human resources: how many, training, time commitment
•National ARV production
•Is coverage adequate?
RE – Stabilization of resources Clinical protocols; permanent staffing; regular flow of funds. •Normative regulation of resources and assignment
•Coverage Projection
•Annual Budget Projection
•Resources for ARV provision are part of the amount of resources available to the organization (common fund)
•Temporary nature of the contribution and renewal of materials and human resources
IE – Compatibility of program activities with those of the organization Similarities between program requirements and organizational capabilities – organization personnel able to fully perform program activities •What activities are linked to the provision of ARVs?
•Who is responsible for these activities?
•What was the previous profile (before the provision of ARV) of activities of the implementer (General Office / Department / Ministry of Health)?
•New activities (if any) that were introduced in the organization (General Office / Department / Ministry of Health) that are compatible with other activities in practical and technical terms?
•Was there discontinuity of the provision of ARV over time? Why?
JIR – Adaptation of activities Changes to existing protocols and procurement models; competition between existing activities •What adjustments were necessary to in the implementation of ARV provision over time?
•Was there or is there competition among old and new tasks / activities within the implementer, or specific to provision and general to the implementer?
Reinforcement Strategies
JIR – Incentives Training, salary increases, leadership bonuses, awards to well managed health centers, external incentives for the organization of the program •Are there incentives such as donation of additional resources, financial or human, for implementation of new activities (provision of ARV)?
•Is there a career plan, policy of promoting employees and earnings consistent with activities?
JIR – Transparent Communication Comprehensive information and surveillance systems; transparency of decision criteria; information feedback; open communication channels •Are there channels of communication available and known to everyone in the organization?
•Is the surveillance and consumption data periodically collected and published?
•Are the protocols produced with the participation of physicians? Is there adherence?
•How is the communication between the various levels of management?
RE – Risk Taking Ventures into new ARVs and protocols; procuring through international initiatives; engaging in price negotiations. •What innovative practices have been brought by the provision for the organization?
•Have these innovations been disseminated and incorporated into the organization?
JIR – Integration of Rules The laws and regulations governing the provision of ARV apply to the rest of the health program •How do the regulations (norms) of ARVs provision in relation to the rest of the provision of medicines?
•Are there rules governing the provision?
•Guidelines, protocols, EML (Essential Medicines List)?
•Is there integration with the other norms?
•Are all these rules known?
Context and Organizational Culture
JIR – Adjustment of goals (Objectives fit) Adjusted fit vs. discrepancy of goals and purposes between the health system and the provision units •Is the mission of the provision of ARVs adjusted to the mission of the implementer?
•Does the provision of ARV occur in the same way as the provision of medicines in general?
•Are there barriers to the provision, which result from the organization of the health system?
JIR – Sharing Cultural Issues Shared values between program and organization; human rights approach. •Does the provision of ARVs require specific mechanisms for prescribing, dispensing and health care?
•Were health professionals used to work like that? When and in what circumstances?
•Are special groups accepted as recipients of care and reached by coverage and therefore by provision?

* IE – Implementation event

RE – Routinization event

JIR – Joint implementation and routinization event

Two main events are listed as Program ‘Resources’: adequate investment and stabilization of resources. First, it is necessary to acknowledge the existence of resources, either human, financial or material in nature. Stabilization deals with a constant and uninterrupted ‘flow’ that guarantees the occurrence of trained personnel, funds and structure of provision. In this sense, framework guides data collection towards volume and source of financial resources over time, budget planning, program coverage and human resources profile.

‘Activities’ describe the procedures of ARV provision within the organizational context. Two different events are presented: Compatibility of program activities with those of the organization and adaptation of activities. Compatibility suggests that program activities can be absorbed by the parent organization without any disruption of the routines. If the organization does not carry out activities as required by the intervention, adjustment or adaptation of activities is needed, in order to minimize or eliminate competing routines that may lead to activity failure. Examining adaptation of activities will show if this adjustment is happening easily and with flexibility. Interview issues included a description of the host organizations’ activities profile and of those required for performing ARV provision, and the stakeholders’ assessment of their interrelationship.

‘Reinforcement Strategies’ deal with all initiatives to boost program outcomes or to overcome organizational resistance: incentives, transparent communication, risk taking and integration of rules. As incentives, training, salary increases, bonuses, performance recognition and any form of praise that may encourage program success. On the other hand, overwork, little pay, lack of organizational structure and support may burden staff and discourage performance. Transparent communication helps sustainability by paving a common pathway of exchange between coordinators and program staff and fostering an open environment, avoiding misunderstandings. Risk-taking is also a reinforcement strategy, in the sense that it involves innovation, and because it mobilizes group attitudes and behavior in order to learn, adapt and adopt. By integrating rules, organization and program will ‘work on the same page’. Investigation focused on mapping principal-agent relations, especially those concerned with adhesion to program activities, such as incentive policies, career plans, communication channels and development of program guidelines.

Finally, ‘Context and Organizational Culture’ examine if program rationale coincides with cultural values and beliefs of implementation agents, and can be best described by two events: adjustment of goals and sharing of cultural artifacts. Adjustment of goals deals with the perceived missions of the organization and of the program – identity of objectives and activities. Coordination between program and organization are considered to be necessary for program sustainability. Sharing of cultural artifacts involves identity of values between parent organization and program, embedded into daily routines and activities. Interviews examined the existence of specific access to medicines mechanisms for HIV/Aids, care patterns for vulnerable groups, values and attitudes associated with those patterns.

Country experiences in sustainability of ARV provision

The framework was challenged in Bolivia, Mozambique and Peru. Chart 2 describes critical events identified in program history, and according to the framework dimensions.

Chart 2 Critical events in the history of the ARV provision programs according to sustainability framework dimensions and theoretical events. Bolivia, Mozambique, Peru, 2000-2009. 

Bolivia Mozambique Peru
Framework Dimension: Resources
The adequate investment of resources (implementation event)
• From the 1990’s to 2007, the country received ARV donations from Brazil;
• In 2002 the country starts receiving resources from the Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM);
• Between 2002 and 2003, 48 treatments are supplied through litigation;
• In 2003 US$ 20,000 were allocated for ARV purchase by government;
• In 2006, the GFATM grant is renewed.
• A first strategic plan for HIV which dealt with resource investment was established in 1999;
• In 2002, the Clinton Foundation (Clinton Health Access Initiative - CHAI) begins supporting the implementation of World Bank and GFATM projects;
• Hospitals in Maputo started offering treatment in 2003-04
• In 2003-04 NGOs trained staff and channeled resources for diagnosis in Maputo, Beira, Nampula;
• From 2004 to 2009: continuous effort to train health professionals and medical technicians to meet staffing needs for planned care.
• 1995-2000 there was no budget for ARV provision. (out-of-pocket payments for services and medicines);
• 2000: a specific budget allocation for ARVs;
• 2003: First ARV purchase with GFATM resources, expanding coverage to 3800 PWH;
• 2006: MoH becomes responsible for ARV provision (60% of costs).
• 2007: Percentage was upped to 70% of costs;
• 2008: 100% of costs. Coverage expanded to 14,300 PWH.
Resource stabilization (routinization event)
• GFATM contract since 2002;
• GFATM renewed in 2006;
• 1990’s -2007 irregular flow of donations from Brazil. (In 2007, donations are discontinued);
• 2007: HIV / AIDS law (Law 3729/07); no additional legislation has been passed to make the law operational; there is no guarantee of budget allocation for ARVs;
• 2007-2009 many changes in program coordination.
• 2004-08 strategic plan reflects gradual increase-in-coverage choice;
• Highly variable flow of resources for AIDS; funding from the GFATM and renewal of WB not guaranteed;
• Existence of a common health fund, including human resources and medicine provision for the purchase and distribution of all medicines, including ARVs;
• 2006: Brazil (represented by the Oswaldo Cruz Foundation), offers continuous training to health boards, doctors and other professionals.
• 2006-2008, the MoH becomes fully responsible for ARV financing;
• 2008: the government changed the type of employment contracts stipulating fixed-term renewable contracts.
Framework Dimension: Activities
Compatibility of program activities with those of the organization (implementation event)
• Lack of technical capabilities for procurement of ARVs by MoH. External agents (CIESS, UNDP, Ibis-Hivos) involved in procurement. • Since 1999, first MEDIMOC then the Center of Medicines and Medical Items (CMAM) have collaborated with MINSAU routinely with medicines purchases. • 2003: DIGEMID already exercised provision-related activities such as selection and forecasting for other medicines and became responsible for ARVs;
• Procurement, storage and distribution, were taken over by OGA, which already carried out these same activities for other medicines.
Adaptation of activities (joint implementation and routinization event)
• 2004-2006 CIESS involved in ARV provision;
• In 2006, after an evaluation by the GFATM, CIESS is removed from activities;
• In 2006 UNDP temporarily substitutes CIESS;
• In 2006, Ibis / Hivos and PROSALUD are chosen for ARV procurement and distribution activities, after tender carried out in the country.
• Since the Ibis / Hivos Foundation began operations, continuity of provision of ARVs can be observed.
• With Ibis / Hivos Foundation there was a change in procurement strategies resulting in the introduction of new suppliers and the obtaining of lower prices for ARVs.
• From 2007: responsibility of ARV procurement was transferred from MEDIMOC to CMAM (change of status of MEDIMOC from public to privately-held);
• After 2008: the rapid expansion and decentralization of treatment coverage required an upgrade in provincial management and structural capabilities exposing management failures (shortages, surplus stocks and expired medicines).
Framework Dimension: Reinforcement strategies
Incentives (joint implementation and routinization event)
• There was specific hiring of professionals for purchase ARVs in CMAM;
• Until 2008 there were day hospitals, and selective recruitment of physicians for HIV / AIDS care. There were financial resources for extra work, training trips, courses and conferences which led to physicians to desire working exclusively for the program.
• 2008: employees receive career benefits that they did not enjoy previously.
Transparent communication (joint implementation and routinization event)
• Regular meetings are scheduled between ESN, DIGEMID, OGA and DISA for discussion and planning of ARV selection and forecasting.
Risk Taking (routinization event)
• Innovative procedures in entrance clearance for ARVs have streamlined the release of lots by customs, making it possible to make imported medicines available in 24 hours. • 2008: the health system has integrated HIV / AIDS care into general health care which requires greater quantity and quality of human resources. • Since 2003, Peru has participated in ARV price negotiation rounds in order to achieve lower prices and greater coverage.
• 2009: Peru has the best buyer profile among Latin America countries.
Integration of rules (joint implementation and routinization event)
• 2004: official regulations on the use of ARVs were released and all health professionals have to adhere to them.
• 2004-2008 :The strategic plan of 2004-08 reinforced the legitimacy of the protocols.
• Since 1997 a succession of laws, edicts and other legislation have regulated HIV/Aids care within the health system.
Framework Dimension: Context and organizational culture
Adjustment of goals/ Objectives fit (joint implementation and routinization event)
• March 2008: closing of the Day Hospital, replaced by Health Counseling and Testing Service (ATS), for diseases in general, not just AIDS.
• 2008: the decision to expand treatment with the definition of the Health Facility network and of treatment goals.
• In 2008, the MoH becomes fully responsible for ARV provision. New duties assigned to existing MoH agencies are synchronic with their institutional missions.
Sharing Cultural Issues (joint implementation and routinization event)
• 2008: with the closing of the Day Hospitals, many patients felt discriminated against by the newer Health Facilities and left to seek care outside the system.

Few events marked resource investment in Bolivia. Allocation of resources by the Ministry of Health (MoH) was small and maintenance of resources depended on renewal of grants from The Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM). Changes in program coordination and management, and litigation for access to ARVs were also observed. In Mozambique, although flow of resources for AIDS had varied over the years, the government planned to maintain a steady rise in number of people on treatment. Events that have been associated with this are the existence of a common health fund, the involvement of the Clinton Health Access Initiative (CHAI) and the training of health professionals. In Peru we observed a triple phenomenon: increasingly autonomous ARV financing; renewable employment contract schemes; and gradual increase in ARV coverage (Chart 2).

In Bolivia ARV procurement activities had been historically performed by organizations outside the MoH and their intermittent change was observed. In Mozambique government organizations dealt with all medicines purchases, including ARVs. When one organization was privatized, another absorbed its activities immediately without observed disruption. On the other hand, expansion and decentralization of second-line treatment coverage revealed problems in provincial stock management. In Peru few changes regarding ARV provision activities were observed in program history. These activities were carried out by government organizations, which were also responsible for provision of all other medicines (Chart 2).

In relation to Reinforcement strategies, an innovative effort was made in Bolivia in regard to customs clearance procedures in order to speed up availability of imported ARVs. In Mozambique, treatment standards and guidelines were implemented for all health professionals in the country. Selective hiring and better job conditions for health professionals working specifically with HIV/AIDS were observed, before 2008. After this, despite difficulties, HIV/AIDS care was integrated with general healthcare in the country. In Peru, meetings involving different ARV provision organizations provided opportunities for the sharing of information, and career benefits fostered the desire to work for the program. Innovation was observed by the country’s participation in joint ARV price negotiations rounds in Latin America. We also observed an effort in the passing of legislation establishing parameters for the program to follow (Chart 2).

No events regarding Context and Organizational Culture were identified in Bolivia. Reorganization in the process of care for people living with HIV (PLHIV) in Mozambique led to the closing of Day Hospitals, integrating HIV/AIDS care with that of other diseases. PLHIV under treatment and care providers perceived it as a letdown that damaged the bond between PLHIV and the health system, making way for patient discrimination. In Peru, ARV provision duties were absorbed by organizations that presented institutional missions synchronic with program requirements (Chart 2).


The main goal of this study was to challenge a sustainability assessment framework, which involved the identification and hierarchization of key conceptual elements, the building of an operational definition of sustainability, and the translation of those theoretical developments into assessment tools. Guidance by an explicit theoretical framework, adoption of clear working definitions, and use of adequate coherent-to-theory tools are pointed out as critical to further develop sustainability research and evaluation13-15.

The choice for research strategy and the selection of factors to be investigated should be determined by the type of intervention. Scheirer15 proposes six different intervention types, including Interventions Requiring Coordination Among Multiple Staff. Aspects pertaining to the sustainability of this type of intervention comprise: administrative support; the role of program champions; culture and mission compatibility between the intervention and the organization; consonance of specific tasks related to the intervention with other organizational procedures; and continuity of financial resources. This type of intervention as well as the above-mentioned aspects may be recognized in Pluye’s proposal24.

Our study’s object was ARV provision, an intervention that requires coordination between multiple organizations, stakeholders, staff etc. As such, our choice of Pluye’s proposal as a reference for building our framework is also supported by Sheirer’s analysis.

The multiple components involved in ARV provision may or may not be recurrent in different scenarios over time. The three countries in which provision was studied presented different income levels, different health system structure, very different levels of HIV prevalence, different forms of organizing HIV/AIDS initiatives, and therefore of ARV provision3,30-34.

This variability hinders the use of classic theory-driven evaluation standardized logic models for programs16,35,36. We preferred to adopt the idea that the program is “a set of resources and activities directed toward one or more common goals”18, the goal in this case being ARV provision. Furthermore, such variability also requires the need for an appropriate data collection approach15. We believe that in focusing on program history24 we were able to better grasp and understand differing internal developments, rather than using a one-fits-all logic model.

In a literature review on the sustainability of new programs and innovations, which included 125 health related studies, Stirman et al.14 point out that the majority of studies neither explore the nature and reasons for changes, nor the process by which adaptations and key decisions are made. By focusing on critical events that build program history, this approach specifically searches for patterns of change, their reasons – identified through key-stakeholders’ perspectives – and aim at analyzing underlying processes that help to explain program development and trends, i.e. process tracing25.

By assuming implementation as a general process we strove not only to open interventions’ black boxes15,16,18,36, but also to analyze which program components and processes of change had in fact been mobilized37,38. By the adoption of Pluye’s focus on organizational routines39, we were able to assess whether or not program pathways were put in place in a manner that favored program continuation, i.e. building program capacity for sustainability13,17.

The framework also inserts sustainability-related conceptual events – implementation and routinization events24 – into an identifiable interface, which can be more easily perceived at data collection.

The interview process stimulated key informants to retell program history emphasizing each of the four dimensions: Program Resources; Activities; Reinforcement Strategies; and, Context and Organizational Culture. By revisiting their own narratives, informants were able to detail events and reflect upon the way in which context shaped program history. This strategy was enhanced by confrontation with the supplied documentation. As such we were able to detect a considerable number of critical events in each country, which in turn led to greater data consistency and internal validity. Moreover, the detection of more events enhanced the potential to explain sustainability in different scenarios. The exhaustive nature of the framework helped overcome the relative difficulty to identify critical events reported by Pluye24.

Informant selection was based on the best possible information retrieval when process tracing is required40. The key process being traced was the carrying out of activities and flow of resources aimed at ARV provision, i.e. the implementation of ARV provision. Key informants were, or had been, very much involved in ARV provision with central participation in program history.

The framework allowed us to observe the differing trends towards resource sustainability. In Bolivia, the dependence on external grants and various changes in program coordination may have resulted in lack of access to ARVs41. These aspects show insufficient investment and lack of resources stability. In Mozambique, heavy dependence on foreign aid42 put resource stability at risk. Notwithstanding, government agents planned an increase in treatment numbers over the years, avoiding large leaps, which in turn has favored continuity in mobilization of resources from external donors43. In five years (2008-2013) coverage has doubled41,44,45. The involvement of the Clinton Health Access Initiative (CHAI) has been tantamount in helping to better ARV provision, by more efficient procurement schemes, resulting in better resource investment. Initiatives in continuous training of health professionals illustrate constant investment in human resources. In Peru a gradual but constant increase in ARV coverage was observed45, and this was associated with a progressive scaling-up of resource investment over the years46. Also, changes in employment contract schemes and the building of autonomous ARV financing have resulted in greater stability of government staff and of flow of resources46,47.

Low-income countries are more prone to rely on donations and international aid19,42. It is noteworthy that even though the dependence of external aid means a risk to resource stability4, the classical teach a man to fish doctrine of external donor aid19 has suffered severe criticism42,48 and seems to be changing43. The GFATM has adhered to “a new form of sustainability that relies on a combination of domestic resources and predictable, open-ended foreign assistance”43. The increase in demand for all aspects of HIV/AIDS care is bound to produce greater competition for international funds4.

The existence of external agents involved in ARV provision in Bolivia indicates that the activities required for ARV provision were not compatible with the MoH activities. Many changes in management of ARV provision throughout the years indicate successive activity failures. A 2012 UNDP audit of GFATM grants reports persisting difficulties in cooperation and coordination and partial compliance to procurement and supply management of health products49. Initially, the introduction of ARV purchases into Mozambican MoH procurement organizations didn’t change their activity profile (compatibility). In spite of changes in procurement responsibilities, observed continuity in work processes showed successful adaptation of activities. Provincial storage and distribution systems failed to adapt to rapid expansion and decentralization of treatment coverage50. In Peru, no adaptation of activities seemed to be necessary. Organizations involved in ARV provision had been working in partnership and kept true to their original activities profiles, indicating compatibility51.

In Bolivia, risk taking procedures were established by new activities regarding customs clearance, which were enforced by means of new legislation52. In Mozambique, integration of rules may be perceived by heavily centralized treatment standards and guidelines. Health professionals perceived the recruitment policies enforced by the government as incentives. This was considered an important strategy to overcome a health workforce gap in Mozambique53. These incentives were abandoned in 2008 when Day Hospitals were discontinued. From that time on, the government took risks by integrating HIV/AIDS and Primary Health Care. Pfeiffer et al.32, defend this step as a positive measure to enhance all aspects of HIV AIDS care and ARV provision.

In Peru, an adequate level of communication was achieved by cooperation between MoH and other organizations involved in ARV provision51. Career benefits worked as incentives, fostering program routines. Engagement in multinational price negotiations was an innovation that resulted in the definition of ceiling prices for future purchases, which may explain lower prices of ARV procured by MoH54. The success of these price negotiations, however, depend upon coordination of pharmaceutical regulations and policies, and/or pooled procurement schemes55,56. Additionally, the enactment of legislation illustrates initiatives to standardize HIV/Aids response in the country (integration of rules).

The end of Day Hospitals in Mozambique resulted in reduction of professionals’ time for HIV/Aids-related activities. This program reorientation also led to dissatisfaction from local-level professionals and PLHIV57. In Peru, programmatic objectives were adjusted to MoH organizations’ institutional missions.

By selecting representatives of high level bureaucracy as key informants, local dynamics and their feedback effects, that are important in implementation analysis38,58-61, may have become underrepresented in program history accounts. Some particular elements that arise from vertical relations, such as the existence and use of communication channels, may have not been well detected. Additionally, the focus on organizational processes shift attention from the broader context, such as political support, that is important to explain sustainability13,17.

By contrasting the three program histories it was possible to infer relative trends of sustainability. Overall, stronger routines and more established standards found in Peru put this country in a good position regarding sustainability. Distribution of events over time may be interpreted as a trend of strengthening ARV provision sustainability in that country. Many events identified in Mozambique indicate the development of routines and standards, but there are some bottlenecks and difficulties, specially regarding availability of resources and health system structure, which challenge program sustainability. Considering the great deal of effort in program planning, scaling-up and adapting to an ever-changing international donor context, trends over time seem to be positive. ARV provision in Bolivia has been going through many organizational and institutional changes, which has led to unstable routines, few standards and, therefore, weaker capacity for sustainability.


The conceptual framework was useful in recognizing events that may influence sustainable ARV provision in assessed countries. The application of our framework and its tools in the assessment of ARV provision resulted in three thorough accounts of program history, in which country specificities and internal development explanatory elements were explored. This approach was successful in capturing relevant data (critical events). The analytical matrix provided by the framework made it easier to organize and categorize data in a way that country-specific narratives could be translated into explanatory accounts. Processes and mechanisms leading to development of routines and establishment of standards could be, then, identified.

Our objective was not to propose a synthetic judgment of sustainability levels. We believe that the analysis of each program’s dynamics allows for the identification of strengths, challenges and opportunities, which, in turn, can contribute for better planning for sustainable ARV provision.


1. United Nations (UN). United Nations Millennium Declaration, A/RES/55/2. New York: UN; 2000. [ Links ]

2. UNAIDS. Declaration of Commitment on HIV/AIDS. Geneva: UNAIDS; 2001. [ Links ]

3. Joint United Nations Programme on HIV/AIDS (UNAIDS). World AIDS Day Report 2012 - Results. Geneva: UNAIDS; 2012. [ Links ]

4. Eholié S-P, Aoussi FE, Ouattara IS, Bissagnéné E, Anglaret X. HIV treatment and care in resource-constrained environments: challenges for the next decade. J Int AIDS Soc 2012; 15:17334. [ Links ]

5. Médecins Sans Frontières (MSF). Untangling the web of antiretroviral price reductions. Geneve: MSF; 2011. [ Links ]

6. Meiners C, Sagaon-Teyssier L, Hasenclever L, Moatti J-P. Modeling HIV/AIDS Drug Price Determinants in Brazil: Is Generic Competition a Myth? PLoS ONE 6(8):e23478. [ Links ]

7. Médecins Sans Frontières (MSF). Untangling the web of antiretroviral price reductions. Geneve: MSF; 2013. [ Links ]

8. Hoen ET, Berger J, Calmy A, Moon S. Driving a decade of change: HIV/AIDS, patents and access to medicines for all. J Int AIDS Soc 2011; 14:15. [ Links ]

9. Atun R, Bataringaya J. Building a Durable Response to HIV/AIDS: Implications for Health Systems. J Acquir Immune Defic Syndr 2011; 57(Supl. 2):S91-S95. [ Links ]

10. Sivaram S, Celentano DD. Training outreach workers for AIDS prevention in rural India: is it sustainable? Health Policy and Planning 2003; 18(4):411-420. [ Links ]

11. Rowe L, Rechel B. Fighting tuberculosis and HIV/AIDS in Northeast Europe: sustainable collaboration or political rhetoric? Eur J Public Health 2006; 16(6):609-614. [ Links ]

12. Greco DB, Simão M. Brazilian policy of universal access to AIDS treatment: sustainability challenges and perspectives. AIDS 2007; 21(Supl. 4):S37-S45. [ Links ]

13. Schell SF, Luke DA, Schooley MW, Elliott MB, Herbers SH, Mueller NB, Bunger AC. Public health program capacity for sustainability: a new framework. Implementation Science 2013; 8(15):1-9. [ Links ]

14. Stirman SW, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implementation Science 2012; 7(17):1-19. [ Links ]

15. Scheirer MA. Linking Sustainability Research to Intervention Types. Am J Public Health 2013; 103(4):e73-80. [ Links ]

16. Chen HT. Theory-driven evaluations. Newbury: Park Sage Publications; 1990. [ Links ]

17. Shediac-Rizkallah MC, Bone LR. Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Education Research 1998; 13(1):87-108. [ Links ]

18. Scheirer MA. Is Sustainability Possible? A Review and Commentary on Empirical Studies of Program Sustainability. Am J Eval 2005; 26(3):320-347. [ Links ]

19. Bossert TJ. Can they get along without us? Sustainability of donor-supported health projects in Central America and Africa. Soc Sci Med 1990; 30(9):1015-1023. [ Links ]

20. Pluye P, Potvin L, Denis J-L. Making public health programs last: conceptualizing sustainability. Evaluation and Program Planning 2004; 27(2):121-133. [ Links ]

21. Mazmanian D, Sabatier PA. A Framework for implementation analysis. In: Mazmanian D, Sabatier PA, editors. Implementation and public policy. Boston, London: University Press of America; 1989. p. 18-48. [ Links ]

22. Denis J-L, Champagne F. Análise da implantação. In: Hartz ZMA, editor. Avalliação em Saúde: dos modelos conceituais à prática na análise de implantação de programas. Rio de Janeiro: Fiocruz; 1997. p. 49-88. [ Links ]

23. Love A. Implementation Evaluation. In: Wholey JS, Hatry HP, Newcomer KE, editors. Handbook of practical program evaluation. 2nd ed. San Francisco: Jossey-Bass; 2004. p. 63-97. [ Links ]

24. Pluye P, Potvin L, Denis J-L, Pelletier J, Mannoni C. Program sustainability begins with the first events. Evaluation and Program Planning 2005; 28(2):123-37. [ Links ]

25. Steel D. Social Mechanisms and Causal Inference. Philosophy of the Social Sciences 2004; 34(1):55-78. [ Links ]

26. Hacker MA, Kaida A, Hogg RS, Bastos FI. The first ten years: achievements and challenges of the Brazilian program of universal access to HIV/AIDS comprehensive management and care,1996-2006. Cad Saude Publica 2007; 23(Supl. 3):S345-S59. [ Links ]

27. Joint United Nations Programme on HIV/AIDS (UNAIDS). “Three Ones” key principles - “Coordination of National Responses to HIV/AIDS” Guiding principles for national authorities and their partners. Conference Paper 1; Washington Consultation 25.04.042004. [ Links ]

28. Savaya R, Spiro S, Elran-Barak R. Sustainability of Social Programs: A Comparative Case Study Analysis. Am J Eval 2008; 29(4):478-493. [ Links ]

29. Brasil. Ministério da Saúde (MS). Conselho Nacional de Saúde. Resolução nº 196 de 10 de outubro de 1996. Diretrizes e Normas Regulamentadoras de Pesquisas Envolvendo Seres Humanos. Diário Oficial da União 1996; 16 out. [ Links ]

30. Giovanella L, Feo O, Faria M, Tobar S, editors. Sistemas de Salud en Suramérica - desafíos para la universalidad, la integralidad y la equidad. Rio de Janeiro: ISAGS; 2012. [ Links ]

31. The World Bank. The World Development Report online - Income Levels Washington, DC2013 [cited 2013 Nov 18]. Available from: [ Links ]

32. Pfeiffer J, Montoya P, Baptista AJ, Karagianis M, Pugas MdM, Micek M, Johnson W, Sherr K, Gimbel S, Baird S, Lambdin B, Gloyd S. Integration of HIV/AIDS services into African primary health care: lessons learned for health system strengthening in Mozambique - a case study. J Int AIDS Soc 2010; 13:3. [ Links ]

33. WHO Regional Office for Africa (WHO/AFRO). WHO Country Cooperation Strategy, 2009-2013 - Mozambique. Brazzaville: WHO/AFRO; 2009. [ Links ]

34. Bastos FI, Cáceres C, Galvão J, Veras MA, Castilho EA. AIDS in Latin America: assessing the current status of the epidemic and the ongoing response. Int J Epidemiol 2008; 37(4):729-737. [ Links ]

35. Hartz ZMdA, Vieira-da-Silva LM. Avaliação em saúde: dos modelos teóricos à prática na avaliação de programas e sistemas de saúde. Salvador, Rio de Janeiro: EDUFBA, Fiocruz; 2005. [ Links ]

36. Chen H-T. Practica program evaluation: Assessing and improving planning, implementation and effectiveness. Thousand Oaks, London, New Delhi: Sage; 2005. [ Links ]

37. Potvin L, Bisset S. There is more to methodology than method. In: Potvin L, McQueen D, editors. Health Promotion Evaluation Practices in the Americas. New york: Springer; 2008. p. 63-80. [ Links ]

38. Bisset S, Potvin L. Expanding our conceptualization of program implementation: lessons from the genealogy of a school-based nutrition program. Health Education Research 2007; 22(5):737-746. [ Links ]

39. Pluye P, Potvin L, Denis JL, Pelletier J. Program sustainability: focus on organizational routines. Health Promot Int 2004; 19(4):489-500. [ Links ]

40. Tansey O. Process Tracing and Elite Interviewing: A Case for Non-probability Sampling. PS: Political Science and Politics 2007; 40(4):765-772. [ Links ]

41. World Health Organization (WHO). Global update on HIV treatment 2013: results, impact and opportunities: WHO report in partnership with UNICEF and UNAIDS. Geneva: WHO; 2013. [ Links ]

42. Ooms G. Health Development versus Medical Relief: The Illusion versus the Irrelevance of Sustainability. PLoS Medicine 2006; 3(8):1202-1205. [ Links ]

43. Ooms G, Damme WV, Baker BK, Zeitz P, Schrecker T. The ‘diagonal’ approach to Global Fund financing: a cure for the broader malaise of health systems? Global Health 2008; 4:6. [ Links ]

44. Aidspan. Countries most in need of scaling-up ARV provision. 2008 [cited 2013 19 December]. Available from: . [ Links ]

45. The World Bank. Antiretroviral therapy coverage 2013 [cited 2013 19 December]. Available from: [ Links ]

46. The Global Fund to Fight Aids, Tuberculosis and Malaria (GFATM). Grant Portfolio - Peru 2013 [cited 2013 December 19]. Available from: [ Links ]

47. Amaya AB. Blood, aid & sustainability: The case of HIV in Peru. International Health Policies 2013. [ Links ]

48. Swidler A, Watkins SC. ‘‘Teach a Man to Fish”: The Sustainability Doctrine and Its Social Consequences. World Development 2009; 37(7):1182-1196. [ Links ]

49. United Nations Development Program (UNDP). Audit of UNDP Bolivia: Grants from The Global Fund to Fight AIDS, Tuberculosis ans Malaria. Report No. 901. Office of Audit and Investigations 15 August 2012. 2012. [ Links ]

50. Moçambique. Ministério da Saúde (MS). Desempenho do sector da Saúde. Oitava avaliação conjunta anual. Maputo: MS; 2009. [ Links ]

51. Peru. Ministério de Salud. Medición del Gasto en SIDA - MEGAS: Informe final. Lima: Minitério de Salud; 2011. [ Links ]

52. Bolivia. Ley nº 3729, de 8 de agosto de 2007. Ley para la prevencion del vih-sida, proteccion de los derechos humanos y asistencia integral multidisciplinaria para la personas que viven con el VIH-SIDA. La Paz; 2007. [ Links ]

53. Ooms G, Damme WV, Temmerman M. Medicines without Doctors: Why the Global Fund Must Fund Salaries of Health Workers to Expand AIDS Treatment. PLoS Medicine 2007; 4(4):e128. [ Links ]

54. Mucha SM. Compras públicas de antirretrovirales en Perú - Octubre 2008- Julio 2009. Lima: Acción Internacional para la Salud, Oficina de Coordinación América Latina y el Caribe; 2010. [ Links ]

55. Osorio-de-Castro CGS, Crisante M, Miranda ES, Oliveira EA, Oliveira MA. Proposed methodology for monitoring antiretroviral drugs price negotiations in Latin America and the Caribbean. Pan Am J Public Health 2009; 26(2):137-147. [ Links ]

56. Vazquez ES, Monguio RR. Negotiating antiretroviral drug prices: the experience of the Andean countries. Health Policy Plan 2007; 22(2):63-72. [ Links ]

57. Couto Z. Encerramento dos Hospitais Dia: doentes perdem espaços onde se sentiam dignificados e não discriminados. Vertical 2009; 25:1-2. [ Links ]

58. Matland RE. Synthesizing the Implementation Literature: The Ambiguity-Conflict Model of Policy Implementation. Journal of Public Administration Research and Theory 1995; 5(2):145-174. [ Links ]

59. Lipsky M. Street-Level Bureaucracy: Dilemmas of the individual in public services. New York: Russel Sage Foundation; 1980. [ Links ]

60. Elmore RF. Backward mapping. Implementation research and policy decisions. Political Science Quaterly 1980; 94(4):601-616. [ Links ]

61. Barrett SM. Implementation studies: time for a revival? Personal reflections on 20 years of implementation studies. Public Administration 2004; 82(2):249-262. [ Links ]

Received: October 25, 2016; Accepted: February 09, 2017; Revised: February 11, 2017


All authors participated in study design. TB Azeredo, CDB Santos-Pinto, ES Miranda and CGS Osório-de-Castro conducted data collection and preliminary analysis. TB Azeredo, MA Oliveira and CGS Osório-de-Castro refined analysis and drafted the manuscript. All authors revised the final version of the manuscript.

Creative Commons License  This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.