Markov Models in health care

ABSTRACT

Markov Chains provide support for problems involving decision on uncertainties through a continuous period of time. The greater availability and access to processing power through computers allow that these models can be used more often to represent clinical structures. Markov models consider the patients in a discrete state of health, and the events represent the transition from one state to another. The possibility of modeling repetitive events and time dependence of probabilities and utilities associated permits a more accurate representation of the evaluated clinical structure. These templates can be used for economic evaluation in health care taking into account the evaluation of costs and clinical outcomes, especially for evaluation of chronic diseases. This article provides a review of the use of modeling within the clinical context and the advantages of the possibility of including time for this type of study.

Keywords:
Health economics; Markov chains; Models, economic

RESUMO

Os modelos de Markov prestam apoio aos problemas de decisão envolvendo incertezas em um período contínuo de tempo. A maior disponibilidade e o maior acesso no poder de processamento por meio dos computadores permite que esses modelos possam ser utilizados mais frequentemente para representar estruturas clínicas. Os modelos de Markov consideram os pacientes em um estado discreto de saúde, e os eventos representam a transição de um estado para outro. A possibilidade de modelar eventos repetitivos e a dependência temporal das probabilidades e utilidades associadas permitem uma representação mais precisa da estrutura clínica avaliada. Esses modelos podem ser utilizados para avaliações econômicas em saúde levando em consideração a avaliação dos custos e desfechos clínicos (outcomes), especialmente para a avaliação de doenças crônicas. Este artigo oferece uma revisão do uso dessa modelagem dentro do contexto clínico e as vantagens da possibilidade da inclusão temporal para esse tipo de estudo.

Descritores:
Economia da saúde; Cadeias de Markov; Modelos econômicos

INTRODUCTION

Economic decision models have been increasingly used to assess health interventions(¹,²). Advances in this field are mainly due to enhanced processing capacity of computers, availability of specific software to perform these tasks, and sophisticated mathematical techniques, which have become more popular.

Due to the reasons pointed out above, more investigators adopted the Markov models, which historically had already been used in epidemiological and clinical evaluations (³). In health economics, the strength of Markov models is that they take into consideration the use of resources and the outcomes.

In this review the authors discuss the use of Markov models for economic evaluations of the health sector. This work introduces a structure to evaluate health programs, the use of the Markov model, its variables and structure of analysis.

Authors agree that economic evaluations in health care should be carried out to deal with the introduction of new technologies, based on an analytic decision model under conditions of uncertainty^(4-8). This model follows the following decision making process:

1. structure: must adequately reflect the possibility of prognosis that individuals may undergo, and the impact that treatment and health programs have on said prognosis. In this situation, the individuals are usually patients with a specific health condition, but may be healthy or asymptomatic, as in prevention campaigns;

2. evidence: provides an analytical structure in which relevant evidence for the study may be defined. This could be obtained through the model and through the entry parameters;

3. evaluation: provides a mean of translating relevant evidence into cost estimates and comparison of the impact of the options under comparison. The main types of study are cost-effectiveness, cost-benefit and cost-utility. The best option must be treated based on the evidence available;

4. uncertainty and variability: enables an evaluation of numerous types of uncertainty, including those related to the model and the entry parameters. The models must also provide flexibility to characterize heterogeneity through several subgroups of individuals;

5. future research: through the evaluation of uncertainties, it is possible to identify priorities for future research, which will produce evidence to re-evaluate the issue in the future.

Following this decision-making process, the economic evaluation seeks information regarding the process of appropriate data measurement and adequate information about the distribution of resources(⁸) according to the uncertainty at hand. Chart 1 contains a brief description of the types of possible uncertainties and the possible approaches to deal with them.

Chart 1
Types of uncertainties and possible approaches

Type of uncertainty	Possible approach
Methodological	Reference case/sensitivity analysis
Sample variation	Statistical analysis
Extrapolation	Modeling methods
Transferability	Sensitivity analysis

In the present study, the authors addressed the problem of uncertainty through extrapolation using Markov models.

TIME IN MARKOV MODELS

The most important difference between the Markov models and other models of economic evaluation in health science is the state of a patient during a specific moment in time. The factor “time” is explicitly associated with the probability of a patient taking certain states in a series of discrete periods of time. In Markov models, these periods are called “cycles”. In other words, a disease is divided in distinct cycles, and probabilities are attributed to the transition between these states. The duration of these cycles depends on the disease and on the interventions that are being evaluated, and may be monthly or annual cycles, for example. From the point of view of economic evaluation, a cost is associated with each cycle, except in the case of cost-utility studies, in which the value represents the utility associated to each cycle. The average amount of time that a patient spends on the various states of the model is then weighted by cost or utility, which will be used to calculate the expected costs and outcomes. How fast patients move from one state of the model to the next is determined by the probability of the transitions. Thus, by determining the use of resources and outcomes in health, it is possible to evaluate these factors associated to the disease and the intervention that is performed.

The first stage in the construction of a Markov model is defining the different states of the disease. These states must represent the important clinical and economic effects of the disease, and said effects should be included in the model. One important consideration is that these stages of disease are mutually exclusive, because the patient cannot be in more than one state of the disease at the same time.

With the development of chronic diseases, such as hypertension and diabetes in developing countries(⁹–¹²), Markov models became important tools for planning health care programs. Figure 1 is a graphic representation of chronic disease that may be introduced in an economic evaluation model.

Figure 1
Stages of the disease progression until death(¹)

The first state is defined as asymptomatic and indicates that the patient suffers from the disease, but is not experiencing its consequences and the risk of death is not higher than in someone who does not have the disease. From this state of the disease on, the patient may move towards the stage of “death,” based on the probability of transition or progression of the disease. In disease progression, the patient starts to experience the health impairment with an increased risk of death caused by the direct result of the disease on all other causes of mortality.

Absorbing state is a state in the model from which it is technically impossible to move out, and an example is death. The backward arrows indicate the possibility of the patient remaining in this state or, according to the model, it is possible to include improvements in the clinical conditions of the patient, as in the case of disease remission.

The probabilities of transition are considered in each cycle of the model, and may be represented in a matrix of the type “n x n”. The sum of probabilities of transition of each cycle must be equal to 1 (one), because there is only one state in each discrete moment of time. Thus, the probability of remaining in the same state is given by the value 1 (one) minus the probability of transition.

For the purpose of illustration, table 1 presents a matrix of probabilities in a monotherapy state.

Table 1
Probabilities of transition in a monotherapy case(¹³)

Transition from	Transition to
	state A	state B	state C	state D
State A	0.721	0.202	0.067	0.01
State B	0	0.581	0.407	0.012
State C	0	0	0.75	0.25
State D	0	0	0	1

One important observation is the possible confusion in the use of the terms “rate” and “probability”. Rate represents the transition in any given point in time, whereas probability is the proportion of the population at risk in a specific period of time(¹,¹⁴). Therefore, probabilities available in the literature may not reflect the same period of time in the Markov cycle of the model in use. Data retrieved from the academic literature is usually expressed in rates that may vary from 0 to indefinite (example: a mortality rate of 2% a year for disease X), whereas probabilities vary from 0 to 1 during a specific period of time.

The issue of probabilities may be avoided if:

where time is expressed as “t” and rate as “r”(¹⁵).

The attribution of weights to the model is needed in order to estimate costs and outcomes. In the case of life expectancy, the weight 1 (one) is attributed to each state of the model in which the patient is “alive,” whereas 0 (zero) is attributed to the state of “dead”. The average life expectancy in terms of size of the model cycle is obtained through performing the model over a large number of cycles and adding up the values throughout these cycles. This multiplied by the size of the cycle in years produces the life expectancy in years. In economic evaluations, it is common to observe this estimate in terms of quality-adjusted life years (QALY). The main advantage resides in averaging the extension of time and a specific state of health represented by the quality of life during this state of health. This makes QALY appropriate for use in Markov models. The use of cost follows the same method, attributing values spent during each cycle, and the cost is obtained through the sum of cycles. One consideration is the possibility to attribute cost not only to the states, but also to the transitions, that can represent punctual treatments.

Markov models are widely used in science, including areas such as Biology, Mathematics, Social Science, Music, internet, Chemistry and Physics.

The strengths of this type of model are the constant transition probabilities that may be solved by matrix algebra, considering that the matrix of transition contemplates the time spent in each state and the expected value of each outcome in a precise manner. However, the weakness of this type of model is also the statistic probability. As mentioned above, these methods became more popular due to the higher processing capacity of computers, which are able to overcome the statistical limit of matrix analysis.

Below is a review of the two main types of Markov models used to assess health care programs – cohort simulation and individual simulation.

The cohort simulation offers a direct solution. The cohort initiates at the moment 0 in an initial state of disease; in our example of chronic disease, it corresponds to the asymptomatic condition of the disease. For each cycle of the model the transition probabilities are applied and the distribution of patients in each state is adjusted. The execution of several cycles determines the profile of how many patients exist in each state of the model throughout time. Table 2 presents a hypothetical cohort model with 1,000 patients.

Table 2
Simulation by cohort for the example model(¹)

Cycle	State of the disease			Total
	Asymptomatic	Progressive	Death
0	1,000	0	0	1,000
1	976	10	14	1,000
2	943	28	29	1,000
3	902	52	46	1,000
4	854	79	67	1,000
5	799	109	92	1,000
6	740	139	121	1,000
7	678	168	154	1,000
8	614	195	191	1,000
9	551	218	231	1,000
10	488	237	275	1,000

In the Monte Carlo simulations, instead of initiating a cohort of the model, a great number of patients are followed up individually. The main difference is that despite the fact that individual patients are subject to the same transition probabilities, in the Monte Carlo model, they may or may not “move” between the stages of each cycle. Thus, the path followed by different patients will vary, due to the random variability; whereas cost and outcomes are produced according to the path followed in the model.

Despite the growth of economic analysis in clinical testing, it is clear that the main economic advantages may be translated in better distribution of scarce resources.

Economic evaluations are a preliminary stage in clinical studies and Markov models are adequate to evaluate the progression of disease throughout time. Another important advantage is the ability to deal with costs and effects simultaneously. Similar to all models, Markov models have limitations that must be overcome as models become more sophisticated, especially when dealing with time-dependent probabilities of transitions and different states of disease. Another inherent difficulty of this type of model is the greater complexity when compared to the simpler decision-making trees, and the lack of “memory”. This is due to the assumption of Markov regarding the probability of moving between states of health disregarding the experiences of previous cycles(¹⁶). This can be reduced with the “tunnel states” that enable integration of health experiences from the previous cycles(¹⁶,¹⁷). The states of the cycles can be accessed only in a pre-determined sequence, an analogy to passing through a tunnel. The purpose of this approach is to offer a temporary adjustment in the probability transitions that last longer than one cycle.

In health sciences, Markov models are widely used as analytical tools to assess diseases from an economic point of view. According to this technique, a patient may be assessed in a finite number of discrete states of health, in which the important clinical events are modeled as transitions from one state to another. The studies involving the Markov chains may be presented simulations, such as cohort; that is, a trial with multiple subjects, or through a Monte Carlo simulation, involving multiple trials and one subject for each.

These studies are presented as cycle trees that combine the structure of decision-making with the Markov processes. This enables the consideration of clinical problems with continuous risks throughout time within a model. The diffusion of these techniques may contribute in clinical assessments of the current moment, which involves increases in cost and in the prevalence of chronic diseases.

REFERENCES

• ¹
  Briggs A, Sculpher M. An introduction to Markov modelling for economic evaluation. Pharmacoeconomics. 1998;13(4):397-409.
• ²
  Buxton MJ, Drummond MF, Van Hout BA, Prince RL, Sheldon TA, Szucs T, et al. Modelling in economic evaluation: an unavoidable fact of life. Health Econ. 1997;6(3):217-27.
• ³
  Sonnenberg FA, Beck JR. Markov models in medical decision making: a practical guide. Med Decis Making. 1993;13(4):322-38.
• ⁴
  Elsinga E, Rutten FF. Economic evaluation in support of national health policy: the case of The Netherlands. Soc Sci Med. 1997;45(4):605-20.
• ⁵
  Sculpher MJ, Claxton K, Drummond M, McCabe C. Whither trial-based economic evaluation for health care decision making? Health Econ. 2006;15(7):677-87.
• ⁶
  Buxton MJ. Economic evaluation and decision making in the UK. Pharmacoeconomics. 2006;24(11):1133-42.
• ⁷
  Noorani HZ, Husereau DR, Boudreau R, Skidmore B. Priority setting for health technology assessments: a systematic review of current practical approaches. Int J Technol Assess Health Care. 2007;23(3):310-5.
• ⁸
  Drummond MF, Sculpher MJ, Torrance GW, OBrien BJ, Stoddart GL. Methods for the economic evaluation of health care programmes. 3^rd ed. Oxford: University Press; 2005.
• ⁹
  Zimmet P, Alberti KG, Shaw J. Global and societal implications of the diabetes epidemic. Nature. 2001;414(6865):782-7.
• ¹⁰
  Abegunde DO, Mathers CD, Adam T, Ortegon M, Strong K. The burden and costs of chronic diseases in low-income and middle-income countries. Lancet. 2007;370(9603):1929-38.
• ¹¹
  Yach D, Leeder SR, Bell J, Kistnasamy B. Global chronic diseases. Science. 2005;307(5708):317.
• ¹²
  Adeyi O, Smith O, Robles S. Public policy and the challenge of chronic noncommunicable diseases. Washington: The World Bank; 2007.
• ¹³
  Chancellor JV, Hill AM, Sabin CA, Simpson KN, Youle M. Modelling the cost effectiveness of lamivudine/zidovudine combination therapy in HIV infection. Pharmacoeconomics. 1997;12(1):54-66.
• ¹⁴
  Miller DK, Homan SM. Determining transition probabilities: confusion and suggestions. Med Decis Making. 1994;14(1):52-8.
• ¹⁵
  Beck JR, Pauker SG. The Markov process in medical prognosis. Med Decis Making. 1983;3(4):419-458.
• ¹⁶
  Rascati, K. Essentials of pharmacoeconomics. Philadelphia: Wolters Kluwer; 2009.
• ¹⁷
  Hawkins N, Sculpher M, Epstein D. Cost-effectiveness analysis of treatments for chronic disease: using R to incorporate time dependency of treatment response. Med Decis Making. 2005;25(5):511-9.

Publication Dates

• Publication in this collection
  Jul-Sep 2010

History

• Received
  09 Dec 2009
• Accepted
  28 June 2010