Markov Models in health care

Markov Chains provide support for problems involving decision on uncertainties through a continuous period of time. The greater availability and access to processing power through computers allow that these models can be used more often to represent clinical structures. Markov models consider the patients in a discrete state of health, and the events represent the transition from one state to another. The possibility of modeling repetitive events and time dependence of probabilities and utilities associated permits a more accurate representation of the evaluated clinical structure. These templates can be used for economic evaluation in health care taking into account the evaluation of costs and clinical outcomes, especially for evaluation of chronic diseases. This article provides a review of the use of modeling within the clinical context and the advantages of the possibility of including time for this type of study.

Descritores: Economia da saúde; Cadeias de Markov; Modelos econômicos inTrODUcTiOn Economic decision models have been increasingly used to assess health interventions (1,2) .Advances in this field are mainly due to enhanced processing capacity of computers, availability of specific software to perform these tasks, and sophisticated mathematical techniques, which have become more popular.
Due to the reasons pointed out above, more investigators adopted the Markov models, which historically had already been used in epidemiological and clinical evaluations (3) .In health economics, the strength of Markov models is that they take into consideration the use of resources and the outcomes.
In this review the authors discuss the use of Markov models for economic evaluations of the health sector.This work introduces a structure to evaluate health programs, the use of the Markov model, its variables and structure of analysis.
Authors agree that economic evaluations in health care should be carried out to deal with the introduction of new technologies, based on an analytic decision model under conditions of uncertainty (4)(5)(6)(7)(8) .This model follows the following decision making process: 1. structure: must adequately reflect the possibility of prognosis that individuals may undergo, and the impact that treatment and health programs have on said prognosis.In this situation, the individuals are usually patients with a specific health condition, but may be healthy or asymptomatic, as in prevention campaigns; 2. evidence: provides an analytical structure in which relevant evidence for the study may be defined.This could be obtained through the model and through the entry parameters; 3. evaluation: provides a mean of translating relevant evidence into cost estimates and comparison of the impact of the options under comparison.The main types of study are cost-effectiveness, cost-benefit and cost-utility.The best option must be treated based on the evidence available; 4. uncertainty and variability: enables an evaluation of numerous types of uncertainty, including those related to the model and the entry parameters.The models must also provide flexibility to characterize heterogeneity through several subgroups of individuals; 5. future research: through the evaluation of uncertainties, it is possible to identify priorities for future research, which will produce evidence to reevaluate the issue in the future.
Following this decision-making process, the economic evaluation seeks information regarding the process of appropriate data measurement and adequate information about the distribution of resources (8) according to the uncertainty at hand.Chart 1 contains a brief description of the types of possible uncertainties and the possible approaches to deal with them.
expected costs and outcomes.How fast patients move from one state of the model to the next is determined by the probability of the transitions.Thus, by determining the use of resources and outcomes in health, it is possible to evaluate these factors associated to the disease and the intervention that is performed.
The first stage in the construction of a Markov model is defining the different states of the disease.These states must represent the important clinical and economic effects of the disease, and said effects should be included in the model.One important consideration is that these stages of disease are mutually exclusive, because the patient cannot be in more than one state of the disease at the same time.
With the development of chronic diseases, such as hypertension and diabetes in developing countries (9- 12) , Markov models became important tools for planning health care programs.Figure 1 is a graphic representation of chronic disease that may be introduced in an economic evaluation model.In the present study, the authors addressed the problem of uncertainty through extrapolation using Markov models.

TiMe in MarKOv MODels
The most important difference between the Markov models and other models of economic evaluation in health science is the state of a patient during a specific moment in time.The factor "time" is explicitly associated with the probability of a patient taking certain states in a series of discrete periods of time.In Markov models, these periods are called "cycles".In other words, a disease is divided in distinct cycles, and probabilities are attributed to the transition between these states.The duration of these cycles depends on the disease and on the interventions that are being evaluated, and may be monthly or annual cycles, for example.From the point of view of economic evaluation, a cost is associated with each cycle, except in the case of cost-utility studies, in which the value represents the utility associated to each cycle.The average amount of time that a patient spends on the various states of the model is then weighted by cost or utility, which will be used to calculate the  (1) The first state is defined as asymptomatic and indicates that the patient suffers from the disease, but is not experiencing its consequences and the risk of death is not higher than in someone who does not have the disease.From this state of the disease on, the patient may move towards the stage of "death", based on the probability of transition or progression of the disease.In disease progression, the patient starts to experience the health impairment with an increased risk of death caused by the direct result of the disease on all other causes of mortality.
Absorbing state is a state in the model from which it is technically impossible to move out, and an example is death.The backward arrows indicate the possibility of the patient remaining in this state or, according to the model, it is possible to include improvements in the clinical conditions of the patient, as in the case of disease remission.
The probabilities of transition are considered in each cycle of the model, and may be represented in a matrix of the type "n x n".The sum of probabilities of transition of each cycle must be equal to 1 (one), because there is only one state in each discrete moment of time.Thus, the probability of remaining in the same state is given by the value 1 (one) minus the probability of transition.
For the purpose of illustration, table 1 presents a matrix of probabilities in a monotherapy state.
the cycle in years produces the life expectancy in years.In economic evaluations, it is common to observe this estimate in terms of quality-adjusted life years (QALY).The main advantage resides in averaging the extension of time and a specific state of health represented by the quality of life during this state of health.This makes QALY appropriate for use in Markov models.The use of cost follows the same method, attributing values spent during each cycle, and the cost is obtained through the sum of cycles.One consideration is the possibility to attribute cost not only to the states, but also to the transitions, that can represent punctual treatments.
Markov models are widely used in science, including areas such as Biology, Mathematics, Social Science, Music, internet, Chemistry and Physics.
The strengths of this type of model are the constant transition probabilities that may be solved by matrix algebra, considering that the matrix of transition contemplates the time spent in each state and the expected value of each outcome in a precise manner.However, the weakness of this type of model is also the statistic probability.As mentioned above, these methods became more popular due to the higher processing capacity of computers, which are able to overcome the statistical limit of matrix analysis.
Below is a review of the two main types of Markov models used to assess health care programs -cohort simulation and individual simulation.
The cohort simulation offers a direct solution.The cohort initiates at the moment 0 in an initial state of disease; in our example of chronic disease, it corresponds to the asymptomatic condition of the disease.For each cycle of the model the transition probabilities are applied and the distribution of patients in each state is adjusted.The execution of several cycles determines the profile of how many patients exist in each state of the model throughout time.Table 2 presents a hypothetical cohort model with 1,000 patients.
Table 1.Probabilities of transition in a monotherapy case (13)  One important observation is the possible confusion in the use of the terms "rate" and "probability".Rate represents the transition in any given point in time, whereas probability is the proportion of the population at risk in a specific period of time (1,14) .Therefore, probabilities available in the literature may not reflect the same period of time in the Markov cycle of the model in use.Data retrieved from the academic literature is usually expressed in rates that may vary from 0 to indefinite (example: a mortality rate of 2% a year for disease X), whereas probabilities vary from 0 to 1 during a specific period of time.
The issue of probabilities may be avoided if: where time is expressed as "t" and rate as "r" (15) .The attribution of weights to the model is needed in order to estimate costs and outcomes.In the case of life expectancy, the weight 1 (one) is attributed to each state of the model in which the patient is "alive", whereas 0 (zero) is attributed to the state of "dead".The average life expectancy in terms of size of the model cycle is obtained through performing the model over a large number of cycles and adding up the values throughout these cycles.This multiplied by the size of In the Monte Carlo simulations, instead of initiating a cohort of the model, a great number of patients are followed up individually.The main difference is that despite the fact that individual patients are subject to the same transition probabilities, in the Monte Carlo model, they may or may not "move" between the stages of each cycle.Thus, the path followed by different patients will vary, due to the random variability; whereas cost and outcomes are produced according to the path followed in the model.
Despite the growth of economic analysis in clinical testing, it is clear that the main economic advantages may be translated in better distribution of scarce resources.
Economic evaluations are a preliminary stage in clinical studies and Markov models are adequate to evaluate the progression of disease throughout time.Another important advantage is the ability to deal with costs and effects simultaneously.Similar to all models, Markov models have limitations that must be overcome as models become more sophisticated, especially when dealing with time-dependent probabilities of transitions and different states of disease.Another inherent difficulty of this type of model is the greater complexity when compared to the simpler decision-making trees, and the lack of "memory".This is due to the assumption of Markov regarding the probability of moving between states of health disregarding the experiences of previous cycles (16) .This can be reduced with the "tunnel states" that enable integration of health experiences from the previous cycles (16,17) .The states of the cycles can be accessed only in a pre-determined sequence, an analogy to passing through a tunnel.The purpose of this approach is to offer a temporary adjustment in the probability transitions that last longer than one cycle.
In health sciences, Markov models are widely used as analytical tools to assess diseases from an economic point of view.According to this technique, a patient may be assessed in a finite number of discrete states of health, in which the important clinical events are modeled as transitions from one state to another.The studies involving the Markov chains may be presented simulations, such as cohort; that is, a trial with multiple subjects, or through a Monte Carlo simulation, involving multiple trials and one subject for each.
These studies are presented as cycle trees that combine the structure of decision-making with the Markov processes.This enables the consideration of clinical problems with continuous risks throughout time within a model.The diffusion of these techniques may contribute in clinical assessments of the current moment, which involves increases in cost and in the prevalence of chronic diseases.

chart 1 .
Types of uncertainties and possible approaches

Figure 1 .
Figure 1.Stages of the disease progression until death(1)