Remarks on Non-Markov Processes

Abstract

It is emphasized that non-Markovian processes, which occur for instance in the case of colored noise, cannot be considered merely as corrections to the class of Markov processes but require special treatment. Many familiar concepts, such as first-passage times, no longer apply and need be reconsidered. Several methods of dealing with non-Markov processes are discussed. As an example a recent application to the transport of ions through a membrane is briefly mentioned.


** This text corresponds to an invited talk at the "Workshop on the Foundations of Statistical Mechanics and Thermodynamics'' held in Natal, Brazil, in October, 1997.

N.G. van Kampen

Institute for Theoretical Physics, University Utrecht

Princetonplein 5, 3584 CC Utrecht,

The Netherlands

Received March 9, 1998

It is emphasized that non-Markovian processes, which occur for instance in the case of colored noise, cannot be considered merely as corrections to the class of Markov processes but require special treatment. Many familiar concepts, such as first-passage times, no longer apply and need be reconsidered. Several methods of dealing with non-Markov processes are discussed. As an example a recent application to the transport of ions through a membrane is briefly mentioned.

I. Definition of Markov processes

The term 'non-Markov Process' covers all random processes with the exception of the very small minority that happens to have the Markov property.

FIRST REMARK. Non-Markov is the rule, Markov is the exception.

It is true that this minority has been extensively studied, but it is not proper to treat non-Markov processes merely as modifications or corrections of the Markov processes - as improper as for instance treating all nonlinear dynamical systems as corrections to the harmonic oscillator. I therefore have to start by reviewing some general facts [1, 2].

A stochastic process is a collection of random variables Xt, labeled by an index t which may be discrete but more often covers all real numbers in some interval. The stochastic properties of {Xt} are expressed by the joint distribution functions

The process is uniquely defined by the entire set of these distribution functions for n = 1,2,..., which in general is infinite.

When the values are given, the remaining variables obey the conditional probability distribution function

This is a probability distribution of , in which enter as parameters. Let us take the ti in chronological order, then the process is Markov if this conditional probability depends on the latest value xk at tk alone and is independent of the earlier values . This must hold for all n, for any choice of k, and for any and . If this is true, all Pn can be constructed once P1 and P2 are given. For example,

SECOND REMARK. The reason for the popularity of Markov processes is the fact that they are fully determined by these two functions alone. For non-Markov processes the distribution functions(1) must be determined by some other means, usually an entirely different mathematical construction. For M-processes it makes therefore sense to honor the function with the name transition probability.

II. Example

Symmetric random walk in 1 dimension. Here t = 0,1,2, ... and x takes integer values i. The process is Markovian with symmetric transition probability

But suppose the walker has a tendency to persist in his direction: probability p to step in the same direction, and q to return [3]. Then Xt is no longer Markovian since the probability of Xt depends not just on xt-1 but also on xt-2. This may be remedied by introducing the two-component variable {Xt, Xt-1}. This joint variable is again Markovian, with transition probability

THIRD REMARK. A physical process (i.e. some physical phenomenon evolving in time) may or may not be Markovian, depending on the variables used to describe it.

If the memory of our random walk involves more preceding steps, more additional variables are needed. That does no longer work, however, if the memory extends over all previous steps. Example: polymers with excluded volume. This problem is often modelled as a random walk, but it is irremediably non-Markov and has not been solved [4].

III. The master equation

Take a Markov process in which t is the time while Xt takes discrete values i = 0,1, 2, ... In eq. (2) take ,

Sum over i2 and take the limit to obtain the master equation

The are transition probabilities per unit time and are properties belonging to the physical system (such as squares of matrix elements), while P refers to the state of the system. The parameters i1, t1 are often not written, which may lead to the misconception that P in (3) is the same as P1 in (1).

FOURTH REMARK. The master equation is an equation for the transition probability of a Markov process, valid for any initial i1, t1. If one knows in addition P1 (i1, t1) the whole hierarchy (1) and thus the process is uniquely determined (for ³ t1).

Warning. In the literature one occasionally encounters something called a "master equation with memory'',

with the claim that it defines a non-Markov process. Objections. (i) A non-Markov process is not defined when merely P(i, t | i1, t1) is known. (ii) The equation cannot be true for every x1, t1. (iii) The equation is no guarantee that the process is not Markovian [5].

IV. Diffusion

Let x be continuous as well. Then takes the form of an integral kernel W(x | x'). If the process is such that during an infinitely short Dt only infinitely small jumps are possible, then the kernel reduces to a differential operator [6]. The simplest example is the diffusion equation for the coordinate x of a Brownian particle,

The solution of this equation specified by the initial condition P(x, t1) = d(x-x1) is the transition probability P(x, t |x1, t1).

Here the coordinate is treated as Markovian, although the particle has a velocity v as well. One ought therefore to consider the joint variable {x, v} as Markovian, with master equation

This is Kramers' equation [7]; g is a friction coefficient and U(x) an external potential.

In this picture x by itself is not Markov. How is that compatible with (4)? The answer is that for large

g the collisions are so frequent that the velocity distribution rapidly becomes locally Maxwellian,

This makes it possible to eliminate v so that there remains an equation for P(x, t) by itself [7,1], namely

V. First-passage problems

Consider one-dimensional diffusion in a potential field as given by (7). Let it take place in a finite medium xa < x < xc (Figure 1). When the diffusing particle starts at an interior point xb, what are the chances that it will exit through xa or xc, respectively?

Figure 1
1

The answer is obtained by solving (7) with absorbing boundary conditions: P(xa, t) = 0 and P(xc, t) = 0. (The solution can be obtained explicitly thanks to the fact that the time does not occur in the equation for the probabilities.) It is clear that when xb is at the top of a high maximum of U(x) the exit probabilities will be fifty-fifty. It is also possible to find the mean time for either exit [2].

In three dimensions the question is: When I surround the potential maximum by a closed surface, what is the probability distribution of the exit points on that surface and how long does it take? This problem can be formulated in the same way, but cannot usually be solved analytically.

For a particle described by (5), however, the coordinate is not Markovian and it does not suffice therefore to know that x(t1) = xb: one also has to know its preceding history. In the present case, that history is represented by the value of v at t1. Of course it is possible to simply pick an initial P1(x, v, t1), but the correct choice depends on the problem one wants to solve. For instance I want to compute the autocorrelation function of x,

Evidently one needs to know the correct initial distribution P1 (x, v, t). Only in the limit of large g may it be replaced with P1 (x, t1) with the aid of (6).

FIFTH REMARK. For a non-Markov process the initial value problem is not well-defined unless further information about the problem is supplied.

Of more interest is the question of escape from a potential minimum such as xa in Fig. 2. How long does it take to get across the barrier? Here the ambiguity of the initial velocity is harmless because the particle moves around in the potential valley long enough for the Maxwellian (6) to prevail.

Figure 2
2

In the case of diffusion described by (7) one may take the mean time of first arrival at xb - and multiply by 2 because once in xb there is equal probability to escape or go back. The mean first-passage time can again be computed analytically. In more dimensions however, the question is: How long does it take to escape from a minimum xa surrounded by a potential ridge? This mean time is determined by the lowest mountain pass on the ridge and an elegant approximation is available [8].

In Kramers' equation (5), however, it is not enough to compute the time for x(t) to reach xb because x(t) is not Markovian and one cannot tell the probability of subsequent escape without taking into account v. It is necessary to reformulate the problem by picking a large xc and solving (5) with boundary condition: P(xc, v, t) = 0 for v < 0 (Fig. 3). This is the much discussed Kramers problem [7,9].

Figure 3

SIXTH REMARK. For non-Markov processes the first-passage problem may be formulated but its physical relevance is questionable.

VI. Langevin equation and colored noise

We start from Kramers' equation (5) and consider the spatially homogeneous case U' = 0. Then it is possible to integrate over x and obtain an equation for the distribution P(v, t) of v alone,

The Markov process v(t) described by this master equation is the Ornstein-Uhlenbeck process. A mathematically equivalent way of describing this process is the Langevin equation,

where x(t) is a stochastic function, called Gaussian white noise, whose stochastic properties are determined by

As a result, the solution v(t) (with given initial value v(t1 ) = v1) is also a stochastic function and Markovian, its master equation being (8) if one takes C = 2gT. The Langevin equation (9) with (10) is popular, because it is more intuitive than (8), but it is not better.

Numerous authors have generalized (9) by taking x(t) non-white or colored noise, i.e., not delta-correlated [10]

with some even function f.

SEVENTH REMARK. When the noise in the L-equation is not white the variable v(t) is not Markov. Hence for the Langevin equation with colored noise one cannot formulate a meaningful initial condition or a first-passage time.

A different generalization of the Langevin equation (9) is the nonlinear version [11],

When x(t) is white, then x(t) is Markov. Unfortunately the equation as it stands has no well-defined meaning and has to be supplied by an "interpretation rule'', either Itô or Stratonovich. To avoid the vagaries of the Itô calculus we choose the latter. It corresponds to the M-equation,

When x(t) is colored, then x(t) is not Markov and no M-equation exists. Attempts at constructing analogous partial differential equations are doomed to fail. Eq. (12) belongs to the general class of stochastic differential equations. They may be treated by approximation methods developed for the case that x(t) is off-white, i.e. short correlation and sharply peaked f(t-t' ) [12]. The M-equation appears as the first approximation.

An other device, applicable when this time is not short, was used by Kubo [13] and has been rediscovered many times [14]. Suppose x(t) is itself a Markov process governed by an M-equation, for example the O.-U. process governed by (8). Then the joint variable (x, x) is Markov, with M-equation

Again, the non-Markov x has been tamed by introducing an additional variable.

In practice this is of not much help unless one chooses for x(t) an even simpler non-white process, viz., the "dichotomic Markov process''. That is, x has two possible values +1, -1, and jumps with a constant probability g per unit time, as described by the M-equation

With this choice equation (13) reduces to

which is less forbidding, but still the subject of many articles [15]. Other choices for x(t), two-valued but not Markovian, have also been considered [16].

VII. A class of non-Markov processes

In various connections the following type of process occurs [17,18]. Let Xt have two or more possible states j, and in each it has a probability per unit time gij to jump to i. If the gij are constants, {Xt} is a Markov process with M-equation (3). However, suppose they are functions g(t) of the time t elapsed since arrival in j. Examples: Molecules in solution may get stuck to the wall temporarily; a bacterium produces offspring after reaching a certain age. Other examples in [18,1].

The probability that X is still in j at a time t after entering it, is given by

When starting at t = 0 in j0, it may have arrived in the state j at time t through a sequence of s transitions, at times t1, t2, ..., ts, taking it through the states j1 , j2 , ..., js = j. The probability for this particular history to happen is

The probability Pjj0(t) to be at time t in state j is obtained by summing over all histories, that is: summing over all s and all js , ..., js-1 and integrating over all intermediate times t1, ..., ts. The result, written in Laplace transforms, is

where is the transform of (14), and the matrix is

In some cases the result (15) can be evaluated explicitly.

Example: transport of ions through a membrane. The following model has been suggested [19]. A membrane separates two reservoirs and ions may enter from the left or from the right reservoir. The rates at which they enter (i.e., the probabilities per unit time) are determined by the two outside liquids, with the restriction that there can be at no time more than one ion inside the membrane. Once inside, the ion may exit after a while on either side. Thus the interior of the membrane is a system having 5 states:

(0) empty;

(1) one ion that entered at left and is destined to exit at right;

(2) one ion from the left destined to exit at left;

(3, 4) two similar states with ions from the right.

The exits occur with probabilities goj per unit time (j = 1,2,3,4), which depend on time t elapsed since the ion entered. This is a non-Markov process of the type described by (15). The equation can be solved and makes it possible to compute such physically relevant quantities as the average of the transported current, and the spectrum of fluctuations [20].

References

[1] N.G. van Kampen, Stochastic Processes in Physics and Chemistry (North-Holland, Amsterdam 1981, 1992).

[2] C.W. Gardiner, Handbook of Stochastic Methods (Springer, Berlin 1983).

[3] F. Zernike, in Handbuch der Physik III (Geiger and Scheel eds., Springer, Berlin 1926).

[4] M. Doi and S.F. Edwards, The Theory of Polymer Dynamics (Oxford Univ. Press 1986).

[5] For a counterexample see ref. 1, ch IV, eq. (2.9).

[6] W. Feller, An Introduction to Probability Theory and its Applications I (2nd ed., Wiley, New York 1966) p. 323.

[7] H.A. Kramers, Physica 7, 284 (1940).

[8] Z. Schuss and B. Matkowski, SIAM J. Appl. Math. 35, 604 (1970).

[9] P. Hänggi, P. Talkner, M. Berkovich, Rev. Mod. Phys. 62, 251 (1990).

[10] E.g. S. Faetti and P. Grigolini, Phys. Rev. A 36, 441 (1987); M.J. Dykman, Phys. Rev. A 42, 2020 (1990).

[11] P. Hänggi and P. Riseborough, Phys. Rev. A 27, 3379 (1983); C. Van den Broeck and P. Hänggi, Phys. Rev. A 30, 2730 (1984). For an application to lasers see S. Zhu, A.W. Yu, and R. Roy, Phys.. Rev. A 34, 4333 (1986).

[12] N.G. van Kampen, Physica 74, 215 and 239 (1974); Physics Reports 24, 171 (1976); J. Stat. Phys. 54, 1289 (1989); R.H. Terwiel, Physica 74, 248 (1974).

[13] R. Kubo, in Stochastic Processes in Chemical Physics (K.L. Shuler ed., Interscience, New York 1969).

[14] E.g. A.J. Bray and A.J. McKane, Phys. Rev. Letters 62, 493 (1989); D.T. Gillespie, Am. J. Phys. 64, 1246 (1996).

[15] Much literature is given in A. Fulinski, Phys. Rev. E 50, 2668 (1994).

[16] R.F. Pawula, J.M. Porrà, and J. Masoliver, Phys. Rev. E 47, 189 (1993); J.M. Pontà, J. Masoliver, and K. Lindenberg, Phys. Rev. E 48, 951 (1993).

[17] G.H. Weiss, J. Stat. Phys. 8, 221 (1973); K.J. Lindenberg and R. Cukier, J. Chem. Phys. 67, 568 (1977).

[18] N.G. van Kampen, Physica A 96, 435 (1979).

[19] E. Barkai, R.S. Eisenberg, and Z. Schuss, Phys. Rev. E 54, 1 (1996).

[20] N.G. van Kampen, Physica A 244, 414 (1997).

  • [1] N.G. van Kampen, Stochastic Processes in Physics and Chemistry (North-Holland, Amsterdam 1981, 1992).
  • [2] C.W. Gardiner, Handbook of Stochastic Methods (Springer, Berlin 1983).
  • [3] F. Zernike, in Handbuch der Physik III (Geiger and Scheel eds., Springer, Berlin 1926).
  • [4] M. Doi and S.F. Edwards, The Theory of Polymer Dynamics (Oxford Univ. Press 1986).
  • [6] W. Feller, An Introduction to Probability Theory and its Applications I (2nd ed., Wiley, New York 1966) p. 323.
  • [7] H.A. Kramers, Physica 7, 284 (1940).
  • [8] Z. Schuss and B. Matkowski, SIAM J. Appl. Math. 35, 604 (1970).
  • [9] P. Hänggi, P. Talkner, M. Berkovich, Rev. Mod. Phys. 62, 251 (1990).
  • [10] E.g. S. Faetti and P. Grigolini, Phys. Rev. A 36, 441 (1987);
  • M.J. Dykman, Phys. Rev. A 42, 2020 (1990).
  • [11] P. Hänggi and P. Riseborough, Phys. Rev. A 27, 3379 (1983);
  • C. Van den Broeck and P. Hänggi, Phys. Rev. A 30, 2730 (1984). For an application to lasers see S. Zhu, A.W. Yu, and R. Roy, Phys.. Rev. A 34, 4333 (1986).
  • [12] N.G. van Kampen, Physica 74, 215 and 239 (1974);
  • Physics Reports 24, 171 (1976); J. Stat. Phys. 54, 1289 (1989);
  • [13] R. Kubo, in Stochastic Processes in Chemical Physics (K.L. Shuler ed., Interscience, New York 1969).
  • [14] E.g. A.J. Bray and A.J. McKane, Phys. Rev. Letters 62, 493 (1989);
  • [15] Much literature is given in A. Fulinski, Phys. Rev. E 50, 2668 (1994).
  • [16] R.F. Pawula, J.M. Porrŕ, and J. Masoliver, Phys. Rev. E 47, 189 (1993);
  • [17] G.H. Weiss, J. Stat. Phys. 8, 221 (1973);
  • [18] N.G. van Kampen, Physica A 96, 435 (1979).
  • [19] E. Barkai, R.S. Eisenberg, and Z. Schuss, Phys. Rev. E 54, 1 (1996).
  • [20] N.G. van Kampen, Physica A 244, 414 (1997).

  • *
    This text corresponds to an invited talk at the "Workshop on the Foundations of Statistical Mechanics and Thermodynamics'' held in Natal, Brazil, in October, 1997.

Publication Dates

  • Publication in this collection
    03 May 1999
  • Date of issue
    June 1998

History

  • Received
    09 Mar 1998
Sociedade Brasileira de Física Caixa Postal 66328, 05315-970 São Paulo SP - Brazil, Tel.: +55 11 3091-6922, Fax: (55 11) 3816-2063 - São Paulo - SP - Brazil
E-mail: sbfisica@sbfisica.org.br