Generalized approximating splitting iterations for solving linear least squares problems in finite dimensions

Generalized approximating splitting iterations for solving linear least squares problems in finite dimensions

Felipe Marti-Lopez

Neurophysics Department, Cuban Neuroscience Center, P.O. Box 6412, Havana City, Cuba, E-mail: [e-mail]

ABSTRACT

In this paper, a new special class of splitting iterations for solving linear least squares problems in finite dimensions is defined and their main properties of strong global convergence to any problem solution are derived. The investigation results prove the new splitting iterations to be a generalization of the approximating splitting iterations for solving linear least squares problems in finite dimensions, suggesting their suitability for the robust approximate solution of such problems.

Mathematical subject classification: 93Exx, 93E24.

Key words: Linear stationary splitting iterations, linear least squares problems in finite dimensions, approximating splitting iterations, strong global convergence of iterations, hierarchical mathematical programming.

1 Introduction

Linear stationary splitting iterations [4, 7, 13] for solving linear least squares problems in finite dimensions [3, 6] are commonly used as a sort of sequential calculation engine for approximating solutions of linear equation systems [2, 7, 12, 13] long since. For that reason, they are, maybe, the most emblematic special class of the well-known successive approximations iterations for solving linear equation systems [2, 13] today.

Consider the general linear least squares problem in finite dimensions

where m Î , n Î , A Î ^m×n, and b Î ^m.

Thus, the splitting iterations for solving the problem (1) are properly those for solving the associate normal equation system [3, 6],

whose formula looks, after being derived in the usual way from (2), through Q Î ^n×n with rank(Q) = n [4, 7, 13],

In accordance with [4], important strong¹ global² convergence properties are guaranteed for those splitting iterations (3), whose splitting matrix Q makes the matrix Q + Q¢ - A¢A to be positive definite; that is to say, "Dx Î ⁿ, Dx ¹ 0,

In this concern, an interesting special sub-class of splitting iterations for solving the problem (1) was recently investigated. Those splitting iterations, named approximating splitting iterations [9, 10], were formulated with a splitting matrix

where the matrix V, V Î ^n×n, is positive definite, what obviously makes (4) to hold, providing various remarkable convergence properties for such iterations.

In efect, in accordance with [9, 10], every approximating splitting iteration sequence {x^[k]}_k=0,1,2,¼Ì ⁿ satisfies

and

" A Î ^m×n, "b Î ^m and "x^[0]Î ⁿ.

So, the approximating splitting iteration convergence to a solution is not merely strong, but even global and guarantees the objective function descent along the generated sequence and the monotonous sequential approximation to its limit.

Besides, every approximating splitting iterate continuously depends on the singular values of the problem matrix at each finite step, as a function of that matrix [10].

Now, it must be clear enough why approximating splitting iterations were effectively introduced in the context of the investigation of robust approximations of linear least squares solutions in finite dimensions³ and which peculiar role the positive definiteness of the matrix V plays there, concerning the robustness of those approximate solutions [8, 10-12].

In consequence, the attention here is mainly focused on the positiveness of the matrix V and on the structure of the splitting iteration formulas [4, 7, 9, 13] to strengthen further the abovementioned convergence results, already achieved for the approximating splitting iterations.

2 Convergence properties of certain approximating-like splitting iterations for solving linear least squares problems in finite dimensions

Let V be now a positive semi-definite matrix [6], V ¹ 0, such that V + A¢A is positive definite; that is to say, "Dx Î ⁿ, Dx ¹ 0,

and

Obviously, one may properly re-write the main approximating splitting iteration formulas [9, 10] with such a matrix V (9)-(10) instead, because, in accordance with [6],

So, one has the fixed-point formula

of a broader class of splitting iterations (3) for solving the problem (1) with the new splitting matrix Q (5) and, consequently, the one-step and the (k + 1)-step calculation formulas respectively

and

where A⁺ denotes the Moore-Penrose pseudo-inverse matrix of A [7, 3, 6].

2.1 Sequential descent of the objective function

It is not too hard to prove that the approximating-like splitting iterations (12)-(14) for solving the problem (1) make the objective function to descend along their generated sequence, as the approximating splitting iterations do.

Indeed, expand the objective function ||Ax - b|| in a Taylor series around the k-th iterate

evaluate that expansion at x^[k+1]¹ x^[k], replacing there x^[k+1] with the right-hand side of (13).

Hence, making some equivalent transformations, one has that

Therefore, whereas V is a positive semi-definite matrix (9), one has that

2.2 Strong global convergence to a solution

At this point, consider, before anything else, the following theorem and its proof.

Theorem [Theorem of the invariance]. Let m Î , n Î , A Î ^m×n, b Î ^m and Î ⁿ. If V is a matrix V Î ^n×n, such that and "Dx Î ⁿ, Dx ¹ 0,

Dx¢V Dx > 0

and

Dx¢(V + A¢A) Dx > 0 ,

then, "a Î ,

Proof. Recall that [3, 6], "A Î ^m×n and "b Î ^m,

if and only if

A¢Ax = A¢b,

due to the convexity of the problem

and to the analytical properties of the class of functions involved there.

So, "a Î , the respective feasible solution set of the goal-attainment bi-level linear least squares problem [1, 14]

and of the corresponding constrained convex goal-attainment linear-quadratic programming problem [6]

coincide; that is to say,

If V is a matrix V Î ^n×n, such that "a Î and "Dx Î ⁿ, Dx ¹ 0,

Dx¢V Dx > 0 and Dx¢(V + A¢A) Dx > 0,

then, on the one hand, "a Î and "Dx Î ⁿ, Dx ¹ 0,

Dx¢(V + a²A¢A) Dx > 0 ;

and, on the other hand, "Dx Î ⁿ, Dx ¹ 0, such that A¢ADx = 0,

Dx¢V Dx > 0 .

Hence "a Î and "Dx Î ⁿ, Dx ¹ 0, such that A¢ADx = 0,

Dx¢(V + a²A¢A) Dx > Dx¢V Dx > 0 .

Therefore, the sufficient condition for a minimum of the convex problem [5]

holds for any a; that is to say, "a Î ,

So, one can determine

for any a just by finding the saddle point of the associate Lagrange functional [5]

(x, l) = (x - )¢(V + a²A¢A) (x - ) + l¢A¢(Ax - b) .

Hence,

Ñ_x(x, l) = 2 (V + a²A¢A) (x - ) + A¢Al = 0

and

D_l(x, l) = A¢Ax - A¢b = 0 .

Whereas the matrix V + A¢A is positive definite, after some equivalent transformations of the two latter equations, "a Î , one has the following linear equation system

where

whose minimal-norm solution [7, 3, 6] leads to the one to be found for any a; that is to say,

and

Therefore, "a Î ,

Now, assume the Singular Value Decomposition (SVD) [3, 6] of the matrix to yield the factorization

Recall that, since V satisfies (9)-(10),

here because, in accordance with [6], < 1.

So, one has that

and, as a consequence,

and

After the replacement of the right-hand sides of equalities (19)-(21) whom it corresponds in both the approximating-like splitting iteration formulas (13) and (14) and after making some equivalent transformations there, one has that

and

Here, notice that, since (18) holds, one has that

So,

Hence, after making some equivalent transformations of the right-hand side of (25) and introducing W,

where it corresponds there, one has that

Therefore, in accordance with the theorem of the invariance,

what proves the extension of the homologous strong global convergence property (7) of the approximating splitting iterations to a broader class of approximating-like splitting iterations (12)-(14), whose formulas are with positive semi-definite matrices V, such that (9)-(10) hold.

2.3 Monotonous successive approximation of any solution

Notice that, the approximating-like splitting iterations (12)-(14) with V (9)-(10) also monotonously approximate any solution of the problem (1) along their generated sequence, as the approximating splitting iterations do.

Indeed, whereas (12) holds for every solution x*,

So, after subtracting (29) from (13),

and replacing in (30) whom it corresponds with the right-hand sides of equalities (20)-(21), one has that

Hence,

Therefore, since (11) and (18) hold, $P = (R¢()) Î ^n×n with rank(P) = n, such that x^[k]¹ x*,

2.3.1 Continuous dependence of every finite-step iterate on the singular values

It is not difficult to prove that every approximating-like splitting iterate (22)-(23) continuously depends on the singular values of the problem matrix at each finite step.

Indeed, "t Î

⁺, t < +¥,

Hence, from (23), one has that

Therefore,

where p Î |p < rank(A).

3 Convergence properties of further approximating-like splitting iterations

Let g Î and q Î be such that

and

Besides, let V be a matrix, that satisfies (9)-(10). Hence, in accordance with [6], V + g²A¢A is a positive definite matrix and, consequently,

for any g, that satisfies (37)-(38).

So, the problem matrix A¢A can be properly split [4, 7, 13] this way:

and one may thus write the fixed-point formula

of a yet broader class of splitting iterations (3) for solving the problem (1), with the splitting matrix

Of course, the one- and the (k + 1)-step calculation formulas of such approximating-like splitting iterations are respectively

and

Since identities

and

hold, then after replacing the right-hand sides of equalities (44) and (45) in both the formulas (42) and (43), and after making there the equivalent transformations

and

the approximating-like splitting iterations (42) and (43) look now as follows

and

where, since the matrix V satisfies (9) and (10), the matrix [] (48) also satisfies them for any g and q, that satisfy (37)-(38).

Hence, the generated sequence {x^[k]}_k=0,1,2,¼Ì ⁿ of any approximating-like splitting iteration (42)-(43) satisfies the convergence properties (16), (28) and (33); specially the one of the global convergence limit; that is to say,

Therefore, following the theorem of the invariance, one has that

Now, the definition of generalized approximating splitting iterations for solving linear least squares problems in finite dimensions may be properly stated, as follows.

Definition. Let m Î and n Î ; and let A Î ^m×n and b Î ^m. Besides, let g Î and q Î be such that

g² + q²¹ 0 and g²> q² .

A generalized approximating splitting iteration for solving the linear least squares problem in finite dimensions is a linear stationary splitting iteration

for solving the problem

whose splitting matrix is

where V is a matrix V Î ^n×n, such that "Dx Î ⁿ, Dx ¹ 0,

Dx¢V Dx > 0

and

Dx¢(V + A¢A)Dx> 0 .

4 Conclusions

Here, a new special class of splitting iterations for solving linear least squares problems in finite dimensions, named generalized approximating splitting iterations, has been defined and their main properties of strong global convergence to any problem solution has been properly derived.

The investigation results proved:

1. The objective function descends along the generated iteration sequence.

2. Every iteration sequence strongly and globally converges to a solution.

3. Every iteration sequence monotonously approximates its convergence limit.

4. Each finite-step iterate continuously depends on the singular values of the problem matrix.

The generalized approximating splitting iterations are certainly a generalization of the approximating splitting iterations [9], [10], not merely because the latter ones are a special case of the former ones (when both the coefficients g and q are equal to the unit and the matrix V is positive definite), but specially because the investigation results proved the generalized approximating splitting iterations to extend the favorable convergence features of those to a quite broader variety of approximating-like splitting iteration formulas (like, for example, the one of the iterated Tikhonov regularization with a non-necessarily full-rank smoothing matrix [7], [12], [9], [10]), what substantiates their suitability for the robust approximation of linear least squares solutions in finite dimensions [10] and hints their possible use in solving multi-level problems [1], [14].

Finally, it should be noticed that the requirements to be fulfilled by the matrix V of generalized approximating splitting iterations are just equivalent to the well-known sufficient conditions for a minimum of any constrained convex goal-attainment linear-quadratic programming problem [5].

Acknowledgement. The author sincerely wishes to thank the anonymous reviewer for his patience and time.

Received: 25/III/04. Accepted: 02/VI/04

#604/04.

• [1] G. Anandalingam and T. Friesz,''Hierarchical optimization: an introduction'', Annals of Operations Research, 34 (1992), pp. 1-11.
• [2] I.S. Berezin and N.P. Zhidkov, Computing Methods (volume 2). MIR Publishers, Moscow, (1965).
• [3] Ĺ. Bjorck, Numerical Methods for Least Squares Problems, SIAM Publishers, New York, (1996).
• [4] A. Dax, ''The convergence of linear stationary iterative processes for solving singular unstructured systems of linear equations'', SIAM Reviews, 32 (4) (1990), pp. 611-635.
• [5] R. Fletcher, Practical Methods of Optimization (second edition). John Wiley and Sons, New York, (1987).
• [6] G.H. Golub and C.F. Van Loan, Matrix Computations (second edition). John Hopkins Pub. Co., Baltimore, (1989).
• [7] M. Hanke and M. Neumann, ''Preconditionings and splittings for rectangular systems'', Numerische Mathematik, 57 (1990), pp. 85-95.
• [8] T. Koenig, F. Marti-Lopez and P. Valdes-Sosa, ''Topographic time-frequency decomposition of spontaneous EEG'', NeuroImage, 1 (14) (2001), pp. 117-134.
• [9] F. Marti-Lopez, ''Global convergence properties of the splitting iterations for solving linear least squares problems'', Investigación Operativa (ALIO), 7 (3) (1999), pp. 133-142.
• [10] F. Marti-Lopez, Splitting Iterations for the Robust Approximation of Linear Least Squares Solutions in Finite Dimensions. (in Spanish), PhD Thesis. Cuban Neuroscience Center, Havana City, Cuba, 2003. http://es.geocities.com/fmartilopez
• [11] F. Marti-Lopez and T. Koenig, ''Approximating method of frames'', Digital Signal Processing (DSP), 13 (2003), pp. 519-529.
• [12] A. Neumaier, ''Solving ill-conditioned and singular linear systems: a tutorial on regularization'', SIAM Reviews, 40 (3) (1998), pp. 636-666.
• [13] J.R. Stoer and R. Bulirsch, Introduction to Numerical Analysis (third edition). Springer, New York, Berlín, Heidelberg, (2002).
• [14] L.N. Vicente and P.H. Calamai, ''Bilevel and multilevel programming: a bibliography review'', Journal of Global Optimization, 5 (1994), pp. 291-306.

1

The iteration converges for any matrix A (with arbitrary rank, dimensions, etc.) and any vector b.

2

The iteration converges from any initial value

x

^[0]

3

The finite-dimension least squares approximation continuously depends on the singular values of the problem matrix.

Publication Dates

• Publication in this collection
  07 Nov 2005
• Date of issue
  Aug 2005

History

• Accepted
  02 June 2004
• Received
  25 Mar 2004