A CONSTRUCTIVE GLOBAL CONVERGENCE OF THE MIXED BARRIER-PENALTY METHOD FOR MATHEMATICAL OPTIMIZATION PROBLEMS

. In this paper we develop a generic mixed bi-parametric barrier-penalty method based upon barrier and penalty generic algorithms for constrained nonlinear programming problems. When the feasible set is deﬁned by equality and inequality functional constraints, it is possible to provide an explicit barrier and penalty functions. If such case, the continuity and differentiable properties of the restrictions and objective functions could be inherited to the penalized function. The main contribution of this work is a constructive proof for the global convergence of the sequence generated by the proposed mixed method. The proof uses separately the main results of global convergence of barrier and penalty methods. Finally, for some simple nonlinear problem, we deduce explicitly the mixed barrier–penalty function and illustrate all functions deﬁned in this work. Also we implement MATLAB code for generate iterative points for the mixed method.


INTRODUCTION
The mathematical optimization is one of the concepts widely used to analyze many complex decision or allocation problems. In order to better use available resources, optimization techniques allow the selection of values for a certain number of interrelated variables, and with them we could measure the performance and quality of a decision by focusing on some objective functions.
Specifically, a mathematical optimization problem consists of minimizing or maximizing an objective function f (x) subject to restrictions x ∈ Ω, where f is a real valued continuous function defined on Ω ⊂ R n . In this work, we consider the feasible set Ω having three types of restrictions where Ω 1 can be whatever restriction set that is difficult to handle, Ω 2 is a robust set and Ω 3 could be a simple set such as signal or boundary restrictions. The robust set means that it has a dense nonempty interior subset. In other words, the set has an interior, and it is possible to get any boundary point by approaching it from a sequence of interior points, Luenberger & Ye (2008).
According to specifications above, we consider the following optimization problem, where the restriction sets are given by Ω 1 = {x ∈ R n : h(x) = 0}, Ω 2 = {x ∈ R n : g(x) ≤ 0} and For many decades, many authors proved some theoretical results and proposed several algorithms in order to solve nonlinear optimization problems considering penalty or barrier function methods. Luenberger & Ye (2008), Fiacco & McCormick (1990) state convergence for both methods, Polyak Polyak (1971) showed convergence rate for penalty function method within Hilbert space, Bertsekas (1976) obtained convergence and rate of convergence results for the sequences of primal and dual variables generated on penalty and Lagrange multiplier methods, he showed that the multiplier method is faster than the pure penalty method. Fiacco & McCormick (1990) demonstrate by contradiction the global convergence for mixed penalty-barrier method, also Breitfeld & Shanno (1995) proposed composite algorithm of augmented Lagrangian, modified log-barrier, and classical log-barrier methods for that they demonstrated global convergence to a first-order stationary point for the constrained problem which was based on .
In this work, we develop the mixed barrier-penalty method for solving a general nonlinear problem (2); and we provide a generic bi-parametric algorithm. The main contribution is a constructive proof of global convergence of sequence generated by that mixed method as an alternative proof to existing ones with slightly different assumptions. Suñagua & Oliveira (2017) showed that computational experiments for NETLIB problems work successfully for large scale linear optimization problems.
where f is a continuous function and Ω is a robust restriction set. This kind of set often arises from the inequality constraints, that is, Ω = {x ∈ R n : g(x) ≤ 0}, for which there is a point x ∈ Ω such that g(x) < 0.
Barrier methods work by establishing a barrier on the boundary of the restriction set that prevents a search procedure from leaving the feasible region. A barrier function is a function B(·) defined on the interior set x approaches the boundary of Ω. For inequality constraints g i (x) ≤ 0, i = 1, 2, · · · , p in many practical applications, the barrier functions commonly used are the logarithmic or inverse barrier function. They are defined on Int(Ω) respectively by Now, the problem (4) can be transformed into a penalized subproblem where µ > 0 is called barrier parameter and we take µ small (going to zero). In this approach, the main assumption is that the original problem (4) has a global solution x * . Let x(µ) be a global solution of subproblem (5). When µ k → 0, we hope x(µ k ) converges to x * .
Proof. Since {µ k } is a monotone decreasing sequence, x k+1 is a global minimizer of (6) and recalling (ii) of barrier condition, B is non-negative function, then For establishes the second inequality, we also have now, using (8) and (7), we get eliminating the common factor µ k − µ k−1 < 0, we prove the item 2.
Finally, by previous inequality, The global convergence of the barrier method, in the sense that any limit point of the sequence is a solution of problem (4), can be verified from the previous Lemma.
Theorem 2.1. Let {x k } be a sequence generated by Algorithm 1, in which µ k → 0. Then, any limit point of the sequence is a global minimizer of problem (4).
Proof. Let f k = min{φ (x, µ k ) : x ∈ Int(Ω)} be global minimum value of φ (·, µ k ) on Int(Ω), whose solution is x k+1 . By Lemma 2.1, f k ≥ f k+1 for all k. If f * = min{ f (x) : x ∈ Ω}, then First of all, we prove only the sequence { f k } converges to f * , next we continue with the demonstration of convergence of some subsequence that will converge to some global minimizer.
Indeed, { f k } is a bounded below monotone decreasing sequence, hence it converges to its infimum, we sayf . Iff = f * , thenf > f * . Recalling x * the global minimizer of (4) and since f is a continuous function, then there is an open ball B centered at x * such that, for all x ∈ B ∩ Int(Ω) we have, Since B(x) ≥ 0 for all x ∈ Int(Ω), and 0 < µ k+1 < µ k , we have 0 < µ k+1 B(x) < µ k B(x), ∀x ∈ Int(Ω). Therefore, lim Thus, for any x ∈ B ∩ Int(Ω), and k large enough, we get Then, from (9) and (11), we have which contradicts to f k →f . Therefore,f = f * . That is Now, letx ∈ Ω be any subsequential limit of {x k }, more precisely, there is a subsequence {x k l } such that x k l →x.
. Thus, every limit point generated by Algorithm 1 is a global solution of the problem (4).

PENALTY METHODS OVERVIEW
Given f : R n → R a continuous function, we consider the problem where Ω 1 and Ω 2 are any arbitrary subsets of R n . In most applications Ω 1 is defined implicitly by functional restrictions as h(x) = 0, where h : R n → R m . In some cases, we assume that f and h are twice differentiable functions. A basic assumption is that problem (GP) admits global minimizer, some theoretical results were established by Polyak (1971), Breitfeld & Shanno (1995), Nash (2010), and Luenberger & Ye (2008).
Given a restriction set Ω 1 , a penalty function is defined as a function P : In order to solve the problem (13), the penalty function method solves the following penalized subproblem where ρ > 0 is a constant called penalty parameter. For ρ large, is clear that a solution of (14) will be in a region where P is small. Thus, when ρ → ∞ is expected that the corresponding optimal points will approach the feasible set Ω 1 .
For C 2 class functions, h : R n → R m and g : R n → R p , some useful penalty functions P based on the type of restrictions h(x) = 0 or g(x) ≤ 0 may be in the first item the quadratic penalty function preserves C 2 property, but in the last three items P is only C 1 .
1. Test the optimality of x k for (13) and stop if it's satisfied.
In general, one of the suggestions to compute ρ k is taking ρ 0 = 1 and ρ k+1 = 10ρ k , Fletcher (2013). However, when Ω 1 is the set of equality constraints h(x) = 0, a basic rule that works in , then ρ k+1 = 10ρ k , otherwise ρ does not change. That approach was successfully tested for linear programming problems, Suñagua & Oliveira (2017).
The following Lemma gives a set of inequalities that follow directly from Algorithm 2 steps. A proof is based in Martınez & Santos (1995) and Luenberger & Ye (2008).
Lemma 3.1. Let {x k } be a sequence generated by Algorithm 2, which x k+1 is global solution of problem (Q k ). Then Proof. Since {ρ k } is a monotone increasing sequence and x k is a global minimizer of subproblem (15), then To establish the second inequality, recalling the optimalities of x k and x k+1 , we have using (17) and (16), we get as ρ k−1 < ρ k , then P(x k ) ≥ P(x k+1 ). Finally, using this inequality Consequently, x k ∈ Ω 1 , if and only if, x k is the global solution of (GP).

A CONSTRUCTIVE GLOBAL CONVERGENCE OF THE MIXED BARRIER-PENALTY METHOD
Proof. Since ρ k > 0 and P(x) ≥ 0 ∀x ∈ R n and x k is the global minimizer of (Q k−1 ), then The global convergence of the penalty method, in the sense any limit point of the sequence is a solution, can be verified from the two previous Lemmas.
Theorem 3.1 (Global convergence for penalty method). Let {x k } be a sequence of global minimizers of (Q k ) generated by Algorithm 2 in which ρ k → +∞. Then, any limit point of the sequence is a global minimizer of problem (13).
Proof. With a slight change of notation, the proof is based on Let f * be an optimal value of problem (GP). By Lemma 3.1 and Lemma 3.2, the sequence {ψ(x k , ρ k−1 )} is nondecreasing and bounded above by f * , then Thus, using (18) and (19), yields Since P(x k l ) ≥ 0 and ρ k l → ∞, we conclude that lim l→∞ P(x k l ) = 0. Using the continuity of P, P(x) = 0, therebyx ∈ Ω 1 . To prove the optimality ofx, just note that by Lemma 3.
which completes the proof, because obviously f * ≤ f (x), and then f (x) = f * .

4 MIXED BARRIER-PENALTY METHOD
For continuous function f : R n → R, we consider the general programming problem where Ω 1 , Ω 2 and Ω 3 are restriction sets that are defined in (1).
As in the previous sections, we assume that problem (22) admits a global minimizer. Now, let P a penalty function related to Ω 1 and B a barrier function related to Ω 2 . Then, taking the penalty parameter ρ > 0 and the barrier parameter µ > 0, we have the associate mixed barrier-penalty subproblem, Since the general problem (NLP) admits global minimizer, then the problem (BP ρ,µ ) in (23) also admits a global solution for any feasible parameter values. Therefore, we define In order to solve the general problem (22), we provide a generic algorithm given in Algorithm 3, that works iteratively updating ρ and µ parameters before solving the penalized subproblem (23).
To establish the global convergence of the Algorithm 3, firstly we can associate the additive terms in two convenient ways Therefore, fixing respectively ρ and µ, we define F ρ (x) = f (x) + ρP(x) and G µ (x) = f (x) + µB(x), then we associate to (NLP) the following two problems Since the problem (NLP) admits a global minimizer, both (GP ρ ) and (GP µ ) in (27) also admit global minimizers. Therefore, defining We have respectively the barrier and penalty subproblems By fixing one of the parameters according to (27), the two problems in (28) are equivalent to (BP ρ,µ ). In fact therefore, we can apply the results obtained in the preceding two sections.
In order to understand more clearly the ideas of the mixed problem, we consider the following particular quadratic problem According to the contours of the objective function and graph of restrictions in Figure 1, the optimal point is x * = (1, 2). First, if we consider the Lagrangian function L (x 1 , whose unique solutions for variables and Lagrangian parameters are x * 1 = 1, x * 2 = 2, u * 1 = 2, u * 2 = 0, λ * = −4. Now, we associate to (30) the mixed barrier-penalty subproblem where the penalized objective function is where M is a large enough positive number such that x 1 > 1, x 2 > −1 and (x 1 − 1)(x 2 + 1) < M, surely this region lies within inequality constraints 1 − x 1 < 0 and −1 − x 2 < 0. This condition ensures that the barrier function is non-negative in the region that contains optimal point.
It is easy to see that Φ is a smooth function, thereby from first-order necessary conditions for optimal points, we have Solving this nonlinear system, subject to x 1 > 1 and x 2 > −1, by the substitution method, we obtain Thus, for each optimal point in (33), the optimal values of problem (QP ρ,µ ) in (31) is θ (ρ, µ) = Φ(x 1 , x 2 , ρ, µ), whose graph is shown in Figure 2 with M = 2. We can see that for fixed µ, θ (ρ, µ) is an increasing function and for fixed ρ, θ (ρ, µ) is a decreasing function. This fact will be showed theoretically in Theorem 4.1.
Furthermore, using (33), when µ → 0 and ρ → ∞, the following gradient's coefficients in (32) converge to optimal Lagrangian parameters where A = 4µ(2 + ρ) + (2 + 3ρ) 2 . In addition, the Hessian matrix for Φ is Then ∇ 2 Φ is a positive definite matrix, that guarantees the minimality of x 1 and x 2 in (33). Moreover, the approximate condition number of this matrix is hence ∇ 2 Φ is ill-conditioned for very small µ and small ρ, however for large ρ, that condition number could be reduced.
Next, we have the global convergence theorem for the mixed barrier-penalty algorithm.
Theorem 4.1 (Global convergence for mixed method). Let {x k } be a sequence of global minimizers of (BP k ) problem in (25) generated by mixed Algorithm 3 in which ρ k → +∞ and µ k → 0.
Then any limit point of sequence is a global minimizer of the (NLP) problem.
Proof. In order to apply the results of the preceding sections, the idea is to fix, one of the parameters in the (BP ρ,µ ) subproblem in (23) one at a time, and apply the corresponding results for each subproblems in (27).
Firstly, to fix ρ, let {x ρ k } be the sequence generated by Algorithm 1 for solving the (GP ρ ) subproblem in (27). By applying Lemma 2.1, we get By the monotonicity in (34) and by (12) the sequence {φ ρ (x ρ k , µ k−1 )} converges to global optimal value of the problem (GP ρ ) in (27), that is, Similarly, to fix µ, let {x µ k } be the sequence generated by the associated Algorithm 2 for solving the (GP µ ) subproblem in (27). By applying Lemma 3.1, we get By the monotonicity in (36) and by (21), the sequence {ψ µ (x µ k , ρ k−1 )} converges to global optimal value of the (GP µ ) problem of (27), that is, In addition, from Theorem 3.1 all convergent subsequence of {x µ k } converges to a global minimizer of the problem (GP µ ) in (27). And by Lemma 3.2, we get Now let {x k } be a sequence of minimizers obtained by Algorithm 3 for a mixed problem. More precisely, let x k+1 = x(ρ k , µ k ) which also minimizes (28), because according to (29), we have Since (35) and (37), we have Giving x(ρ k , µ k ) the solution of (BP ρ,µ ) for µ = µ k and ρ = ρ k . For µ k < µ k−1 , additionally we solve (BP ρ,µ ) for µ = µ k−1 and ρ = ρ k , which solution is called x(ρ k , µ k−1 ). Using (34) and by (29), for x = x(ρ k , µ k ) and y = x(ρ k , µ k−1 ), we have Using (39) and (40), we get Similarly, for ρ k > ρ k−1 , additionally we consider a solution of (BP ρ,µ ) for ρ = ρ k−1 and µ = µ k , which solution is called x(ρ k−1 , µ k ). Using (36) and by (29) for x = x(ρ k , µ k ) and z = x(ρ k−1 , µ k ), we have Using (41) and (42), we get Let x * be a global minimizer of (NLP). Recalling x µ k * a solution of the problem (GP µ k ) in (27), with the additional assumption x } is a monotone nonincreasing sequence that is bounded below by f (x * ) and using (12) this sequence converges to its infimum f (x * ). Also, {F ρ k (x ρ k * )} is monotone nondecreasing sequence that is bounded above by f (x * ) and by (21) that sequence converges to its supremum f (x * ), that is, By applying squeeze theorem 1 to (38) and (43), we show Let {x k l } be any subsequence of {x k } such that x k l →x. By the continuity of f , we get f (x k l ) → f (x). The final demonstration will be done by contradiction under the assumptionx = x * with f (x) > f (x * ). Using (10) for the problem (GP ρ ), we have µ k l −1 B(x) → 0, for any x ∈ Int(Ω 2 ). Furthermore, using (20) for the problem (GP µ ), also we have ρ k l −1 P(x k l ) → 0, and by continuity of f , the sequence { f (x k l ) − f (x * ) + ρ k l P(x k l ) + µ k l B(x k l )} cannot converge to zero, which contradicts (44).

Barrier-Penalty applied to convex problem
The Algorithm 4 is an algorithm based on generic Algorithm 3 in order to solve the nonlinear problem (30).
For ρ 0 = 1, µ 0 = 1 and x 0 = (1.5, 1) we write a MATLAB script for Algorithm 4 in order to compute a sequence of optimal points that approach to x * = (1, 2). The iterative results is shown in Table 1.
The path following points is shown in Figure 3, where last points are close to x * . The exact results also solved by MATLAB are x * = (1.000000, 2.000000) and f (x * ) = 5.000000.
where A is m × n matrix, c, x ∈ R n , b ∈ R m , and E is formed by rows of n × n identity matrix corresponding to bounded variables, thereby Ex is the vector of bounded variables for which u is the vector of upper bounds. In this case is usual to add the slack variable v such that Ex where v ≥ 0. In the most computational packages that implement Interior Point Methods for solving linear programming problems only barrier parameter is considered.
In order to solve the LP problem (46), by using the quadratic penalty and logarithmic barrier functions, the objective function is penalized as follow where µ and ρ are respectively the barrier and penalty parameters and n b is the number of bounded variables. Then the associated mixed barrier-penalty subproblem is Since Φ(x, v, ρ, µ) is a smooth function on open set (x, v) > 0. By applying the first-order necessary condition, we have Defining y = ρ(b − Ax), z = µX −1 e, w = µV −1 e, we get .
Therefore, the optimality conditions for subproblem (LPP ρ,µ ) on (x, v) > 0 and (z, w) > 0 are In Interior Point Methods reviewed on Suñagua & Oliveira (2017), we find a search direction by applying Newton's Method for solving nonlinear system (49). In fact, the Newton's directions satisfy solving this block linear equations, we find up replacing this on third group of equations using (52) and first group of equations of (50), we get the normal equations (ADA T + δ I)dy = AD(r d − X −1 r c + E T V −1 r s − E T V −1 W r u ) close to optimal point, D matrix is very bad scaled and then ADA T is also very ill-conditioned. In this case, the penalty parameter δ improves that condition number, which is helpful for solving the symmetric positive definite system by applying conjugate gradient method for instance.
Alternatively to (52) and (53), dx and dy also obtain by solving the following augmented system where r 1 = r d − X −1 r c + E T V −1 r s − E T V −1 W r u . This system is also symmetric, indefinite and better condition number due to penalty parameter.
For computational experiments, we use the open source package PCx (Czyzyk et al., 1997), that implements the Mehrotra's Predictor-Corrector algorithm in which the barrier parameter µ is already incorporated in order to solve linear programming problems. By adding an appropriate code to PCx, we achieve to incorporate the penalty parameter δ , thus, we obtain the modified PCx called the Predictor-Corrector mixed algorithm with barrier and penalty parameters. The numerical results for several NETLIB LP problems were computed for approaches proposed in Suñagua & Oliveira (2017), where the goodness of the approaches were compared according to Dolan & More (2002) performance profile criteria.

CONCLUSIONS
Firstly, we present a brief summary of the main concepts and results upon the barrier and penalty methods, where for each method we show the global convergence theorems in order to use these strategies in the proof of the global convergence theorem for mixed algorithm.
In the Section 4, we provide a mixed algorithm for solving mixed barrier-penalty subproblem (23), and we provide a constructive proof on global convergence theorem for mixed barrierpenalty methods as an alternative showed in Fiacco & McCormick (1990) and Breitfeld & Shanno (1995). For simple convex nonlinear problems we write MATLAB code in order to generate iterative points that illustrate penalty and barrier functions.
Finally, we develop an application for nonlinear programming problems with equality and inequality functional constraints, such as, quadratic programming problem, and a standard linear programming problem. Since the functions involved have the smooth property on an open set, then the optimality conditions for each class of problems are stated, those can be solved by applying interior point methods.