A CONSTRUCTIVE GLOBAL CONVERGENCE OF THE MIXED BARRIER-PENALTY METHOD FOR MATHEMATICAL OPTIMIZATION PROBLEMS

ABSTRACT

In this paper we develop a generic mixed bi-parametric barrier-penalty method based upon barrier and penalty generic algorithms for constrained nonlinear programming problems. When the feasible set is defined by equality and inequality functional constraints, it is possible to provide an explicit barrier and penalty functions. If such case, the continuity and differentiable properties of the restrictions and objective functions could be inherited to the penalized function.

The main contribution of this work is a constructive proof for the global convergence of the sequence generated by the proposed mixed method. The proof uses separately the main results of global convergence of barrier and penalty methods. Finally, for some simple nonlinear problem, we deduce explicitly the mixed barrier-penalty function and illustrate all functions defined in this work. Also we implement MATLAB code for generate iterative points for the mixed method.

Keywords:
nonlinear programming; mixed barrier-penalty methods; convergence of mixed algorithm

1 INTRODUCTION

The mathematical optimization is one of the concepts widely used to analyze many complex decision or allocation problems. In order to better use available resources, optimization techniques allow the selection of values for a certain number of interrelated variables, and with them we could measure the performance and quality of a decision by focusing on some objective functions.

Specifically, a mathematical optimization problem consists of minimizing or maximizing an objective function f (x) subject to restrictions $x\in \mathrm{\Omega }$, where f is a real valued continuous function defined on $\Omega \subset {\mathrm{ℝ}}^n$. In this work, we consider the feasible set Ω having three types of restrictions

$x\in {\Omega }_1,x\in {\Omega }_2,x\in {\Omega }_3$ (1)

where Ω₁ can be whatever restriction set that is difficult to handle, Ω₂ is a robust set and Ω₃ could be a simple set such as signal or boundary restrictions. The robust set means that it has a dense nonempty interior subset. In other words, the set has an interior, and it is possible to get any boundary point by approaching it from a sequence of interior points, ^{Luenberger & Ye (2008}).

According to specifications above, we consider the following optimization problem,

$$\begin{gathered} \min f\mathit{}\left(x\right) \\ \mathrm{s}\mathit{.}\mathit{}\mathrm{t}\mathit{.}\mathit{}x\in {\Omega }_1,x\in {\Omega }_2,x\in {\Omega }_3. \end{gathered}$$ (2)

One of the most common nonlinear programming problems formulation is when the restrictions are characterized by equality and inequality functional constraints, ^{Bazaraa et al. (2013}), ^{Luenberger & Ye (2008}), ^{Wright & Nocedal (1999}), ^{Griva et al. (2009}). In which given the continuous functions $f\mathit{}:\mathit{}{\mathrm{ℝ}}^{\mathit{n}}\mathit{\to }\mathrm{ℝ}\mathit{,}\mathit{}h:\mathit{}{\mathrm{ℝ}}^{\mathit{n}}\mathit{\to }{\mathrm{ℝ}}^{\mathit{m}}\mathit{,}\mathit{}g:\mathit{}{\mathrm{ℝ}}^{\mathit{n}}\mathit{\to }{\mathrm{ℝ}}^{\mathit{p}}$, the classical nonlinear optimization problem is

$$\begin{gathered} \min f\left(x\right) \\ \mathrm{s}\mathit{.}\mathit{}\mathrm{t}\mathit{.}\mathit{}h\left(x\right)\mathit{=}\mathit{0} \\ \mathit{}\mathit{}\mathit{}g\left(x\right)\le 0, \end{gathered}$$ (3)

where the restriction sets are given by ${\Omega }_1=\left\{x\in {\mathrm{ℝ}}^n:h\left(x\right)=0\right\},{\Omega }_2=\left\{x\in {\mathrm{ℝ}}^n:g\left(x\right)\le 0\right\}\text{and}{\Omega }_3={ℝ}^n$.

For many decades, many authors proved some theoretical results and proposed several algorithms in order to solve nonlinear optimization problems considering penalty or barrier function methods. ^{Luenberger & Ye (2008}), ^{Fiacco & McCormick (1990}) state convergence for both methods, ^{Polyak Polyak (1971}) showed convergence rate for penalty function method within Hilbert space, ^{Bertsekas (1976}) obtained convergence and rate of convergence results for the sequences of primal and dual variables generated on penalty and Lagrange multiplier methods, he showed that the multiplier method is faster than the pure penalty method. Fiacco & McCormick (1990) demonstrate by contradiction the global convergence for mixed penalty-barrier method, also ^{Breitfeld & Shanno (1995}) proposed composite algorithm of augmented Lagrangian, modified log-barrier, and classical log-barrier methods for that they demonstrated global convergence to a first-order stationary point for the constrained problem which was based on ^{Breitfeld & Shanno (1994}).

In this work, we develop the mixed barrier-penalty method for solving a general nonlinear problem (2); and we provide a generic bi-parametric algorithm. The main contribution is a constructive proof of global convergence of sequence generated by that mixed method as an alternative proof to existing ones with slightly different assumptions. ^{Suñagua & Oliveira (2017}) showed that computational experiments for NETLIB problems work successfully for large scale linear optimization problems.

2 BARRIER METHODS OVERVIEW

Barrier methods are also called interior point or internal penalty methods. Some theoretical results of them were developed by ^{Martınez & Santos (1995}), ^{Luenberger & Ye (2008}), ^{Nash & Sofer (1993}), and ^{Wright (1992}). These methods are applicable to problems of the form

$$\begin{gathered} \min f\left(x\right) \\ s.t.x\in \Omega \end{gathered}$$ (4)

where f is a continuous function and Ω is a robust restriction set. This kind of set

often arises from the inequality constraints, that is, $\Omega =\left\{x\in {\mathrm{ℝ}}^n:g\left(x\right)\le 0\right\}$, for which there is a point $\overline{x}\in \Omega$ such that $g\left(\overline{x}\right)<0$.

Barrier methods work by establishing a barrier on the boundary of the restriction set that prevents a search procedure from leaving the feasible region. A barrier function is a function B(·) defined on the interior set $Int(\Omega )=\left\{x:g\left(x\right)<0\right\}$ of Ω such that (i) B is continuous,(ii) $B\left(x\right)\ge 0$, (iii) $B\left(x\right)\to \infty$ as x approaches the boundary of Ω. For inequality constraints $g_i\left(x\right)\le 0,i=1,2,...,p$ in many practical applications, the barrier functions commonly used are the logarithmic or inverse barrier function. They are defined on Int(Ω) respectively by

$B\left(x\right)=-\sum _{i=1}^p\log \left(-g_i\left(x\right)\right)andB\left(x\right)=-\sum _{i=1}^p\frac{1}{g_i\left(x\right)}.$

Now, the problem (4) can be transformed into a penalized subproblem

$$\begin{gathered} \left(P_{\mu }\right)\min f\left(x\right)+\mu B\left(x\right) \\ s.t.x\in \mathrm{Int}(\Omega ) \end{gathered}$$ (5)

where µ > 0 is called barrier parameter and we take µ small (going to zero). In this approach, the main assumption is that the original problem (4) has a global solution x ^∗. Let x(µ) be a global solution of subproblem (5). When ${\mu }_k\to 0$, we hope x(µ _k ) converges to x ^∗.

Given $\varphi (x,\mu )=f\left(x\right)+\mu B\left(x\right)$, we have a generic barrier algorithm given in Algorithm 1.

Algorithm 1
Barrier Algorithm

The following Lemma gives a set of inequalities that follow directly from Algorithm 1 steps. A proof is based from ^{Luenberger & Ye (2008}) and ^{Martınez & Santos (1995}).

Lemma 2.1.Let {x _k } be a sequence generated byAlgorithm 1, then

1. $\varphi (x_{k+1},{\mu }_k)\le \varphi (x_k,{\mu }_{k-1})$

2. $B\left(x_{k+1}\right)\ge B\left(x_k\right)$

3. $f\left(x_{k+1}\right)\le f\left(x_k\right)$.

Proof. Since {µ _k } is a monotone decreasing sequence, x _k+1 is a global minimizer of (6) and recalling (ii) of barrier condition, B is non-negative function, then

$$\begin{gathered} \varphi (x_{k+1},{\mu }_k)=f\left(x_{k+1}\right)+{\mu }_{k}B\left(x_{k+1}\right) \\ \le f\left(x_k\right)+{\mu }_{k}B\left(x_k\right) \\ \le f\left(x_k\right)+{\mu }_{k-1}B\left(x_k\right)=\varphi (x_k,{\mu }_{k-1}). \end{gathered}$$

For establishes the second inequality, we also have

$\varphi (x_{k+1},{\mu }_k)=f\left(x_{k+1}\right)+{\mu }_{k}B\left(x_{k+1}\right)\le f\left(x_k\right)+{\mu }_{k}B\left(x_k\right)$ (7)

$\varphi (x_k,{\mu }_{k-1})=f\left(x_k\right)+{\mu }_{k-1}B\left(x_k\right)\le f\left(x_{k+1}\right)+{\mu }_{k-1}B\left(x_{k+1}\right),$ (8)

now, using (8) and (7), we get

$({\mu }_k-{\mu }_{k-1})B\left(x_{k+1}\right)\le ({\mu }_k-{\mu }_{k-1})B\left(x_k\right)$

eliminating the common factor ${\mu }_k-{\mu }_{k-1}<0$, we prove the item 2.

Finally, by previous inequality,

$f\left(x_{k+1}\right)+{\mu }_{k}B\left(x_{k+1}\right)\le f\left(x_k\right)+{\mu }_{k}B\left(x_k\right)\le f\left(x_k\right)+{\mu }_{k}B\left(x_{k+1}\right),$

hence $f\left(x_{k+1}\right)\le f\left(x_k\right)$. □

The global convergence of the barrier method, in the sense that any limit point of the sequence is a solution of problem (4), can be verified from the previous Lemma.

Theorem 2.1.Let {x _k } be a sequence generated byAlgorithm 1, in which${\mu }_k\to 0$. Then, any limit point of the sequence is a global minimizer of problem (4).

Proof. Let $f_k=\min \left\{\varphi (x,{\mu }_k):x\in \mathrm{Int}(\Omega )\right\}$ be global minimum value of $\mathrm{\varphi }(·,{\mu }_{\mathrm{k}})$ on Int(Ω), whose solution is x _k+1 . By Lemma 2.1, $f_k\ge f_{k+1}$ for all k. If $f^{\ast }=\min \left\{f\left(x\right):x\in \Omega \right\}$, then

$f_0\ge f_1\ge ···\ge f_k\ge f_{k+1}\ge ···\ge f^{\ast }.$

First of all, we prove only the sequence {f _k } converges to f ^∗, next we continue with the demonstration of convergence of some subsequence that will converge to some global minimizer.

Indeed, {f _k } is a bounded below monotone decreasing sequence, hence it converges to its infimum, we say $\overline{f}$. If $\overline{f}\ne f^{\ast }$, then $\overline{f}>f^{\ast }$. Recalling x ^∗ the global minimizer of (4) and since f is a continuous function, then there is an open ball ℬ centered at x ^∗ such that, for all $x\in ℬ\cap \mathrm{Int}(\Omega )$ we have,

$f\left(x\right)<\overline{f}-\frac{1}{2}(\overline{f}-f^{\ast }).$ (9)

Since $B\left(x\right)\ge 0$ for all $x\in \mathrm{Int}(\Omega )$, and $0<{\mu }_{k+1}<{\mu }_k$, we have $0<{\mu }_{k+1}B\left(x\right)<{\mu }_{k}B\left(x\right),\forall x\in \mathrm{Int}(\Omega )$. Therefore,

$\underset{\mathrm{k}\to \infty }{\lim }{\mu }_{k}B\left(x\right)=0,\forall x\in \mathrm{Int}(\Omega ).$ (10)

Thus, for any $x\text{'}\in ℬ\cap \mathrm{Int}(\Omega )$, and k large enough, we get

${\mu }_{k}B(x\text{'})<\frac{1}{4}(\overline{f}-f^{\ast }).$ (11)

Then, from (9) and (11), we have

$\varphi (x\text{'},{\mu }_k)<\overline{f}-\frac{1}{2}(\overline{f}-f^{\ast })+\frac{1}{4}(\overline{f}-f^{\ast })=\overline{f}-\frac{1}{4}(\overline{f}-f^{\ast })<\overline{f},$

which contradicts to $f_k\to \overline{f}$. Therefore, $\overline{f}=f^{\ast }$. That is

$f_{k+1}=\varphi (x_{k+1},{\mu }_k)\to f^{\ast }.$ (12)

Now, let $\overline{x}\in \Omega$ be any subsequential limit of {x _k }, more precisely, there is a subsequence $\left\{x_{k_l}\right\}$ such that $x_{k_l}\to \overline{x}$. If $\overline{x}\ne x^{\ast }$ with $f\left(\overline{x}\right)>f\left(x^{\ast }\right)$, then by continuity of f a subsequence $\left\{f\left(x_{k_l}\right)-f\left(x^{\ast }\right)+{\mu }_{k_l}B\left(x_{k_l}\right)\right\}$ cannot converge to zero, which contradicts $f_k-f^{\ast }\to 0$. Therefore, $\overline{x}=x^{\ast }$ or $\overline{x}\ne x^{\ast }$, but $f\left(\overline{x}\right)=f\left(x^{\ast }\right)$. Thus, every limit point generated by Algorithm 1 is a global solution of the problem (4). □

3 PENALTY METHODS OVERVIEW

Given $f:{\mathrm{ℝ}}^n\to \mathrm{ℝ}$ a continuous function, we consider the problem

$$\begin{gathered} \left(GP\right)minf\left(x\right) \\ s.t.x\in {\Omega }_1 \\ x\in {\Omega }_2. \end{gathered}$$ (13)

where Ω₁ and Ω₂ are any arbitrary subsets of ℝⁿ . In most applications Ω₁ is defined implicitly by functional restrictions as $h\left(x\right)=0$, where $h:{\mathrm{ℝ}}^n\to {\mathrm{ℝ}}^m$. In some cases, we assume that f and h are twice differentiable functions. A basic assumption is that problem (GP) admits global minimizer, some theoretical results were established by ^{Polyak (1971}), ^{Breitfeld & Shanno (1995}), ^{Nash (2010}), and ^{Luenberger & Ye (2008}).

Given a restriction set Ω₁, a penalty function is defined as a function $𝒫:{\mathrm{ℝ}}^n\to \mathrm{ℝ}$ satisfying (i) 𝒫 is continuous, (ii) $𝒫\left(x\right)=0\mathrm{if}x\in {\Omega }_1$, and (iii) $𝒫\left(x\right)>0\mathrm{if}x\notin {\Omega }_1$.

In order to solve the problem (13), the penalty function method solves the following penalized subproblem

$$\begin{gathered} \left(Q_{\rho }\right)\min f\left(x\right)+\rho \mathcal{P}\left(x\right) \\ \mathrm{s}.\mathrm{t}.x\in {\mathrm{\Omega }}_2 \end{gathered}$$ (14)

where $\rho >0$ is a constant called penalty parameter. For ρ large, is clear that a solution of (14) will be in a region where 𝒫 is small. Thus, when $\rho \to \infty$ is expected that the corresponding optimal points will approach the feasible set Ω₁.

For C ² class functions, $h:{\mathrm{ℝ}}^n\to {\mathrm{ℝ}}^m\text{and}g:{\mathrm{ℝ}}^n\to {\mathrm{ℝ}}^p$, some useful penalty functions 𝒫 based on the type of restrictions $h\left(x\right)=0\text{or}g\left(x\right)\le 0$ may be

1. $𝒫\left(x\right)=\frac{1}{2}{||h\left(x\right)||}_2^2$, quadratic penalty,

2. $𝒫\left(x\right)={||h\left(x\right)||}_1$,

3. $\mathcal{P}\left(x\right)=\sum _{i=1}^p{\left[max\left\{0,g_i\left(x\right)\right\}\right]}^2,$

4. $𝒫\left(x\right)=\frac{1}{2}{||h\left(x\right)||}_2^2+\sum _{i=1}^p{\left[\max \right\{0,g_i\left(x\right)\left\}\right]}^2$,

in the first item the quadratic penalty function preserves C ² property, but in the last three items 𝒫 is only C ¹.

Given $\psi (x,\rho )=f\left(x\right)+\rho 𝒫\left(x\right)$, we have a generic penalty algorithm given in Algorithm 2 for solving the problem (13), that works iteratively updating the parameter ρ before solving the penalized subproblem (14)

Algorithm 2
Penalty Algorithm

In general, one of the suggestions to compute ρk is taking ${\rho }_0=1$ and ${\rho }_{k+1}=10{\rho }_k$, ^{Fletcher (2013}). However, when Ω₁ is the set of equality constraints $h\left(x\right)=0$, a basic rule that works in practice is that if $||h\left(x_k\right)||\ge 0.1||h\left(x_{k-1}\right)||$, then ${\rho }_{k+1}=10\rho k$, otherwise ρ does not change. That approach was successfully tested for linear programming problems, ^{Suñagua & Oliveira (2017}).

The following Lemma gives a set of inequalities that follow directly from Algorithm 2 steps. A proof is based in ^{Martınez & Santos (1995}) and ^{Luenberger & Ye (2008}).

Lemma 3.1.Let {x _k } be a sequence generated byAlgorithm 2, which x _k+1 is global solution of problem (Q _k ). Then

1. $\psi (x_k,{\rho }_{k-1})\le \psi (x_{k+1},{\rho }_k)$

2. $\mathcal{P}\left(x_{k+1}\right)\le \mathcal{P}\left(x_k\right)$

3. $f\left(x_k\right)\le f\left(x_{k+1}\right)$.

Proof. Since {ρ _k } is a monotone increasing sequence and x _k is a global minimizer of subproblem (15), then

$$\begin{gathered} \psi (x_k,{\rho }_{k-1})=f\left(x_k\right)+{\rho }_{k-1}\mathcal{P}\left(x_k\right) \\ \le f\left(x_{k+1}\right)+{\rho }_{k-1}\mathcal{P}(xk+1) \\ \le f\left(x_{k+1}\right)+{\rho }_k\mathcal{P}\left(x_{k+1}\right)=\psi (x_{k+1},{\rho }_k). \end{gathered}$$

To establish the second inequality, recalling the optimalities of x _k and x _k+1 , we have

$\psi (x_k,{\rho }_{k-1})=f\left(x_k\right)+{\rho }_{k-1}\mathcal{P}\left(x_k\right)\le f\left(x_{k+1}\right)+{\rho }_{k-1}\mathcal{P}\left(x_{k+1}\right)$ (16)

$\psi (x_{k+1},{\rho }_k)=f\left(x_{k+1}\right)+{\rho }_k\mathcal{P}\left(x_{k+1}\right)\le f\left(x_k\right)+{\rho }_k\mathcal{P}\left(x_k\right)$ (17)

using (17) and (16), we get

$({\rho }_{k-1}-{\rho }_k)\mathcal{P}\left(x_k\right)\le ({\rho }_{k-1}-{\rho }_k)\mathcal{P}\left(x_{k+1}\right),$

as ${\rho }_{k-1}<{\rho }_k$, then $𝒫\left(x_k\right)\ge 𝒫\left(x_{k+1}\right)$. Finally, using this inequality

$f\left(x_k\right)+{\rho }_{k-1}\mathcal{P}\left(x_k\right)\le f\left(x_{k+1}\right)+{\rho }_{k-1}\mathcal{P}\left(x_{k+1}\right)\le f\left(x_{k+1}\right)+{\rho }_{k-1}\mathcal{P}\left(x_k\right)$

hence $f\left(x_k\right)\le f\left(x_{k+1}\right)$. □

Lemma 3.2.If x^*is a global minimizer of (GP) then for$k=0,1,2,···$

$f\left(x_k\right)\le \psi (x_k,{\rho }_{k-1})\le f\left(x^{\ast }\right).$

Consequently,$x_k\in {\Omega }_1$, if and only if, x_kis the global solution of (GP).

Proof. Since ${\rho }_k>0$ and $𝒫\left(x\right)\ge 0\forall x\in {\mathrm{ℝ}}^n$ and x _k is the global minimizer of (Q _k−1 ), then

$f\left(x_k\right)\le f\left(x_k\right)+{\rho }_{k-1}\mathcal{P}\left(x_k\right)\le f\left(x^{\ast }\right)+{\rho }_{k-1}\mathcal{P}\left(x^{\ast }\right)=f\left(x^{\ast }\right),$

where $\mathcal{P}\left(x^{\ast }\right)=0$. □

The global convergence of the penalty method, in the sense any limit point of the sequence is a solution, can be verified from the two previous Lemmas.

Theorem 3.1 (Global convergence for penalty method). Let {x _k } be a sequence of global minimizers of (Q_k ) generated by Algorithm 2 in which ${\rho }_k\to +\infty$. Then, any limit point of the sequence is a global minimizer of problem (13).

Proof. With a slight change of notation, the proof is based on Martínez & Santos’ demonstration. Indeed, let $\left\{x_{k_l}\right\}$ be a subsequence of {x _k } such that $x_{k_l}\to \overline{x}$. By the continuity of f , we have

$f\left(x_{k_l}\right)\to f\left(\overline{x}\right).$ (18)

Let f* be an optimal value of problem (GP). By Lemma 3.1 and Lemma 3.2, the sequence $\left\{\psi (x_k,{\rho }_{k-1})\right\}$ is nondecreasing and bounded above by f ^∗, then

$\underset{l\to \infty }{\lim }\psi (x_{k_l},{\rho }_{k_l-1})=\underset{l\ge 1}{\sup }\psi (x_{k_l},{\rho }_{k_l-1})=p^{\ast }\le f^{\ast }.$ (19)

Thus, using (18) and (19), yields

$$\begin{gathered} \underset{l\to \infty }{\lim }{\rho }_{k_l-1}\mathcal{P}\left(x_{k_l}\right)=\underset{l\to \infty }{\lim }\left[\left(f\left(x_{k_l}\right)+{\rho }_{k_l-1}\mathcal{P}\left(x_{k_l}\right)\right)-f\left(x_{k_l}\right)\right] \\ =\underset{l\to \infty }{\lim }\left(f\left(x_{k_l}\right)+{\rho }_{k_l-1}\mathcal{P}\left(x_{k_l}\right)\right)-\underset{l\to \infty }{\lim }f\left(x_{k_l}\right)=p^{\ast }-f\left(\overline{x}\right). \end{gathered}$$

Since $\mathcal{P}\left(x_{k_l}\right)\ge 0$ and ${\rho }_{k_l}\to \infty$, we conclude that $\underset{l\to \infty }{\lim }\mathcal{P}\left(x_{k_l}\right)=0$. Using the continuity of 𝒫, $\mathcal{P}\left(\overline{x}\right)=0$, thereby $\overline{x}\in {\mathrm{\Omega }}_1$. To prove the optimality of $\overline{x}$, just note that by Lemma 3.2, $f\left(x_{k_l}\right)\le f^{\ast }$, then

$f\left(\overline{x}\right)=\underset{l\to \infty }{\lim }f\left(x_{k_l}\right)\le f^{\ast },$

which completes the proof, because obviously $f^{\ast }\le f\left(\overline{x}\right)$ and then $f\left(\overline{x}\right)=f^{\ast }$. □

Furthermore, by definition of ψ and (19)

$f\left(x_{x_l}\right)\le \psi (x_{x_l},{\rho }_{k_l}-1)\le p^{\ast }\Rightarrow f\left(\overline{x}\right)\le p^{\ast }\le f^{\ast }.$

Therefore $f\left(\overline{x}\right)=p^{\ast }=f^{\ast }$, then

$\underset{l\to \infty }{\lim }{\rho }_{k_l-1}\mathcal{P}\left(x_{k_l}\right)=0.$ (20)

And using (19)

$\underset{l\to \infty }{\lim }\Psi \left(x_{k_l},{\rho }_{k_l-1}\right)=f^{\ast }.$ (21)

4 MIXED BARRIER-PENALTY METHOD

For continuous function $f:{\mathrm{ℝ}}^n\to \mathrm{ℝ}$, we consider the general programming problem

$$\begin{gathered} \left(NLP\right)\min f\left(x\right) \\ \mathrm{s}.\mathrm{t}.x\in {\Omega }_1,x\in {\Omega }_2,x\in {\Omega }_3. \end{gathered}$$ (22)

where Ω₁, Ω₂ and Ω₃ are restriction sets that are defined in (1).

As in the previous sections, we assume that problem (22) admits a global minimizer. Now, let 𝒫 a penalty function related to Ω₁ and B a barrier function related to Ω₂. Then, taking the penalty parameter $\rho >0$ and the barrier parameter $\mu >0$, we have the associate mixed barrier-penalty subproblem,

$$\begin{gathered} \left(B{P}_{\rho ,\mu }\right)\min f\left(x\right)+\rho \mathcal{P}\left(x\right)+\mu B\left(x\right) \\ \mathrm{s}.\mathrm{t}.x\in \mathrm{Int}\left({\Omega }_2\right),x\in {\Omega }_3. \end{gathered}$$ (23)

Since the general problem (NLP) admits global minimizer, then the problem (BP _ρ,µ) in (23) also admits a global solution for any feasible parameter values. Therefore, we define

$\Phi (x,\rho ,\mu )=f\left(x\right)+\rho \mathcal{P}\left(x\right)+\mu B\left(x\right).$ (24)

In order to solve the general problem (22), we provide a generic algorithm given in Algorithm 3, that works iteratively updating ρ and µ parameters before solving the penalized subproblem (23).

Algorithm 3
Mixed barrier-penalty algorithm

To establish the global convergence of the Algorithm 3, firstly we can associate the additive terms in two convenient ways

$$\begin{gathered} \Phi (x,\rho ,\mu )=\left[f\right(x)+\rho \mathcal{P}(x\left)\right]+\mu B\left(x\right) \\ =\left[f\right(x)+\mu B(x\left)\right]+\rho \mathcal{P}\left(x\right). \end{gathered}$$ (26)

Therefore, fixing respectively ρ and µ, we define $F_{\rho }\left(x\right)=f\left(x\right)+\rho 𝒫\left(x\right)\text{and}{G}_{\mu }\left(x\right)=f\left(x\right)+\mu B\left(x\right)$, then we associate to (NLP) the following two problems

$$\begin{gathered} \left(G{P}_{\rho }\right)\min F_{\rho }\left(x\right)\left(G{P}_{\mu }\right)\min G_{\mu }\left(x\right) \\ \mathrm{s}.\mathrm{t}.x\in {\mathrm{\Omega }}_2\mathrm{s}.\mathrm{t}.x\in {\mathrm{\Omega }}_1,x\in \mathrm{Int}\left({\mathrm{\Omega }}_2\right) \\ x\in {\mathrm{\Omega }}_{3}x\in {\mathrm{\Omega }}_3. \end{gathered}$$ (27)

Since the problem (NLP) admits a global minimizer, both (GP _ρ ) and (GP _µ ) in (27) also admit global minimizers. Therefore, defining

${\varphi }_{\rho }(x,\mu )=F_{\rho }\left(x\right)+\mu B\left(x\right)\text{and}{\psi }_{\mu }(x,\rho )=G_{\mu }\left(x\right)+\rho \mathcal{P}\left(x\right).$

We have respectively the barrier and penalty subproblems

$$\begin{gathered} \left(B{P}_{\rho }\right)\min {\varphi }_{\rho }\left(x,\mu \right)\left(P{P}_{\mu }\right)\min {\psi }_{\mu }\left(x,\rho \right) \\ \mathrm{s}.\mathrm{t}.x\in \mathrm{Int}\left({\mathrm{\Omega }}_2\right)\mathrm{s}.\mathrm{t}.x\in \mathrm{Int}\left({\mathrm{\Omega }}_2\right) \\ x\in {\mathrm{\Omega }}_3,x\in {\mathrm{\Omega }}_3 \end{gathered}$$ (28)

By fixing one of the parameters according to (27), the two problems in (28) are equivalent to (BP _ρ,µ ). In fact

${\varphi }_{\rho }(x,\mu )=\Phi \left(x,\rho ,\mu \right)={\psi }_{\mu }(x,\rho ),$ (29)

therefore, we can apply the results obtained in the preceding two sections.

In order to understand more clearly the ideas of the mixed problem, we consider the following particular quadratic problem

$$\begin{gathered} \min x_1^2+x_2^2 \\ \mathrm{s}.\mathrm{t}.x_2=2 \\ 1-x_1\le 0 \\ -1-x_2\le 0. \end{gathered}$$ (30)

According to the contours of the objective function and graph of restrictions in Figure 1, the optimal point is $x^{\ast }=(1,2)$. First, if we consider the Lagrangian function $ℒ(x_1,x_2,\lambda ,u_1,u_2)=x_1^2+x_2^2+\lambda (x_2-1)+u_1(1-x_1)+u_2(-1-x_2)$, the Karush Kuhn-Tuker conditions (-^{Kuhn & Tucker, 1951}) are

$$\begin{gathered} {ℒ}_{x_1}=2x_1-u_1=0,{ℒ}_{x_2}=2x_2+\lambda -u_2=0 \\ {x}_2-2=0,1-x_1\le 0,-1-x_2\le 0 \\ {u}_1(1-x_1)=0,u_2(-1-x_2)=0 \\ {u}_1\ge 0,u_2\ge 0 \end{gathered}$$

whose unique solutions for variables and Lagrangian parameters are $x_1^{*}=1,x_2^{*}=2,u_1^{*}=2,u_2^{*}=0,{\lambda }^{*}=-4.$

Figure 1
Convex Problem.

Now, we associate to (30) the mixed barrier-penalty subproblem

$$\begin{gathered} \left(Q{P}_{\rho ,\mu }\right)\min \Phi \left(x,\rho ,\mu \right) \\ \mathrm{s}.\mathrm{t}.1-x_1<0 \\ -1-x_2<0. \end{gathered}$$ (31)

where the penalized objective function is

$\Phi \left(x,\rho ,\mu \right)=x_1^2+x_2^2+\frac{\rho }{2}{\left(x_2-2\right)}^2-\mu \left[\log \left(x_1-1\right)+\log \left(x_2+1\right)-\log M\right]$

where M is a large enough positive number such that $x_1>1,x_2>-1$ and $(x_1-1)(x_2+1)<M$, surely this region lies within inequality constraints $1-x_1<0\text{and}-1-x_2<0$. This condition ensures that the barrier function is non-negative in the region that contains optimal point.

It is easy to see that Φ is a smooth function, thereby from first-order necessary conditions for optimal points, we have

$$\begin{gathered} 2x_1-\frac{\mu }{x_1-1}=0, \\ 2x_2+\rho \left(x_2-2\right)-\frac{\mu }{x_2+1}=0. \end{gathered}$$ (32)

Solving this nonlinear system, subject to $x_1>1$ and $x_2>-1$, by the substitution method, we obtain

$$\begin{gathered} x_1=\frac{1+\sqrt{1+2\mu }}{2}\stackrel{\mu \to 0}{\to }1=x_1^{*} \\ {x}_2=\frac{\rho -2+\sqrt{{\left(2-\rho \right)}^2+4\left(2+\rho \right)\left(2\rho +\mu \right)}}{2\left(2+\rho \right)}\stackrel{\stackrel{\mu \to 0}{\rho \to \infty }}{\to }2=x_2^{*}. \end{gathered}$$ (33)

Thus, for each optimal point in (33), the optimal values of problem (QP _ρ,µ ) in (31) is $\theta (\rho ,\mu )=\Phi (x_1,x_2,\rho ,\mu )$, whose graph is shown in Figure 2 with $M=2$.

Figure 2
$\theta (\rho ,\mu ),0<\mu <2.5,0<\rho <20$.

We can see that for fixed $\mu ,\theta (\rho ,\mu )$ is an increasing function and for fixed $\rho ,\theta (\rho ,\mu )$ is a decreasing function. This fact will be showed theoretically in Theorem 4.1.

Furthermore, using (33), when $\mu \to 0$ and $\rho \to \infty$, the following gradient’s coefficients in (32) converge to optimal Lagrangian parameters

$$\begin{gathered} u_1=\frac{\mu }{x_1-1}=1+\sqrt{1+2\mu }\to 2=u_1^{*} \\ {u}_2=\frac{\mu }{x_2+1}=\frac{2\mu \left(2+\rho \right)}{2+3\rho +A}\to 0=u_2^{*} \\ \lambda =\rho \left(x_2-2\right)=\frac{\rho \left(-10-3\rho +A\right)}{2\left(2+\rho \right)}\to -4={\lambda }^{*} \end{gathered}$$

where $A=\sqrt{4\mu (2+\rho )+{(2+3\rho )}^2}$. In addition, the Hessian matrix for Φ is

$$\begin{gathered} {\nabla }^2\Phi =\left(\begin{array}{cc}2+\frac{\mu }{{\left(x_1-1\right)}^2}& 0\\ 0& 2+\rho +\frac{\mu }{{\left(x_2+1\right)}^2}\end{array}\right) \\ =\left(\begin{array}{cc}2+\frac{u_1^2}{\mu }& 0\\ 0& 2+\rho +\frac{u_2^2}{\mu }\end{array}\right)~\left(\begin{array}{cc}2+\frac{4}{\mu }& 0\\ 0& 2+\rho \end{array}\right). \end{gathered}$$

Then ∇²Φ is a positive definite matrix, that guarantees the minimality of x ₁ and x ₂ in (33). Moreover, the approximate condition number of this matrix is

$\kappa \left({\nabla }^2\Phi \right)\approx \frac{2+{\displaystyle \frac{4}{\mu }}}{2+\rho },$

hence ∇²Φ is ill-conditioned for very small µ and small ρ, however for large ρ, that condition number could be reduced.

Next, we have the global convergence theorem for the mixed barrier-penalty algorithm.

Theorem 4.1 (Global convergence for mixed method).Let {x _k } be a sequence of global minimizers of (BP _k ) problem in (25) generated by mixedAlgorithm 3in which${\rho }_k\to +\infty$and${\mu }_k\to 0$. Then any limit point of sequence is a global minimizer of the (NLP) problem.

Proof. In order to apply the results of the preceding sections, the idea is to fix, one of the parameters in the (BP _ρ,µ ) subproblem in (23) one at a time, and apply the corresponding results for each subproblems in (27).

Firstly, to fix ρ, let $\left\{x_k^{\rho }\right\}$ be the sequence generated by Algorithm 1 for solving the (GP _ρ ) subproblem in (27). By applying Lemma 2.1, we get

$$\begin{gathered} {\varphi }_{\rho }\left(x_{k+1}^{\rho },{\mu }_k\right)\le {\varphi }_{\rho }\left(x_k^{\rho },{\mu }_{k-1}\right), \\ {F}_{\rho }\left(x_{k+1}^{\rho }\right)\le F_{\rho }\left(x_k^{\rho }\right). \end{gathered}$$ (34)

By the monotonicity in (34) and by (12) the sequence $\left\{{\varphi }_{\rho }(x_k^{\rho },{\mu }_{k-1})\right\}$ converges to global optimal value of the problem (GP _ρ ) in (27), that is,

${\varphi }_{\rho }(x_k^{\rho },{\mu }_{k-1})\to \underset{k\ge 1}{\inf }{\varphi }_{\rho }(x_k^{\rho },{\mu }_{k-1})=F_{\rho }\left(x_{*}^{\rho }\right).$ (35)

In addition, from Theorem 2.1 all convergent subsequence of $\left\{x_k^{\rho }\right\}$ converges to a global minimizer of the problem (GP _ρ ) in (27).

Similarly, to fix µ, let $\left\{x_k^{\mu }\right\}$ be the sequence generated by the associated Algorithm 2 for solving the (GP _µ ) subproblem in (27). By applying Lemma 3.1, we get

$$\begin{gathered} {\psi }_{\mu }\left(x_k^{\mu },{\rho }_{k-1}\right)\le {\psi }_{\mu }\left(x_k^{\mu },{\rho }_k\right), \\ {G}_{\mu }\left(x_k^{\mu }\right)\le G_{\mu }\left(x_{k+1}^{\mu }\right). \end{gathered}$$ (36)

By the monotonicity in (36) and by (21), the sequence $\left\{{\psi }_{\mu }(x_k^{\mu },{\rho }_{k-1})\right\}$ converges to global optimal value of the (GP _µ ) problem of (27), that is,

${\psi }_{\mu }\left(x_k^{\mu },{\rho }_{k-1}\right)\to \underset{k\ge 1}{sup}{\psi }_{\mu }\left(x_k^{\mu },{\rho }_{k-1}\right)=G\left(x_{*}^{\mu }\right).$ (37)

In addition, from Theorem 3.1 all convergent subsequence of $\left\{x_k^{\mu }\right\}$ converges to a global minimizer of the problem (GP _µ ) in (27). And by Lemma 3.2, we get $G\mu \left(x_k^{\mu }\right)\le G\mu \left(x_{*}^{\mu }\right),\forall k$.

Now let {x _k } be a sequence of minimizers obtained by Algorithm 3 for a mixed problem. More precisely, let $x_{k+1}=x({\rho }_k,{\mu }_k)$ which also minimizes (28), because according to (29), we have

$\Phi (x_{k+1},{\rho }_k,{\mu }_k)=f\left(x_{k+1}\right)+{\rho }_k\mathcal{P}\left(x_{k+1}\right)+{\mu }_{k}B\left(x_{k+1}\right)={\varphi }_{{\rho }_k}(x_{k+1},{\mu }_k)={\psi }_{{\mu }_k}(x_{k+1},{\rho }_k)$

Since (35) and (37), we have

$$\begin{gathered} F_{{\rho }_k}\left(x_{*}^{{\rho }_k}\right)\le {\varphi }_{{\rho }_k}(x_{k+1},{\mu }_k)=\Phi (x_{k+1},{\rho }_k,{\mu }_k) \\ ={\psi }_{{\mu }_k}(x_{k+1},{\rho }_k)\le G_{{\mu }_k}\left(x_{*}^{{\mu }_k}\right). \end{gathered}$$ (38)

Giving x(ρ _k , µ _k ) the solution of (BP _{ρ ,µ} ) for $\mu ={\mu }_k$ and $\rho ={\rho }_k$. For ${\mu }_k<{\mu }_{k-1}$, additionally we solve (BP _{ρ ,µ} ) for $\mu ={\mu }_{k-1}$ and $\rho ={\rho }_k$, which solution is called x(ρ _k , µ _k−1 ). Using (34)

${\varphi }_{{\rho }_k}\left(x\right({\rho }_k,{\mu }_k),{\mu }_k)\le {\varphi }_{{\rho }_k}\left(x\right({\rho }_k,{\mu }_{k-1}),{\mu }_{k-1})$ (39)

and by (29), for $x=x({\rho }_k,{\mu }_k)$ and $y=x({\rho }_k,{\mu }_{k-1})$, we have

$$\begin{gathered} {\varphi }_{{\rho }_k}(x,{\mu }_k)=f\left(x\right)+{\rho }_k\mathcal{P}\left(x\right)+{\mu }_{k}B\left(x\right)={\psi }_{{\mu }_k}(x,{\rho }_k) \\ {\varphi }_{{\rho }_k}(y,{\mu }_{k-1})=F_{{\rho }_k}\left(y\right)+{\mu }_{k-1}B\left(y\right)=f\left(y\right)+{\rho }_k\mathcal{P}\left(y\right)+{\mu }_{k-1}B\left(y\right) \\ =f\left(y\right)+{\mu }_{k-1}B\left(y\right)+{\rho }_k\mathcal{P}\left(y\right)=G_{{\mu }_{k-1}}\left(y\right)+{\rho }_k\mathcal{P}\left(y\right) \\ ={\psi }_{{\mu }_{k-1}}(y,{\rho }_k) \end{gathered}$$ (40)

Using (39) and (40), we get

$$\begin{gathered} {\psi }_{{\mu }_k}\left(x\left({\rho }_k,{\mu }_k\right),{\rho }_k\right)\le {\psi }_{{\mu }_{k-1}}\left(x\left({\rho }_k,{\mu }_{k-1}\right),{\rho }_k\right) \\ \Rightarrow G_{{\mu }_k}\left(x_{*}^{{\mu }_k}\right)\le G_{{\mu }_{k-1}}\left(x_{*}^{{\mu }_{k-1}}\right). \end{gathered}$$

Similarly, for ${\rho }_k>{\rho }_{k-1}$, additionally we consider a solution of (BP _{ρ ,µ} ) for $\rho ={\rho }_{k-1}$ and $\mu ={\mu }_k$, which solution is called x(ρ _k−1 ,µ _k ). Using (36)

${\psi }_{{\mu }_k}\left(x\left({\rho }_{k-1},{\mu }_k\right),{\rho }_{k-1}\right)\le {\psi }_{{\mu }_k}\left(x\left({\rho }_k,{\mu }_k\right),{\rho }_k\right)$ (41)

and by (29) for $x=x\left({\rho }_k,{\mu }_k\right)$ and $z=x\left({\rho }_{k-1},{\mu }_k\right)$, we have

$$\begin{gathered} {\psi }_{{\mu }_k}\left(x,{\rho }_k\right)=f\left(x\right)+{\mu }_{k}B\left(x\right)+{\rho }_k\mathcal{P}\left(x\right)={\varphi }_{{\rho }_k}\left(x,{\mu }_k\right) \\ {\psi }_{{\mu }_k}\left(z,{\rho }_{k-1}\right)=G_{{\mu }_k}\left(z\right)+{\rho }_{k-1}\mathcal{P}\left(z\right)=f\left(z\right)+{\mu }_{k}B\left(z\right)+{\rho }_{k-1}\mathcal{P}\left(z\right) \\ =f\left(z\right)+{\rho }_{k-1}\mathcal{P}\left(z\right)+{\mu }_{k}B\left(z\right)=F_{{\rho }_{k-1}}\left(z\right)+{\mu }_{k}B\left(z\right) \\ ={\varphi }_{{\rho }_{k-1}}\left(z,{\mu }_k\right) \end{gathered}$$ (42)

Using (41) and (42), we get

$$\begin{gathered} {\varphi }_{{\rho }_{k-1}}\left(x\left({\rho }_{k-1},{\mu }_k\right),{\mu }_k\right)\le {\varphi }_{{\rho }_k}\left(x\left({\rho }_k,{\mu }_k\right),{\mu }_k\right) \\ \Rightarrow F_{{\rho }_{k-1}}\left(x_{*}^{{\rho }_{k-1}}\right)\le F_{{\rho }_k}\left(x_{*}^{{\rho }_k}\right). \end{gathered}$$

Let x* be a global minimizer of (NLP). Recalling $x_{*}^{{\mu }_k}$ a solution of the problem $\left(G{P}_{{\mu }_k}\right)$ in (27), with the additional assumption $x_{*}^{{\mu }_k}\in Int\left({\Omega }_2\right)$, we can conclude that $f\left(x*\right)\le G_{{\mu }_k}\left(x_{*}^{{\mu }_k}\right)$. Moreover x* is a feasible point of the problem (GP _ρ ), then $F_{{\rho }_k}\left(x_{*}^{{\rho }_k}\right)\le f\left(x*\right)$. Therefore, $\left\{G_{{\mu }_k}\left(x_{*}^{{\mu }_k}\right)\right\}$ is a monotone nonincreasing sequence that is bounded below by f(x*) and using (12) this sequence converges to its infimum f(x*). Also, $\left\{F_{{\rho }_k}\left(x_{*}^{{\rho }_k}\right)\right\}$ is monotone nondecreasing sequence that is bounded above by f(x*) and by (21) that sequence converges to its supremum f(x*), that is,

$$\begin{gathered} F_{{\rho }_k}\left(x_{*}^{{\rho }_k}\right)\to \underset{k\ge 1}{sup}{F}_{{\rho }_k}\left(x_{*}^{{\rho }_k}\right)=f\left(x*\right), \\ {G}_{{\mu }_k}\left(x_{*}^{{\mu }_k}\right)\to \underset{k\ge 1}{inf}{G}_{{\mu }_k}\left(x_{*}^{{\mu }_k}\right)=f\left(x*\right). \end{gathered}$$ (43)

By applying squeeze theorem¹ to (38) and (43), we show

$\underset{k\to \infty }{lim}\Phi (x_k,{\rho }_{k-1},{\mu }_{k-1})=f(x*).$ (44)

Let $\left\{x_{k_l}\right\}$ be any subsequence of {x _k } such that $x_{k_l}\to \overline{x}$. By the continuity of f, we get $f\left(x_{k_l}\right)\to f\left(\overline{x}\right)$. The final demonstration will be done by contradiction under the assumption $\overline{x}\ne x*$ with $f\left(\overline{x}\right)>f(x*)$. Using (10) for the problem (GP _ρ ), we have ${\mu }_{k_{l^{-1}}}B\left(x\right)\to 0$, for any $x\in Int\left({\Omega }_2\right)$. Furthermore, using (20) for the problem (GP _µ ), also we have ${\rho }_{k_{l^{-1}}}𝒫\left(x_{k_l}\right)\to 0$, and by continuity of f, the sequence $\left\{f\left(x_{k_l}\right)-f\left(x^{\ast }\right)+{\rho }_{k_l}𝒫\left(x_{k_l}\right)+{\mu }_{k_l}B\left(x_{k_l}\right)\right\}$ cannot converge to zero, which contradicts (44). □

5 APPLICATIONS

5.1 Barrier-Penalty applied to convex problem

The Algorithm 4 is an algorithm based on generic Algorithm 3 in order to solve the nonlinear problem (30).

Algorithm 4
Mixed barrier-penalty algorithm

For ${\rho }_0=1,{\mu }_0=1\text{and}{x}_0=(1.5,1)$ we write a MATLAB script for Algorithm 4 in order to compute a sequence of optimal points that approach to $x*=(1,2)$. The iterative results is shown in Table 1.

Table 1
Iterative results.

k	x k	Φ(x k , ρ k , µ k )	ρ k	µ k
0	(1.500000,1.000000)	3.25000000	1	1
1	(1.366025,0.847127)	3.63962871	10	0.1
2	(1.047723,1.669788)	4.63714955	100	0.01
3	(1.004975,1.960817)	4.97372208	103	0.001
4	(1.000500,1.996008)	4.99951984	104	0.0001
5	(1.000054,1.999600)	5.00018097	105	1e-05
6	(1.000009,1.999960)	5.00004320	106	1e-06
7	(1.000101,1.999996)	5.00020113	107	1e-07
8	(1.000020,2.000000)	5.00004029	108	1e-08
9	(1.000001,2.000000)	5.00000209	109	1e-09

The path following points is shown in Figure 3, where last points are close to x*. The exact results also solved by MATLAB are $x*=(1.000000,2.000000)$ and $f(x*)=5.000000$.

Figure 3
Points generated by Algorithm 4.

5.2 Penalized standard linear programming problem

We consider the standard linear programming problem where several variables are upper bounded

$$\begin{gathered} \left(LP\right)min{c}^{T}x \\ s.t.Ax=b \\ Ex\le u \\ x\ge 0, \end{gathered}$$ (46)

where A is m × n matrix, $c,x\in {\mathrm{ℝ}}^n,b\in {\mathrm{ℝ}}^m$, and E is formed by rows of n × n identity matrix corresponding to bounded variables, thereby Ex is the vector of bounded variables for which u is the vector of upper bounds. In this case is usual to add the slack variable v such that $Ex+v=u$, where v ≥ 0.

In the most computational packages that implement Interior Point Methods for solving linear programming problems only barrier parameter is considered.

In order to solve the LP problem (46), by using the quadratic penalty and logarithmic barrier functions, the objective function is penalized as follow

$\Phi \left(x,v,\rho ,\mu \right)=c^{T}x+\frac{\rho }{2}{||b-Ax||}^2-\mu \sum _{j=1}^n\log x_j-\mu \sum _{j=1}^{n_b}\log v_j,$ (47)

where µ and ρ are respectively the barrier and penalty parameters and nb is the number of bounded variables. Then the associated mixed barrier-penalty subproblem is

$$\begin{gathered} \left(LP{P}_{\rho ,\mu }\right)min\Phi (x,v,\rho ,\mu ) \\ s.t.\left(x,v\right)>0. \end{gathered}$$ (48)

Since Φ(x,v,ρ,µ) is a smooth function on open set $\left(x,v\right)>0$. By applying the first-order necessary condition, we have

$c-A^T\rho (b-Ax)-\mu X^{-1}e+\mu E^T{V}^{-1}e=0,$

Defining $y=\rho (b-Ax),z=\mu X^{-1}e,w=\mu V^{-1}e$, we get

$$\begin{gathered} c-A^{T}y+E^{T}w-z=0,Ex+v=u, \\ XZe=\mu e,VWe=\mu e,y=\rho (b-Ax). \end{gathered}$$

Taking $\delta =1/\rho$, we rewrite $\delta y=b-Ax$. Thus $Ax+\delta y=b$.

Therefore, the optimality conditions for subproblem (LPP _ρ,µ ) on $(x,v)>0$ and $(z,w)>0$ are

$$\begin{gathered} Ax+\delta y=b \\ Ex+v=u \\ {A}^{T}y+z-E^{T}w=c \\ XZe=\mu e \\ VWe=\mu e. \end{gathered}$$ (49)

In Interior Point Methods reviewed on ^{Suñagua & Oliveira (2017}), we find a search direction by applying Newton’s Method for solving nonlinear system (49). In fact, the Newton’s directions satisfy

$\left(\begin{array}{ccccc}A& 0& \delta I& 0& 0\\ E& I& 0& 0& 0\\ 0& 0& A^T& I& -E^T\\ Z& 0& 0& X& 0\\ 0& W& 0& 0& V\end{array}\right)\left(\begin{array}{c}dx\\ dv\\ dy\\ dz\\ dw\end{array}\right)=\left(\begin{array}{c}{r}_p\\ r_u\\ r_d\\ r_c\\ r_s\end{array}\right)\begin{array}{c}{r}_p=b-Ax-\delta y\\ r_u=u-Ex\\ r_d=c-A^{T}y-z+E^{T}w\\ r_c=\mu e-XZe\\ r_s=\mu e-VWe,\end{array}$ (50)

solving this block linear equations, we find up

$dz=X^{-1}(r_c-Zdx),dw=V^{-1}(r_s-Wdw),dv=r_u-Edx$ (51)

replacing this on third group of equations

$A^{T}dy-D^{-1}dx=r_d-X^{-1}{r}_c+E^T{V}^{-1}{r}_s-E^T{V}^{-1}W{r}_u$

where $D^{-1}=X^{-1}Z+E^T{V}^{-1}WE$, then

$dx=D(A^{T}dy-r_d+X^{-1}{r}_c-E^T{V}^{-1}{r}_s+E^T{V}^{-1}W{r}_u)$ (52)

using (52) and first group of equations of (50), we get the normal equations

$(AD{A}^T+\delta I)dy=AD(r_d-X^{-1}{r}_c+E^T{V}^{-1}{r}_s-E^T{V}^{-1}W{r}_u)$ (53)

close to optimal point, D matrix is very bad scaled and then ADA ^T is also very ill-conditioned. In this case, the penalty parameter δ improves that condition number, which is helpful for solving the symmetric positive definite system by applying conjugate gradient method for instance.

Alternatively to (52) and (53), dx and dy also obtain by solving the following augmented system

$\left(\begin{array}{cc}-D^{-1}& A^T\\ A& \delta I\end{array}\right)\left(\begin{array}{c}dx\\ dy\end{array}\right)=\left(\begin{array}{c}{r}_1\\ r_p\end{array}\right)$ (54)

where $r_1=r_d-X^{-1}{r}_c+E^T{V}^{-1}{r}_s-E^T{V}^{-1}W{r}_u$. This system is also symmetric, indefinite and better condition number due to penalty parameter.

For computational experiments, we use the open source package PCx (^{Czyzyk et al., 1997}), that implements the Mehrotra’s Predictor-Corrector algorithm in which the barrier parameter µ is already incorporated in order to solve linear programming problems. By adding an appropriate code to PCx, we achieve to incorporate the penalty parameter δ, thus, we obtain the modified PCx called the Predictor-Corrector mixed algorithm with barrier and penalty parameters. The numerical results for several NETLIB LP problems were computed for approaches proposed in ^{Suñagua & Oliveira (2017}), where the goodness of the approaches were compared according to ^{Dolan & More (2002}) performance profile criteria.

6 CONCLUSIONS

Firstly, we present a brief summary of the main concepts and results upon the barrier and penalty methods, where for each method we show the global convergence theorems in order to use these strategies in the proof of the global convergence theorem for mixed algorithm.

In the Section 4, we provide a mixed algorithm for solving mixed barrier-penalty subproblem (23), and we provide a constructive proof on global convergence theorem for mixed barrierpenalty methods as an alternative showed in ^{Fiacco & McCormick (1990}) and ^{Breitfeld & Shanno (1995}). For simple convex nonlinear problems we write MATLAB code in order to generate iterative points that illustrate penalty and barrier functions.

Finally, we develop an application for nonlinear programming problems with equality and inequality functional constraints, such as, quadratic programming problem, and a standard linear programming problem. Since the functions involved have the smooth property on an open set, then the optimality conditions for each class of problems are stated, those can be solved by applying interior point methods.

ACKNOWLEDGEMENTS

Thanks to CNPq, FAPESP (grant number 2010/06822-4) and Universidad Mayor de San Andrés (UMSA) for their financial support.

References

• ¹
  BAZARAA MS, SHERALI HD & SHETTY CM. 2013. Nonlinear programming: theory and algorithms. John Wiley & Sons.
• ²
  BERTSEKAS DP. 1976. On penalty and multiplier methods for constrained minimization. SIAM Journal on Control and Optimization, 14(2): 216-235.
• ³
  BREITFELD MG & SHANNO DF. 1994. A globally convergent penalty-barrier algorithm for nonlinear programming and its computational performance. Rutgers University. Rutgers Center for Operations Research [RUTCOR].
• ⁴
  BREITFELD MG & SHANNO DF. 1995. A Globally Convergent Penalty-Barrier Algorithm for Nonlinear Programming. In: Operations Research Proceedings 1994. pp. 22-27. Springer.
• ⁵
  CZYZYK J, MEHROTRA S, WAGNER M & WRIGHT SJ. 1997. PCx user guide (Version 1.1). Optimization Technology Center, Northwestern University,.
• ⁶
  DOLAN ED & MORE JJ. 2002. Benchmarking optimization software with performance profiles. Mathematical programming, 91(2): 201-213.
• ⁷
  FIACCO AV & MCCORMICK GP. 1990. Nonlinear programming: sequential unconstrained minimization techniques. vol. 4. Siam.
• ⁸
  FLETCHER R. 2013. Practical methods of optimization. John Wiley & Sons, Chichester.
• ⁹
  GRIVA I, NASH SG & SOFER A. 2009. Linear and nonlinear optimization. vol. 108. Siam.
• ¹⁰
  KUHN HW & TUCKER AW. 1951. Nonlinear Programming. In: Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability. pp. 481-492. Berkeley, California: University of California Press. Available at: http://projecteuclid.org/euclid.bsmsp/1200500249
  » http://projecteuclid.org/euclid.bsmsp/1200500249
• ¹¹
  LUENBERGER DG & YE Y. 2008. Linear and nonlinear programming. 3th ed.. Springer New York.
• ¹²
  MARTINEZ JM & SANTOS SA. 1995. Métodos computacionais de otimização. Colóquio Brasileiro de Matemática, Apostilas, 20. Available at: https://www.ime.unicamp.br/~martinez/mslivro.pdf
  » https://www.ime.unicamp.br/~martinez/mslivro.pdf
• ¹³
  NASH SG. 2010. Penalty and barrier methods. Wiley Encyclopedia of Operations Research and Management Science,.
• ¹⁴
  NASH SG & SOFER A. 1993. A barrier method for large-scale constrained optimization. ORSA Journal on Computing, 5(1): 40-53.
• ¹⁵
  POLYAK BT. 1971. The convergence rate of the penalty function method. USSR Computational Mathematics and Mathematical Physics, 11(1): 1-12.
• ¹⁶
  SUÑAGUA P & OLIVEIRA AR. 2017. A new approach for finding a basis for the splitting preconditioner for linear systems from interior point methods. Computational Optimization and Applications, 67(1): 111-127.
• ¹⁷
  WRIGHT MH. 1992. Interior methods for constrained optimization. Acta Numérica, 1: 341-407.
• ¹⁸
  WRIGHT SJ & NOCEDAL J. 1999. Numerical optimization. vol. 2. Springer New York.

• 1
  formulated in modern terms by Carl Friedrich Gauss

Publication Dates

• Publication in this collection
  18 May 2020
• Date of issue
  2020

History

• Received
  07 Dec 2018
• Accepted
  31 Oct 2019