INTERIOR-POINT METHODS VIA SELF-CONCORDANCE

OR RELATIVE LIPSCHITZ CONDITION

Habilitationsschrift der Fakultät für Mathematik

der Bayerischen Julius-Maximilians-Universität Würzburg für das Fach Mathematik vorgelegt von Florian Jarre

Würzburg, März 1994 Korrigierte Version vom 7. Juli 1994

Contents

Introduction 3 Outline 4 Notation 6

1. Motivation 7 1.1 A convex problem and assumptions 7 1.2 The affine scaling method 7 1.3 The logarithmic barrier approach 8 1.4 The method of centers 8

2. Self-concordance 10 2.1 The first question, Newton’s method 10 2.1.1 Derivation of self-concordance 10 2.1.2 Some examples 11 2.1.3 The H-norm 13 2.1.4 Relative Lipschitz condition 14 2.1.5 Newton’s method 16 2.2 The second question, outer ellipsoidal approximations 18 2.2.1 Equivalent formulations 19 2.2.2 Some examples 20 2.2.3 Properties 20 2.2.4 Optimal barriers 23 2.2.5 Further examples of self-concordant functions 24

3. A basic algorithm 26 3.1 A model algorithm with polynomial rate of convergence 26 3.2 Phase 1 28 3.3 Logarithmic barrier and affine scaling method 30 4. KKT Approach 32 4.1 The search direction 33 4.2 A “primal dual” algorithm 35 4.3 Relationship to logarithmic barrier methods 37

5. Application: A min-max eigenvalue problem 41 5.1 The problem 41 5.2 Relationship to other methods 42 5.3 A barrier approach 43 5.4 A specific barrier method 44 5.5 Some numerical results 51

Extensions of the basic model of self-concordant barrier functions 55

6. Self-concordance of order two 57 6.1 The problem 57 6.2 A simple example 59 6.3 Self-concordance of order two 60 6.4 Additional results 61 6.5 Some numerical examples 66

7. Fractional programs with convex constraints 68 7.1 The problem 68 7.2 Basic assumptions 71 7.3 A method of centers 72 7.4 Convergence analysis 81 7.5 Stopping criterion 85 7.6 A sketch of a practical implementation 87 7.7 Numerical examples 89

8. Appendix, two basic lemmas 92

Bibiliography 94

Introduction

In 1984, Karmarkar’s paper [39] triggered extensive research on polynomial-time interiorpoint methods for linear programming problems and certain classes of convex and pseudoconvex programming problems. The new developments in this area form one of the most important and most extensive contributions to optimization theory. A recent bibliography [45] counts more than 1500 articles in the field of interior-point methods. Goal of this monograph is to summarize some of the main theoretical developments of interior-point methods for nonlinear programs and to provide some numerical examples that illustrate the potentials of the interior-point approach.

Karmarkar’s algorithm relies on the function “− log x” as a barrier function for the constraint x ≥ 0. While the use of the log function in the context of linear or convex programming was not new [19, 11], Karmarkar was the first one who succeeded to prove polynomial convergence of his method by a rigorous exploitation of the properties of the log function. Important generalizations and improvements over Karmarkar’s original method include the definition and analysis of the analytic center for convex sets [82], the proof of an O(\sqrt{n})-iteration complexity of the method of centers for linear programs [75], the analysis of interior-point methods for linear complementarity problems [43], the proof of an O(n^{3}L) overall complexity [92, 24] for linear programs and the extension of the O(\sqrt{n})-iteration method to convex quadratic constraints [29, 53]. However, it was not until 1988 that the crucial properties of the function log x have been identified [62] and carefully analyzed in the monograph [64]. The essential properties identified in [62] may be regarded as two simple forms of local Lipschitz continuity, the local Lipschitz continuity of f(x) = − log x and of f′′(x), both measured in the norm defined by f′′(x). A more precise and intuitive explanation of these properties is given in Chapter 2 below.

Outline

Chapters 1 and 2 are intended to give a self-contained and easy-to-read introduction to self-concordant functions. In Chapter 1 a convex problem is defined and three elementary methods for solving this problem are listed. For each method some crucial questions are stated that determine its efficiency. Based on these questions the notion of self-concordance is naturally derived in Chapter 2. Chapter 2 also presents the basic theoretical results needed in the subsequent Chapters. The notion of self-concordance as well as many of the theoretical results about self-concordant functions are due to Nesterov and Nemirovsky [66].

In Chapter 3 a short proof of polynomiality for the method of centers and the logarithmic barrier method (both modifications of the conceptual methods described in Chapter 1) is given. The proof is very simple once the results of Chapter 2 are known.

In Chapter 4 a different approach, namely solving convex programs via the KKT-system is presented. Even though this approach appears quite different form the logarithmic barrier method of the previous Chapters, it is shown that both methods are closely related.

In Chapter 5 an example for a practical application of the logarithmic barrier method is given. The problem considered here, namely the minimization of the maximum eigenvalue of an affine combination of symmetric matrices has important applications in mechanics and systems analysis. The results of this Chapter appeared in slightly different form in [31].

Chapter 6 is aimed to further improve the theoretical results for self-concordant functions by making a somewhat more restrictive assumption on the problem. We define the notion of “self-concordance of order 2”, which is a stronger assumption than the self-concordance condition (2.2) of Chapter 2, but is nevertheless satisfied by the standard examples for (2.2). We show that for certain questions, self-concordance of order 2 implies stronger theoretical results than (2.2), and point out how these results might be used during a line-search. The results of this Chapter are to appear in [33].

In Chapter 7 we present an extension of an interior-point method to solving fractional programs with convex constraints.

The basic step in the method of centers of Chapter 1 corresponds to a parallel shift of some linear constraint. Here, we show that—with a very similar argument as used for the analysis of the method of centers—also the rotation of some linear constraint about the origin allows the proof of a polynomial rate of convergence. Moreover, the rate of convergence is of the same order as for the method of centers. This result is applied to fractional programs, and used to show that they are solvable up to a specified accuracy ϵ in polynomial time.

A new element of this approach is the fact that the problem under consideration is slightly non-convex; (“slightly” since it is still pseudo-convex). We conclude this Chapter with some numerical examples for the (non-convex) problem of minimizing the condition number of some affine combination of symmetric matrices which is coverd by the above analysis. Apart from polynomial solvability, the numerical examples indicate that a modification of our method may be a viable approach to solve this problem numerically. The results of this Chapter are joint work with Roland Freund and have been submitted for publication, [16].

Acknowledgements

The author would like to thank all colleagues who helped with numerous discussions and by sending early versions of their research reports; Professor Kees Roos, Professor Tamas Terlaky, Dr. Dick den Hertog, Professor Jean-Philippe Vial, Professor M. Kojima, Professor Takashi Tsuchiya, Professor Yurii Nesterov, Professor Arkadii Nemirovsky, Dr. Margaret Wright, Professor B. Jongen and Professor K. Anstreicher. It was Professor György Sonnevend who led me to the topic of interior point methods and was of great help for my understanding of these methods. Professor Mike Saunders and Professor Stephen Boyd contributed to this work in many stimulating discussions during my visit in Stanford. I am grateful to Professor Shinji Mizuno who was proofreading part of this monograph. Dr. Thomas Huckle helped me with important hints in many discussions. Dr. Roland Freund also helped in many discussions and with our joint research, part of which is summarized in Chapter 7. I am also thankful to Mr. Jürgen Launer for typing parts of this work. Finally, I am greatly indebted to my supervisor Professor J. Stoer who supported me throughout my stay in Würzburg.

Notation

The following main notation is used throughout this monograph: S The feasible set. c The objective function, either linear c^{T} x, or nonlinear c(x). c_{i} The ith constraint function. m The number of constraint functions. n The dimension of the space, S ⊂ IR^{n}. λ Bound on the objective function, c(x) ≤ λ. μ Weight for the objective function, c(x)/μ. ϕ Logarithmic self-concordant barrier function for S. θ Self-concordance parameter. φ ϕ “combined” with some term for the objective function. φ(x, λ) ϕ(x) − κ log(λ − c(x)). φ(x, μ) ϕ(x) + c(x)/μ. x(λ) The minimum of φ(x, λ). x(μ) The minimum of φ(x, μ). f The restriction of ϕ or φ to a one-dimensional subspace. D The gradient, Dϕ(x) e.g. stands for a row vector. ∇ The transpose of the gradient, ∇ϕ(x) e.g. stands for a column vector. H_{x} The Hessian D^{2}ϕ(x) of ϕ at x. H_{x,λ} The Hessian of φ( . , λ) at x. H_{x,μ} The Hessian of φ( . , μ) at x. log Always denotes the natural logarithm with basis e.

Chapter 1

Motivation

In this Chapter we present a convex problem and several approaches for solving this problem. For each approach we point out some critical points that are crucial for its performance. These points will then be used in the next Chapter to derive the notion of self-concordance [63].

1.1 A convex problem and assumptions

Let a convex set S ⊂ IR^{n},

S = {x| r_{i}(x) ≤ 0 for 1 ≤ i ≤ m}  (1.1)

be given by m restrictions r_{i} : S → IR. We assume that S has a nonempty interior S^{o}, and that the constraint functions r_{i} are continuous on S and three times continuously differentiable on S^{o}. The problem under consideration is to minimize a linear objective function c^{T} x subject to the constraint x ∈ S, i.e. we search for x^{opt} ∈ S and λ^{opt} such that

c^{T} x^{opt} = λ^{opt} := min{c^{T} x| x ∈ S}.  (1.2)

(Note that a smooth convex ojective function can be rewritten as a linear objective function by adding one more constraint and one more variable.) Throughout we assume that the set S^{opt} of optimal solutions is nonempty and bounded. Next we briefly outline three primal methods for solving (1.2), and point out certain critical aspects of these methods.

1.2 The affine scaling method

Problem (1.2) is to minimize a linear objective function over some convex set S. Since the shape of the set S is not known one would like to approximate S by a simple set E. An obvious choice is an ellipsoid. In this case, E is given by its center x̂ and a positive definite matrix H,

E = {z| (z − x̂)^{T} H(z − x̂) ≤ 1}.

Since E is intended to approximate S we require x̂ ∈ S^{o}. Minimizing the objective function c^{T} x over E is “easily” possible if H and x̂ are given; it is equivalent to solving a set of linear equations involving H. Let us denote the objective value at the center by λ = c^{T} x̂, and the sub level set S ∩ {x| c^{T} x ≤ λ} by S(λ). If the ellipsoid E well approximates the set S(λ), the resulting point x′ := arg min{c^{T} x| x ∈ E} will be a useful approximation to x^{opt} = arg min{c^{T} x| x ∈ S}.

To guarantee that x′ ∈ S^{o} holds true one would like to construct E such that E ⊂ S^{o}. In this case we may choose x′ as the center of a new ellipsoid

E′ := {z| (z − x′)^{T} H′(z − x′) ≤ 1}

with some new positive definite matrix H′, and iterate the above process. An algorithm that results from this approach is the affine scaling algorithm originally due to Dikin (1967) [7], rediscovered in 1986 by Vanderbei, Meketon and Fredman [93], and carefully analyzed by Tsuchiya [91].

A “good” approximation of S(λ) by E can be guaranteed, for example, if there exists a concentric Ellipsoid Ê = {z| (z − x̂)^{T} H(z − x̂) ≤ r^{2}} such that S(λ) ⊂ Ê holds true, and the smaller the ratio r of inner ellipsoid E for S and outer ellipsoid Ê for S(λ), the better the approximation of S(λ) by E.

The existence of such ellipsoidal approximations will be a key issue in Chapter 2.

1.3 The logarithmic barrier approach

Let

ϕ(x) = − ∑_{i=1}^{m} log(−r_{i}(x))  (1.3)

be the logarithmic barrier function for the set S. We assume that the functions r_{i} are such that ϕ is convex. The solution of problem (1.2) can be approximated by the solutions of a sequence of unconstrained problems based on the barrier function ϕ, namely

min\{\frac{c^{T} x}{μ_{k}} + ϕ(x)\},

where μ_{k} > μ_{k+1} → 0. This idea leads to the SUMT algorithm by Fiacco and McCormick (1968) [11]. The efficiency of the SUMT method clearly depends on the efficiency by which the subproblems are solved and on the speed at which the solutions of the subproblems converge to the solution of (1.2). The subproblems are usually solved by some variant of Newton’s method (with line search), and thus, one of our main concerns regarding logarithmic barrier methods are the properties of Newton’s method when applied to minimize ϕ(x)—or more generally to minimize ϕ(x) + \frac{c^{T} x}{μ}. Our investigations in Chapter 2 will also address which considerations lead to the choice of the logarithmic barrier function rather than some other barrier function, see also [9].

1.4 The method of centers

A third method for solving (1.2) is the method of centers of Huard (1966), [26] which is best suited to point out the two crucial aspects when solving (1.2).

Let some λ > λ^{opt} be given (if some point x̂ ∈ S is known we may choose for example λ = 1 + c^{T} x̂). As before, we define

S(λ) := S ∩ {z| c^{T} z ≤ λ}  (1.4)

and consider the following barrier function for S(λ):

φ(x, λ) := −κ log(λ − c^{T} x) + ϕ(x)  (1.5)

for some fixed κ ≥ 1. The minimum of φ( . , λ) will be denoted by x(λ). (Such a minimizer x(λ) exists if S(λ) is bounded, i.e. if S^{opt} is nonempty and bounded, and it is unique if ϕ is strictly convex.) The simplest form of the method of centers proceeds in two steps.

Method of centers Initialization: Let some value λ = λ^{0} > λ^{opt} be given and some approximation x^{(0)} ∈ S^{o} to x(λ^{0}) with c^{T} x^{(0)} < λ^{0}. Set k = 0. Until some stopping criterion is satisfied repeat

1. Reduce λ to λ^{k+1} = \frac{1}{2}(λ^{k} + c^{T} x^{(k)}).

2. Approximate x(λ^{k+1}) by Newton’s method (for minimizing φ( . , λ^{k+1})) starting at x^{(k)}.

3. Set k = k + 1.

End.

The performance of this method depends on two crucial questions.

1. How well does Newton’s method perform when applied to minimizing φ( . , λ)?

2. How large is the “distance” \frac{1}{2}(λ^{k} − c^{T} x^{(k)}) by which λ^{k} is reduced compared to the “distance” c^{T} x^{(k)} − λ^{opt} of x^{(k)} to optimality?

It is intuitively clear that the method of centers will interesting if and only if both questions allow a satisfactory answer. These two questions will be used to motivate two forms of local Lipschitz continuity in the next Chapter.

Self-Concordance

We give an answer to the two crucial questions of the previous section by introducing the notion of self-concordance. Self-concordant functions are defined and examined in great detail by Nesterov and Nemirovsky in [62, 63], and while our derivation is different, most results presented in this Chapter are due to [63].

2.1 The first question, Newton’s method

In this section we state a condition that enables us to analyze some theoretical properties of Newton’s method for minimizing the barrier function ϕ for the set S. These results will be applied to φ( . , λ) of (1.5) and S(λ) later on. (Note that S(λ) has the same structure as S being given by the intersection of finitely many smooth contraints.)

2.1.1 Derivation of self-concordance

The self-concordance condition of this section can be derived by the following straightforward argument.

• Newton’s method for minimizing ϕ starting at x^{(k)} is based on an approximation to ϕ, the approximation having a constant Hessian H = D^{2}ϕ(x^{(k)}) (quadratic model).

• Intuitively it is clear that Newton’s method is “good” if the relative change of D^{2}ϕ(x) is small for small changes in x.

• The absolute change of D^{2}ϕ(x) is determined by D^{3}ϕ(x).

• Thus, D^{3}ϕ(x) should be small relative to D^{2}ϕ(x).

• Let us consider a simple example: ϕ(x) := − log x, the logarithmic barrier function

for the positive real axis. In this case, for x > 0, ϕ′′(x) = \frac{1}{x^{2}} and ϕ′′′(x) = −\frac{2}{x^{3}} .

The natural condition to bound ϕ′′′ relative to ϕ′′ is to require |ϕ′′′(x)| ≤ 2ϕ′′(x)^{3/2}. Of course, the constant “2” appears somewhat arbitrary, and also the exponent 3/2 needs further justification. But as we will see next, this choice of condition makes sense indeed.

• The generalization to n dimensions of the above condition results in the self-concor-dance condition given in [62]; for any x ∈ S^{o} ⊂ IR^{n} and any direction h ∈ IR^{n} we require

|D^{3}ϕ(x)[h, h, h]| ≤ 2(D^{2}ϕ(x)[h, h])^{3/2}

to hold true. From this formulation it becomes evident that the exponent 3/2 on the right hand side is natural in that it ensures independence of this relation of the norm of h.

• Note that the quantities involved in this relation are just the second and third directional derivatives of ϕ at x in direction h. Thus, the above relation can equivalently be rewritten as follows.

Definition 2.1 (Self-concordance) Let x ∈ S^{o} be some strictly feasible point for ϕ and let h ∈ IR^{n}, h ≠ 0 be some direction. Define the function f : I → IR by

f(t) = f_{x,h}(t) = ϕ(x + th).  (2.1)

The function f hence depends on the barrier function ϕ as well as on x and h. Here, I = {t| x + th ∈ S^{o}} is an open interval containing 0. The function ϕ is self-concordant if it is convex, three times continously differentiable, and if for any x ∈ S^{∘} and h ∈ IR^{n}, the function f satisfies

|f′′′(0)| ≤ 2f′′(0)^{3/2}.  (2.2)

Throughout we will assume that ϕ is a barrier function for S, that is for any point y ∈ ∂S on the boundary of S we assume that lim_{x→y, x∈S^{o}} ϕ(x) = ∞. (In [63] the barrier property is called “strong self-concordance”.)

Note that inequality (2.2) is not invariant under multiplication of f—respectively ϕ—by a positive constant. Condition (2.2) essentially requires that the supremum

sup_{x∈S^{o}, h∈IR^{n}} \frac{|f′′′_{x,h}(0)|}{(f′′_{x,h}(0))^{3/2}}

is finite, and that ϕ is multiplied by sufficiently large constant such that the supremum is less or equal to 2.

(Our definition 2.1 is in slight variation to the original definition in [63] who denote the above supremum by 2/\sqrt{α} and call ϕ self-concordant with parameter α. However, in [63] it is also assumed (without loss of generality!) that α = 1, so that the definitions are more or less the same.)

Before proving that our incentive—of finding some criterion which guarantees that Newton’s method for minimizing ϕ converges well—is indeed fulfilled by (2.2), we show that there are a number of functions that satisfy (2.2).

2.1.2 Some examples

In trying to construct functions that satisfy (2.2) let us start with the function “− log x” which of course satisfies (2.2).

• Summation. Let us observe first that condition (2.2) is closed with respect to summation, that is if ϕ_{1} and ϕ_{2} satisfy (2.2), then so does ϕ_{1,2} := ϕ_{1} + ϕ_{2} (as long as the intersection of the domains of ϕ_{1} and ϕ_{2} is not empty). Indeed, |ϕ′′′_{1} + ϕ′′′_{2}| ≤ |ϕ′′′_{1}| + |ϕ′′′_{2}| ≤ 2(ϕ′′_{1})^{3/2} + 2(ϕ′′_{2})^{3/2} ≤ 2(ϕ′′_{1} + ϕ′′_{2})^{3/2}. Here, we

denote by ϕ′′ the second directional derivative ϕ′′ := \frac{d^{2}}{dt^{2}} ϕ(x+th)|_{t=0}, and analogously for ϕ′′′. ⊓ ⊔

• Affine transformations. Similarly, (2.2) is invariant under affine transformations. Let A(x) := Ax + b be an affine mapping with some matrix A ∈ IR^{p×q} and some vector b ∈ IR^{p}. If ϕ( . ) : IR^{p} → IR satisfies (2.2) then so does ϕ(A( . )) : IR^{q} → IR (as long as there exists some x such that ϕ(Ax + b) is defined at all). Indeed,

\frac{d^{k}}{dt^{k}} ϕ(A(x + th)) = \frac{d^{k}}{dt^{k}} ϕ((Ax + b) + t(Ah)) = \frac{d^{k}}{dt^{k}} ϕ(x̃ + th̃)

with x̃ = Ax + b and h̃ = Ah. Thus, if ϕ satisfies (2.2) with f defined by x̃ and h̃, then so does ϕ(A( . )) with f defined by x and h. ⊓ ⊔

• Polyhedron. In particular, if we choose A = a_{i}^{T} and b = β_{i} in the previous observation we may conclude that − log(a_{i}^{T} x + β_{i}) satisfies (2.2), and by the above closedness under summation we may further conclude that

− ∑_{i=1}^{m} log(a_{i}^{T} x + β_{i})  (2.3)

is a self-concordant barrier function for the polyhedron {x| a_{i}^{T} x + β_{i} ≥ 0 for 1 ≤ i ≤ m} (if it has non-empty interior).

Let us give two further brief examples of self-concordant barrier functions.

• Convex quadratic constraints. First note that the logarithmic barrier function

− log(−q(x))  (2.4)

of the constraint q(x) ≤ 0 with a convex quadratic function q : IR^{n} → IR satisfies (2.2). Indeed, the restriction f(t) := − log(−q(x + th)) can be split into two linear parts (since q is convex and x is feasible): f(t) = − log(r_{1}t + s_{1}) − log(r_{2}t + s_{2}) with s_{1} > 0, s_{2} > 0, r_{1}, r_{2} ∈ IR, and these satisfy condition (2.2) as we have just seen. ⊓ ⊔

• Semidefiniteness constraint. The second example regards the logarithmic barrier function − log det X for the cone of positive definite matrices. Given a positive definite n × n-matrix X and a symmetric matrix Y, we may consider the restriction f of ϕ,

f(t) = − log det(X + tY).  (2.5)

In order to evaluate its derivatives we rewrite f as

f(t) = − log det(X^{1/2}(I + tX^{−1/2}Y X^{−1/2})X^{1/2})

= −2 log det X^{1/2} − log det(I + tX^{−1/2}Y X^{−1/2}) = − log det X − ∑_{i=1}^{n} log(1 + tλ_{i}),

where λ_{i} are the eigenvalues of X^{−1/2}Y X^{−1/2} (independent of t). For k ≥ 1 we thus obtain

f^{(k)}(t) = (−1)^{k}(k − 1)! ∑_{i=1}^{n}(\frac{λ_{i}}{1 + tλ_{i}} )^{k},  (2.6)

and conclude that

4f′′(0)^{3} = 4(∑_{i=1}^{n} λ_{i}^{2})^{3} = 4∥λ⃗∥_{2}^{6} ≥ 4∥λ⃗∥_{3}^{6} ≥ (2 ∑_{i=1}^{n} λ_{i}^{3})^{2} = f′′′(0)^{2},  (2.7)

that is (2.2) holds true. (Here, λ⃗ denotes the vector of all eigenvalues λ_{i}, and is not to be confused with the number λ in (1.4).) ⊓ ⊔

2.1.3 The H-norm

Our analysis of Newton’s method heavily depends on the choice of the norm in which the analysis is carried out. By convexity of ϕ, its Hessian H_{x} := D^{2}ϕ(x) is positive semidefinite,

and we may thus define a semi-norm ∥z∥_{H_{x}} := (z^{T} H_{x}z)^{1/2}. By our assumption on problem (1.2), the set of optimal solutions S^{opt} is bounded, hence S does not contain a straight line, and therefore ϕ is strictly convex^{1} by the observation following Lemma 2.2 below. Thus, ∥ . ∥_{H_{x}} is a norm—referred to as H-norm in the sequel—and as it will turn out, this norm is a natural and very suitable choice for our analysis. Indeed, it will turn out that the H-norm is closely related to the shape of the set S.

Lemma 2.2 (Inner ellipsoid) Assume that ϕ is a self-concordant barrier function and set H_{x} := D^{2}ϕ(x). Let x ∈ S^{o} and h ∈ IR^{n} be arbitrary. If δ := ∥h∥_{H_{x}} ≤ 1 then x + h ∈ S.

Simple proof: As in Definition 2.1 denote by f = f_{x,h} the restriction of ϕ to the line {x + th| t ∈ I}. We note that the differential inequality (2.2) is assumed to hold only at the argument t = 0, since this is easier to verify. However, since (2.2) is assumed to hold for any x ∈ S^{o} and h ∈ IR^{n} in (2.1), the more general inequality

|f′′′(t)| ≤ 2f′′(t)^{3/2}  (2.8)

in fact holds true for all t ∈ I. To prove the Lemma it suffices to show that the points ±δ^{−1} (±∞ if δ = 0) are in the domain of f or at its boundary. Here, δ^{2} = ∥h∥_{H_{x}}^{2} = f′′(0). We consider the function u(t) = f′′(t). Note that u(t) ≥ 0 ∀t ∈ I by convexity of f. By finding the poles of u for t ≥ 0 we may determine the domain of f. Let v be the “extremal” solution of the differential inequality u′ ≤ 2u^{3/2} in (2.8) with the same initial values as u (i.e. as f′′),

v′(t) = 2v(t)^{3/2}, v(0) = δ^{2}.

1If this assumption is violated and S does contain a straight line, then the null space of D^{2}ϕ(x) is nontrivial but independent of x, and straightforward modifications are possible to generalize the results of this Chapter.

We deduce that u(t) ≤ v(t). (Straightforward, by exploiting the differential inequality (2.8), see for example [40], Theorem 3.1, page 19.) Since v is given by v(t) = δ^{2}(δt − 1)^{−2} and has its pole at t = δ^{−1} the claim follows.

Observe that in the case that f′′(0) = 0 it follows from v(t) ≡ 0 that f′′(t) = 0 ∀t, i.e. the domain of f in (2.1) is I = IR, and S contains the straight line {x + th|t ∈ IR}.

Lemma 2.2 was first proved in [63], and simple examples are given there (e.g. the function f(t) = log t) that show that the bound t < δ^{−1} on the maximum feasible step length for f is tight. The above proof is taken from [32]. Note that only scalar inequalities (such as (2.2)) are needed to provide an inner ellipsoid in n-dimensional space.

The close relation of the H-norm to the shape of the feasible set S will become more apparent in Section 2.2 where it is shown that a small multiple of the inner ellipsoid is an outer ellpsoid for a certain subset of S, namely there exists a small number r > 1 such that for any x ∈ S^{o}, if δ = ∥h∥_{H_{x}} ≥ r then either x + h ∉ S or Dϕ(x)h < 0.

2.1.4 Relative Lipschitz condition

In [63], condition (2.2) is rewritten in an equivalent finite difference version,

∀x ∈ S^{∘} ∀h ∈ IR^{n} ∀Δx ∈ IR^{n} with δ := ∥Δx∥_{H_{x}} < 1 :

x + Δx ∈ S^{o} and

(1 − δ)∥h∥_{H_{x}} ≤ ∥h∥_{H_{x+Δx}} ≤ \frac{1}{1 − δ} ∥h∥_{H_{x}}.  (2.9)

Before proving that the finite difference version (2.9) follows from (2.2) we use (2.9) to derive a relative Lipschitz condition. By subtracting ∥h∥_{H_{x}} from (2.9) we obtain

−δ∥h∥_{H_{x}} ≤ ∥h∥_{H_{x+Δx}} − ∥h∥_{H_{x}} ≤ \frac{δ}{1 − δ} ∥h∥_{H_{x}},

which implies

|∥h∥_{H_{x+Δx}} − ∥h∥_{H_{x}} | ≤ \frac{δ}{1 − δ} ∥h∥_{H_{x}},

and hence for the squares of the norms

|h^{T} (∇^{2}ϕ(x + Δx) − ∇^{2}ϕ(x))h| ≤ δ \frac{2 + δ/(1 − δ)}{1 − δ} h^{T} (∇^{2}ϕ(x))h.  (2.10)

which may be regarded as a relative Lipschitz condition on the Hessian of ϕ similar to the one developed in [29]^{2}. This condition precisely states our original intention that the relative change of ∇^{2}ϕ be small if the change in its argument ∥Δx∥_{H_{x}} = δ is small. However, since condition (2.10) involves two directions Δx and h it seems less practical than the scalar condition (2.2) which can easily be checked.

By choosing h = Δx/δ it follows ∥h∥_{H_{x}} = 1, and when dividing both sides of (2.10) by δ and taking the limit as δ → 0 we obtain that |D^{3}ϕ(x)[h, h, h]| ≤ 2 which is just (2.2). The converse direction of showing (2.10) via (2.2) is slightly more difficult as (2.10) involves two vectors Δx and h while (2.2) is a scalar inequality. The following basic lemma due to [63] allows a proof of this implication:

2In [29], the term \frac{2+δ/(1−δ)}{1−δ} is replaced by some constant M, and δ is restricted to δ ≤ \frac{1}{2} . Clearly, this modification is equivalent (up to a constant factor) to (2.10). In [29], however, the condition was not applied to the logarithmic barrier function ϕ = − ∑ log(−r_{i}( . )), but in a modified form to the constraint functions r_{i}. The resulting condition implies (2.10), but as shown in [30], the converse is not true in general.

Lemma 2.3 (Spectral Radius for Symmetric Trilinear Forms) Let M : IR^{n} ×IR^{n} × IR^{n} → IR be a symmetric homogenous trilinear form, let A : IR^{n} × IR^{n} → IR be a symmetric bilinear form, and let further μ > 0 be a scalar such that

M[h, h, h]^{2} ≤ μA[h, h]^{3}, ∀h ∈ IR^{n},

then

M[x, y, z]^{2} ≤ μA[x, x]A[y, y]A[z, z], ∀x, y, z ∈ IR^{n}.  (2.11)

The proof of (2.11) is given in Chapter 8.

Proof of (2.9) by (2.2): (The outline of this proof is due to [64].)

Let a self-concordant function ϕ, a point x ∈ S^{o}, the gradient Dϕ(x), the Hessian matrix D^{2}ϕ(x) = H_{x}, an arbitrary vector Δx ∈ IR^{n} with δ = ∥Δx∥_{H_{x}} < 1 and an arbitrary vector h ∈ IR^{n} be given. From Lemma 2.2 follows that that x + Δx ∈ S^{o}.

To evaluate how the H-norm of the vectors Δx and h changes for different matrices H_{x+tΔx}, with t ∈ [0, 1], let us define

u(t) := ∥Δx∥_{H_{x+tΔx}}^{2} = Δx^{T} D^{2}ϕ(x + tΔx)Δx ≥ 0

and

v(t) := ∥h∥_{H_{x+tΔx}}^{2} = h^{T} D^{2}ϕ(x + tΔx)h ≥ 0.

Note that u(t) = f′′_{x,Δx}(t) is as in the proof of Lemma 2.2. From the proof of Lemma 2.2 we thus obtain that

0 ≤ u(t) ≤ \frac{δ^{2}}{(δt − 1)^{2}} .  (2.12)

Relation (2.9) can be expressed in terms of v, i.e. we have to show:

(1 − δ)\sqrt{v(0)} ≤ \sqrt{v(1)} ≤ \frac{1}{1 − δ} \sqrt{v(0)}.

The change of v can be estimated by its derivative v′(t) using the estimate (2.11). From self-concordance of ϕ it follows with (2.11) that

|D^{3}ϕ(x)[h_{1}, h_{2}, h_{3}]| ≤ 2\sqrt{D^{2}ϕ(x)[h_{1}, h_{1}] D^{2}ϕ(x)[h_{2}, h_{2}] D^{2}ϕ(x)[h_{3}, h_{3}]},

and in particular,

|v′(t)| = D^{3}ϕ(x + tΔx)[Δx, h, h]
≤ 2\sqrt{Δx^{T} D^{2}ϕ(x + tΔx)Δx} h^{T} D^{2}ϕ(x + tΔx)h = 2u(t)^{1/2}v(t).

Inserting (2.12) in this inequality we obtain a differential inequality,

|v′(t)| ≤ \frac{2δ}{1 − tδ} v(t).

The “extremal” solutions ν of this differential inequality satisfying

ν′(t) = ±\frac{2δ}{1 − tδ} ν(t), ν(0) = v(0),

are given by ν(t) = v(0)(1−tδ)^{∓2}, and thus (again by the comparison theorem for differential inequalities [40])

v(0)(1 − tδ)^{2} ≤ v(t) ≤ \frac{v(0)}{(1 − tδ)^{2}}

which is just relation (2.9) (after taking square roots).

2.1.5 Newton’s method

This Section is concerned with answering the first crucial question (about Newton’s method for minimizing ϕ). First note that ϕ has a unique minimum x^{∗} if S is bounded. The minimum is called analytic center of S (Sonnevend [82]). This notation is commonly used but somewhat misleading since the analytic center does not depend on the set of points S, but rather on the barrier function ϕ describing S.

Next we show that self-concordance of ϕ further implies that Newton’s method converges “well” when applied to minimize ϕ. Note that a linear perturbation of ϕ does not influence the self-concordance condition (2.2), so that our results below can also be applied to the unconstrained minimization problem

min\{\frac{c^{T} x}{μ} + ϕ(x)| x ∈ S^{∘}\}.  (2.13)

For simplicity of presentation we disregard the linear term (i.e. set μ = ∞), and for x ∈ S^{∘} let

Δx := −H_{x}^{−1}∇ϕ(x)  (2.14)

denote the Newton step for obtaining the next Newton iterate x̃ := x + Δx. The following Lemma due to [63] states the main result about Newton’s method.

Lemma 2.4 (Newton) If ϕ is a self-concordant barrier function for S and x ∈ S^{o} such that ∥Δx∥_{H_{x}} < 1, then ϕ has a minimum x^{∗} ∈ S^{o}. Further, the next Newton iterate x̃ satisfies x̃ ∈ S^{∘}, and with Δx̃ denoting the next Newton step starting at x̃, we have

∥Δx̃∥_{Hx̃} ≤ \frac{∥Δx∥_{H_{x}}^{2}}{(1 − ∥Δx∥_{H_{x}})^{2}} .  (2.15)

For ∥Δx∥_{H_{x}} ≤ 1/4 this implies that Newton’s method (without line search) is guaranteed to be quadratically convergent with constant at most \frac{16}{9}.

Proof: In [63] it is shown that ϕ has a minimum. We only prove (2.15) and follow the outline of [63].

Define x(s) := x+sΔx for s ∈ [0, 1] where Δx = −D^{2}ϕ(x)^{−1}Dϕ(x)^{T} is the Newton step starting at x to minimize ϕ. By (2.9), x(s) ∈ S^{o} for all s ∈ [0, 1] and

|z^{T} (D^{2}ϕ(x) − D^{2}ϕ(x(s)))z| ≤ (\frac{1}{(1 − s∥Δx∥_{H_{x}})^{2}} − 1) z^{T} D^{2}ϕ(x)z.

Using the generalized Cauchy-Schwarz inequality (8.1) and defining δ := ∥Δx∥_{H_{x}} we obtain for z ∈ IR^{n}

|\frac{d}{ds}Dϕ(x(s))z − Δx^{T} D^{2}ϕ(x)z| = |Δx^{T} (D^{2}ϕ(x(s)) − D^{2}ϕ(x))z|

≤ (\frac{1}{(1 − sδ)^{2}} − 1) \sqrt{Δx^{T} D^{2}ϕ(x)Δx}\sqrt{z^{T} D^{2}ϕ(x)z} = (\frac{1}{(1 − sδ)^{2}} − 1) δ∥z∥_{H_{x}}.

The left-hand side is the absolute value of the derivative K′(s) where K is defined K(s) := Dϕ(x(s))z − (1 − s)Dϕ(x)z. By integration (K(0) = 0!) we can thus bound

|K(s)| ≤ ∥z∥_{H_{x}}δ ∫_{0}^{s} \frac{1}{(1 − tδ)^{2}} − 1 dt = ∥z∥_{H_{x}} \frac{s^{2}δ^{2}}{1 − sδ} .

For s = 1, x(1) = x + Δx this implies

|K(1)| = |Dϕ(x + Δx)z| ≤ \frac{δ^{2}}{1 − δ} ∥z∥_{H_{x}}.

Choosing z = Δx̃ = −D^{2}ϕ(x + Δx)^{−1}Dϕ(x + Δx)^{T} as the next Newton step we obtain

∥Δx̃∥_{H_{x+Δx}}^{2} = |Dϕ(x + Δx)Δx̃| ≤ \frac{δ^{2}}{1 − δ} ∥Δx̃∥_{H_{x}} ≤ \frac{δ^{2}}{(1 − δ)^{2}} ∥Δx̃∥_{H_{x+Δx}},

the last inequality following from (2.9). Now the claim follows when dividing the last line by ∥Δx̃∥_{H_{x+Δx}}.

An immediate consequence of (2.9) and (2.15) is the following corollary:

Corollary 2.5 Under the assumptions of Lemma 2.4 let x^{∗} be the minimum of (2.13). Newton’s method starting at x is quadratically convergent if x ∈ x^{∗} + \frac{1}{4} · E(x^{∗}), where E(x^{∗}) = {z| z^{T} H_{x^{∗}}z ≤ 1}.

The importance of these results on Newton’s method is that they do not depend at all on μ in (2.13) or on the data of the problem (1.2)—(as long as ϕ is self-concordant). Furthermore, for μ = ∞, the above corollary implies that the domain of quadratic convergence is one quarter of the inner ellipsoid of S which in turn is a fixed fraction of the outer ellipsoid of S as we will see in Section 2.2.

Without proof we further quote the following result of [63]. If ∥Δx∥_{H_{x}} < 1/3 then the distance of x to x^{∗} is bounded as follows:

∥x − x^{∗}∥_{H_{x}} ≤ 1 − (1 − 3∥Δx∥_{H_{x}})^{1/3} (≈ ∥Δx∥_{H_{x}}).  (2.16)

The previous results depend on the H-norm of the Newton step for minimizing ϕ starting at x. Clearly, the H-norm is a continous function of x, and as will be shown in Section 2.2, it is uniformly bounded for the examples considered here.

Nevertheless, the requirement that the starting point x of Newton’s method be close to the center in the sense that ∥Δx∥_{H_{x}} < 1 can often not be satisfied in practice. For ∥Δx∥_{H_{x}} > 1, the result x + Δx of the Newton step may lie outside S. The following simple step length rule for a line-search, however, seems to be an apparently favourably damped version of Newton’s method ([33]).

Let a search direction Δx be given with Dϕ(x)Δx < 0, for example Δx may be the Newton direction (2.14) for ϕ. We are interested in a search step s such that ϕ(x + sΔx) is as small as possible. (Line-search problem.) For this purpose define f(t) = ϕ(x+tΔx) as in (2.1) and denote the minimizer of f by t^{∗} and the Newton step for f by Δt = −f′(0)/f′′(0). A possible rule for the search step s referred to as the reduced Newton step for f and for which [63] proved global convergence is the following:

s = \frac{Δt}{1 + |Δt|\sqrt{f′′(0)}}.  (2.17)

(Note that the Newton step for f refers to a one-dimensional Newton’s method that is not to be confused with the n-dimensional Newton direction for ϕ.)

Lemma 2.6 Assume the barrier function ϕ for S is self-concordant. A line-search for ϕ starting at x in direction Δx using the reduced Newton step s (2.17) is monotonically convergent, i.e. s is “a little too short”, s ≤ t^{∗}. Further, s is the largest step possible that is guaranteed to satisfy s ≤ t^{∗} if all that is known are the first two derivatives of f at the current point t = 0.

Proof: We first prove monotonicity of the line-search. Let a = f′(0) < 0, b = f′′(0) > 0, and t^{∗} be the zero of g(t) := f′(t). We show that s ≤ t^{∗} holds true by writing the differential inequality (2.2) (self-concordance of f) in terms of the function g = f′. We obtain

g(0) = a,  g′(0) = b,  g′′(t) ≤ 2g′(t)^{3/2}.  (2.18)

The “extremal” solution ν of (2.18) that solves the initial value problem

ν(0) = a,  ν′(0) = b,  ν′′(t) = 2ν′(t)^{3/2}

is an upper bound for g: Since g′ satisfies a first-order initial value inequality it follows that g′ ≤ ν′ (see, for example, [40], Theorem 3.1), and this implies that g ≤ ν. The function ν is given by ν(t) = (b^{−1/2} − t)^{−1} + a − b^{1/2} and has the (unique) zero s = ab^{−1}/(1 + ab^{−1/2}). By construction of ν, we may conclude that s ≤ t^{∗}.

On the other hand, since g = ν is possible, any step larger than s may exceed t^{∗}.

Observe that if Δx is the Newton direction (2.14) of ϕ at x, then the reduced Newton step in direction Δx simply yields Δx/(1 + ∥Δx∥_{H_{x}}).

2.2 The second question, outer ellipsoidal approximation

The results of the previous section state that Newton’s method for minimizing a selfconcordant function converges quadratically in “one quarter of the inner ellipsoid around the analytic center”. This result looks nice, but it is meaningful only if the inner ellipsoid is sufficiently large; for example if a small multiple of the inner ellipsoid forms an outer ellipsoid for S.

Similarly, we may link the second crucial question (about the “distance” c^{T} x(λ) − λ^{opt}) to ellipsoidal approximations of S(λ) around the analytic centers x(λ). If φ( . , λ) is selfconcordant for fixed λ the results of the previous Chapter imply that

E(λ) := {z| z^{T} H_{x,λ}z ≤ 1}

is an inner ellipsoid for S(λ) in the sense that x(λ) + E(λ) ⊂ S(λ). (Here, by H_{x,λ} we denote the Hessian D_{x}^{2}φ(x, λ) of φ( . , λ).) If there exists a small number r > 1 such that rE(λ) yields an outer ellipsoid for S(λ), i.e. S(λ) ⊂ x(λ) + rE(λ), then we may conclude that

λ − c^{T} x(λ) ≥ \frac{1}{r}(c^{T} x(λ) − λ^{opt}),

and thus answer the second question.

It turns out that a second property of the log function is needed to allow such ellipsoidal approximations. (Self-concordance by itself is not sufficient—as can easily be seen from the example ϕ(x) := − log x − σ log(1 − x) : (0, 1) → IR; this function is self-concordant for σ ≥ 1, the inner ellipsoids around the minimizer \frac{1}{1+σ}, however, become smaller and smaller as σ → ∞.)

A naive derivation of a second property needed to provide outer ellipsoids can again be obtained from the function “− log x”. To prevent the minimum of ϕ from being close to the boundary of the domain of ϕ one might impose a condition that bounds the growth of ϕ (i.e. that bounds Dϕ(x))—relative to the canonical norm ∥ . ∥_{H_{x}}. For the function ϕ(x) := − log x : IR_{+} → IR we may observe that

ϕ′(x)^{2} ≤ ϕ′′(x) ∀x > 0

holds true. More generally we assume a second differential property of ϕ:

Definition 2.7 (Self-limitation) Let the function f be defined as in (2.1), and a constant parameter θ ≥ 1 be given. The barrier function ϕ : S ⊂ IR^{n} → IR is called θ-self-concordant barrier function if f satisfies (2.2) and

|f′(0)| ≤ \sqrt{θ}f′′(0)^{1/2} ∀x ∈ S^{∘}, h ∈ IR^{n}  (2.19)

for all x ∈ S^{o} and all h ∈ IR^{n}.

The number \sqrt{θ} may be interpreted as a local Lipschitz constant for ϕ (or f), where the change in the argument x is measured in the H-norm.

This definition coincides with the one given in [64]. In some cases however, we need to refer to property (2.19) independently of (2.2). We therefore allocate an extra name and refer to (2.19) (without assuming that (2.2) holds) as θ-self-limiting.

Note that just as (2.2), also (2.19) is assumed to hold only for the argument t = 0 since this is easier to verify. However, as before, (2.19) in fact holds true for all t ∈ I.

Note further that (in contrast to (2.2)) condition (2.19) is not invariant when adding a linear perturbation \frac{c^{T} x}{μ} to ϕ. (This perturbation is used in the logarithmic barrier approach.)

2.2.1 Equivalent formulations

As before for (2.2), there are also other equivalent formulations for (2.19).

The first reformulation of (2.19) is the requirement that the function Ψ : S^{o} → IR defined by Ψ(x) := e^{−ϕ(x)/θ} is concave.

(The proof that this is an equivalent condition is trivial.) Note that Ψ(x) > 0 for x ∈ S^{o}, and Ψ can be extended contiuously to the boundary of S by setting Ψ(x) = 0 for x ∈ ∂S. The resulting function is related to the multiplicative barrier function in [27].

The Newton step allows another equivalent formulation of the condition (2.19), namely the condition that the Newton step (2.14) is to satisfy

∥Δx∥_{H_{x}} ≤ \sqrt{θ} ∀x ∈ S^{∘}.  (2.20)

The derivation of this formulation is also straightforward, e.g. by applying the KKT-Theorem to max{Dϕ(x)h| h^{T} D^{2}ϕ(x)h ≤ 1}. Observe that this formulation uses our assumption that ϕ is strictly convex, i.e. that the Newton step exists at all, while the previous two formulations are slightly more general.

Condition (2.20) is remarkable, as for ∥Δx∥_{H_{x}} < 1 Lemma 2.4 about Newton’s method is applicable.

2.2.2 Some examples

Let us briefly verify that the above examples (2.3) – (2.5) satisfy (2.19) as well.

• Affine transformations. We begin with the invariance of property (2.19) with respect to affine transformations and observe as for (2.2) that if A(x) := Ax + b is an affine mapping with some matrix A ∈ IR^{p×q} and some vector b ∈ IR^{p}, then with ϕ( . ) : IR^{p} → IR, also ϕ(A( . )) : IR^{q} → IR satisfies (2.19) with the same parameter θ (as long as there exists some x such that ϕ(Ax + b) is defined at all). (The proof of this statement is exactly as for (2.2).)

• Summation. Similarly, we observe that if ϕ_{1}, ϕ_{2} satisfy (2.19) for some self-concor-dance parameters θ_{1}, θ_{2}, then so does ϕ_{1,2} := ϕ_{1} +ϕ_{2} with self-concordance parameter θ_{1,2} = θ_{1} + θ_{2} (as long as the intersection of the domains of ϕ_{1} and ϕ_{2} is not empty). (Straightforward)

• Linear, quadratic or semi-definiteness constraints. The proofs for (2.3) and (2.5) can easily be modified to show the following results. The logarithmic barrier function of a linear constraint is a θ = 1-self-concordant barrier function, the logarithmic barrier function − log X of a symmetric positive definite n × n matrix X is θ = n-self-concordant. Moreover it is straightforward to see that the logarithmic barrier function − log(−r(x)) of any convex constraint function r(x) ≤ 0 is θ = 1-self-limiting (but not necessarily self-concordant) as long as there exists some x for which r(x) < 0. In particular, the logarithm of a convex quadratic constraint is θ = 1-self-concordant.

2.2.3 Properties

We note here that for θ < 1 there is no solution that satisfies both, (2.2) and (2.19) except for constant functions. (This is proved in Lemma 2.12 below.)

One of our main concerns in our derivation of (2.19) was the desire for an outer ellipsoid. The following lemma shows that (2.19) indeed provides such an ellipsoid.

Lemma 2.8 (Outer ellipsoid) Let ϕ be a θ-self-concordant barrier function for S and x^{∗} be the analytic center of S. Let h be some vector with δ = ∥h∥_{H_{x^{∗}}} > (θ + 2\sqrt{θ}). Then the points x^{∗} ± h are outside S.

Proof: We show that the points ±d(θ + 2\sqrt{θ}) are not feasible for f = f_{x^{∗},h} in (2.1), where d := 1/\sqrt{f′′(0)}. We consider the functions w(t) = f′(t) and u(t) = f′′(t). To determine the domain of f we investigate the poles of w for t ≥ 0. By (2.19), w is a solution of

w^{2}(t) ≤ θw′(t),  w(0) = 0,  w′(0) = f′′(0) > 0.  (2.21)

Because of the initial values, the inequality w^{2} ≤ θw′ is “inactive” near t = 0. For small values of t ≥ 0 we therefore apply inequality (2.2) again. Let

t̄:= d\sqrt{θ}.

If f is not defined at t̄ there is nothing to show. Hence we assume it is, and conclude analogously as in the proof of Lemma 2.2 that (2.2): w′′(t) ≥ −2w′(t)^{3/2} implies

w′(t) ≥ r(t) := (t + d)^{−2} for t ∈ [0, t̄]

(since r satisfies r′′(t) = −2r′(t)^{3/2} and has the same initial value at t = 0). With the variable t̃:= t − t̄ and w̃(t̃) := w(t̃+ t̄) = w(t) relation (2.21) implies

w̃(t̃)^{2} ≤ θw̃′(t̃) for t̃≥ 0.  (2.22)

We find that the initial values for (2.22) satisfy

w̃(0) = ∫ t̄ ; 0w′(τ) dτ ≥ ∫ t̄ ; 0r(τ) dτ = \frac{\sqrt{θ}}{d(1 + \sqrt{θ})} =: d_{1},

and

w̃′(0) = w′(t̄) ≥ r(t̄) = d^{−2}(1 + \sqrt{θ})^{−2} =: d_{2}.

Observe that d_{1}^{2} = θd_{2}. It follows that w̃(t̃) ≥ s(t̃), where s satisfies

s(t̃)^{2} = θs′(t̃) and s(0) = d_{1}.

The function s is given by s(t̃) = (d_{1}^{−1} − t̃/θ)^{−1}, and has its pole at t̃ = θd_{1}^{−1}. The corresponding value of t is t = t̄+ t̃= dθ(1 + 2/\sqrt{θ}). By construction, s(t̃) ≤ w(t), so that the pole of w and hence of f must lie before this point.

Note that in the proof of Lemma 2.8, the function r can be continued beyond the point t̄ in such a way that r = s′ for t > t̄. The second integral R of r is twice continuously differentiable, and satisfies the relations (2.2) and (2.19) almost everywhere (except at t̄). This shows that also the bound d(θ +2\sqrt{θ}) obtained from R cannot be improved for general self-concordant functions.

We point out that the proof of Lemma 2.8 may be generalized to yield an outer ellipsoid centered at other points x̃ ≠ x^{∗} if the corresponding function f satisfies df′(x̃) ≤ α < 1 independent of h. (In this case of course the constants will change.)

An immediate consequence of Lemma 2.2 and Lemma 2.8 is the following Theorem:

Theorem 2.9 Let ϕ be a θ-self-concordant barrier function for S and let the ellipsoid E(x) = {h| h^{T} D^{2}ϕ(x)h ≤ 1} be defined by the Hessian of ϕ at x. For all x ∈ S^{o} we have

x + E(x) ⊂ S,

and if the minimum x = x^{∗} of ϕ exists we further have

x^{∗} + (θ + 2\sqrt{θ})E(x^{∗}) ⊃ S.

A two-sided ellipsoidal approximation of this type is also proved in [64] with a slightly larger ratio of inner and outer ellipsoid.

It is clear that two-sided ellipsoidal approximations of S by concentric ellipsoids with a fixed similarity ratio r = θ + 2\sqrt{θ} are only possible at or near the analytic center x^{∗} of S. For arbitrary x ∈ S^{o} however, we can show the following new corollary relating to our question posed for the affine scaling method. By “cutting off” from S a suitable half-space H through the current point x, a small multiple of the inner ellipsoid E(x) yields an outer ellipsoid for the remaining part of S ∩ H.

Corollary 2.10 Let ϕ be a θ-self-concordant barrier function for S and let x ∈ S^{o} be arbitrary. Define H := {y| Dϕ(x)(y − x) ≥ 0} and E(x) := {h| h^{T} D^{2}ϕ(x)h ≤ 1}, then

x + E(x) ⊂ S,

and

x + (θ + 2\sqrt{θ})E(x) ⊃ S ∩ H.

Proof: The proof of Lemma 2.8 can be applied to the Corollary. ⊓ ⊔ As pointed out above, the two-sided ellipsoidal approximations proved above can be used

to find an estimate for our second question regarding the distance λ − c^{T} x(λ) compared to c^{T} x(λ) − λ^{opt}. However, for κ > 1 in (1.5) the resulting answer is not optimal. Below we list a stronger estimate taken from [16]. (The result in [16] is slightly more general than what is needed to answer the second “crucial question” in Chapter 1.4.)

Lemma 2.11 Let c : S ↦ IR be a continuously differentiable convex function such that the set

S_{c} := { x ∈ S | c(x) ≤ 0 }

is bounded and has a nonempty interior S_{c}^{∘}. (Thus, c(x) = c^{T} x − λ and S_{c} = S(λ) in the case of (1.4), (1.5).) Let ϕ be a θ-selfconcordant barrier function associated with S, let κ ≥ 1 be a constant, and let x̂ ∈ S_{c} be the unique optimal solution of

min_{x∈S_{c}} ϕ(x) − κ log(−c(x)).  (2.23)

Then :

(1 + \frac{θ}{κ}) c(x̂) ≤ c(x) ≤ 0 for all x ∈ S_{c}.  (2.24)

Proof. First, we recall that ϕ is strictly convex on the interior S^{∘} of S, and that ϕ = ∞ on the boundary of S. Clearly, the function in (2.23) has the same properties on the set S_{c}, and this guarantees that (2.23) has a unique optimal solution x̂ and x̂ ∈ S_{c}^{∘}. Let x ∈ S_{c} be arbitrary, but fixed, and set h := x − x̂. Since c is convex, it follows that

c(x̂) − c(x) = c(x̂) − c(x̂ + h) ≤ −h^{T} ∇c(x̂).  (2.25)

We will prove that

μ ≤ θ, where μ := κ\frac{−h^{T} ∇c(x̂)}{−c(x̂)}.  (2.26)

Combining (2.25) and (2.26) then gives the desired result (2.24).

If μ ≤ 0, then (2.26) is trivially satisfied. Assume that μ > 0. Let ψ(β) = ϕ(x̂ + βh), ψ : I → IR, where I ⊃ [0, 1) is an open interval such that x̂ + βh ∈ S_{c}^{∘} for all β ∈ I. In view of (2.19), we have

ψ′′(β) ≥ \frac{(ψ′(β))^{2}}{θ} for all β ∈ I.  (2.27)

Moreover, since x̂ is the optimal solution of (2.23), it follows that ψ′(0) = μ. Using the monotonicity of the differential inequality (2.27) (see, e.g., [40, Theorem 3.1]) and the observation that the function

χ(β) := (\frac{1}{μ} − \frac{β}{θ} )^{−1}

satisfies χ(0) = ψ′(0) and χ′(β) = χ^{2}(β)/θ, we conclude that

ψ′(β) ≥ χ(β).

Clearly, χ has its pole at β = θ/μ > 0, implying that ψ′ and hence also ψ has a pole at some β_{0} with 0 < β_{0} ≤ θ/μ. Since ψ has no pole in the interval I ⊃ [0, 1), it follows that 1 ≤ β_{0} ≤ θ/μ. This is just the claim (2.26).

We now direct our effort to proving for which types of constraints the logarithmic barrier function (1.3) is “optimal”.

2.2.4 Optimal Barriers

The previous section illustrated the importance of self-concordance of a barrier with a small parameter θ. Here, we are interested in finding the “best” (with minimal θ) barrier of a convex set S.

In particular, we would like to find barrier functions that have a self-concordance parameter θ lower than the straightforward barrier functions defined in the previous sections. A very interesting theoretical result is the following due to [64].

For any closed convex set S ⊂ IR^{n} with nonempty interior there exists a universal θ = O(n)-self-concordant barrier function. If S does not contain a straight line, the universal barrier function is given by

const log |{h| h^{T} (y − x) ≤ 1 ∀ y ∈ S}| ,

where | . | is the Lebesque measure in IR^{n}, and const is some positive number independent of n. In most cases, this barrier function is not practical, and its evaluation is much more costly than the solution of the original optimization problem (1.2). In some cases however, we may evaluate the universal barrier explicitly; for the positive orthant of IR^{2} for example, it is straightforward to derive that the (above) universal barrier is given by − log(x_{1}) − log(x_{2}), i.e. we obtain the standard logarithmic barrier.

One might suspect that − log(x_{1}) − log(x_{2}) is “optimal” (with respect to θ) for this set, and indeed if there was a better barrier function ϕ^{∗}, say with self-concordance parameter θ < 2, then for x ∈ IR_{+}^{2n} one could construct the barrier function

ϕ^{∗∗}(x) := ∑_{i=1}^{n} ϕ^{∗}(( \begin{matrix}x_{i} \\ x_{i+1} \end{matrix} ))

with self-concordance parameter nθ. For large n it follows that nθ + 2\sqrt{nθ} < 2n − 1, and this contradicts the ellipsoidal approximation of inner and outer ellipsoid by a ratio of 1 : nθ + 2\sqrt{nθ} (since by preapplying the affine mapping A : IR^{2n−1} → IR^{2n} with

y → ( \begin{matrix}y \\ 1 − ∑_{i=1}^{2n−1} y_{i} \end{matrix} ) ,

the function ϕ^{∗∗}(A( . )) becomes a nθ-self-concordant barrier for the 2n − 1-dimensional simplex {y_{i} ≥ 0, ∑_{i=1}^{2n−1} y_{i} ≤ 1}, and inner and outer ellipsoid cannot approximate the simplex in IR^{2n−1} with a ratio that is better than 1 : 2n − 1).

With a similar consideration one can show the following Lemma.

Lemma 2.12 Any self-concordant barrier function for a convex set S has self-concordance parameter at least k if there exists an affine subspace U such that S ∩ U contains a vertex at which precisely k linearly independent smooth constraints are active.

Thus, “− log det X” is optimal as one might choose the linear subspace U that fixes all off-diagonal elements of a matrix X to zero; of course, the diagonal elements X_{ii}, of a positive definite diagonal matrix X must satisfy X_{ii} > 0 so that there are precisely n linearly independent constraints active in U at X = 0.

A direct derivation of this lower bound for barrier functions is given in [64] for polyhedra, and that result is easily generalizable to the case of nonlinear constraints as well.

We used the quality of the ellipsoidal approximations to prove the lower bound on θ, and since this bound is sharp (as there exist barrier functions that attain this bound), our derivation also implies that asymptotically, for large θ, the ellipsoidal approximations of the sets S are optimal. Nevertheless, for special classes of barrier functions (like − ∑ log x_{i} or − log det X), the ratio of the ellipsoidal approximations can be improved to 1 : θ − 1, see e.g [82, 5].

Above we have seen that the logarithm of a single linear or convex quadratic constraint is optimal. Further, it is straightforward to verify that the logarithm of a single linear constraint is the unique—up to an additive constant—optimal barrier function. In contrast, a convex quadratic constraint q(x) ≤ 0 has more than one optimal barrier function; for example if x̄ = arg min q(x) exists (and q(x̄) < 0), then \frac{−1}{4q(x̄)}q(x)−log(−q(x)) is also a θ = 1self-concordant barrier function for the set {x| q(x) ≤ 0}. (The proof is straightforward.)

Finally we point out that the “optimal” self-concordance parameter θ is not a smooth function of the constraints. The set {x| ∥x∥_{2} ≤ 1, x_{1} ≤ 1 + ϵ} has a θ = 1-self-concordant barrier function for ϵ ≥ 0, and for −2 < ϵ < 0 this set has a vertex at which precisely two linearly independent constraints are active, so that θ ≥ 2 must hold.

Another aspect disregarded in this Section is the cost of evaluating ϕ and its derivatives which is an important issue when looking for an implementable barrier function.

2.2.5 Further examples of self-concordant functions

Here, we quote some further examples from [67] of convex domains that have “easily computable” self-concordant barrier functions.

Let α ≥ 1 be fixed and ζ be some function with |ζ′′′(t)| ≤ −3αζ′′(t)/t for all t > 0. (For example α = 1 and ζ(t) := t^{p} for some 0 < p < 1.) Then the function

−α^{2}(log x + log(ζ(x) − y))  (2.28)

is a θ = 2α^{2}-self-concordant barrier function for the set {(x, y)| x ≥ 0, y ≤ ζ(t)} ⊂ IR^{2}.

In particular (see also [8]), the two-dimensional sets {(x, y)| y ≥ e^{x}}, {(x, y)| y ≥ (x_{+})^{p}} for some p ≥ 1 (and x_{+} := max{x, 0}), {(x, y)| y ≥ x^{p}, x > 0} for some p ≤ −1, or {(x, y)| y ≥ x log x, x ≥ 0}

have the 2-self-concordant barrier functions − log(log(y) − x) − log y, − log(y^{1/p} − x) − log y, − log(x − y^{1/p}) − log y, respectively − log(y − x log(x)) − log x.

Another example is the set {(X, t) |X ∈ IR^{p×q}, t > 0, ∥X∥_{2} ≤ t} with the q + 1-self-concordant barrier function

− log det(t^{2}I − X^{T} X) + (q − 1) log t.  (2.29)

Here, I is the q×q identity matrix, and ∥ . ∥_{2} is the lub_{2} norm. (Note that by ∥X∥_{2} = ∥X^{T} ∥_{2} we may assume q < p.)

Compatibility

We conclude this Chapter with a criterion that allows us to treat some nonlinear convex objective function c : S → IR as a constraint by intruducing an n + 1-st variable x_{n+1}, and setting x_{n+1} − c(x) ≤ 0. The criterion guarantees that the resulting barrier function − log(x_{n+1} − c(x)) + ϕ(x) is a self-concordant barrier function for the set

S^{+} := {(x, x_{n+1})| x ∈ S, c(x) ≤ x_{n+1}}.

This result (taken from [8]) is a special case of a more general theory developed in [66], .

Proposition 2.13 Let ϕ be some θ-self-concordant barrier function for the set S ⊂ IR^{n}, and let c : S → IR be some convex function. For x ∈ S^{o} and h ∈ IR^{n} let

f_{1}(t) := ϕ(x + th) and f_{2}(x) := c(x + th).

The function c is called β-compatible with ϕ for some β ≥ 1 iff for all x and h as above,

f′′′_{2}(0) ≤ 3βf′′_{2}(0)\sqrt{f′′_{1}(0)}.

If c is β-compatible with ϕ then the function

−β^{2} log(x_{n+1} − c(x)) + β^{2}ϕ(x)

is a self-concordant barrier function for the set S^{+} with parameter β^{2}(1 + θ).

The proof of this proposition follows from Theorem 3.1 in [66] by setting Γ := S^{+}, G := IR_{+}(= {t ∈ IR| t ≥ 0}), F (t) := − log t, Π(x, x_{n+1}) := ϕ(x), and A(x, x_{n+1}) := x_{n+1} − c(x).

A basic algorithm

In this Chapter we show that the conditions identified in the previous Chapter are indeed sufficient to guarantee a polynomial rate of convergence of the method of centers and the logarithmic barrier method when certain parameters are suitably chosen.

3.1 A model algorithm with polynomial rate of convergence

We present a “polynomial” version of the method of centers for solving problem (1.2). Here, “polynomial” refers to the rate of convergence; the exact solution of problem (1.2), of course, is not computable in general.

Let ϕ be a θ-self-concordant barrier function for S. Under the assumptions of Chapter 1.1 let a point x^{(0)} ∈ S^{o} and some number λ^{0} > λ^{opt} be given such that

∥Δx^{(0)}∥_{H_{x^{(0)},λ^{0}}} = ∥Dφ(x^{(0)}, λ^{0})^{T} ∥_{H_{x^{(0)},λ^{0}}^{−1}} ≤ 1/5.

Here, Δx^{(0)} is the Newton step −D^{2}φ(x^{(0)}, λ^{0})^{−1}Dφ(x^{(0)}, λ^{0})^{T}, H_{x,λ} = D_{x}^{2}φ(x, λ), and φ is given as in (1.5) with κ = θ.

Algorithm 1

1. k := 0; σ := 1/(8\sqrt{θ}); ε := desired accuracy.

2. x^{(k+1)} := x^{(k)} + Δx^{(k)} where Δx^{(k)} := −D^{2}φ(x^{(k)}, λ^{k})^{−1}Dφ(x^{(k)}, λ^{k})^{T} .

3. If λ^{k} − c^{T} x^{(k+1)} ≤ \frac{13}{15}ε stop, else

4. λ^{k+1} := λ^{k} − σ(λ^{k} − c^{T} x^{(k+1)}).

5. k := k + 1; go to 2.

Convergence Analysis We prove feasibility and convergence of the objective function value c^{T} x^{(k)} to the optimal value. By induction, we assume that x^{(k)} is strictly feasible, x^{(k)} ∈ S^{o}, and satisfies

∥Dφ(x^{(k)}, λ^{k})^{T} ∥_{H_{x^{(k)},λ^{k}}^{−1}} ≤ 1/5.

(This means that the iterates remain “close” to the centers x(λ^{k}).) We analyze the algorithm step by step.

• Step 2: By Lemma (2.4), the result x^{(k+1)} of step 2. satisfies ∥Dφ(x^{(k+1)}, λ^{k})∥_{H_{x^{(k+1)},λ^{k}}^{−1}} ≤ 1/16.

• Step 3: By (2.16) and (2.9), and the above result we may conclude that

∥x^{(k+1)} − x(λ^{k})∥_{H_{x(λ^{k}),λ^{k}}} ≤ 1/14  (3.1)

i.e. x^{(k+1)} lies in 1/14 of the inner ellipse around x(λ^{k}). For the center x(λ^{k}) we can apply Lemma 2.11 and obtain λ^{k} − c^{T} x(λ^{k}) ≥ c^{T} x(λ^{k}) − λ^{opt}. By combining these two estimates we conclude

λ^{k} − c^{T} x^{(k+1)} ≥ \frac{14 − 1}{14 + 1}(c^{T} x^{(k+1)} − λ^{opt}) = \frac{13}{15}(c^{T} x^{(k+1)} − λ^{opt}).  (3.2)

Therefore the stopping criterion in step 3 guarantees c^{T} x^{(K)} − λ^{opt} < ε if K is the index k at which the method terminates.

• Step 4: Relation (3.2) implies that the gap λ^{k} − λ^{opt} between the upper bound λ^{k} for c^{T} x^{(k)} and the (unknown) optimal value λ^{opt} is reduced by a factor of at least \frac{13}{28}σ in step 4. To complete our induction we verify that after the update of λ^{k+1} in step 4, the iterate x^{(k+1)} again satisfies

∥Dφ(x^{(k+1)}, λ^{k+1})^{T} ∥_{H_{x^{(k+1)},λ^{k+1}}^{−1}} ≤ 1/5.  (3.3)

From the definition of the Hessian H_{x,λ} = D^{2}ϕ(x) + κ\frac{cc^{T}}{(λ−c^{T} x)^{2}} follows that H_{x^{(k+1)},λ^{k+1}} ≥ H_{x^{(k+1)},λ^{k}}, and therefore the inverses satisfy H_{x^{(k+1)},λ^{k+1}}^{−1} ≤ H_{x^{(k+1)},λ^{k}}^{−1}. (Here, two symmetric matrices A, B satisfy A ≥ B iff A − B is positive semidefinite.) We may thus neglect the change in H caused by the update of λ^{k+1} and restrict ourselves to bound the change in Dφ. The change of Dφ is given by

∥Dφ(x^{(k+1)}, λ^{k})^{T} − Dφ(x^{(k+1)}, λ^{k+1})^{T} ∥_{H_{x^{(k+1)},λ^{k+1}}^{−1}}
= ∥σc/(λ^{k+1} − c^{T} x^{(k+1)})∥_{H_{x^{(k+1)},λ^{k+1}}^{−1}} ≤ 1/8 < 1/5 − 1/14.

The first inequality is an immediate consequence of the Sherman Morrison update formula for inverse matrices applied to H = D^{2}ϕ+θ \frac{cc^{T}}{(λ−c^{T} x)^{2}} , and the second inequality implies (3.3). ⊓ ⊔

For the above choice of κ = θ and σ = 1/(8\sqrt{θ}) these relations imply that at each iteration the distance λ^{k} − λ^{opt} is reduced by 13/(28 · 8\sqrt{θ}), and from this it is straightforward to derive that the number K of iterations is bounded by

K ≤ 17\sqrt{θ} log(\frac{λ^{0} − λ^{opt}}{ϵ}).

Each iteration involves the computation of the functions r_{i}, their first and second derivatives as well as the solution of a linear system in IR^{n}.

3.2 Phase 1

The above algorithm assumes that an initial point close to the center of some sub level set S(λ) is given. For linear programs and certain classes of linear complementarity problems it is well-known that this assumption is unnecessary and that infeasible interior-point methods are possible as well.

For general (nonlinear) convex programs with self-concordant barriers however, the situation is more difficult than in the case of linear programs. A restriction r_{i} of (1.2) may be convex in the interior of S and undefined or nonconvex outside S (while linear functions of course are also linear outside S). Elegant approaches that combine phase 1 of finding a feasible starting point and phase 2 of solving the actual minimization problem (1.2) are therefore more difficult to devise in general, and may depend on the particular form of a given problem.

An approach commonly used for linear programs is to relax a constraint of the form r_{i}(x) = a_{i}^{T} x − b_{i} ≤ 0 to r_{i}(x) ≤ tβ_{i} for some fixed β_{i} ≥ 0, such that a given initial point x^{(0)} is strictly feasible for the relaxed constraint with t = 1, i.e. “− log(tβ_{i} + r_{i}(x^{(0)}))” is well defined. During the course of the algorithm the duality gap and the infeasibility t simultaneously tend to zero. It is clear that this approach needs to be modified, for example, if r_{i} has a singularity somewhere in {x| r_{i}(x) ≥ 0}. Another example is the function “− log det X” which is self-concordant, but the function “− log(tβ+det X)” is not for tβ > 0. Even for linear programs the analysis of infeasible interior point methods is a difficult topic; the rate of convergence of most infeasible interior-point methods is worse (namely O(m)) than for feasible methods (which is O(\sqrt{m})), see e.g. [15, 41, 54, 55, 72, 73, 90, 98].

Under the assumption that S^{opt} is bounded and that an arbitrary point x̂^{(0)} ∈ S^{o} is available, we briefly sketch a phase I to find a point that satisfies the initial conditions of the previous algorithm

Set

ϕ̂(x) := ϕ(x) − κ log(λ̄ − c^{T} x), Ĥ_{x} := D^{2}ϕ̂(x),

where λ̄ = c^{T} x̂^{(0)} + d^{(0)}. Here, d^{(0)} may be set to 1, (or, more appropriately to d^{(0)} := min{1, ∥c∥_{D^{2}ϕ(x̂^{(0)})^{−1}}/∥Dϕ(x̂^{(0)})∥_{D^{2}ϕ(x̂^{(0)})^{−1}}}/(5κ)). Observe that ϕ̂(x) is equal to φ(x, λ̄). Set ĉ:= −Dϕ̂(x̂^{(0)}), θ̂:= 2θ, κ̂ := θ̂, and

φ̂(x, λ̂) := ϕ̂(x) − κ̂ log(λ̂ − ĉ^{T} x), Ĥ_{x,λ̂} := D^{2}φ(x, λ̂).

The new “objective function” ĉ^{T} x is simply chosen to yield a zero gradient of φ̂ at x̂^{(0)} and λ̂^{0} := κ̂ + ĉ^{T} x̂^{(0)}. To render the objective constraint “ĉ^{T} x̂ ≤ λ̂” inactive we may run the previous model algorithm in “opposite direction”(i.e. increase λ̂ slowly).

Phase I

1. k := 0; σ := 1/(12\sqrt{θ̂});

2. x̂^{(k+1)} := x̂^{(k)} + Δx̂^{(k)} where Δx̂^{(k)} := −D^{2}φ̂(x̂^{(k)}, λ̂^{k})^{−1}Dφ̂(x̂^{(k)}, λ̂^{k})^{T} .

3. If ∥Dϕ̂(x̂^{(k)})∥Ĥ_{x̂^{(k)}}^{−1} ≤ \frac{1}{5} stop, else

4. λ̂^{k+1} := λ̂^{k} + σ(λ̂^{k} − ĉ^{T} x̂^{(k+1)}).

5. k := k + 1; go to 2.

The analysis of this algorithm coincides with the analysis of the model algorithm in the previous Section. The only difference in the analysis is that now, the matrices Ĥ_{x̂,λ̂} become “smaller” by the update of λ̂ in step 4., and thus the definition of σ is slightly different. (Note that Ĥ_{x̂,λ̂} and Ĥx̂ only differ by a rank-one matrix, so that the stopping criterion in step 2. can be verified at little extra cost.)

If S^{opt} is bounded Phase I stops at some iteration K̂. Here, K̂ is bounded by a linear function of \sqrt{θ} and of log(θρ) where ρ measures the “asymmetry” of the initial point with respect to Ŝ:= S ∩ {y| c^{T} y ≤ λ̄},

ρ(x̂^{(0)}, Ŝ) := sup_{y∈∂Ŝ}{ρ| x̂^{(0)} + ρ (x̂^{(0)} − y) ∈ Ŝ} ≥ 1

where ∂Ŝ denotes the boundary of Ŝ. To see this let y^{∗} := arg max{ĉ^{T} y| y ∈ Ŝ}, λ^{∗} := max{λ| x̂^{(0)} + λ(x̂^{(0)} − y^{∗}) ∈ Ŝ}, and ỹ:= arg min{ĉ^{T} y| y ∈ Ŝ} ∈ ∂Ŝ. Then by definition of ρ = ρ(x̂^{(0)}, Ŝ), ρ ≥ λ^{∗} and ρ ≥ 1/λ^{∗}, and by Lemma 2.11,

λ̂^{0} − ĉ^{T} x̂^{(0)} ≥ ĉ^{T} x̂^{(0)} − ĉ^{T} ỹ ≥ λ^{∗}ĉ^{T} (y^{∗} − x̂^{(0)}).

Since the “distance” λ̂^{k} − ĉ^{T} x̂^{(k)} is multiplied by (1 + 1/(12\sqrt{θ})) at each iteration of Phase I, it follows like in Chapter 3.1 for k ≥ 26\sqrt{θ} log ρ that λ̂^{k} ≥ ĉ^{T} y^{∗} must hold true, i.e. the constraint ĉ^{T} x̂ ≤ λ̂ no longer intersects Ŝ. After another 26\sqrt{θ} log(5θρ) iterations the constraint ĉ^{T} x̂ ≤ λ̂ does not intersect any inner ellipsoid for Ŝ enlarged by 5θ,

λ̂ − ĉ^{T} x̂ ≥ 5θ max{ĉ^{T} y| y^{T} Ĥx̂y ≤ 1} = 5θ∥ĉ∥Ĥ−1 ; x̂,

and Phase I will stop at this point.

Summarizing we note that Phase I stops after at most K̂ = 26\sqrt{θ} log(6θρ) iterations. With x^{(0)} := x̂^{(K̂)} and λ^{0} := λ̄ we obtain a starting point for the model algorithm

satisfying the initial conditions of the previous model algorithm. ⊓ ⊔

Since Phase I still requires that some initial point in the interior of S be known, we close this Section by mentioning some auxiliary problem of how to generate an initial point in S^{o}.

min{t| r_{i}(ty^{i} + x) ≤ 0, for 1 ≤ i ≤ m, t ≥ 0}.  (3.4)

Here, we assume that for each i some point y^{i} ∈ IR^{n} is known such that r_{i}(y^{i}) < 0. As strictly feasible starting point for the auxiliary problem (3.4) we set t^{0} = 1, x^{(0)} = 0. Note that the set of optimal solutions of problem (3.4) is S ×{t = 0} which may be unbounded. Technical modifications like considering a “big M” auxiliary problem with objective tM +c^{T} x instead, or how to deal with the case that S = ∅ or S^{o} = ∅ and are omitted here. Note that for linear constraints r_{i}(x) = a_{i}^{T} x − b_{i}, problem (3.4) is equivalent to solving

min{t| r_{i}(x) ≤ tβ_{i}, t ≥ 0}  (3.5)

with β_{i} = −a_{i}^{T} y^{i}, that is it corresponds to the relaxation mentioned at the beginning of this Section. For nonlinear r_{i}, however, the approach via (3.4) guarantees that the argument x + ty^{i} remains inside S while the one via (3.5) does not.

3.3 Logarithmic barrier and affine scaling method

The analysis in Chapter 3.1 referred to a modification of the method of centers in Chapter 1.4. Here, we briefly mention in how far the analysis can be transferred to the logarithmic barrier method of Chapter 1.3 and the affine scaling method of Chapter 1.2.

• Under the assumptions of Chapter 3.1, the logarithmic barrier method also converges at a polynomial rate if the parameters μ_{k} are chosen appropriately.

Let ϕ be a θ-self-concordant barrier function for S. Under the assumptions of Chapter 1.1 let a point x^{(0)} ∈ S^{o} and some number μ^{0} > 0 be given such that

∥Δx^{(0)}∥_{H_{x^{(0)}}} = ∥\frac{c}{μ^{0}} + Dϕ(x^{(0)})∥_{H_{x^{(0)}}^{−1}} ≤ 1/5.

Here, Δx^{(0)} is the Newton step −D^{2}ϕ(x^{(0)})^{−1}(\frac{c}{μ^{0}} + Dϕ(x^{(0)})^{T}).

Algorithm 2

1. k := 0; σ := 1/(9\sqrt{θ}); ε = desired accuracy.

2. x^{(k+1)} := x^{(k)} + Δx^{(k)} where Δx^{(k)} := −D^{2}ϕ(x^{(k)})^{−1}(\frac{c}{μ^{k}} + Dϕ(x^{(k)})^{T}).

3. If θμ^{k} ≤ \frac{13}{15}ε stop, else

4. μ^{k+1} := μ^{k}(1 − σ).

5. k := k + 1; go to 2.

The convergence analysis of this modification may be carried out by induction like in Chapter 3.1. We show that ∥\frac{c}{μ^{k}} + Dϕ(x^{(k)})^{T} ∥_{D^{2}ϕ(x^{(k)})} ≤ \frac{1}{5} for all k. After completion of step 2., it follows from Lemma 2.4 that

∥\frac{c}{μ^{k}} + Dϕ(x^{(k+1)})^{T} ∥_{D^{2}ϕ(x^{(k+1)})} ≤ \frac{1}{16}.  (3.6)

By property (2.20) we may thus conclude that

∥\frac{c}{μ^{k}} ∥_{D^{2}ϕ(x^{(k+1)})} ≤ \sqrt{θ} + \frac{1}{16}.

Hence, after the update of μ^{k+1} in Step 4., it follows

∥\frac{c}{μ^{k+1}} + Dϕ(x^{(k+1)})^{T} ∥_{D^{2}ϕ(x^{(k+1)})} ≤ \frac{1}{16} + \frac{\sqrt{θ} + 1/16}{9\sqrt{θ} − 1} ≤ \frac{1}{5},

since θ ≥ 1. Finally, to verify that the stopping test in Step 3. is accurate observe that the centers x(λ) minimizing φ in (1.5) and the points x(μ) minimizing ϕ(x) + c^{T} x/μ coincide for μ = κ/(λ − c^{T} x(λ)). (Here, in slight abuse of notation we use the name (λ or μ) of the parameter to distinguish to which minimization problem the curve x( . ) refers to (i.e. to φ(x, λ) or to ϕ(x) + c^{T} x/μ).) Again, a relation of the form (3.1) follows from (3.6) and implies that x^{(k+1)} lies within \frac{1}{14} of the inner ellipsoid around x(μ^{k}) = x(λ^{k}) with μ^{k} = (λ^{k} −c^{T} x(λ^{k})/κ. Thus, as in Chapter 3.1, the desired bound

θμ^{k} ≥ \frac{13}{15}(c^{T} x^{(k+1)} − λ^{opt})

follows. ⊓ ⊔ • The affine scaling algorithm, in contrast, does not admit the same straightforward

analysis. Even though self-concordant barrier functions allow the computation of large inner ellipsoids that are “asymptotically optimal” in a certain sense (see Corollary 2.10 and the comments following Lemma 2.12), it is difficult to prove polynomiality—or merely convergence—of the affine scaling method without making additional nondegeneracy assumptions, see e.g. [91].

The above algorithms are primal algorithms in that dual variables are not explicitly defined. In Chapter 4 we will establish a close relationship of primal-dual algorithms to the method of centers.

KKT Approach

In the previous chapters we considered logarithmic barrier functions for solving (1.2). Another approach that has been proposed in the literature for solving convex programs is to solve the system of KKT conditions by some method for solving a set of nonlinear equations and inequalities, see eg. [96]. We note that the converse approach of solving nonlinear equations by solving some associated optimization problem has been proposed as well, see eg. [60]. Here, we state the first order necessary conditions for optimality as a certain system of nonlinear equations and inequalities and link this system to the logarithmic barrier function. This approach may be regarded as a derivation of primal-dual interior-point methods for solving convex programs with nonlinear constraints. We do not explicitly define a dual problem, implicitly however, “dual” variables are used. If we consider the standard Lagrange dual to a convex minimization problem with nonlinear constraints we observe that is suffers from a certain asymmetry: For a primal problem of the form

min{c(x)| r_{i}(x) ≤ 0 for 1 ≤ i ≤ m}

with convex functions c, r_{i}, 1 ≤ i ≤ m, the dual is

min{ min_{x}{c(x) + ∑_{i=1}^{m} y_{i}r_{i}(x)} | y_{i} ≥ 0 for 1 ≤ i ≤ m }.

Nesterov and Nemirovsky [64] noted that it is difficult to extract the primal problem from the dual one and therefore propose a conic formulation of constrained optimization problems. Using this formulation the dual is completely symmetric. (The duality results stated in the original form in terms of min-max problems for convex-concave functions are symmetric as well, see e.g. [89].) Here, we avoid the “lack” of symmetry by not regarding the original optimization problem but only the problem of solving the necessary first order conditions— assuming some constraint qualification. The necessary first order conditions are stated below and will be referred to as KKT conditions in the sequel. The variables involved in the KKT system are of course what is usually referred to as the dual variables and the error-bound that can be obtained from a primal-dual pair of feasible solutions is immediately obvious even without defining a dual problem.

In this Chapter we consider the following problem:

min c(x)  (4.1)
r_{i}(x) ≤ 0 for 1 ≤ i ≤ m
s_{j}(x) = 0 for 1 ≤ j ≤ m_{2},

where the functions c, r_{i} and s_{j}: IR^{n} → IR are twice continously differentiable.

We assume that the problem satisfies the constraint qualification by Slater, i.e. if the problem is convex we assume that there is a point x̄ such that r_{i}(x̄) < 0 for all nonlinear r_{i}. We derive a primal-dual method that can be applied to both, convex or non-convex problems. All of our results, however, will refer to the case that (4.1) is convex, i.e. that s_{j} are linear for 0 ≤ j ≤ m_{2} and r_{i} are convex for 0 ≤ i ≤ m.

Finally we assume that the set of optimal solutions of (4.1) is nonempty and bounded. As before, an optimal solution will be referred to by x^{opt}.

4.1 The search direction

The necessary conditions for a local minimum x of (4.1) are that there exist vectors y ∈ IR^{m}, v ∈ IR^{m_{2}} such that

r_{i}(x) ≤ 0, s_{i}(x) = 0,  (4.2)
y_{i} ≥ 0, ∇c(x) + ∑_{i=1}^{m} y_{i}∇r_{i}(x) + ∑_{i=1}^{m_{2}} v_{i}∇s_{i}(x) = 0,  (4.3)
y_{i}r_{i}(x) = 0.  (4.4)

(4.2) will sometimes be referred to as primal feasibility, (4.4) as complementarity, and (4.3) as dual feasiblity, even though we avoid the explicit use of a dual problem in this Chapter. Similarly, x will be referred to as primal variable, and y, v as dual variables. If we denote the constraint vectors and their Jacobians by

( r_{1}(x) ; ⋮ ; r_{m}(x) ) = R(x), ( s_{1}(x) ; ⋮ ; s_{m_{2}}(x) ) = S(x),

( ∇r_{1}(x)^{T} ; ⋮ ; ∇r_{m}(x)^{T} ) = J(x), ( ∇s_{1}(x)^{T} ; ⋮ ; ∇s_{m_{2}}(x)^{T} ) = K(x),

then system (4.2)–(4.4) is equivalent to finding vectors x ∈ IR^{n}, y, z ∈ IR^{m}, v ∈ IR^{m_{2}} such that y ≥ 0, z ≥ 0 and

( \begin{matrix}S(x) \\ R(x) + z \\ ∇c(x) + J(x)^{T} y + K(x)^{T} v \\ Zy \end{matrix} ) = 0,  (4.5)

where

Z = diag(z_{1}, …, z_{m}).

The variable z is merely for simplification of the presentation and derivation of our results. An initial point that at least satisfies the inequality constraints of (4.5) is readily obtained. (See also below.) An obvious idea is to approximate (4.5) by (damped) Newton’s method.

Since Newton’s method cannot consider the inequalities y ≥ 0, z ≥ 0, the system is relaxed by some parameter μ > 0, and we consider systems of the form

( \begin{matrix}S(x) \\ R(x) + z \\ ∇c(x) + J(x)^{T} y + K(x)^{T} v \\ Zy \end{matrix} ) = ( \begin{matrix}0 \\ 0 \\ 0 \\ μe \end{matrix} ) , e = ( 1 ⋮ 1 ) ∈ IR^{m}.  (4.6)

The “positive” solutions of (4.6) (i.e. those satisfying y > 0, z > 0) will be denoted by x(μ), y(μ), v(μ) and z(μ). As we will see below, if (4.1) is of the form (1.2) and satisfies the assumptions of Chapter 3.1, then for fixed μ > 0 the positive solution of (4.6) exists and is unique, (and this is true in fact for slightly more general problems (4.1)).

Locally, near the positive solution of (4.6), Newton’s method converges and the Newton successor does not violate the inequalities y ≥ 0, z ≥ 0. This observation suggests an iterative method that approximates a sequence of systems (4.6) with decreasing parameters μ^{k} > 0 such that strict positivity of y and z is maintained throughout. If the parameters μ^{k} satisfy μ^{k} → 0, the systems (4.6) and (4.5) are equivalent in the limit.

Linearization of (4.6) is straightforward. The first two rows result in

S(x + Δx) ≈ S(x) + K(x)Δx = 0,
R(x + Δx) + z + Δz ≈ R(x) + J(x)Δx + z + Δz = 0.

Let us define

H(x, y, v) = ∇^{2}c(x) + ∑_{i=1}^{m} y_{i}∇^{2}r_{i}(x) + ∑_{j=1}^{m_{2}} v_{j}∇^{2}s_{j}(x),

then the linearization of the third and fourth row yield

∇c(x) + J(x)^{T} y + J(x)^{T} Δy + K(x)^{T} v + K(x)^{T} Δv + H(x, y, v)Δx = 0
(Z + ∇Z)(y + Δy) ≈ Zy + ZΔy + YΔz = μe.

The search direction (Δx, Δy, Δz, Δv) resulting from this linearization is the Newton direction for (4.6) and is given by the following system of linear equations

( \begin{matrix}K(x) \\ J(x) & & I \\ H(x, y, v) & J(x)^{T} & & K(x)^{T} \\ & Z & Y \end{matrix} ) ( \begin{matrix}Δx \\ Δy \\ Δz \\ Δv \end{matrix} ) = ( \begin{matrix}a \\ b \\ g \\ d \end{matrix} ) ,  (4.7)

where

( \begin{matrix}a \\ b \\ g \\ d \end{matrix} ) := ( \begin{matrix}−S(x) \\ −R(x) − z \\ −∇c(x) − J(x)^{T} y − K(x)^{T} v \\ μe − Zy \end{matrix} )  (4.8)

Reordering the equations and omitting the arguments x, y, v of H, J and K we obtain

( \begin{matrix}H & J^{T} & K^{T} & 0 \\ J & 0 & 0 & Z \\ K & 0 & 0 & 0 \\ 0 & Z & 0 & YZ \end{matrix} ) ( \begin{matrix}Δx \\ Δy \\ Δv \\ Z^{−1}Δz \end{matrix} ) = ( \begin{matrix}g \\ b \\ a \\ d \end{matrix} ) .  (4.9)

(We symmetrized this system by multiplying the last column by Z and the corresponding variable by Z^{−1}.) For convenience we may take the Schur complement with respect to YZ,

( \begin{matrix}H & J^{T} & K^{T} \\ J & −Y^{−1}Z & 0 \\ K & 0 & 0 \end{matrix} ) ( \begin{matrix}Δx \\ Δy \\ Δv \end{matrix} ) = ( \begin{matrix}g \\ b − Y^{−1}d \\ a \end{matrix} ) ,  (4.10)

Δz = Y^{−1}(d − ZΔy).

If J and H are dense, another Schur complement with respect to −Y^{−1}Z may be meanigful—the resulting system is

( \begin{matrix}H + J^{T} Z^{−1}YJ & K^{T} \\ K & 0 \end{matrix} ) ( \begin{matrix}Δx \\ Δv \end{matrix} ) = ( \begin{matrix}g + J^{T} YZ^{−1}b − J^{T} Z^{−1}d \\ a \end{matrix} ) ,  (4.11)

Δy = YZ^{−1}(JΔx + Y^{−1}d − b).

If H is sparse, possible fill-in has to be considered. Also, the condition number of (4.11) may be worse than the one of (4.10). The numerical solution of systems in the form (4.10), (4.11) has recently been investigated in the context of interior-point methods for solving linear programs, see e.g. [52, 20, 21, 18].

4.2 A “primal dual” algorithm

A conceptual primal-dual algorithm may be outlined as follows: Conceptual algorithm: Given a point x^{0}, y^{0}, v^{0}, z^{0}, with y^{0} > 0, z^{0} > 0, set k = 0 and repeat until some stopping criterion is satisfied.

• Choose μ_{k} ≤ \frac{y^{k}^{T} z^{k}}{m} (μ_{k} ≥ 0) and compute the Newton direction (Δx^{k}, Δy^{k}, Δv^{k}, Δz^{k}) defined by system (4.6).

• Choose a step length α_{k} ∈ (0, 1] and set

(x^{k+1}, y^{k+1}, v^{k+1}, z^{k+1}) = (x^{k}, y^{k}, v^{k}, z^{k}) + α(Δx^{k}, Δy^{k}, Δv^{k}, Δz^{k}).

Here, α_{k} must be sufficiently small such that y^{k+1} > 0 and z^{k+1} > 0.

End

4.2.1 Initialization

Any point x, y, v, z such that y > 0, z > 0 and any parameter μ > 0 may be chosen as initial data for the algorithm in this Chapter. However, the choice of the starting point will influence the number of iterations. A heuristic that is commonly used for computing a “good” starting point when solving linear programs (see e.g. [48]) can be generalized to problems of the form (4.1). Let some x̂^{0} (e.g. x̂^{0} = 0) be given and let

M = max_{1≤i≤m, 1≤j≤m_{2}} {0, −r_{i}(x̂^{0}), |s_{j}(x̂^{0})|}

be the maximum constraint violation at x̂^{0}. By linearizing r_{i}, s_{j} find a point x^{0} nearby x̂^{0} such that x^{0} approximately satisfies the equality and inequality constraints:

x^{0} := arg min{∥R(x̂^{0}) + J(x̂^{0})(x − x̂^{0}) + Me∥_{2} | x : S(x̂^{0}) + K(x̂^{0})(x − x̂^{0}) = 0}.

Choose ϵ > 0 and M̃ ≥ ϵ. Set z_{i} = max{−r_{i}(x), ϵ}. Set v = arg min ∥∇c + K^{T} v∥_{2} where ∇c and K are evaluated at x^{0}, and choose μ_{0} ∈ [ϵ, M] such that ∥∇c + K^{T} v + μ_{0}J^{T} Z^{−1}e∥_{2} becomes minimal. Then set y_{i} = μ_{0}/z_{i}. This defines a starting point (x, y, v, z) with y > 0 and z > 0.

4.2.2 The step-length

In Step 2, a suitable number α ∈ (0, 1] is to be determined. This can be done by a line search. We require the merit function for the line search to have a minimum or an infimum at the solution of (4.5), furthermore, it must guarantee that y and z remain positive. The following merit function for the line search appears to be a natural choice

Ψ(x, y, z, v) = − ∑_{i=1}^{m} log y_{i} − ∑_{i=1}^{m} log z_{i} + p log ∥ ( \begin{matrix}−S(x) \\ −R(x) − z \\ −∇c(x) − J(x)^{T} y − K(x)^{T} v \\ Zy \end{matrix} ) ∥, (4.12)

where p > m, and ∥.∥ is some norm. If ∥.∥ is a weighted 1-norm, i.e. ∥ξ∥ = ∑ w_{i}|ξ_{i}| for some weights w_{i} > 0, this function is a special case of a class of potential functions proposed and analyzed in [44, Chapter 4.3]. The analysis in [44] applies to monotone nonlinear complementarity problems and proves a finite number of iterations for either finding an optimal solution, or detecting the case that there does not exist an optimal solution in some given compact set. However, since system (4.5) is not monotone (even if (4.1) is convex), the analysis of [44] does not apply directly. Remark: For linear programs it is common to use different step lengths for the primal and dual variables, since the residual can be separated into a primal and a dual component. Here, the residual is a more complicated function of the step lengths, and the best choice of different step lengths in x, y, v and z involves the solution of a four dimensional minimization subproblem.

4.2.3 A stopping criterion for convex problems

In this subsection we present a simple and well-known stopping criterion assuming that problem (4.1) is convex and that S^{opt} is not empty.

Lemma 4.1 Let the point (x, y, z, v) be primal and dual feasible, i.e. (x, y, z, v) satisfy the first 3 rows of the system (4.6). The optimal value of (4.1) then satisfies c^{opt} ≥ c(x) − z^{T} y.

Proof: Let x^{∗} be any feasible point for (4.1) (e.g. x^{∗} = x^{opt} if x^{opt} exists). Let h = x^{∗}−x, then by convextiy of c:

c(x^{∗}) = c(x + h) ≥ c(x) + h^{T} ∇c(x) = c(x) − h^{T} (J(x)^{T} y + K(x)^{T} v).

Since x and x^{∗} are feasible and s_{j} are linear it follows K(x)h = 0, and since R(x + h) ≤ 0 it follows from convexity of r_{i} that −J(x)h ≥ R(x), so that

c(x^{∗}) ≥ c(x) + R(x)^{T} y.

Since z = −R(x) the claim follows.

The following equivalent statement of Lemma 4.1 is slightly more convenient when testing a point x, y, v generated by the conceptual algorithm for optimality.

Lemma 4.2 Let x ∈ IR^{n}, y ∈ IR^{m}, y ≥ 0, v ∈ IR^{m_{2}} be given. Define the residual vectors π^{(1)} (≥ 0) and π^{(2)}, π^{(3)}, π^{(4)} such that

r_{i}(x) ≤ π_{i}^{(1)}, s_{i}(x) = π_{i}^{(2)},  (4.13)
∇c(x) + ∑_{i=1}^{m} y_{i}∇r_{i}(x) + ∑_{i=1}^{m_{2}} v_{i}∇s_{i}(x) = π^{(3)},  (4.14)
y_{i}r_{i}(x) − y_{i}π_{i}^{(1)} = π_{i}^{(4)},  (4.15)

then x is “almost optimal” to a perturbation of problem (4.1) in the following sense. The point x is feasible for the perturbed problem

Find c̄^{opt} := inf c(x) − π^{(3)}^{T} x  (4.16)
r_{i}(x) ≤ π_{i}^{(1)} for 1 ≤ i ≤ m
s_{j}(x) = π_{j}^{(2)} for 1 ≤ j ≤ m_{2},

and “close” to optimality in the sense that c̄(x) ≤ c̄^{opt} + π^{(4)}^{T} e.

4.3 Relationship to logarithmic barrier methods

A feature of the previous algorithm distinguishing it from the barrier method in Chapter 1 is the fact that it does not assume a strictly feasible initial point (more precisely a point x with r_{i}(x) < 0 for 1 ≤ i ≤ m) to be known. Further, it leaves some freedom in the choice of μ. For μ = y^{T} z/m the strictly positive solution of a relaxed system (4.6) is approximated, and for μ = 0, the solution of problem (4.5) is approximated.

To compare the previous algorithm with the logarithmic barrier method in Chapter 1.3 we modify the barrier method to handle infeasible starting points, and extend it to a predictor corrector method (as in [35]). The predictor step is aimed towards the optimal solution of problem (4.1), (corresponding to the choice μ = 0 above) and the corrector step is aimed towards centrality, i.e. strict fulfillment of all positivity constraints (corresponding to the choice μ = y^{T} z/m above).

To guarantee that the logarithmic barrier method is well defined we make the following assumptions. We assume that the constraints r_{i} are convex and defined on all of IR^{n}, and that the set of optimal solutions is nonempty and bounded. For simplicity of presentation, we also assume that there are no equality constraints s_{j}, that is m_{2} = 0, and that the objective function c(x) = c^{T} x is linear. (The latter assumptions can easily be dropped.) Given some initial point x^{0}, choose some vector β > R(x^{0}), (β ≥ 0) and define the logarithmic barrier function for the (enlarged) set S(μ) := {x| r_{i}(x) ≤ μβ_{i}} by

ϕ(x, μ) := − ∑_{i=1}^{m} log(μβ_{i} − r_{i}(x)).

(As pointed out in Chapter 3.2, the function ϕ( . , μ) might be self-concordant for μ = 0, but not so for μ > 0. In particular, if the above assumptions are violated, and the domain of r_{i} is not all of IR^{n}, the approach below may not be possible.)

We decribe a predictor corrector barrier method for following the path

x(μ) := arg min \frac{c^{T} x}{μ} + ϕ(x, μ)

which leads to an optimal solution as μ → 0. The main difference to the logarithmic barrier method of Chapter 1.3 is that ϕ is not the barrier function for S but for a slightly enlarged set S(μ) with S(μ) → S uniformly in every compact set in IR^{n} as μ → 0 (see e.g. [35]). Note that here, μ is a parameter for both, feasibility and optimality. The norm ∥c∥ determines the ratio of “infeasibility” to “optimality” at the points x(μ), (and may be adjusted to our needs).

The predictor step computes a tangent to a perturbed curve x̃(μ) which also leads to an optimal solution. The perturbed curve x̃(μ) is defined by

\frac{c}{μ} + ∇_{x}ϕ(x̃(μ), μ) = const.  (4.17)

Here, “const” is a constant chosen such that the current iterate x^{(k)} generated by the algorithm lies on the perturbed curve. (Thus, to be precise, by x̃(μ) we denote a whole family of perturbed curves parameterized by the n-dimensional vector “const”. If const = 0 (and β = 0), the perturbed curve coincides—up to the parameterization—with the path of centers x(λ) or x(μ) of Chapters 1.3 and 1.4.) The gradient ∇_{x}ϕ(x, μ) has the form

∇_{x}ϕ(x, μ) = ∑_{i=1}^{m} \frac{∇r_{i}(x)}{μβ_{i} − r_{i}(x)}.

Differentiating (4.17) with respect to μ yields the predictor direction

−\frac{c}{μ^{2}} + ∇_{x}^{2}ϕ(x̃, μ) ˙ x̃(μ) − ∑_{i=1}^{m} β_{i} \frac{∇r_{i}(x̃)}{(μβ_{i} − r_{i}(x̃))^{2}} = 0.

It is convenient to express ˙ x̃(μ) with some vectors y and z defined by z_{i} := μβ_{i} − r_{i}(x̃) and y_{i} = μ/z_{i},

˙x̃(μ) = (μ∇_{x}^{2}ϕ(x̃, μ))^{−1}(\frac{c}{μ} + J(x̃)^{T} YZ^{−1}β).  (4.18)

The corrector step is Newton’s method for finding the solution of (4.17) with const = 0, and is given by

Δx̃ = −∇_{x}^{2}ϕ(x̃, μ)^{−1}(\frac{c}{μ} + ∇_{x}ϕ(x̃, μ)) = (μ∇_{x}^{2}ϕ(x̃, μ))^{−1}(−c − μ∇_{x}ϕ(x̃, μ)). (4.19)

We compare (4.18) and (4.19) with the directions defined by system (4.11). Assume a point x, y with y > 0 and some parameter μ̄ > 0 are given. By using row 4 of system (4.6) we define the variable z introduced in (4.6) by z = μ̄Y^{−1}e. We further define β = (z +R(x))/μ̄. It follows b = −μ̄β and d = (μ − μ̄)e (in (4.8)). Since no equality constraints are present, the submatrix K is empty, and the right hand side of (4.11) is given by

g + J^{T} YZ^{−1}b − J^{T} Z^{−1}d = −c − J^{T} y − μ̄J^{T} YZ^{−1}β − J^{T} Z^{−1}(μ − μ̄)e
= −c − μ̄∇_{x}ϕ(x, μ̄) − μ̄J^{T} YZ^{−1}β − (μ − μ̄)J^{T} Z^{−1}e.  (4.20)

Since c(x) = c^{T} x is linear, the matrix

H + J^{T} Z^{−1}YJ = ∇^{2}c(x) + μ̄ ∑_{i=1}^{m} \frac{∇^{2}r_{i}(x)}{μ̄β_{i} − r_{i}(x)} + μ̄ ∑_{i=1}^{m} \frac{∇r_{i}(x)∇r_{i}(x)^{T}}{(μ̄β_{i} − r_{i}(x))^{2}}

is just a multiple of the Hessian, H + J^{T} Z^{−1}YJ = μ̄∇^{2}ϕ(x, μ̄).

If μ = 0 (4.20) reduces to

−c − μ̄J^{T} YZ^{−1}β − J^{T} (y − Z^{−1}μ̄e) = −c − μ̄J^{T} YZ^{−1}β

which is a multiple (4.18). (In (4.18) the step length is −μ to “reach” x̃(0), in (4.11), (4.20) it is 1.)

For μ = μ̄(:= y^{T} z/m) we obtain

= −c − μ̄∇_{x}ϕ(x, μ̄) − μ̄J^{T} YZ^{−1}β.  (4.21)

As noted above, we may adjust ∥c∥ to our needs. In particular, if we replace c by c/2 in (4.17) (and subsequently in (4.18) and (4.19)) we may observe that (4.21) is just the sum of −μ times (4.18) and (4.19). (Alternatively, we may replace the term “const” in (4.17) by “μ const” to obtain (4.21) as the tangent vector of some other curve (4.17) (which also leads to the optimal set).

We may thus observe that the span of the search directions of the logarithmic barrier method and of the conceptual KKT-based method is the same. Furthermore, the x-components of the solutions of system (4.6) coincide with the minimizers of \frac{c^{T} x}{μ} + ϕ(x) and are therefore unique if ϕ is self-concordant and S^{opt} is bounded (see chapters 2 and 3.1). From this it is straightforward to see that also the y, z-components exist and are unique. The two methods essentially differ in two points. First, the update Δy_{i} of y_{i} in (4.11) and the update of the “implicit” variable y_{i} := μ̄/(μ̄β_{i} −r_{i}(x)) in the logarithmic barrier method are different. Second, the barrier method fixes z = μY^{−1}e, while z is treated as a variable in (4.5). Note here, that our motivation for introducing the variable z in (4.5) was for simplicity of presentation, however, the observed superiority of primal-dual methods over primal methods suggests that the additional variable z may also have some computational benefits.

One benefit might be the fact that the inequalities y ≥ 0, z ≥ 0 are linear in (4.5) while the inequality R(x) ≤ μβ is nonlinear. The effect of nonlinear inequalities can best be explained for the case β = 0. If a point x̃ is far from optimality but close to some constraint j (i.e. 0 < −r_{j}(x) ≪ r_{i}(x) for some i with r_{i}(x^{opt}) = 0 for all optimal solutions x^{opt}) then ˙ x̃ is nearly parallel to r_{j} (i.e. |∇r_{j}(x̃)^{T} ˙ x̃| ≪ ∥∇r_{j}(x̃)∥_{2}∥ ˙x̃∥_{2}). If r_{j} is linear this implies that a “rather long” step t along x̃+t ˙x̃ does not violate the constraint r_{j}(x̃+t ˙x̃) < 0. However, if r_{j} has some (convex) curvature this is no longer true, and only a short step t in direction ˙ x̃ may be possible without violating the inequalities. (Recall that the inequalities must be satisfied throughout the algorithm to be able define the search direction at all!).

Another heuristical explanation^{1} for the benefits of the variable z is as follows. Near optimality, y and z approximate the y-z-component of the solution of (4.5). (Let us assume for the moment that this solution is unique.) The matrices Z^{−1}Y defining parts of both, left hand side and right hand side of (4.11) are the geometric mean of μZ^{−2} and Y^{2}/μ. One might hope that the geometric mean using the two variables y and z deviates less from the limit than Y^{2}/μ depending on the single variable y and being used in (4.18) and (4.19). (Note again, that the variable z occurring (4.18) and (4.19) is not an “independent” variable but defined as z = μ̄Y^{−1}e.)

1This explanation was presented at some conference, but we were unable to trace back when and by whom.

Application: A min-max eigenvalue problem

In this Chapter we give an example of an interesting class of problems that has a selfconcordant barrier function. For this example we also outline a modification of the method of centers that appears to be practical for a numerical implementation. Some promising numerical results obtained from a preliminary implementation are included.

The problem under consideration is to minimize the largest eigenvalue of an affine combination of symmetric matrices. This nonsmooth problem is transformed into an equivalent smooth constrained problem, which is solved by a predictor-corrector interior-point method taking full advantage of the differentiability and convexity.

5.1 The problem

Problems in mechanics and systems analysis can often be expressed in the following (or a closely related) form: Find

λ^{opt} := inf_{x∈IR^{m}}{ λ_{max}(A(x)) }  (5.1)

and an optimal solution x^{opt} if it exists. Here λ_{max}(A(x)) is the largest eigenvalue of the matrix

A(x) := A^{(0)} + ∑_{i=1}^{m} x_{i}A^{(i)},  (5.2)

for given symmetric matrices A^{(i)} ∈ IR^{n×n}, 0 ≤ i ≤ m.

Since λ_{max}(A(x)) is convex, problem (5.1) is a convex but non-differentiable optimization problem. It arises in a variety of applications; for example, the stability analysis of dynamical systems [4, 6], or in combinatorial applications [1]; see also [12, 69, 71] for further applications.

Below we state some simple facts that illustrate the nature of problem (5.1) and are easily verified.

• If λ^{opt} > −∞, it lies in the interval [λ_{min}(A^{(0)}), λ_{max}(A^{(0)})].

• There is no x such that ∑_{i} x_{i}A^{(i)} is positive definite iff λ^{opt} is finite.

• There is no x ≠ 0 such that ∑_{i} x_{i}A^{(i)} is semidefinite iff problem (5.1) has a nonempty and bounded set of optimal solutions

5.1.1 Basic assumptions

Without loss of generality we assume that the matrices I, A^{(i)}, 0 ≤ i ≤ m are linearly independent, i.e. r_{0}I + ∑_{i≥1} r_{i}A^{(i)} = 0 iff r_{i} = 0, 0 ≤ i ≤ m. We further assume that λ_{min}(A^{(0)}) = 0 (by adding a suitable multiple of the identity to A^{(0)}). For convenience we also assume that all A^{(i)} have Frobenius norm 1.

5.2 Relationship to other methods

The need for efficient methods for solving minimax eigenvalue problems like (5.1) is expressed, for example, in [71]. In this section we briefly introduce different methods for solving (5.1) and outline the development that led to the method of this paper.

5.2.1 “Classical” methods

Standard methods for nondifferentiable optimization, e.g. Shor’s subgradient method, Shor’s ellipsoid method or Kelly’s cutting plane method can be used for solving (5.1); for a survey of these methods we refer to [4, ch.13-14]. Other papers on methods for nondifferentiable optimization include [46, 57, 81, 99]. Specialized methods for (5.1) are developed in e.g. [6, 23, 37, 38, 68, 69, 74].

Overton’s method [68, 69] is locally quadratically convergent. His method builds on earlier work of Fletcher [12] for semi-definite programming. Goh and Teo’s method [23] is most closely related to the approach of this paper, since it involves transforming (5.1) into a smooth nonlinear programming problem by means of determinants.

5.2.2 Development of interior-point methods

In [84] Sonnevend compared different barrier functions for the cone of positive definite matrices and showed why the barrier function − log detX is the “principal actor” for such optimization problems. A detailed complexity analysis of interior-point methods for convex optimization problems based on the notion of self-concordance is given in Nesterov and Nemirovsky’s work [62]. As already mentioned in Chapter 2, their analysis also applies to the function − log detX and provides the theoretical tools for a proof of a polynomial rate of convergence. An independent (and simultaneous) derivation of a polynomial primal-dual method for the problem considered here was also given by Alizadeh in [1]. In [1], Alizadeh also gives an elegant proof for a duality result concerning minimax eigenvalue problems and applies this result to solve combinatorial problems. His method is particularly attractive for the case m ≫ n. Alizadeh [1] and also Ringertz [77] further present implementations of interior-point methods for solving eigenvalue problems based on the function log detX that is used here. Ringertz describes an implementation of a barrier method for minimizing the largest eigenvalue of a matrix subject to nonlinear equality and inequality constraints. His problems arise in the optimal design of shell structures for airplanes. They are not convex, and hence more difficult in nature than problem (5.1). Ringertz does not touch the topic of estimating the complexity of his method but presents numerical experiments on a Cray XMP with several thousand nonlinear constraints. His results support the obvious conjecture (see also [35]) that interior-point methods are also very efficient for much more difficult problems than the one analyzed here, and for which a complexity analysis is not yet possible. Nevertheless, the complexity analysis of interior-point algorithms is an integral part of the method, not only since it led to their development, but also since it explains their behaviour (weak dependence on the data, affine invariance) and gives important conclusions for implementational aspects (line search, stopping test).

As in [23], problem (5.1) is transformed to a nonlinear optimization problem. This is solved by a modification of the logarithmic barrier method taking full advantage of the structure (differentiability and convexity) of the transformed problem. The efficiency of the barrier method is what makes the approach attractive. As shown in Chapter 3.1 the logarithmic barrier method is globally linearly convergent and the guaranteed rate of convergence depends only on the dimension of the problem and not on the data of the matrices A^{(i)}. More precisely, there after at most O(\sqrt{n}) iterations the error λ_{max}(A(x)) − λ^{opt} is reduced by a factor of 2. Furthermore, recent analyses of interior point methods for linear programs by Ye et. al. in [97] showed that predictor-corrector methods related to the one presented here are quadratically convergent under weak assumptions.

5.3 A barrier approach

5.3.1 An equivalent problem

We rewrite problem (5.1) as another convex differentiable problem with a positive definite constraint. In the following we will write A > 0 (A ≥ 0) if the matrix A is positive definite (positive semidefinite). Positive definiteness can easily be verified numerically (via Cholesky decomposition). Moreover, the cone of positive definite matrices A > 0 is characterized by the domain of the self-concordant barrier function

− log det_{+}(A).

Here, det_{+} of a symmetric matrix is simply defined as

det_{+}(A) := \{ \begin{matrix}det(A) & if A is positive definite, \\ 0 & otherwise. \end{matrix}  (5.3)

Let us introduce an (m + 1)-st variable λ and denote

x̃ := (x^{T} , λ)^{T} ∈ IR^{m+1}.  (5.4)

Note that a number λ is larger than λ_{max}(A(x)) iff λI − A(x) is positive definite. This observation can be used to rewrite (5.1) and (5.2) as

inf_{x̃∈IR^{m+1}}{λ| det_{+}(λI − A(x)) > 0}.  (5.5)

The domain of problem (5.5) will be denoted by the open set S^{o}. Define the logarithmic barrier function for S^{o},

ϕ(x̃) := − log det_{+}(λI − A(x)).  (5.6)

Under the basic assumptions of Section 5.1 the function ϕ is strictly convex and selfconcordant with parameter θ = n.

Before outlining our modification of the logarithmic barrier method next, we briefly state a nice duality theorem for problem (5.5) presented by Alizadeh in [1].

Adopting the notation in [1], we define the “symmetric scalar product” of two matrices by

A • B := ∑_{i,j} A_{ij}B_{ij} = trace(A^{T} B).  (5.7)

A slightly more general form of problem (5.5), in that it allows a general linear objective function and does not fix A^{(m+1)} = −I, is the problem

max_{x∈IR^{m+1}}{b^{T} x | C − ∑_{i} x_{i}A^{(i)} ≥ 0},  (Primal)

where C = A^{(0)}. Alizadeh gives a short and self-contained proof that for this problem there is a dual problem

min_{Y ≥0}{C • Y | A^{(i)} • Y = b_{i}},  (Dual)

for which the following duality relations hold: (The dual variable is a positive semidefinite matrix Y.) Optimal solutions x^{opt} and Y^{opt} exist and the optimal values are the same if both problems have strictly feasible solutions. Furthermore, b^{T} x ≤ C • Y for any primal and dual feasible variables x and Y. Alizadeh points out a surprising structural similarity of these problems to linear programming and develops a potential reduction method for the dual problem, and applies it to approximate the solution of combinatorial optimization problems.

The above duality result implies that the dual of problem (5.5) is given by

max_{Y ≥0}{A^{(0)} • Y | Y • I = 1, A^{(i)} • Y = 0}.

5.4 A specific barrier method

5.4.1 Notation and initialization

We will apply a modifiaction of the logarithmic barrier method to problem (5.5). Recall the definition (5.4) of x̃ and define

A(x̃) := λI − A(x) = λI − A^{(0)} − ∑_{i=1}^{m} x_{i}A^{(i)}.  (5.8)

For a given vector w̃ in IR^{m+1} and for μ ∈ (0, 1] consider the barrier function

φ(x̃, μ) := \frac{1}{μ}λ − log det_{+}(A(x̃)) − w̃^{T} x̃.  (5.9)

Note that the feasible domain of (5.5) is given by the domain of φ, S^{o} = {x̃| φ(x̃, μ) < ∞}.

The vector w̃ is a linear perturbation to the barrier function, and while it does not influence self-concordance of φ it worsens the self-limitation property (2.19). We will choose w̃ so that our initial point is a minimizer of φ( . , 1).

To initialize the algorithm let λ_{0} = 2λ_{max}(A^{(0)}) > 0 (recall λ_{min}(A^{(0)}) = 0), and let x^{0} = 0 ∈ IR^{m}. This implies that x̃^{0} = (0, …, 0, 2λ_{max}(A^{(0)}))^{T} is a strictly feasible starting point (and that cond(A(x̃^{0})) = 2). Set μ = λ_{max}(A^{(0)})/n for example. The vector w̃ is chosen such that

Dx̃φ(x̃^{0}, 1) = 0,  (5.10)

and is constant throughout our algorithm. Let e_{m+1} := (0, …, 0, 1)^{T} ∈ IR^{m+1}. Then another representation of φ is given by

φ(x̃, μ) = (\frac{1}{μ}e_{m+1} − w̃)^{T} x̃ − log det_{+}(A(x̃)).  (5.11)

5.4.2 The derivatives of φ

The first and second derivatives of φ are defined in terms of

B(x̃) := A(x̃)^{−1}.  (5.12)

To abbreviate the notation we refer to x̃_{m+1} = λ and A^{(m+1)} = −I. It is known (and easy to verify) that the partial derivatives of “log det” are

\frac{∂}{∂X_{ij}} log det(X) = (X^{−T})_{ij}

(for X ∈ IR^{n×n} with det(X) > 0, [13]), so that the derivative D log detX = X^{−T} where X^{−T} is to be interpreted as a linear functional via the scalar product (5.7), X^{−T} [A] := X^{−T} • A. Thus by the chain rule one obtains

ĝ_{i} := ĝ_{i}(x̃) := \frac{∂}{∂x̃_{i}} log det_{+}(A(x̃)) = B(x̃) • (\frac{d}{dx̃_{i}} A(x̃)).

Hence

ĝ_{i} = −B(x̃) • A^{(i)}, 1 ≤ i ≤ m + 1.  (5.13)

The derivative g(x̃, μ) = Dx̃φ(x̃, μ)^{T} of φ now follows from (5.11),

g(x̃, μ) = \frac{1}{μ}e_{m+1} − ĝ(x̃) − w̃.  (5.14)

The derivative Dx̃g(x̃, μ) of g in turn is given by the partial derivatives

\frac{∂}{∂x̃_{j}} (A^{(i)} • B(x̃)) = ∑_{k,l} A_{kl}^{(i)} \frac{∂}{∂x̃_{j}} B(x̃)_{kl} = ∑_{k,l} A_{kl}^{(i)}(−B_{kq}B_{rl})_{qr} • (\frac{∂}{∂x̃_{j}} A(x̃))

for 1 ≤ i, j ≤ m + 1. (Here and in the following we occasionally omit the argument of B = B(x̃).) Summarizing, we find

\frac{∂}{∂x̃_{j}} g_{i}(x̃, μ) = ∑_{k,l,q,r} A_{kl}^{(i)}A_{qr}^{(j)}B_{kq}B_{rl} = BA^{(i)} • A^{(j)}B = trace(A^{(i)}BA^{(j)}B).

It is straightforward to compute the Hessian

H(x̃) := Dx̃g(x̃, μ) = (\frac{∂}{∂x̃_{j}} g_{i}(x̃, μ))_{ij}  (5.15)

in O(mn^{3} + m^{2}n^{2}) multiplications. For our examples below with m, n < 100 this was sufficient, we note however that if the A^{(i)} are sparse or of low rank (and n is large, while m is moderate), this can be done much more efficiently. In this case, the Hessian is a dense (m by m) matrix, even if the the (n by n) matrix A(x̃) is sparse. Also note that the Hessian does not depend on μ or w̃.

5.4.3 Perturbed centers

Definition A minimizing point of the function φ( . , μ) will be called the perturbed center and will be denoted by x̃(μ). If it exists, the perturbed center with perturbation w̃ = 0 is the analytic center (see Chapter 2.1.5).

Lemma 5.1 Let some point x^{0} ∈ S^{o} be given and w̃ be such that Dφ(x^{0}, 1) = 0. Then the functions φ( . , μ) have unique minimizers x̃(μ) for all μ ∈ (0, 1] iff λ^{opt} is finite.

The analytic center exists for μ = 1 iff it exists for all μ ≥ 0, or equivalently, iff the set of optimal solutions for (5.1) is nonempty and bounded.

(Note, if λ^{opt} is finite, the set of optimal solutions for (5.1) can be empty or unbounded.) Proof of Lemma 5.1.

1. Suppose λ^{opt} is finite, i.e. λ^{opt} ≥ 0 by assumption on A^{(0)}. We show that the barrier function φ( . , μ) has unique minimizers. By definition of w̃, the point x̃^{0} is a unique minimum of φ( . , 1). (Its gradient is zero at x̃^{0} and φ( . , 1) is strictly convex.) The existence of a unique minimum for the convex function φ( . , 1) implies that φ(x̃, 1) → ∞ as x̃ → ∞. Now let μ ∈ (0, 1). Since λ^{opt} ≥ 0 it follows that φ(x̃, μ) = ∞ if λ ≤ 0. For λ > 0 it follows trivially that φ(x̃, μ) > φ(x̃, 1); hence φ(x̃, μ) → ∞ as x̃ → ∞. Thus, the existence of a unique minimum for φ( . , μ) follows again from strict convexity of φ( . , μ).

2. Conversely, if λ^{opt} = −∞ then there exists an x̄ ∈ IR^{m} such that λ_{max}(∑ x̄_{i}A^{(i)}) < −1. For ¯ x̄ := (x̄, −1)^{T} and μ < 1/(w̃^{T} ¯ x̄) we find that φ(t¯x̄, μ) → −∞ as t → ∞, i.e. a perturbed center doesn’t exist for this μ.

3. Suppose the analytic center exists for some μ > 0. Then there is no x ≠ 0 such that ∑ x_{i}A^{(i)} ≤ 0. (The A^{(i)} are linearly independent, and φ( . , μ) decreases along the ray given by x̃ = (x^{T} , 0)^{T} if ∑ x_{i}A^{(i)} ≤ 0.) Hence, there is an ϵ > 0 such that λ_{max}(∑ x_{i}A^{(i)}) ≥ ϵ for all x with ∥x∥ = 1. Also, λ^{opt} is finite. For ∥x∥ ≥ 1 + λ^{opt}/ϵ it follows that λ_{max}(A(x)) = λ_{max}(A^{(0)} + ∑ x_{i}A^{(i)}) ≥ λ_{min}(A^{(0)}) + λ_{max}(∑ x_{i}A^{(i)}) ≥ 0+ϵ(1+λ^{opt}/ϵ) = λ^{opt} +ϵ. Therefore, the compact domain ∥x∥ ≤ 1+λ^{opt}/ϵ contains all optimal solutions, and the set of optimal solutions is not empty.

4. Suppose the set of optimal solutions is finite and nonempty. Let x̄ be an optimal solution. Then there exists a circle of radius r that contains the the optimal set S_{opt} ⊂ x̄ + K_{r} and λ_{max}(A(x)) ≥ λ^{opt} + ϵ for x ∈ ∂(x̄ + K_{r}), the boundary of x̄ + K_{r}. Let ∥g∥ = r, then by convexity λ_{max}(A(x̄ + θg)) ≥ λ^{opt} + θϵ for θ ≥ 1, hence, λ > λ^{opt} + θϵ if λI − A(x̄ + θg) > 0. Therefore, for any fixed μ > 0 the quantity λ/μ grows at least linearly with θ while log det_{+}(A(x̃)) is sublinear in θ (the determinant is a multinomial in the matrix coefficients). For w̃ = 0 this implies that φ(x̃, μ) → ∞ as x̃ → ∞, i.e. φ( . , μ) has unique minimizers, i.e. the analytic centers exist for all μ > 0.

It is intuitive (and has been shown e.g. in [11, 35]) that if (5.1) has an optimal solution the perturbed centers x̃(μ) converge to a solution x̃^{opt} as μ → 0. This result holds for any strictly feasible starting point x̃^{0} and the corresponding w̃. Thus Lemma 5.1 describes a whole family of curves of perturbed centers, each of which starts somewhere in S^{o} and leads to an optimal solution.

As stated in (2.9), the analytic center allows a two-sided ellipsoidal approximation of some level set {x̃ ∈ S| f_{0}(x̃) ≤ τ} for a suitable τ = τ(μ). The inner ellipsoid describes the domain in which the path of centers x̃(μ) is well approximated by its tangent. Further, points on the path of analytic centers define dual feasible variables that can be used in a stopping test. Similar results hold in a somewhat weaker form for the perturbed center if the perturbation w̃ is small. For large perturbations w̃ (for w̃^{T} D2 ; x̃φ(x̃^{0}, 1)^{−1}w̃ ≥ 1), however, this is no longer true. The vector w̃ guarantees the existence of the perturbed center and that the method below is well defined even if the set of optimal solutions is unbounded. However, it is our goal to keep w̃ small. To decrease the norm of w̃ it is advantageous in some instances to perform a phase one, and to introduce an additional bound, for example ∥x̃∥_{2} ≤ 10^{6}. Then find the analytic center of this set (the existence of it is guaranteed by the additional bound) and define w̃ as in (5.10) for this starting point after deleting the additional bound again.

The algorithm in this paper is a modification of the one described in [35]. The situation here is particularly simple, since a strictly feasible starting point can easily be constructed. Below we explain one iteration of the method.

5.4.4 One iteration of the algorithm

The following description is a modified version of the barrier method outlined in chapters 1 and 3.1. At the beginning of each iteration we assume that a strictly feasible current iterate x̃^{k} and an associated parameter μ_{k} are given, where x̃^{k} is considered as an approximation to x̃(μ_{k}). We repeat the following iteration until some convergence criterion is satisfied.

1. Determine the tangent to the curve of perturbed centers x̃^{k}(μ), that passes through x̃^{k}.

The tangent to the curve x̃(μ) is easily obtained by differentiating the characteristic equation

g(x̃(μ), μ) ≡ 0  (5.16)

with respect to μ. The differentiation yields

Dx̃g(x̃(μ), μ)^{T} x̃′(μ) + \frac{∂}{∂_{μ}} g(x̃(μ), μ)^{T} = H(x̃(μ))x̃′(μ) − \frac{1}{μ^{2}} e_{m+1} = 0.

The tangent can thus be computed from a Cholesky decomposition of the Hessian. Note that the direction of the tangent x̃′(μ) does not depend on the vector w̃ or on μ, but only on the current point x̃. Thus the above computation precisely yields the direction of the tangent to the (unique) curve of perturbed centers starting at x̃ and ending in an optimal solution. Also, the last component (i.e. λ) of x̃′(μ) is positive (since H is positive definite) so that −x̃′(μ) is obviously a descent direction for the objective value λ.

2. Perform a linear extrapolation in the direction of the tangent:

x̂^{k+1} := x̃^{k} − β_{k}x̃′(μ).

The steplength β_{k} for the extrapolation is chosen as r_{k} times the maximum steplength β_{max} such that A(x̃^{k} − β_{max}x̃′(μ)) is still positive semidefinite. (The exact computation of β_{max} involves computing the maximum eigenvalue of a matrix.) Here, r_{k} ∈ (0, 1) can be chosen adaptively: Set r_{1} = 0.9 and (for k ≥ 2) if Newton’s method (in step 4 below) converged in less than 3 steps in the previous iteration, increase r (as r_{k} = (1 + r_{k−1})/2); if it converged in more than four Newton steps, decrease r (as r_{k} = max{r_{k−1}/2, 2r_{k−1} − 1}).

3. The parameter μ is controlled as follows. Given the k-th iterate x̃^{k}, an associated parameter μ_{k} and an extrapolated point x̂^{k+1} ∈ IR^{m+1} for the (k + 1)-st iterate, determine the (unique) value μ̂_{k+1} that “best fits” x̂^{k+1}, i.e. such that the last component of the gradient g at x̂^{k+1} is zero. Set μ_{k+1} = min{μ_{k}/2, μ̂_{k+1}}

Remark: For a moderate stepsize β for the (linear) extrapolation one can show that μ̂_{k+1} < μ_{k} Proof:

We show that the last component of Dφ(x̃ − βx̃′, μ) is a decreasing function of β for small |β|. Clearly,

\frac{d}{dβ} Dφ(x̃ − βx̃′, μ)e_{m+1} = −x̃^{′T} H(x̃ − βx̃′)e_{m+1} = −e_{m+1}^{T}H^{−1}(x̃)H(x̃ − βx̃′)e_{m+1}

is negative (and equal to −1) for β = 0. (Here, H(x̃) = D^{2}φ(x̃, μ) is independent of μ as noted in (5.15).) From the relative Lipschitz condition 2.10 it follows that this is negative if ∥βx̃′∥_{H(x̃)} < 1/2.

We note that there are (very “artificial”) examples that μ̂_{k+1} > μ_{k} for large β < β_{max} even when the extrapolation started on the central path (and μ̂_{k+1} < μ_{k} for smaller values of β > 0). In general however, it is true that lim_{β→βmax} μ̂_{k+1} = 0, and in our program the safeguard μ_{k+1} ≤ μ_{k}/2 was never needed.

4. Perform Newton’s method (with line search) to minimize the barrier function φ( . , μ_{k+1}). The stopping criterion for Newton’s method is if the H-norm (see Chapter 2.1.3) with H = D_{x}^{2}φ(x, μ) of the Newton step Δx = H^{−1}Dφ(x, μ_{k+1})^{T} is less than 0.2.

In our program below we implemented a simple modification of Newton’s method by using the “old” factorization of the Hessian that was computed for the last Newton step (or in the extrapolation) for four further “inexact” Newton steps with line search before recomputing the new Hessian for the next true Newton step.

5.4.5 Stopping test

The following Lemma may be used for determining when to stop.

Lemma 5.2 If for a given point x̃^{k} and H := H(x̃^{k}) (see (5.15)) there is a μ such that the Newton step Δx̃ := H^{−1}Dφ(x̃^{k}, μ)^{T} for finding the analytic center x̃(μ) (with w̃ = 0) satisfies

∥Δx̃∥_{H} ≤ \frac{1}{4}

then (5.1) has a bounded optimal solution set, and x̃^{k} is almost optimal in the following sense: λ − λ^{opt} ≤ 2nμ.

The assumption that there is such a μ can be interpreted as x̃^{k} being moderately close to the path of analytic centers.

Proof: This lemma can either be shown as in the analysis of the logarithmic barrier method in Chapter 3.1, or more elegantly using the duality results of [1]. We show the second approach. By the results on Newton’s method, Lemma 2.4, the assumption on x̃ implies that the analytic center exists, and therefore by Lemma 5.1, the set of optimal solutions of (5.1) is nonempty and bounded. First, suppose that x̃ is on the central path, x̃ = x̃(μ) for some μ > 0 and w̃ = 0. Then, by definition, the partial derivatives of φ are all zero, that is (by (5.13) and (5.14)):

\frac{∂}{∂x̃_{i}} φ(x̃, μ) = A^{(i)} • B(x̃) = 0 for i = 1, …, m  (5.17)

and

\frac{∂}{∂λ}φ(x̃, μ) = \frac{1}{μ} − traceB(x̃) = 0.  (5.18)

Also, the dual problem as given in [1] is

max_{Y ≥0}{A^{(0)} • Y | traceY = 1, A^{(i)} • Y = 0}.

Hence, the matrix μB(x̃) is dual feasible and the “duality gap” can easily be computed as

λ − μB(x̃) • A^{(0)} = −μ ∑_{i=1}^{m+1} x̃_{i}(A^{(i)} • B(x̃)) − μB(x̃) • A^{(0)}

(by substituting the equations (5.17) and (5.18) and using A^{(m+1)} = −I)

= −μ(A^{(0)} + ∑_{i=1}^{m+1} x̃_{i}A^{(i)}) • B(x̃) = μA(x̃) • B(x̃) = nμ,

and is an upper bound for λ − λ^{opt}.

If ∥Δx̃∥_{H} ≤ 1/4 then ∥x̃ − x̃(μ)∥_{H} ≤ 0.371 and ∥x̃ − x̃(μ)∥_{H(x̃(μ))} ≤ 0.59 by the results of Chapter 2.1.5, and by the inner ellipsoidal approximation this implies that λ − λ^{opt} ≤ nμ(1 + 0.59) ≤ 2nμ.

• The assumption of Lemma 5.2 is equivalent to ∥w̃ + g(x̃, μ)∥_{H^{−1}} ≤ \sqrt{α}/4, where g(x̃, μ) is the norm of the gradient of the perturbed barrier function defined in (5.14) for w̃ ≠ 0. Often (for example if the optimal solution x^{opt} is unique) one can show that lim_{μ→0} ∥w̃∥_{H(x(μ))^{−1}} = 0 so that the above stopping criterion is also useful when following a path of perturbed centers.

5.4.6 Ill-conditioning

The barrier method of the previous sections encounters ill-conditioned matrices (as μ tends to zero). We will briefly discuss the two critical points in our computations.

1. First, as μ tends to zero, the matrix A(x̃) approaches the singular matrix A(x̃^{opt}), and is therefore ill-conditioned. Simple considerations show that the minimum eigenvalue of A(x̃(μ)) is of order O(μ). It was our experience that for the small examples below with μ ≥ 10^{−10} the inversion of the matrix A(x̃) could be carried out to sufficient accuracy (Matlab is using double precision arithmetic), provided the maximum eigenvalue of A(x̃^{opt}) was O(1). For many practical applications it suffices to approximate the optimal solution up to low accuray only (e.g. μ ≥ 10^{−6}), and in these cases we expect the barrier method to work well even for larger problems. We did not examine any methods for stabilizing the computation of A(x̃)^{−1} in the case that a very high accuracy of the optimal solution is desired, and we suspect that further difficulties may arise.

However, interior-point methods that use a centering procedure at each iteration have an “implicit stability” that can be explained as follows: Note that the evaluation of the Hessian yields a positive (semi)-definite matrix H, no matter how inaccurate the computation of the matrix B = A(x̃)^{−1} is. (This is true since the ijth entry of the Hessian H is given by H_{ij} = BA^{(i)} • A^{(j)}B and

z^{T} Hz = B(∑ z_{i}A^{(i)}) • (∑ z_{j}A^{(j)})B = ∥(∑ z_{i}A^{(i)})B∥_{F}^{2} ≥ 0 .)

In our algorithm we use a uniformly positive definite approximation Ĥ of the Hessian H that is obtained from a simple “stabilization procedure” described below. Since inexact Newton’s method with a line-search converges for any uniformly positive definite approximation Ĥ of the Hessian, the center will be found eventually, even if H was a random positive definite matrix. Similarly, the extrapolation is a descent direction for the objective value λ for any positive definite approximation of H (see Step 1 of Section 5.4.4). This property suffices to prove that a method as outlined above always converges to an optimal solution x̃^{opt}, even if the computation of the Hessian is inaccurate. We point out that such a statement is not true for methods that do not use a centering procedure. If the Hessian is computed inaccurately such a method may converge to a non-optimal point. Furthermore, in general, the Hessians that are evaluated at points on the central path are less ill-conditioned than the Hessians that are evaluated at points far away from the central path. (For barrier functions of linear programs with a unique optimal solution, one can prove, for example, that the condition number of the Hessians near the central path is globally bounded independent of μ, while in general, the condition number of the Hessian becomes unbounded as the argument approaches the boundary of the feasible set.) These facts may also explain our observation in Section 5.5.3 below.

2. Second, as μ tends to zero, the Hessian H(x̃(μ)) = D^{2}φ(x̃(μ), μ) becomes ill-con-ditioned. This type of ill-conditioning also occurs when solving (degenerate) linear programs by interior-point methods. Lustig et. al. [48], Gill et. al. [20] and others present solutions for such instabilities (via iterative refinement) and show with a large number of numerical experiments that their strategies are effective. In our program we used the following simple strategy: Given the Hessian H = H(x̃) at some point x̃, set Ĥ := H + ϵ∥H∥_{∞}I (with ϵ = 10^{−10}) and denote the Newton step by Δx̃ = H^{−1}g and the “stabilized Newton step” by Δx̂ = Ĥ^{−1}g. As motivated in Chapter 2.1.3, the canonical norm associated with this method at the point x̃ is the norm ∥h∥_{H} = (h^{T} Hh)^{1/2}. It is straightforward to verify (e.g. Theorem 2.11 of Stewart and Sun, [88] page 124) that

\frac{∥Δx̂ − Δx̃∥_{H}}{∥Δx̃∥_{H}} ≤ ϵ\frac{lub_{∞}(H)}{lub_{2}(H^{−1})} ≤ \sqrt{n} ϵ cond_{2}(H).

Since Δx̂ is just an iterative direction for a line-search during Newton’s method, this error is not crucial as long as cond_{2}(H) ≪ 1/ϵ. For very ill-conditioned H though this stabilization considerably increases the number of Newton steps.

5.5 Some numerical results.

The method outlined in Section 5.4.4 has been implemented in Matlab [56]. We present a few preliminary examples for which the method shows the same convergence behavior as interior-point methods applied to linear programs. This—along with the work of Alizadeh— strongly suggests that the method may be very efficient for solving minimax eigenvalue problems and for more general convex programming problems.

5.5.1 Random examples

In Table 1 we give results for minimizing the largest eigenvalue of affine combinations of m random symmetric matrices of size n for different values of m and n. Our stopping criterion was nμ ≤ 10^{−4}. In each iteration we performed one extrapolation and a small number of Newton and “inexact Newton” steps with a line search. We list m, n and the sample size s_{m,n} of how many random problems were tested for each pair m, n. We further list the average and the maximum number of iterations needed, and the average and maximum number of evaluations of the Hessian and total (inexact) Newton steps. Finally we record the average multiplicity of the maximum eigenvalue in the optimum. For n = 5, only about 20% of all problems had a finite λ^{opt}. For the unbounded problems a negative definite combination was found in the first iteration^{1}, so that we only list the statistics for the bounded problems. For n = 10, four problems were unbounded; for n > 10, none.

n m s_{m,n} iterations Hessians inexact Newton steps multiplicity
5 10 500  7.8/9  9.5/14  23.7/37  3.0
10 20 100 9.0/11 11.4/15  29.6/42  3.8
20 40 100 10.0/12 12.5/15  33.2/43  5.1
40 80 20 11.1/13 14.0/16  34.4/42  6.7

Table 1 This table does not allow any conclusions regarding the effect of the condition numbers of the linear systems on algorithm stability for large problems. We believe, however, that it gives some impression about how the number of iterations depends on the size of the problem. We can observe the same slow growth of the number of iterations with the dimensions m and n as has been observed throughout for interior-point methods for linear programs, see e.g. [48, 20]. This makes interior-point methods particularly suitable for parallel computation, since the sequential part of the algorithm (the number of iterations) is small compared to the parallelizable part of solving a (large) system of linear equations at each iteration.

1With 2 exceptions where it was found in the second iteration. For the first iteration we added the extra bound ∥x∥ ≤ 1000 to avoid divergence for the unbounded problems.

5.5.2 Boyd and Yang’s example

Here we use our method to solve two examples given in [6, ch.8]. Boyd and Yang modelled a simple two-input two-output control system, and reduced the question of whether there exists a quadratic Lyapunov function that establishes stability of this control system to a problem of the form (5.1), involving 22 matrices A^{(i)} of size 40 × 40. In these examples A^{(0)} = 0 and the problem was to find whether there exists a definite linear combination of the A^{(i)}, i ≥ 1. For problem one there does, but problem two is bounded, i.e. λ^{opt} = 0 for x = 0. When solving these problems with our implementation it took one iteration, 3 evaluations of the Hessian and 12 steps of (inexact) Newton’s method to find a negative definite combination for problem one, and 6 iterations, 9 Hessians and 24 inexact Newton steps for problem two to find out that the optimal solution for problem two is within 10^{−4} of 5.7 ∗ 10^{−5}. (It is exactly 0.) We emphasize that also for problem two we could have stopped after one iteration, since by Lemma 5.1, the existence of the first analytic center already implies boundedness of (5.1). A more careful implementation that determines only whether the first center exists or not would suffice for this type of problem and would be more efficient. (The case that there exists a nonzero positive semidefinite but not a positive definite combination would be hard to detect, however.) Boyd and Yang [6] solve a variation of problem (5.1)

min_{|x_{i}|≤1} λ_{max}(∑_{i=1}^{n} x_{i}A^{(i)}).

This problem has a nonempty and finite set of optimal solutions for any set of symmetric A^{(i)} while problem (5.1) does not. But it is not well-posed in the sense that a definite combination of the two matrices A^{(1)} := diag(1, −1, −1) and A^{(2)} := diag(−1 + ϵ, 1 + ϵ, 1) might not be detected for very small ϵ > 0. Boyd and Yang solved the problem by the cutting plane algorithm and by the subgradient method. In Tables 2 and 3 we list their results and the results that we obtained from an interior-point method by imposing the same bounds on the problem as they did. It is straightforward (but not efficient!) to modify the interior-point algorithm to this situation. The tables show the number of iterations necessary to determine that there is a negative combination, or to find the optimal value up to an accuracy of 10^{−4} and of 10^{−6}.

Problem one λ_{max} < 0 ϵ = 10^{−4} ϵ = 10^{−6}
Interior Point  5  8  11
Cutting Plane  109  242  —
Subgradient  5507  —  —

Table 2 It should be noted that each iteration of the subgradient method involves an eigenvalue computation of a 40 by 40 matrix (this consists of five 8 by 8 blocks in this example and is thus relatively cheap). In addition, the i-th iteration of the cutting plane algorithm involves the solution of a linear program with m + 1 unknowns and 2m + i constraints. It is difficult to compare the computational effort of all three of these methods under consideration of the sparsity pattern of the matrices A^{(i)}, but for m < n and n sparse, as is the case here, we believe that an efficient implementation of the interior-point method may be very fast.

For problem two Boyd and Yang only determine that the problem has a largest eigenvalue that is larger than −ϵ. We list the results for two different values of ϵ. Since the subgradient method does not provide a lower bound on the optimal solution it failed for this example.

Problem two ϵ = 10^{−4} ϵ = 10^{−6}
Interior Point  8  10
Cutting Plane  113  —
Subgradient  —  —

Table 3

5.5.3 Overton’s example

This is an example taken from Overton [68]. The problem is to find the diagonal of a 10 by 10 matrix M to minimize its maximum eigenvalue in absolute value. The off-diagonal elements of M are given. There are obvious modifications of the algorithm in this paper that do not increase the size of the problem and that take the “extreme sparsity” of the matrices A^{(i)}—one nonzero element on the diagonal for 1 ≤ i ≤ 10—into account. Here, however, we simply applied our algorithm with the matrices

Â^{(i)} := ( \begin{matrix}A^{(i)} & 0 \\ 0 & −A^{(i)} \end{matrix} ) , 0 ≤ i ≤ 10,

thus doubling the size of the problem and working with 11 matrices of size 20. In order to test the limitations of our algorithm, we deliberately set the stopping criterion to 1.0·10^{−20}. The optimal value λ^{opt} that our program computed for this problem was 22.36612164584166. This result was obtained in iteration 17 for μ = 1.0 · 10^{−15}, and it is accurate to at least 13 digits. The program stopped in the next iteration with the Matlab-message that the matrix (A(x̃)) to be inverted has a condition number of 10^{17}, and our algorithm then terminated with an error message. In Table 4, we ran the problem with stopping criterion nμ ≤ 10^{−4} and list the iteration number, λ_{max} in that iteration, the value of μ and the rates by which λ_{max} − λ^{opt} was reduced compared to the previous iteration. Note that the final accuracy is higher than the 4 digits guaranteed by Lemma 5.2.

iteration  λ_{max}  rate of convergence
0  37.7  –
1  33.9  1.3
2  25.2  4.1
3  22.83  6.1
4  22.435  6.7
5  22.3763  6.8
6  22.3680  5.4
7  22.36642  6.2
8  22.366144  13.3

Table 4

In these iterations, the Hessian was recomputed 10 times, and a total of 25 (inexact) Newton steps were performed. Overton [68] presents an algorithm for solving this problem based on successive quadratic programming (QP). In each iteration his algorithm solves a small number of quadratic programs with m + 1 unknowns and several equality constraints, the number depending on the estimated multiplicity of λ_{max} at the solution. Overton also presents numerical experiments for the above problem, and noted that “this problem is quite difficult to solve, since at the optimal solution the interior eigenvalues are nearly equal to λ_{max}”. His program computed the optimal solution with 10 digits of accuracy in 14 iterations solving a total of 26 QP’s. We tested the interior-point method for different settings of the various parameter values but we did not observe any irregular behaviour of the interior-point method as long as the centering componenet was used. A modification of our algorithm, however, that did not use any centering steps converged only to 5 digits accuracy (and took 23 iterations to get there).

The observed robustness of the interior-point method coincides with the fact that the proof of convergence of the method is independent of the data, and only depends on the dimensions m and n.

Overton [69] also suggests a successive linear programming algorithm for problem (5.1) that is better suited for large problems. Since our current implementation does not exploit sparsity, we could not compare the two methods for large problems. (For small problems, Overton favours his SQP algorithm.) We note that both of Overton’s methods compute “dual matrix” information which is relevant for sensitivity analysis of the solution; see [69].

Extensions of the basic model of self-concordant barrier functions

In the previous chapters we presented some of the principal developments of self-concordant barrier functions and their application to solving convex problems of the form (1.2).

Next we outline two extensions, the first one is aimed to further improve the theoretical results for self-concordant functions by making a somewhat more restrictive assumption on the problem, and the second one is aimed to enlarge the class of problems under consideration, and to cover a “slightly” non-convex problem by the concept of barrier methods with self-concordant functions.

• In Chapter 6 we define the notion of self-concordance of order 2, which is a stronger assumption than (2.2), but is nevertheless satisfied by the standard examples for (2.2). We show that for certain questions, self-concordance of order 2 implies stronger theoretical results than (2.2), and point out how these results might be used during a line search.

• In Chapter 7 we present an extension of an interior-point method to solving fractional

programs of the form

min\{\frac{x_{2}}{x_{1}} | x ∈ S\}.  (P)

Recall that the method of centers is based on a sequence of parameters λ^{k} > λ^{∗} := min{c^{T} x| x ∈ S}, with λ^{k} → λ^{∗} as k → ∞. At each step the analytic center x(λ^{k}) of the sub level set S ∩ {x| c^{T} x ≤ λ^{k}} is approximated. Self-concordance of ϕ allows a proof of “polynomial” convergence of the method of centers, where “polynomial” indicates that the rate of convergence is polynomial; more precisely, at each step of the algorithm, the “distance” λ − λ^{∗} is reduced by a factor 1 − 1/(17\sqrt{θ}) where θ is the self-concordance parameter of ϕ The step “λ^{k} ↦ λ^{k+1}” in the method of centers corresponds to a parallel shift of the constraint “c^{T} x ≤ λ”. Here, we show that—with a very similar argument as used for the analysis of the method of centers—also the rotation of some linear constraint about the origin allows the proof of a polynomial rate of convergence. Moreover, the rate of convergence is of the same order as for the method of centers. This result is applied to problem (P) showing that (P) is solvable up to a specified accuracy ϵ in polynomial time. (Here, the polynomial depends on \sqrt{θ} and on | log ϵ|.) A new element of this approach is the fact that (P) is slightly nonconvex; (“slightly” since it is still pseudo-convex). We conclude this Chapter with some numerical examples for the (non-convex) problem of minimizing the condition number of some affine combination of matrices which is a special case of problem (P). Apart from polynomial solvability, the numerical examples indicate that a modification of our method may be a viable approach to solve this problem numerically.

Self-concordance of order two

This Chapter is concerned with improvements of the model of self-concordance for the special case of a line-search problem. The development of an efficient subroutine for a linesearch is an important detail of the implementation of interior-point methods for solving linear or nonlinear convex programs that has received little attention so far.

Even though the line-search is “only” a one-dimensional problem, it is far from trivial, and our analysis below shows a new approach in which the structure of the line-search problem can be exploited. In particular, higher-order directional derivatives can be used. While higher-order derivatives of multivariate functions are often too expensive to compute, higher-order directional derivatives are available at low computational cost for a number of problems. This observation is the motive to define and exploit the notion of self-concordance of order two, relating the second and fourth derivative of a barrier function φ. Our theoretical results (taken from [33]) allow the computation of sharp lower bounds on the distance from a given point t to the minimum of φ along a given line. For functions φ satisfying the self-concordance condition of order two and with cheap third-order directional derivatives we devise a cubically convergent procedure that is faster than an apparently favourably damped version of Newton’s method. We note that if the domain of φ is bounded along the search direction, and if its boundary is easy to compute, efficient search steps may also be derived based on the knowledge of the boundary; see [58, 59].

6.1 The Problem

We are concerned with line-search procedures for a class of problems with certain nonlinear constraints. The class includes as special cases linear and convex quadratic programming problems, entropy programming problems and minimization problems over the cone of positive semidefinite matrices, [1, 64, 32]. We briefly recall the basic notation used in this Chapter.

For solving a constrained optimization problem of the form

min{c(x) | r_{i}(x) ≤ 0 for 1 ≤ i ≤ m}  (6.1)

with four times continously differentiable functions r_{i}, we consider logarithmic barrier functions of the form

φ(x, μ) = \frac{c(x)}{μ} − ∑_{i=1}^{m} log(−r_{i}(x))  (6.2)

that depend on the variable x and on a fixed positive parameter μ. Occasionally, we will consider the logarithmic barrier function without objective function c, that is with μ = +∞. We assume that φ is convex and self-concordant. Our analysis is not influenced by additional linear constraints Ax = b in (6.1); for convenience, however, we omit linear equalities in this Chapter. Let x be some strictly feasible point for (6.1) and h be some search direction for finding the minimum of φ. Without loss of generality we assume for the rest of this Chapter that h is a descent direction for φ. In slight variation to the definition of the function f in (2.1), let

f(t) = f_{x,h}(t) = φ(x + th, μ).  (6.3)

By the assumption on h: f′(0) < 0. In the following we will also assume that f satisfies (2.2). By t̄we denote the minimizer of f, and under assumption (2.2), the number t̄∈ (0, ∞] is unique.

We consider line-search procedures of the following general form: Given a point t_{k} in the domain of f, a search step s_{k} for minimizing f is generated and the next point t_{k+1} is given by t_{k+1} = t_{k} + s_{k}. In our analysis, however, we omit the index k and will always assume that our current point t is the point t = 0. This assumption is possible without loss of generality. Indeed, after computing a step s for minimizing f we may consider a “new” function, say f̃(t) := f(s + t) defined via (6.3) at the “new” point x̃ = x + sh by f̃ = f_{x̃,h}.

Recall that the canonical norm induced by the function φ at a point x is given by ∥v∥_{H} := \sqrt{v^{T} D_{x}^{2}φ(x, μ)v}. Here, H = H_{x,μ} = D^{2}φ(x, μ) stands for the Hessian of φ at x. We will use a similar notation here (associating H with the “Hessian” f′′(0)) and denote the H-length of some number Δt by

|Δt|_{H} := |Δt|\sqrt{f′′(0)}.

It is our goal to perform a line-search, i.e. to find a point t̄ > 0 where f is minimized. To avoid confusion, here and in the following we will refer to the one-dimensional step Δt as the Newton step (for f), and to −D_{x}^{2}φ(x, μ)^{−1}D_{x}φ(x, μ) as the Newton direction (for φ).

We start our derivations by recalling the definition of the reduced Newton step

s = \frac{Δt}{1 + |Δt|_{H}} .  (6.4)

As shown in Lemma 2.6, a line-search using the reduced Newton step s (6.4) is monotonically convergent, i.e. s is “a little too short”, s ≤ t̄. Further, s is the largest step possible that is guaranteed to satisfy s ≤ t̄ if all that is known are the first two derivatives of f at the current point t = 0.

In the following we will further exploit the properties of the logarithmic barrier function and obtain additional results that are unpractical in higher dimensions but appear to be useful for a line-search.

First we illustrate with a simple example the difficulties that arise when trying to devise a heuristic for a line-search.

6.2 A simple example

Consider the logarithmic barrier function

f(t) = −(m − 1) log(1 − t) − log t,

that arises if the feasible region [0, 1] is described by m − 1 (identical) constraints of the form t ≤ 1 and the constraint t ≥ 0. Clearly, the minimum of this function is at t̄ = 1/m. For m ≫ 1 the situation arising here is typical for the problems that are encountered during a line-search in that the minimum of f lies rather close to a pole of f (see our remarks at the end of this Chapter). If we choose a starting point t_{0} for Newton’s method for finding t̄ that is fairly close to t̄, e.g. t_{0} = 1/\sqrt{m}, we find that the (first) Newton step Δt has length ≈ 1/2. This means, the Newton step is too long by a factor of about \sqrt{m}/2 , and the resulting point is outside the feasible set. If we choose a starting point that is further away from t̄, e.g. t_{0} = 1/2, we find to our surprise that the Newton step is exact. And if we move even farther away from t̄ it turns out that the Newton step becomes much too short. Below we show a graph of this situation for m = 100 by depicting the following quantities in Figure 1:

– the H-length of the Newton step Δt (dotted line)

– the “relative accuracy” of the Newton step, i.e. the quotient “Newton step / true distance to the minimum” (solid line)

– the relative accuracy of the reduced Newton step (dashed line). The graph and other numerical results are obtained using Matlab [56].

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Figure 1 Relative accuracy (and H-length of Δt) 0 1 2 3 4 5 6 7 8 9 10 ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.} Newton step^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.}_{.} reduced Newton step .................................................................................................................................................................................................................................. H-length

Moving from t_{0} = 1 to t_{0} = 0, we see that the Newton step is first too short (i.e. the quotient is very close to zero), then (much) too long and then again too short. In particular, the information that the H-length of the Newton step is large (≫ 1) does not give any information whether the Newton step is too long or too short. Thus it is hard to identify the domain in which the Newton step contains some useful information (about the step length for the line-search), and the domain in which for example bisection (and backtracking) would be a better choice to minimize f. Note further, that a seemingly “good” Newton step starting at t_{0} = 0.45 that is too long by a factor of only 1.1 still yields an infeasible point and is rather useless. We are therefore interested in search steps that are guaranteed not to be too long. (Note that in general, the H-length of the Newton step Δt reveals no information about the usefulness of the direction h defining f via (6.3).)

In the next section we describe a property that may help us to further analyze f and to develop new results that improve the rate of convergence.

6.3 Self-concordance of order two

Our further analysis is based on the observation that in addition to (2.2) the logarithmic barrier function of several convex programming problems satisfies a second property, namely that

|f′′′′(0)| ≤ 6f′′(0)^{2}, f′′(0) ≥ 0.  (6.5)

In particular, the examples mentioned in the introduction satisfy this condition. Property (6.5) will be referred to as self-concordance of order two, and like condition (2.2), it may be interpreted as a condition for f, or for φ, or for problem (6.1). Condition (6.5) is stated in terms of f, however, by the definition of f, it may also be regarded as a condition on φ, or, by (6.2), as a condition on problem (6.1).

We first show that this estimate is valid for the logarithmic barrier function of linear constraint functions: Clearly, if f(t) satisfies property (6.5) then so does f(rt + s) with r, s constants, i.e. just like (2.2), the property is invariant under affine transformations. Also, if f_{1} and f_{2} satisfy (6.5) then so does f_{1} + f_{2}, (and for r > 0, rf satisfies a similar property with 6 replaced by 6/r). Since relation (6.5) is trivially true for f(t) = log t this completes the proof for linear constraints. By the same observation as in Chapter 2, the logarithmic barrier function of a convex quadratic constraint function q(x) ≤ 0 also satisfies (6.5).

Similarly, also the function

f(t) = − log det(X + tY)  (6.6)

of a positive definite matrix X and symmetric Y satisfies (6.5). Note that, for the special cases, linear, quadratic and semi-definite programming, property (6.5) is generalizable in the sense that if k is an even number and l > k, then a suitable multiple of the k-th derivative raised to the power l/k is a global upper bound for the l-th derivative of the function f.

While the previous three examples have identified certain constraints whose logarithmic barrier functions satisfy (6.5), we briefly mention a composition rule that allows us to identify possible objective functions c in (6.2). Let g(t) := c(x + th) and f(t) := − ∑ log(−r_{i}(x + th)) be the line-search function (6.3) with μ = ∞. If for all strictly feasible x and all h, the function f satisfies (6.5) and if g is convex and satisfies

g′′′′(0) ≤ 12f′′(0)g′′(0),  (6.7)

then the function φ in (6.2) satisfies (6.5) for any μ > 0. This statement is analogous to the “compatibility” condition for self-concordant functions in [64]; its proof is straightforward. We may note here, that many features of self-concordant barriers—affine invariance, invariance properties with respect to addition and multiplication, compatibility—carry over to self-concordant functions of order two in slightly modified form. We also believe that many functions that have been identified to satisfy (2.2) satisfy (6.5) as well.

Entropy programming problems with objective function

c(x) = c^{T} x + ∑_{i} x_{i} log x_{i}

subject to the constraint x ≥ 0 (and possibly further constraints satisfying (6.5)) are an application of the compatibility relation (6.7). To verify (6.5) for the associated function φ in (6.2) it suffices to consider g(t) = (t_{0} + t) log(t_{0} + t) and f(t) = − log(t_{0} + t). Again it is straightforward to check that these functions satisfy (6.7). (For the extended entropy programming problem we may also derive a compatibility condition similar to the one in [8].)

6.4 Additional results

This section is motivated by the fact that in a number of cases higher-order derivatives are easily computable for functions f generated by the logarithmic barrier function.

If f is given by

f(t) = c^{T} (x + th) − ∑_{i=1}^{m} log(b_{i} − a_{i}^{T} (x + th)),

the derivatives of f are

f′(t) = c^{T} h + ∑_{i=1}^{m} \frac{a_{i}^{T} h}{b_{i} − a_{i}^{T} (x + th)}  (6.8)

and

f^{(k)}(t) = (k − 1)! ∑_{i=1}^{m} \frac{(a_{i}^{T} h)^{k}}{(b_{i} − a_{i}^{T} (x + th))^{k}}  (6.9)

for k ≥ 2. The “expensive” part (O(mn)) is to compute all the scalar products a_{i}^{T} x and a_{i}^{T} h, but once these are known, it is comparatively cheap (O(m)) to compute higher-order derivatives. (For linear and quadratic constraints, it is also cheap to evaluate the function f and its derivatives for other values of t, once f is evaluated for t = 0. The products a_{i}^{T} x can be updated in the course of an algorithm, so that only the evaluation of a_{i}^{T} h needs to be carried out. But note that for sparse data a_{i} or for computations on parallel machines the evaluation of the inner products is cheaper than O(mn).) Similarly, the function f of (6.6) allows the cheap computation of its derivatives via (2.6). Higher-order derivatives are cheap also for quadratic constraints or for the entropy programming problem. In general, results of automatic differentiation imply that the cost of obtaining higher-order directional derivatives is never more than a small multiple of that needed for the directional derivative itself.

If f′′′ can be computed cheaply, this suggests to exploit condition (6.5) by a procedure for calculating a reduced third-order search step s̄ that is guaranteed not to be larger than the minimum t̄, and that is maximal with respect to this property. The reduced third-order search step is based on the Weierstraß ℘-function which satisfies the differential equation

℘^{′2} = 4℘^{3} − 1,  ℘′′ = 6℘^{2},  (6.10)

and by relating (6.10) to (6.5), we are led to the third-order search step. Note here, that the first part of (6.10) implies that

|℘′| = (4℘^{3} − 1)^{1/2} < 2℘^{3/2},

i.e. the second integral of ℘ satisfies the self-concordance relation (2.2).

We digress at this point to recall some known results (taken from [87]) about the Weierstraß ℘-function that will be used below. In particular, we reproduce some power series expansions of ℘ and related functions that we used for our numerical examples; we believe, however, that it is more efficient to evaluate these functions by other methods as for example using tables with precomputed values. In any case, for the determination of our search step below, one evaluation of each function will suffice, and each evaluation will cost a constant number (less than thirty) of multiplications and additions.

The Weierstraß ℘-function is determined by two parameters g_{2} and g_{3}. We are only interested in the case that ℘ is defined by the invariants g_{2} = 0 and g_{3} = 1 (this is referred to as the equianharmonic case in the literature) and in real arguments for ℘. For this case, the description of ℘ involves the following constants:

ω_{2} = \frac{Γ^{3}(\frac{1}{3})}{4π} ≈ 1.529954037, e_{2} = \frac{1}{\sqrt[3]{4}}, η_{2} = \frac{π}{2ω_{2}\sqrt{3}} ≈ 0.59276269754.  (6.11)

The function ℘ is periodic with period 2ω_{2}, and has poles at t = 2nω_{2}, n ∈ ZZ. Further, for t ∈ (0, 2ω_{2}), it holds that ℘(t) ≥ e_{2}. Numerically, ℘ can easily be evaluated by its power series expansion. We list two possible expansions for ℘. First,

℘(t) = \frac{1}{t^{2}} + \frac{1}{28}t^{4} + \frac{1}{10192}t^{10} + \frac{1}{5422144}t^{16} + O(t^{22})  (6.12)

which may be used for t ∈ (0, 1) e.g., and second

℘(t + ω_{2}) = e_{2} + 3e_{2}x[1 + x + x^{2} + \frac{6}{7}x^{3} + \frac{5}{7}x^{4} + \frac{4}{7}x^{5} + O(x^{6})]  (6.13)

where x = e_{2}t^{2}. (Both ℘(t) and ℘(ω_{2} + t) are even functions.) The above expansions also yield the expansions for ℘′ and for the Weierstraß ζ-function, with ζ′ = −℘. (The constant term of ζ in (6.12) is zero; in (6.13) it is η_{2}.) Finally, the power series expansions of the inverse functions to ℘ and to ζ are known. Let ℑ : [e_{2}, ∞) → (0, ω_{2}] be the inverse function of ℘; then

ℑ(z) = z^{−1/2}[1 + \frac{u}{7} + \frac{3u^{2}}{26} + \frac{5u^{3}}{38} + \frac{7u^{4}}{40} + \frac{63u^{5}}{248} + \frac{231u^{6}}{592} + \frac{429u^{7}}{688} + O(u^{8})],

where u = z^{−3}/8 and z ≥ 0.9. For z ∈ [e_{2}, 0.9) the following expansion may be used:

ℑ(e_{2} + z) = ω_{2} − \frac{1}{3e_{2}} \sqrt{3z − \frac{z^{2}}{e_{2}} + \frac{z^{3}}{3e_{2}^{2}} − \frac{2z^{4}}{21e_{2}^{3}} + O(z^{5})}.

The inverse function ℜ : IR → (0, 2ω_{2}) to ζ has the power series

ℜ(z) = z^{−1}[1 − \frac{γ}{7} + \frac{17γ^{2}}{143} − \frac{496γ^{3}}{3553} + O(γ^{4})]  (6.14)

for z ≥ 0.75 and γ = 1/(20z^{6}), and

ℜ(z) = ω_{2} − \frac{z − η_{2}}{e_{2}} [1 − y + \frac{12y^{2}}{5} − \frac{267y^{3}}{35} + \frac{139y^{4}}{5} − \frac{30192y^{5}}{275} + \frac{1634208y^{6}}{3575} + O(y^{7})]

where y = (z − η_{2})^{2}/e_{2} and |z − η_{2}| is small. Finally, it holds that ζ(z + 2ω_{2}) = ζ(z) + 2η_{2} (℘ is periodic but ζ is not!) and that ζ is an odd function. From this we obtain the relation ℜ(z) = 2ω_{2} − ℜ(2η_{2} − z) which allows us to evaluate ℜ for z ≤ 0.45 e.g. via (6.14).

We now return to the problem of finding sharp lower bounds on the location of the minimum of f. Let the values

f′(0) = a < 0, f′′(0) = b > 0, and f′′′(0) = c  (6.15)

be given and define v(t) := f′′(t). Self-concordance of order two implies that

v′′(t) ≤ 6v(t)^{2}, v(0) = b, v′(0) = c.  (6.16)

We intend to determine the domain of the function v for positive t as well as a lower bound on the location of the minimum of f. As in the proof of Lemma 2.2, the structure of the differential inequality (6.16) implies that the solution w of the initial value problem

w′′(t) = 6w(t)^{2}, w(0) = b > 0, w′(0) = c  (6.17)

is an upper bound for v. More precisely, let [0, t^{+}] be the intersection of the domains of v and w, then

w(t) ≥ v(t) ≥ 0  and  w′(t) ≥ v′(t)  (6.18)

for t ∈ [0, t^{+}].

The above statement holds in a more general setting; for the special case needed here we give a short proof: By construction of v and (6.5) we have v(t) ≥ 0. From (6.19) below it follows that also w(t) ≥ 0. Assume that (6.18) is not true and there exists t^{∗} ∈ [0, t^{+}) and ϵ ∈ (0, t^{+} − t^{∗}) such that w(t^{∗}) ≥ v(t^{∗}) and w′(t^{∗}) ≥ v′(t^{∗}) and either w(t) < v(t) or w′(t) < v′(t) for all t ∈ (t^{∗}, t^{∗} + ϵ] (or both). (We may think of t^{∗} as the smallest point t ≥ 0 where (6.18) starts being violated.) For convenience we assume t^{∗} = 0 in the sequel.

Case 1), assume w(t) < v(t): Let Δ(t) := v(t) − w(t), then Δ(0) ≤ 0, Δ′(0) ≤ 0 and Δ′′(t) ≤ 6v^{2}(t) − 6w^{2}(t) = 6Δ(t)(v(t) + w(t)). By definition of t^{∗} it follows that Δ(t) > 0 for all t ∈ (0, ϵ]. Thus,

0 < Δ(t) = Δ(0) + tΔ′(0) + \frac{1}{2}t^{2}Δ′′(η) ≤ 3t^{2}Δ(η)(v(η) + w(η))

for some η = η(t) ∈ (0, t). Given an initial value t_{0} ∈ (0, ϵ] we may consider the (monotonically decreasing) sequence t_{k+1} = η(t_{k}). The previous inequality can be written as

Δ(t_{k+1}) ≥ Δ(t_{k})\frac{1}{3t_{k+1}^{2}(v(t_{k}) + w(t_{k}))} > 0,

and for sufficiently small t_{0} we obtain Δ(t_{k}) → ∞ since v and w are bounded in [0, ϵ]. This is a contradiction.

Case 2), assume w′(t) < v′(t) and w(t) ≥ v(t): Let Δ(t) be as above. From 0 < Δ′(ϵ) = Δ′(0) + ϵΔ′′(η) ≤ 6ϵΔ(η)(v(η) + w(η)) ≤ 0 we obtain an immediate contradiction.

The general solution of (6.17) is the ℘-function with invariants g_{2} = 0 and g_{3} ∈ IR. In our case, the solution can be reduced to the equianharmonic case g_{2} = 0, g_{3} = 1 by the transformation

w(t) = α^{2}℘(αt + β)  (6.19)

with

α = ^{6} \sqrt{4b^{3} − c^{2}} and β = \{ \begin{matrix}ℑ(b/α^{2}) & c ≤ 0 \\ 2ω_{2} − ℑ(b/α^{2}) & c > 0 \end{matrix} .  (6.20)

Here, 4b^{3} ≥ c^{2} is guaranteed by condition (2.2). (The case α = 0 yields the invariants g_{2} = g_{3} = 0 which correspond to the Weierstraß function ℘(t) = 1/(1 − t)^{2}.)

Using (6.10) it is easily verified that w indeed satisfies (6.17). Since v ≤ w, the pole of v must lie behind the one of w, i.e. 0 < w < v if w and v denote the poles of w and v. In the next lemma we thus obtain an inner approximation of the feasible domain of f, which is slightly better than the bound (2.2). (As noted above, the second integral of the ℘-function satisfies (2.2), and since the bound (2.2) holds for all functions satisfying (2.2), it follows that the domain of ℘ yields an estimate that is equally good or better than (2.2).)

Lemma 6.1 Let f satisfy (2.2) and (6.5), and let α and β be given by (6.15) and (6.20). If t ≥ 0 satisfies

|t|_{H} = t\sqrt{b} < \frac{\sqrt{b}}{α}(2ω_{2} − β)

then t is strictly feasible^{1}.

More important yet, we may also derive a lower bound on the minimum t̄ of f. Let

W (t) = −αζ(α(t + β))  (6.21)

be an antiderivative to w (6.19). The value s̄ with W (s̄) − W (0) = ∫ s̄ ; 0 w(t) dt = −a is a lower bound for t̄ that is better^{2} than the bound derived in Lemma 2.6 and can easily be computed using the quantities computed in (6.15), (6.20), and the function ℜ given in (6.14). We obtain

s̄ = α^{−1}(ℜ(\frac{a}{α} + ζ(β)) − β),  (6.22)

and will refer to s̄ as the reduced third-order search step. Note that the evaluation of α in (6.20) is cheap once the first three derivatives a, b, c of f in (6.15) are known; the computation of β and s̄ involves the evaluation of ℑ, ζ and ℜ (once each). If these functions are implemented with sufficient care (see [87]), the cost of their evaluation is comparable to the cost of evaluating , say, the log-function once. The construction of s̄ implies the following lemma.

Lemma 6.2 Under the assumptions of Lemma 6.1, the reduced third-order search step s̄ (6.22), the reduced Newton step s (6.4), and the minimum t̄ of f satisfy the relation

0 < s ≤ s̄ ≤ t̄.

Furthermore, the reduced third-order search step is the largest step possible that is guaranteed not to exceed the minimum of f if all that is known are the first three derivatives of f at t = 0.

Let C be the class of functions f satisfying (6.5) and with f′(0) = a, f′′(0) = b, f′′′(0) = c. The previous lemma stated that s̄ is the largest step in the sense

s̄ = max{s | s ≤ t̄(f) ∀f ∈ C}  (6.23)

1For the examples considered here this bound is merely for illustration since the exact domain is known.

2Again, the fact that the new bound is actually better follows from the fact that the second integral of the ℘-function satisfies (2.2).

where t̄(f) := arg min f. Clearly, if the class C comprised just one function then s̄ = t̄, and a line-search based on the reduced third-order search step would converge in one step. As the class C grows, the convergence subject to (6.23) becomes slower. Below we show that the reduced third-order search step is cubically convergent, i.e. the step is locally third-order accurate, and thus its name is justified. Proof: It is straightforward to verify that t̄ is the limit of the reduced third-order search step iteration. Since f′′(t̄) > 0, it is sufficient to show that f′ decreases cubically. Note that by the definition of s̄ we have the following Taylor expansions of W and of f′

a + s̄W′(0) + \frac{1}{2}s̄^{2}W′′(0) + \frac{1}{6}s̄^{3}W′′′(ξ_{1}) = 0,

a + s̄f′′(0) + \frac{1}{2}s̄^{2}f′′′(0) + \frac{1}{6}s̄^{3}f′′′′(ξ_{2}) = f′(s̄),

with ξ_{1}, ξ_{2} ∈ [0, s̄] ⊆ [0, t̄]. By construction, f′′(0) and W′(0), as well as f′′′(0) and W′′(0), concide, so that by taking the difference we obtain

\frac{1}{6}s̄^{3}(f′′′′(ξ_{2}) − W′′′(ξ_{1})) = f′(s̄),

and using (6.5) and (6.17) yields

s̄^{3}(f′′(ξ_{2})^{2} + W′(ξ_{1})^{2}) ≥ |f′(s̄)|.  (6.24)

By observing the continuous dependence of α and β in (6.20) on t and by definition (6.21) we may conclude that W′ in (6.24) is bounded. Since f′′(ξ_{2}) → f′′(t̄), also f′′ in (6.24) is bounded so that the claim follows.

In Figure 2 we illustrate the result of Lemma 6.2 and compare the relative accuracy of the reduced Newton step (dashed line) and of the reduced third-order search step (solid line) for the example in Section 3. The improvement of our reduced third-order search step over the reduced Newton step is most significant in the domain t ∈ (0, 0.01) (close to a single constraint). For better resolution we have restricted t in Figure 2 to the interval [0, 0.1]. For large t (e.g. for t > 0.5) both search steps are poor and need to be modified.

0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1 Figure 2 Relative accuracy 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{reduced third-order step} ........................................................................................................................^{reduced Newton step}

Even though the reduced third-order search step s̄ is guaranteed to be better than the reduced Newton step, it does not give a satisfactory rate of convergence for all starting points t_{0} (e.g. not for t_{0} > 0.5 in the above example) as the relative accuracy is close to zero. It may be a good initial trial step though if the current iterate is close to the boundary, if the feasible domain is unbounded, or if its boundary is hard to compute. To give an idea of the efficiency of this search step we present some numerical examples in the next section.

6.5 Some numerical examples

In Table 1 we list results for the repeated iteration of the reduced Newton step (6.4) and the reduced third-order search step (6.22). Each row lists the starting point t_{0} and the average number of iterations needed by each method for 20 random examples with 5000 linear constraints and without objective function. In order to move the minimum of f close to the boundary of the feasible domain we chose an unbalanced distribution of the constraints; 100 constraints are of the form t ≥ β_{i} with β_{1} = 0 and β_{i} uniformly in [−10, 0] for i ≥ 2. The remaining 4900 constraints are of the form t ≤ γ_{i} with γ_{1} = 1 and γ_{i} ∈ [1, 2] (uniformly) for i ≥ 2. The minima of the barrier functions lie around t̄ ≈ 0.0003. As final accuracy we require |Δt|_{H} ≤ 0.01. (This corresponds to about 10 digits relative accuracy for these examples.)

t_{0} red. Newton red. third-order
10^{−8}  29.2  5.1
0.1  9  8
0.999  127.7  104.7

Table 1 “average number of iterations”

Since we are interested in the global convergence behaviour we chose starting points that all have a poor accuracy (relative to the feasible domain [0, 1]). Clearly, the last row has a very bad starting point; it is remarkable, however, that for the first starting point the reduced third-order search step is quite efficient even though it starts from very close to the singularity (zero).

In Table 2 we list an example with 5000 constraints and the linear objective t. This time the constraints are “evenly” distributed, i.e. there are 2500 constraints t ≥ β_{i} with β_{i} in [−10, 0] and 2500 of the form γ_{i} ≥ t with γ_{i} in [1, 11]. The value t̄ is the minimum of f, and is almost constant for the 20 random examples in each row.

μ  t_{0} approx. t̄ red. Newton red. third-order
10^{−5} 0.1  10^{−5}  12.9  11.5
10^{−5} 10^{−8}  10^{−5}  21  3.6
10^{−10} 0.1  10^{−10}  13.4  11.9
10^{−10} 10^{−14} 10^{−10}  27  3

Table 2 “average number of iterations”

We showed that the “reduced Newton step” underestimates the distance to the exact minimizer of a self-concordant function f, and is best possible in this respect assuming only second-order information. Nevertheless, the above examples clearly indicate that the reduced Newton step may severely underestimate the distance and is thus much too slow for a practical implementation. To improve over the reduced Newton step we define a “reduced third-order search step” with analogous underestimation properties for functions f that are “self-concordant of order two”. The reduced third-order search step is shown to be better than the reduced Newton step and best possible assuming only third-order information.

To motivate our numerical examples we recall that the barrier functions considered here are used as a means to solve constrained optimization problems. Since the solutions of the underlying optimization problems lie at the boundary (in general), the minima of the barrier functions are typically very close to the boundary. For a line-search function f : (a, b) → IR as investigated here, it follows that its minimum t̄ lies typically very close to one end of its domain, and the other end is far away (or infinity), say |a − t̄| ≪ |b − t̄|. In our numerical examples we simulated this situation and we could observe that the reduced Newton step is unsatisfactory (i.e., much too short) at both ends a and b of the domain. In some cases the reduced third-order search step is considerably better. More precisely, it appears to be quite satisfactory in (a, t̄) even if the starting point happens to lie close to the pole a of f, but it is still bad near the “far end” b.

Fractional programs with convex constraints

Here, we extend the approach of Chapter 3.1, and we present a polynomial method for fractional programs with convex constraints. Only few applications of interior-point techniques to classes of fractional programs have appeared in the literature. Anstreicher [2] was the first to present an interior-point algorithm for linear-fractional programs. In [66], Nesterov and Nemirovsky studied an interior-point method for a class of generalized linear-fractional programs. While the linear-fractional programs in [2] are equivalent to linear programs, the problems considered in [66] and in [16] include “truly nonconvex” problems, such as minimizing the condition number of a positive definite matrix. Nesterov and Nemirovsky do not give a complete proof of polynomiality of their method in [66], the recent paper [61], however includes a full complexity analysis of a modified method for the same class of problems. The analyses in [61, 66] refer to a more general class of problems than those considered in [16], but the assumptions are different and no numerical examples are given. This Chapter is a minor modification of the article [16]. The modification being that the order of the rate of convergence has been improved by a factor \sqrt{θ} by introducing some artificial constraints. (This modification is merely to improve the theoretical bound on the number of iterations, it does not reduce the number of iterations in the practical implementation.) We include some numerical experiments indicating that the method may be attractive in practice.

7.1 The problem

We consider fractional programs with convex constraints of the form

minimize_{x∈IR^{n}} \frac{x_{2}}{x_{1}}  subject to r_{i}(x) ≤ 0, i = 1, 2, …, s,  (7.1)

or

maximize_{x∈IR^{n}} \frac{x_{2}}{x_{1}}  subject to r_{i}(x) ≤ 0, i = 1, 2, …, s,  (7.2)

with constraint functions r_{i} : IR^{n} ↦ IR ∪ ∞. We denote by

S := { x ∈ IR^{n} | r_{i}(x) ≤ 0, i = 1, 2, …, s }  (7.3)

the feasible domain of (7.1) and (7.2). We always assume that S is a bounded convex set with a nonempty interior, and that

(x_{1}, x_{2}) ≠ (0, 0) for all x ∈ S.  (7.4)

Condition (7.4) guarantees that (7.1) has an optimal solution if the objective function is bounded below on S. We note that our algorithm below is still well defined, and also our convergence analysis still holds if condition (7.4) is violated, but in the absence of optimal solutions for (7.1), our convergence result no longer allows the interpretation given in Section 7.1.2.

We stress that the objective function c(x) = x_{2}/x_{1} is not convex (and neither concave), and hence neither (7.1) nor (7.2) are convex programming problems. However, the function c is pseudo-convex if the feasible set S satisfies the additional condition

x_{1}, x_{2} ≥ 0 for all x ∈ S.  (7.5)

We will show below that any fractional program (7.1) can always be reduced to a problem of the same form (7.1) such that S fulfills (7.5). Furthermore, we will see that problem (7.2) can be rewritten in the form (7.1).

7.1.1 Applications

Problems of the form (7.1) and (7.2) include a number of important special cases. First, consider optimization problems of the type

maximize_{x∈IR^{n}} \frac{g(x)}{h(x)}  subject to x ∈ S,  (7.6)

where the denominator function h satisfies h(x) > 0 for all feasible points x ∈ S. The problem (7.6) is called a concave-convex fractional program [78] if the numerator function g is concave, h is convex, and the constraint functions r_{i}, i = 1, 2, …, s, in (7.3) are all convex. Concave-convex fractional programs arise in many applications; see [78, 79, 80] and the references given there. Clearly, the concave-convex fractional program (7.6) is equivalent to the problem

maximize_{α, β∈IR, x∈IR^{n}} \frac{β}{α}
subject to h(x) − α ≤ 0, β − g(x) ≤ 0, r_{i}(x) ≤ 0, i = 1, 2, …, s,

which is a fractional program of the type (7.2).

Second, consider the problem of minimizing the Euclidean condition number of positive definite matrices M(x) that depend affinely on x ∈ IR^{n}. Let M_{0}, M_{1}, …, M_{n} be given real symmetric l × l matrices, and set

M(x) := M_{0} + ∑_{j=1}^{n} x_{j}M_{j}.  (7.7)

Moreover, we require that

M(x) ≠ 0 for all x ∈ IR^{n}  (7.8)
and M(x^{(0)}) > 0 for at least one x^{(0)} ∈ IR^{n}.  (7.9)

Here and in the sequel, the notation M > 0 (M ≥ 0) is used to indicate that M is positive definite (positive semidefinite). The problem then is to find, among all positive semidefinite matrices M(x), one with minimal condition number:

minimize_{x∈IR^{n}} \frac{λ_{max}(M(x))}{λ_{min}(M(x))}  subject to M(x) ≥ 0.  (7.10)

Here, λ_{max}(M) and λ_{min}(M) denote the largest and smallest eigenvalue of M. Applications that lead to problems of the type (7.10) are described in [10, 25]. Setting, for symmetric matrices M,

det_{+}(M) := \{ \begin{matrix}det(M) & if M ≥ 0, \\ −∞ & otherwise, \end{matrix}  (7.11)

we can rewrite (7.10) as the following optimization problem:

minimize_{α, β∈IR, x∈IR^{n}} \frac{β}{α}
subject to − α ≤ 0, −det_{+}(M(x) − αI) ≤ 0, −det_{+}(βI − M(x)) ≤ 0.

(7.12)

Here, I denotes the l ×l identity matrix. Clearly, the problem (7.12) is a fractional program of the type (7.1). The condition (7.9) guarantees that the feasible set of (7.12) has a nonempty interior. Note that, in (7.12), α = β = 0 is feasible if, and only if, there exists a point x ∈ IR^{n} with M(x) = 0. Therefore, the assumption (7.8) guarantees that the condition (7.4) is satisfied for (7.12). If M(x) is a linear combination of matrices, i.e., M_{0} = 0 in (7.7), then we can fix α = 1 in (7.12). Thus, for this special case, the problem (7.12) is actually a convex program, and it can be solved by the interior-point methods described in [64, 31], or by the dual approach in [1].

7.1.2 Notation and main result

For vectors x ∈ IR^{n}, n ≥ 2, we always denote by x_{1} and x_{2} the first two components of x, and

x_{1,2} := [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] ∈ IR^{2}

is the truncated vector with components x_{1} and x_{2}.

In the next Section, we summarize our assumptions on (7.1) and (7.2). Then we present our interior-point method for fractional programs, and show that this algorithm computes strictly feasible iterates x^{(k)}.

To motivate our convergence result, we observe that the objective function c of (7.1) only depends on the first two components of the variable x. Given a point x that is feasible for (7.1), it is thus natural to measure the “distance” of x to optimality by its first two components x_{1} and x_{2}. For this purpose, we define two-dimensional convex sets C_{k} containing the projection x_{1,2}^{(k)} of the kth iterate x^{(k)} on its first two components, as well as the origin (of IR^{2}) and the projections x^{opt}_{1,2} of all optimal solutions of (7.1). It is then shown that after every 71\sqrt{θ} iterations the area ν(C_{k}) of C_{k} is reduced by at least 50 percent. Here, θ ≥ 1 is the self-concordance parameter of a barrier function associated with the feasible set S.

This result may become more intuitive by the observation that for small values of ν(C_{k}) the points x^{(k)} are “almost optimal” in the following sense.

Proposition 7.1 Let x^{opt} be an optimal solution of (7.1) with finite objective value c^{opt} := x^{opt}_{2}/x^{opt}_{1}, and let ϵ_{k} := 2ν(C_{k})/∥x^{opt}_{1,2}∥. Given a feasible point x^{(k)} with x_{1,2}^{(k)} ∈ C_{k}, there exists a perturbation vector δx^{(k)} with norm ∥δx^{(k)}∥ ≤ ϵ_{k} such that c(x^{(k)} + δx^{(k)}) = c^{opt}. Thus, an ϵ_{k}-perturbation of x^{(k)} is sufficient to obtain a point with optimal objective value.

We note that ϵ_{k} is not the Euclidean distance of x^{(k)} to an optimal solution of (7.1), since x^{(k)} + δx^{(k)} may be (slightly) infeasible.

At the end of this Chapter we sketch a practical implementation and report results of a few numerical experiments.

7.2 Basic assumptions

We now state our assumptions on the fractional program (7.1). Since max (x_{2}/x_{1}) = − min (−x_{2}/x_{1}), the problem (7.2) can always be written as a fractional program (7.1) by setting x_{2} := −x_{2}. Therefore, from now on, we only consider problems of the form (7.1).

Throughout, we assume that the feasible domain S of (7.1) is a bounded convex set in IR^{n}. Furthermore, we always require that S has a nonempty interior, i.e.,

S^{∘} ≠ ∅,  (7.13)

and that S satisfies the condition (7.4). We also assume a θ̂-self-concordant barrier function ϕ̂ for S to be given.

Reduction to the positive quadrant

The four quadrants of IR^{2} induce the following decomposition of the feasible domain of (7.1):

S = S_{I} ∪ S_{II} ∪ S_{III} ∪ S_{IV} ,  (7.14)

where

\begin{aligned}S_{I} & := \{ x ∈ S | x_{1} ≥ 0, x_{2} ≥ 0 \}, S_{II} := \{ x ∈ S | x_{1} ≤ 0, x_{2} ≥ 0 \}, \\ S_{III} & := \{ x ∈ S | x_{1} ≤ 0, x_{2} ≤ 0 \}, S_{IV} := \{ x ∈ S | x_{1} ≥ 0, x_{2} ≤ 0 \}. \end{aligned}  (7.15)

From (7.4) and the convexity of S, it follows that at most three of the sets (7.15) are nonempty. Determining whether the set S_{J}, J ∈ { I, II, III, IV }, is nonempty is a convex programming problem that can be solved in polynomial time (up to a given accuracy) by using any of the algorithms described in [64]. Therefore, by solving at most four convex programming problems, we can find out which of the four components of S in (7.14) are nonempty.

Once the nonempty components of S are known, the search for an optimal solution of (7.1) can be restricted to precisely one of the four sets (7.15). This is trivially true if only one set is nonempty. For the case that two or three sets are nonempty, note that the objective function c(x) = x_{2}/x_{1} of (7.1) is nonnegative on S_{I} and S_{III}, and c is nonpositive on S_{II} and S_{IV}. Hence, if S is contained in either S_{I} ∪S_{IV} or in S_{II} ∪S_{III}, then we only need to consider S_{IV} or S_{II}, respectively. In all other cases, it follows that S^{∘} ∩ { x ∈ IR^{n} | x_{1} = 0 } ≠ ∅, and as a result, the infimum of the objective function c on S is −∞. In this case, we can stop.

These considerations show that, without loss of generality, we can always assume that the feasible domain S of (7.1) is one of the four sets (7.15). We can even assume

S = S_{I}.  (7.16)

Indeed, since min (x_{2}/x_{1}) = −1/ min (−x_{1}/x_{2}), the case S = S_{II} can always be transformed into (7.16) by redefining the variables x_{1} and x_{2} as follows: x_{2} := −x_{1}, x_{1} := x_{2}. Similarly, the case S = S_{III} and S = S_{IV} can be written in the form (7.16) by setting x_{1} := −x_{1}, x_{2} := −x_{2} and x_{2} := x_{1}, x_{1} := −x_{2}, respectively. Therefore, from now on, we assume that S = S_{I}. This, together with (7.4), guarantees that the feasible domain S always satisfies the following conditions:

x_{1}, x_{2} ≥ 0 and (x_{1}, x_{2}) ≠ (0, 0) for all x ∈ S.  (7.17)

For technical reasons (explained below) we modify the barrier function ϕ̂ for S and consider the following barrier function

ϕ(x) := ϕ̂(x) − θ̂log x_{1} − θ̂log x_{2}  (7.18)

which is self-concordant with parameter θ = 3θ̂ ≥ 3. (In [16] the analysis is based on ϕ̂.)

7.3 A method of centers

In this section, we describe our interior-point method for fractional programs, and we derive some of its basic properties. We stress that this algorithm is merely of theoretical interest since it allows a simple proof of polynomial convergence. A practical implementation of the method will be sketched in Section 7.6.

7.3.1 An equivalent formulation

Let us recall that the objective function c of (7.1) only depends on the first two components of x. Furthermore, note that, in the x_{1}, x_{2}-plane, the objective function c(x) = x_{2}/x_{1} is constant on all half-lines emanating from the origin. Thus, rather than performing a parallel shift of some linear objective constraint c^{T} x ≤ λ as in the method of centers, we may rotate some half-line L about the origin such that L bounds the objective value c(x) = x_{2}/x_{1}. It is natural to parameterize the rotation of L by the unit circle of the x_{1}, x_{2}-plane. Formally this is possible by the following reformulation of the fractional program (7.1). We set

v(t) := [ − sin t ; cos t ; 0 ; ⋮ ; 0 ] ∈ IR^{n} and u(x, t) := x^{T} v(t), x ∈ IR^{n}, t ∈ [0, π/2].  (7.19)

In view of (7.17), we have, for all x ∈ S and t ∈ [0, π/2],

u(x, t) ≤ 0 ⟺ \frac{x_{2}}{x_{1}} ≤ tan t,  (7.20)

and thus (7.1) is equivalent to the following problem:

minimize_{t∈[0,π/2], x∈IR^{n}} t  subject to u(x, t) ≤ 0, x ∈ S.  (7.21)

In the sequel, we always denote by t^{opt} ∈ [0, π/2] the optimal value of (7.21). Furthermore, for each fixed t, we denote by

S(t) := { x ∈ S | u(x, t) ≤ 0 },

the set of points x that are feasible for (7.21), and by

S^{∘}(t) := { x ∈ S^{∘} | u(x, t) < 0 }  (7.22)

the interior of S(t). Note that u(x, π/2) = −x_{1}, and by (7.17) and (7.13), we have

S(π/2) = S and S^{∘}(π/2) ≠ ∅.  (7.23)

Obviously, if t = t^{opt}, then equality must hold on both sides of (7.20). It follows that

S(t^{opt}) = { x ∈ S | u(x, t^{opt}) = 0 }, S^{∘}(t^{opt}) = ∅,  (7.24)

and S(t^{opt}) is just the set of optimal solutions of the fractional program (7.1). From (7.23) and (7.24), it follows that 0 ≤ t^{opt} < π/2. Finally, we remark that

S^{∘}(t) ≠ ∅ if t^{opt} < t ≤ π/2, and S(t) = ∅ if 0 ≤ t < t^{opt}.  (7.25)

In the following, let t with t^{opt} < t ≤ π/2 be arbitrary, but fixed. Clearly, S(t) is a bounded convex set whose interior, by (7.25), is nonempty. We define a logarithmic barrier function φ(·, t) for S^{∘}(t) by setting

φ(x, t) := ϕ(x) − θ log(−u(x, t)) for all x ∈ S^{∘}(t).  (7.26)

Here, ϕ is the logarithmic barrier function for S given by (7.18). We stress that the factor θ of the second term in (7.26) could be replaced by an arbitrary constant κ > 0. However, the choice κ = θ leads to the best constant in the convergence result presented in Theorem 7.8, and therefore, we only consider the case κ = θ.

Note that ϕ is assumed to be θ-self-concordant. The function −θ log(−u(·, t)) is obviously θ-self-concordant. Therefore, the function φ(·, t) is (2θ)-self-concordant. Consequently, φ(·, t) satisfies all the properties listed in Chapter 2 (with S replaced by S(t) and θ replaced by 2θ). In particular, the analytic center—denoted by x(t) in the sequel—of the set S(t) is the unique optimal solution of the unconstrained optimization problem

minimize_{x∈S(t)} φ(x, t).  (7.27)

It can be shown (see, e.g., [11]) that x(t) converges to a point in S(t^{opt}) as t approaches t^{opt}, and thus

x^{opt} := lim_{t→t^{opt}+0} x(t)  (7.28)

is an optimal solution of the original fractional program (7.1).

Finally, we note that, in view of (7.26) and (7.19), the gradient and the Hessian of φ(·, t) are given by

g(x, t) := ∇_{x}φ(x, t) = ∇ϕ(x) − θ \frac{v(t)}{u(x, t)},  (7.29)

H(x, t) := ∇_{x}^{2}φ(x, t) = ∇^{2}ϕ(x) + θ \frac{v(t)v(t)^{T}}{(u(x, t))^{2}} .  (7.30)

7.3.2 The method

The property (7.28) of the analytic centers x(t) suggests the following procedure for computing an optimal solution of the fractional program (7.1). We choose a sequence

{t_{k}}_{k=0}^{∞}, where t^{opt} < t_{k+1} < t_{k} ≤ π/2 for all k,

and we determine for each k an approximation—denoted by x^{(k)} in the sequel—to the analytic center x(t_{k}) of S(t_{k}). We remark that quantities corresponding to the kth iteration of this procedure are always marked by subscripts k, with the exception of the iterates x^{(k)} for which superscripts (k) are used. This is done to avoid confusion with the notation x_{1} and x_{2} for the first and second component of a vector x.

If lim_{k→∞} t_{k} = t^{opt} and if we chose x^{(k)} = x(t_{k}) then x^{(k)} would converge to an optimal solution of (7.1). In our method, we use approximations x^{(k)} that are obtained by a single Newton step at each iteration. We also propose a simple strategy for choosing the sequence {t_{k}}_{k=0}^{∞}. As we will demonstrate in Section 4, this procedure still converges in a certain sense to the optimal solutions of the fractional program (7.1). More precisely, the rate of convergence is measured in terms of the area of two-dimensional convex sets C_{k} containing x_{1,2}^{(k)} and the first two components of all optimal points of (7.1). We will show that the area of C_{k} is reduced by a constant factor at each iteration.

At the start of our method, we choose a t_{0}, 0 < t_{0} ≤ π/2, such that S^{∘}(t_{0}) ≠ ∅. For example, by (7.23), one can set t_{0} = π/2. Then, we determine a point x^{(0)} ∈ S^{∘}(t_{0}) that is “close” to the analytic center of S(t_{0}) in the following sense:

∥g(x^{(0)}, t_{0})∥_{(H(x^{(0)},t_{0}))^{−1}} ≤ \frac{1}{16}.  (7.31)

A point x^{(0)} satisfying (7.31) can easily be obtained in polynomial time by using any of the algorithms for convex programming in [64]. At each kth iteration, our algorithm decreases the current value t_{k} by setting t_{k+1} = t_{k} − Δt_{k}. Here the step size Δt_{k} > 0 is chosen suitably, based on the kth iterate x^{(k)}. Then, we perform one step of Newton’s method for minimizing φ(·, t_{k+1}) over S(t_{k+1}) with the current iterate x^{(k)} as starting point. The resulting Newton iterate is x^{(k+1)}. In Theorem 7.6 below, we will show that the iterates x^{(k)} generated by this procedure remain close to the analytic center x(t_{k}) in the sense that

∥g(x^{(k)}, t_{k})∥_{(H(x^{(k)},t_{k}))^{−1}} ≤ \frac{1}{16} for all k = 0, 1, ... .  (7.32)

Note that, in (7.32), the kth gradient is measured in the (H(x^{(k)}, t_{k}))^{−1}-norm. Moreover, we remark that, in view of (7.27), g(x(t_{k}), t_{k}) = 0, and hence the left-hand side in (7.32) would be zero if x^{(k)} were chosen as the analytic center x(t_{k}) of S(t_{k}).

In the sequel, it will be convenient to use the following notation:

g_{k} := g(x^{(k)}, t_{k}), ḡ_{k} := g(x^{(k)}, t_{k+1}),  (7.33)
H_{k} := H(x^{(k)}, t_{k}), H̄_{k} := H(x^{(k)}, t_{k+1}),  (7.34)
B_{k} := H_{k}^{−1}, B̄_{k} := H̄_{k}^{−1},  (7.35)
u_{k} := u(x^{(k)}, t_{k}), ū_{k} := u(x^{(k)}, t_{k+1}).  (7.36)

The proposed algorithm can then be formulated as follows.

Algorithm 7.2 (Interior-Point Method for Fractional Programs.)4pt] 0) Choose t_{0} ∈ (0, π/2] such that S^{∘}(t_{0}) ≠ ∅. Determine a point x^{(0)} ∈ S^{∘}(t_{0}) such that

∥g_{0}∥_{B_{0}} ≤ \frac{1}{16}.

For k = 0, 1, …, do : 4pt] 1) Choose σ_{k} > 0 and set

Δt_{k} = \frac{σ_{k}|u_{k}|}{\sqrt{θ}∥x_{1,2}^{(k)}∥}.  (7.37)

2) Set

t_{k+1} = t_{k} − Δt_{k}.  (7.38)

3) Set

x^{(k+1)} = x^{(k)} − B̄_{k}ḡ_{k}.  (7.39)

We remark that the quantities generated in Algorithm 7.2 are only defined if we can guarantee that t^{opt} < t_{k} and x^{(k)} ∈ S^{∘}(t_{k}) for all k. As we will show in part a) of Theorem 7.6 in the next section, t_{k} and x^{(k)} indeed satisfy these properties if σ_{k} ≤ 1/23 for all k. Obviously, it is desirable to always choose the maximal value σ_{k} = 1/23 in (7.37) so that the sequence {t_{k}}_{k=0}^{∞} decreases as fast as possible. As our main result, namely Theorem 7.8 below, we will prove that Algorithm 7.2 converges at a polynomial rate if σ_{k} = 1/23 for all k.

7.3.3 Properties

In this subsection, we establish some basic properties of Algorithm 7.2. First, we prove two auxiliary results that will be needed later on.

Proposition 7.3 Let x ∈ S^{∘}, and set H_{x} = ∇^{2}ϕ(x). Then :

∥h_{1,2}∥ ≤ \sqrt{3/θ} ∥x_{1,2}∥ for all h ∈ IR^{n} with ∥h∥_{H_{x}} ≤ 1,  (7.40)
∥h∥_{H_{x}^{−1}} ≤ \sqrt{\frac{3}{θ}} ∥h∥ · ∥x_{1,2}∥ for all h = [ \begin{matrix}h_{1,2} \\ 0 \end{matrix} ] ∈ IR^{n}, h_{1,2} ∈ IR^{2}. (7.41)

Proof. Letting M denote the leading principal 2×2 submatrix of H_{x}^{−1}, the inequality (7.41) is equivalent to the relation

h_{1,2}^{T}Mh_{1,2} ≤ \frac{3}{θ}∥h_{1,2}∥^{2} · ∥x_{1,2}∥^{2} for all h_{1,2} ∈ IR^{2}.  (7.42)

To show (7.42), note that, by construction of ϕ via ϕ̂ in (7.18), we have

H_{x} ≥ θ̂diag(\frac{1}{x_{1}^{2}} , \frac{1}{x_{2}^{2}} , 0, …, 0) ≥ \frac{θ̂}{∥x_{1,2}∥^{2}} diag(1, 1, 0, …, 0).  (7.43)

It is easy to verify that (with θ = 3θ̂) (7.43) implies (7.40) and

M^{−1} ≥ \frac{θ̂}{∥x_{1,2}∥^{2}} [ \begin{matrix}1 & 0 \\ 0 & 1 \end{matrix} ] or, equivalently, M ≤ \frac{∥x_{1,2}∥^{2}}{θ̂} [ \begin{matrix}1 & 0 \\ 0 & 1 \end{matrix} ] .  (7.44)

The second relation in (7.44) is equivalent to the inequality (7.42).

Lemma 7.4 Let w, Δw ∈ IR^{n} and γ > 0 be given, and let M and M̄ be the matrices defined by

M := I + ww^{T} and M̄ = I + w̄w̄^{T} , where w̄ := \frac{1}{γ} (w + Δw),  (7.45)

and I is the n × n identity matrix. Then :

M ≤ d M̄, where d := max{1, γ^{2}} + ∥Δw∥ (γ + ∥Δw∥) .  (7.46)

Proof. First, we remark that M ≤ dM̄ if, and only if,

\frac{y^{T} My}{y^{T} M̄y} ≤ d for all y ∈ IR^{n} with ∥y∥ = 1.  (7.47)

Let y ∈ IR^{n} with ∥y∥ = 1 be arbitrary, and set λ := y^{T} w̄. With (7.45), it follows that

\frac{y^{T} My}{y^{T} M̄y} = \frac{1 + (y^{T} w)^{2}}{1 + (y^{T} w̄)^{2}} = \frac{1 + (γy^{T} w̄ − y^{T} Δw)^{2}}{1 + (y^{T} w̄)^{2}}

= \frac{1 + γ^{2}λ^{2}}{1 + λ^{2}} + y^{T} Δw (\frac{y^{T} Δw}{1 + λ^{2}} − \frac{2γλ}{1 + λ^{2}} ) .  (7.48)

Next, we note that, for all λ ∈ IR, one has

\frac{1 + γ^{2}λ^{2}}{1 + λ^{2}} ≤ max\{1, γ^{2}\}, \frac{1}{1 + λ^{2}} ≤ 1, and \frac{2|λ|}{1 + λ^{2}} ≤ 1.  (7.49)

Using (7.49) and the estimate |y^{T} Δw| ≤ ∥Δw∥, we obtain from (7.48) that

\frac{y^{T} My}{y^{T} M̄y} ≤ max\{1, γ^{2}\} + ∥Δw∥ (γ + ∥Δw∥) .  (7.50)

In view of (7.46), the upper bound in (7.50) is just d, and thus (7.50) is the desired inequality (7.47).

In the following theorem, we establish some first properties of the kth iterate x^{(k)} of Algorithm 7.2.

Theorem 7.5 Let 0 < t_{k} ≤ π/2, x^{(k)} ∈ S^{∘}(t_{k}), and assume that

∥g_{k}∥_{B_{k}} ≤ \frac{1}{16}.  (7.51)

Let σ_{k} > 0, and let t_{k+1} be defined by (7.38) and (7.37). Then :

a)

|\frac{ū_{k}}{u_{k}} − 1| ≤ \frac{σ_{k}}{\sqrt{θ}} .  (7.52)

b) If σ_{k} < \sqrt{θ}, then :

t^{opt} < t_{k+1} < t_{k} and x^{(k)} ∈ S^{∘}(t_{k+1}).  (7.53)

c) If σ_{k} ≤ 1/23, then :

∥ḡ_{k}∥B̄_{k} ≤ \frac{1}{5}.  (7.54)

Proof. For simplicity, we omit indices k and set x = x^{(k)}, t = t_{k}, Δt = Δt_{k}, σ = σ_{k}, g = g_{k}, ḡ = ḡ_{k}, and B̄ = B̄_{k}. Furthermore, we set

v := v(t) and v̄ = v(t − Δt),  (7.55)

and we denote by H_{x} = ∇^{2}ϕ(x) the Hessian of ϕ at x. Remark that, by (7.30), we have 0 < H_{x} ≤ H̄_{k}, and thus

B̄ = H̄_{k}^{−1} ≤ H_{x}^{−1}.  (7.56)

Using (7.19), (7.55), and standard trigonometric identities, it follows that

v − v̄ = 2 sin(Δt/2)v̇, where v̇ := [ − cos(t − Δt/2) − sin(t − Δt/2) 0 ⋮ 0 ] ∈ IR^{n}.  (7.57)

Note that ∥v̇∥ = 1, and by (7.41) and (7.56), we have

∥v̇∥B̄ ≤ ∥v̇∥_{H_{x}^{−1}} ≤ \sqrt{\frac{3}{θ}}∥x_{1,2}∥.  (7.58)

By multiplying (7.57) from the left by x^{T} and by using the estimate

2 sin(Δt/2) ≤ Δt,  (7.59)

we obtain

u − ū = 2 sin(Δt/2)x^{T} v̇ and |u − ū| ≤ Δt∥x_{1,2}∥.  (7.60)

Combining the second relation in (7.60) and (7.37) gives

|u − ū| ≤ \frac{σ|u|}{\sqrt{θ}}.  (7.61)

Note that, by assumption, x ∈ S^{∘}(t), and thus, by (7.22), we have

|u| = −u > 0.  (7.62)

In particular, |u| ≠ 0, and dividing (7.61) by |u| gives (7.52). This concludes the proof of part a).

Next,we turn to part b), and we assume that 0 < σ < \sqrt{θ}. Moreover, we set t̄:= t_{k+1} = t − Δt. From (7.61)–(7.62) and since 1 − σ/\sqrt{θ} > 0, it follows that

ū ≤ u (1 − \frac{σ}{\sqrt{θ}} ) < 0.  (7.63)

Note that, by (7.22), x ∈ S^{∘}(t) implies that x ∈ S^{∘}. Thus, by (7.63) and (7.22), we conclude that x ∈ S^{∘}(t̄), which is the second claim in (7.53). Clearly, t̄ < t, and therefore, it only remains to show that t^{opt} < t̄. To this end, we first note that ū = u(x, t̄) depends continuously on t̄, and that, by (7.19), u(x, 0) = x_{2} > 0. Since, by (7.63), ū < 0 for all 0 < σ < \sqrt{θ}, we conclude that t̄ > 0. Suppose that t̄ ≤ t^{opt}. Recall from (7.24) and (7.25) that then S^{∘}(t̄) = ∅, in contradiction to x ∈ S^{∘}(t̄).

Finally, we turn to part c), and we assume that 0 < σ ≤ 1/23. We will prove the following two inequalities:

∥g∥B̄ ≤ \frac{1}{14},  (7.64)
∥ḡ − g∥B̄ ≤ \frac{9}{70} = \frac{1}{5} − \frac{1}{14}.  (7.65)

By the triangle inequality, (7.64) and (7.65) imply the desired estimate (7.54). To show (7.64), we will derive the following more general result:

B̄ ≤ \frac{64}{49} B.  (7.66)

Clearly, from (7.66) and (7.51) it follows that

∥g∥B̄ ≤ \sqrt{\frac{64}{49}} ∥g∥_{B} ≤ \frac{8}{7} · \frac{1}{16} = \frac{1}{14},

which is just (7.64). Since B̄ > 0, the claim (7.66) is equivalent to

B^{−1} ≤ \frac{64}{49} B̄^{−1}.  (7.67)

Recall that the matrix H_{x} is symmetric positive definite, and hence it has a symmetric positive definite square root H_{x}^{1/2}. By multiplying (7.67) from the left and the right by H_{x}^{−1/2}, we obtain the relation

M ≤ \frac{64}{49} M̄, where M := H_{x}^{−1/2}B^{−1}H_{x}^{−1/2}, M̄ := H_{x}^{−1/2}B̄^{−1}H_{x}^{−1/2}.  (7.68)

Obviously, (7.68) and (7.67) are equivalent; we will prove (7.68). Using (7.30), (7.34)–(7.36), and (7.55), one easily verifies that the matrices M and M̄ in (7.68) can be represented as follows:

M = I + ww^{T} , where w := \frac{\sqrt{θ}}{u} H_{x}^{−1/2}v,
M̄ = I + w̄w̄^{T} , where w̄ := \frac{\sqrt{θ}}{ū} H_{x}^{−1/2}v̄.

(7.69)

By (7.57), the vectors w and w̄ satisfy

w̄ = \frac{1}{γ} (w + Δw), where γ := \frac{ū}{u}, Δw := −\frac{\sqrt{θ}}{u} 2 sin(Δt/2)H_{x}^{−1/2}v̇.

From the first relation in (7.60) and the definition (7.57) of v̇, it follows that u − ū < 0. Moreover, by (7.62), u < 0, and hence we have

γ < 1.  (7.70)

Similarly, using (7.59), (7.37), and (7.58), we obtain that

∥Δw∥ ≤ \frac{\sqrt{θ} Δt}{|u|} ∥v̇∥_{H_{x}^{−1}} ≤ σ\sqrt{\frac{3}{θ}} ≤ \frac{1}{23}.  (7.71)

We now apply Lemma 7.4 to the matrices M and M̄ in (7.69). Together with (7.70) and (7.71), it follows from (7.46) that

M ≤ d M̄, d ≤ \frac{23^{2} + 23 + 1}{23^{2}} < \frac{64}{49},

which is just the claim (7.68).

It remains to prove the estimate (7.65). Note that, by (7.29) and (7.33), we have

ḡ − g = θ (\frac{v}{u} − \frac{v̄}{ū}) = (1 − \frac{u}{ū}) \frac{θ}{u}v + \frac{θ}{ū}(v − v̄),

and thus

∥ḡ − g∥B̄ ≤ |1 − \frac{u}{ū}| \frac{θ}{|u|}∥v∥B̄ + \frac{θ}{|ū|}∥v − v̄∥B̄.  (7.72)

From (7.29), we get

\frac{θ}{u}v = ∇ϕ(x) − g.  (7.73)

Using (7.56) and the bound (2.20) on the length of the Newton step, we obtain

∥∇ϕ(x)∥B̄ ≤ ∥∇ϕ(x)∥_{H_{x}^{−1}} = ∥H_{x}^{−1}∇ϕ(x)∥_{H_{x}} ≤ \sqrt{θ}.  (7.74)

By (7.73), (7.74), and (7.64), it follows that

\frac{θ}{|u|}∥v∥B̄ ≤ \sqrt{θ} + \frac{1}{14}.  (7.75)

From (7.57)–(7.59) and (7.37), we get the estimate

∥v − v̄∥B̄ ≤ \sqrt{\frac{3}{θ}} Δt ∥x_{1,2}∥ = \frac{\sqrt{3} σ|u|}{θ}.  (7.76)

Next, we note that (7.61) implies

\frac{|u|}{|ū|} ≤ \frac{1}{1 − σ/\sqrt{θ}} and |1 − \frac{u}{ū}| ≤ \frac{σ/\sqrt{θ}}{1 − σ/\sqrt{θ}} .  (7.77)

Finally, by combining (7.72) and (7.75)–(7.77), we obtain

∥ḡ − g∥B̄ ≤ \frac{σ/\sqrt{θ}}{1 − σ/\sqrt{θ}} (\sqrt{θ} + \frac{1}{14} + \sqrt{3θ}) = \frac{σ}{1 − σ/\sqrt{θ}} (1 + \frac{1}{14\sqrt{θ}} + \sqrt{3}) .

Since θ ≥ 3 and σ ≤ 1/23, it follows that

∥ḡ − g∥B̄ ≤ \frac{σ}{1 − σ} (1 + \frac{1}{14} + \sqrt{3}) < \frac{9}{70},

which is the desired estimate (7.65). This concludes the proof of Theorem 7.5.

We now show that, if σ_{k} ≤ 1/23, then t^{opt} < t_{k} and x^{(k)} ∈ S^{∘}(t_{k}) for all k, and consequently, all quantities involved in Algorithm 7.2 are always well defined.

Theorem 7.6 Let t_{k} and x^{(k)} be the iterates generated by Algorithm 7.2, and assume that 0 < σ_{k} ≤ 1/23 for all k. Then, for all k = 0, 1, …, the following properties are satisfied :

a)

t^{opt} < t_{k} ≤ π/2 and x^{(k)} ∈ S^{∘}(t_{k}),  (7.78)
∥g(x^{(k)}, t_{k})∥_{(H(x^{(k)},t_{k}))^{−1}} ≤ \frac{1}{16}.  (7.79)

b)

∥x^{(k)} − x(t_{k})∥_{H(x^{(k)},t_{k})} ≤ \frac{1}{14},  (7.80)
∥x^{(k)} − x(t_{k})∥_{H(x(t_{k}),t_{k})} ≤ \frac{1}{13}.  (7.81)

Proof. We use induction on k to show part a). For k = 0, both properties (7.78) and (7.79) are satisfied. Indeed, the initial point x^{(0)} of Algorithm 7.2 is chosen such that (7.79) holds. Furthermore, we have 0 < t_{0} ≤ π/2, and thus, by part b) of Theorem 7.5 (with k = 0), t^{opt} < t_{0}. Now assume that (7.78) and (7.79) hold for some k ≥ 0. We need to show that

t^{opt} < t_{k+1} ≤ π/2,  (7.82)
x^{(k+1)} ∈ S^{∘}(t_{k+1}),  (7.83)
∥g_{k+1}∥_{B_{k+1}} ≤ \frac{1}{16}.  (7.84)

Note that, by (7.78)–(7.79), the assumptions of Theorem 7.5 are fulfilled, and from (7.53), we immediately get (7.82). We now set x := x^{(k)} and x̃ := x^{(k+1)}. By the second part of (7.53), we have x ∈ S^{∘}(t_{k+1}). Recall that x̃ = x + Δx is obtained by performing one step—starting at x—of Newton’s method (without line search) for minimizing φ(·, t_{k+1}) over S(t_{k+1}). Moreover, by (7.39) and since B̄_{k} = H̄_{k}^{−1}, we have

Δx = −B̄_{k}ḡ_{k} = −H̄_{k}^{−1}ḡ_{k}.

Together with part c) of Theorem 7.5, it follows that

∥Δx∥H̄_{k} = ∥ḡ_{k}∥B̄_{k} ≤ \frac{1}{5}.  (7.85)

Thus, we can apply the results on Newton’s method in Chapter 2 (with S replaced by S(t_{k+1}) and ϕ(·) replaced by φ(·, t_{k+1})). From Lemma 2.4, we get x̃ ∈ S^{∘}(t_{k+1}), which is just (7.83). Using (7.85), we obtain

∥Δx̃∥_{Hx̃} ≤ \frac{∥Δx∥2 ; H̄_{k}}{(1 − ∥Δx∥H̄_{k})^{2}} ≤ \frac{1/25}{(1 − 1/5)^{2}} = \frac{1}{16}.  (7.86)

Note that since x̃ = x^{(k+1)}, we have

∥Δx̃∥_{Hx̃} = ∥g_{k+1}∥_{B_{k+1}},

and thus (7.86) is equivalent to (7.84). This completes the proof of part a).

We now turn to part b). Let k ≥ 0 be fixed, and set x = x^{(k)}, t = t_{k}, and H = H(x^{(k)}, t_{k}). To prove (7.80), we make use of property (2.16) of Newton’s method in Chapter 2 (with S replaced by S(t) and ϕ(·) replaced by φ(·, t)). Since B_{k} = H^{−1}, the Newton step at x for finding x(t) is given by

Δx̃ = −H^{−1}∇_{x}φ(x, t) = −B_{k}g_{k}.

Thus, by (7.79), we have ∥Δx̃∥_{H} ≤ 1/16 < 1/3, and by applying (2.16) (with x_{⋆} = x(t)), we get

∥x − x(t)∥_{H} ≤ 1 − (1 − \frac{3}{16})^{1/3} < \frac{1}{14}.

This is just the desired estimate (7.80). Note that Algorithm 7.2 does not compute the above step Δx̃, but rather updates t_{k+1} = t_{k} − Δt_{k}, and then computes the Newton step for finding x(t_{k+1}).

Finally, the inequality (7.81) follows immediately from (7.80) by means of a general result on the norms induced by self-concordant functions. Indeed, by applying [64, Theorem 1.1], we obtain

∥x − x(t)∥_{H(x(t),t)} ≤ \frac{∥x − x(t)∥_{H}}{1 − ∥x − x(t)∥_{H}} ≤ \frac{1/14}{1 − 1/14} = \frac{1}{13}.

This completes the proof of Theorem 7.6.

7.4 Convergence analysis

In this section, we present the main result of this Chapter, namely the polynomial convergence of Algorithm 7.2.

7.4.1 Main result

We continue to use the notation introduced in Section 3. In addition, for t^{opt} ≤ t ≤ t_{0}, we denote by

Q(t) := { z ∈ IR^{2} | z = x_{1,2} for some x ∈ S(t) }  (7.87)

the set in IR^{2}, which consists of all truncations x_{1,2} of the vectors x ∈ S(t). We denote by Q^{∘}(t) the interior of Q(t). Recall from (7.24) that

Q(t^{opt}) = { x_{1,2} ∈ IR^{2} | x is an optimal solution of (7.1) }  (7.88)

just consists of the truncations of all optimal solutions of the fractional program (7.1). For all k = 0, 1, …, we define sets

T_{k} := \{ z ∈ IR^{2} | 0 ≥ z^{T} v_{1,2}(t_{k}) ≥ −d_{k} \}, where d_{k} := \frac{13}{6} |u(x^{(k)}, t_{k})| .  (7.89)

Here, in view of (7.19), the vector v_{1,2}(t) is given by

v_{1,2}(t) = [ \begin{matrix}− sin t \\ cos t \end{matrix} ] .  (7.90)

We remark that T_{k} is a stripe in IR^{2} of width d_{k}. Finally, we set

D := \{ z ∈ IR^{2} | z_{1}, z_{2} ≥ 0 and ∥z∥ ≤ (1 + \sqrt{2θ})^{2} ∥x_{1,2}(t_{0})∥ \},  (7.91)

where x(t_{0}) is the analytic center of S(t_{0}). Note that D is the quarter of a disk about the origin.

Both sets Q(t_{k}) and T_{k} contain the truncation x_{1,2}^{(k)} of the kth iterate of Algorithm 7.2, as well as the set Q(t^{opt}) of truncations of all optimal solutions of the fractional program (7.1). In the next theorem, we summarize these results.

Theorem 7.7 Let t_{k} and x^{(k)} be the iterates generated by Algorithm 7.2, and assume that 0 < σ_{k} ≤ 1/23 for all k. Then, for all k = 0, 1, …, the following properties are satisfied :

x_{1,2}^{(k)} ∈ Q^{∘}(t_{k}),  (7.92)
Q(t^{opt}) ⊆ Q(t_{k+1}) ⊆ Q(t_{k}) ⊆ D,  (7.93)
Q(t_{k}) ⊂ T_{k}.  (7.94)

Proof of Theorem 7.7. From the second relation in (7.78), we immediately get (7.92). We now turn to (7.93). First, we show that

Q(t_{0}) ⊆ D.  (7.95)

Let z ∈ Q(t_{0}) be arbitrary, but fixed. Since x_{1}, x_{2} ≥ 0 for all x ∈ S, we have z_{1}, z_{2} ≥ 0. It remains to show the second condition in (7.91). We set x := x(t_{0}) and H_{x} := ∇^{2}ϕ(x). Note that, in view of (7.30), we have H_{x} ≤ H(x, t_{0}). Using Lemma 2.8 (applied to the (2θ)-self-concordant function φ(·, t_{0})), we conclude that

z = x_{1,2} + 2 (θ + \sqrt{2θ}) h_{1,2} for some h ∈ IR^{n} with ∥h∥_{H_{x}} ≤ 1.  (7.96)

From (7.96) and (7.40), it follows (with θ ≥ 3) that

∥z∥ ≤ (1 + 2(θ + \sqrt{2θ})) ∥x_{1,2}∥ = (1 + \sqrt{2θ})^{2} ∥x_{1,2}∥,

and thus z ∈ D.

Next, we note that v_{1,2}(t) given by (7.90) clearly satisfies

z^{T} v_{1,2}(t′) ≥ z^{T} v_{1,2}(t) for all 0 ≤ t′ < t ≤ \frac{π}{2} and z = [ \begin{matrix}z_{1} \\ z_{2} \end{matrix} ] , z_{1}, z_{2} ≥ 0.

Consequently, we have Q(t′) ⊆ Q(t) for all 0 ≤ t′ < t ≤ π/2. Since t_{k} is monotonically decreasing, this—along with (7.95)—implies (7.93).

We now turn to (7.94). By the definitions of Q(t) and T_{k} in (7.87) and (7.89), and since |u(x^{(k)}, t_{k})| = −(x^{(k)})^{T} v(t_{k}), the inclusion (7.94) is equivalent to the following estimates:

0 ≥ x^{T} v(t_{k}) ≥ \frac{13}{6} (x^{(k)})^{T} v(t_{k}) for all x ∈ S(t_{k}).  (7.97)

In order to prove (7.97), we apply Lemma 2.11 with c(x) := x^{T} v(t_{k}), S_{c} := S(t_{k}), and κ := θ. By (7.27) and (7.26), the unique optimal solution x̂ of (2.23) is just the analytic center x(t_{k}) of S(t_{k}), and from (2.24), we obtain that

0 ≥ x^{T} v(t_{k}) ≥ 2 (x(t_{k}))^{T} v(t_{k}) for all x ∈ S(t_{k}).  (7.98)

It only remains to show that

(x(t_{k}))^{T} v(t_{k}) ≥ \frac{13}{12} (x^{(k)})^{T} v(t_{k}),  (7.99)

since (7.98) and (7.99) imply the desired estimate (7.97). To this end, note that, by inequality (7.81) from Theorem 7.6, the vector

h := 13 (x^{(k)} − x(t_{k}))  (7.100)

satisfies ∥h∥_{H(x(t_{k}),t_{k})} ≤ 1. Thus, by the inner ellipsoidal approximation of Lemma 2.2 (with S replaced by S(t_{k})), we have

x(t_{k}) + h ∈ S(t_{k}).

Together with (7.100), it follows that

0 ≥ (x(t_{k}) + h)^{T} v(t_{k}) = 13 (x^{(k)})^{T} v(t_{k}) − 12 (x(t_{k}))^{T} v(t_{k}).

This inequality is equivalent to (7.99).

After these preparations, we are now in a position to prove our convergence result for Algorithm 7.2. We set

C_{k} := D ∩ T_{0} ∩ T_{1} ∩ ⋯ ∩ T_{k},

where D and T_{j}, j = 0, 1, …, k, are the sets defined in (7.91) and (7.89). Furthermore, for compact sets C ⊂ IR^{2}, we denote by ν(C) := area(C) the area of C. Our convergence result can then be formulated as follows.

Theorem 7.8 Let x^{(k)} be the kth iterate generated by Algorithm 7.2 with σ_{k} = 1/23 for all k. Let θ be the self-concordance parameter of the logarithmic barrier function (7.18) associated with the feasible set S of the fractional program (7.1). Then, for all k = 0, 1, …, the following properties are satisfied :

x_{1,2}^{(k)} ∈ C_{k}^{∘},  (7.101)
0 ∈ C_{k} and x_{1,2} ∈ C_{k} for all optimal solutions x of (7.1),  (7.102)
ν(C_{k+1}) ≤ (1 − \frac{1}{102 \sqrt{θ}} ) ν(C_{k}).  (7.103)

Proof. The properties (7.101) and (7.102) follow immediately from the relations (7.92)– (7.94) in Theorem 7.7 and from (7.88).

We now turn to (7.103). Note that, in view of (7.89), each stripe T_{j} is bounded by two parallel lines with slope tan t_{j}. Moreover, the left line passes through the origin 0 in IR^{2}. Recall that 0 < t_{j+1} < t_{j} ≤ π/2 for all j. It follows that the upper left boundary of the set C_{k} consists of a segment of the form [0, p_{k}] of the left borderline of T_{k}. Here, p_{k} is given by

p_{k} := ρ_{k} [ \begin{matrix}cos t_{k} \\ sin t_{k} \end{matrix} ] , where ρ_{k} := max \{ ρ ∈ IR | ρ [ \begin{matrix}cos t_{k} \\ sin t_{k} \end{matrix} ] ∈ C_{k} \}.

Let p_{k+1} be the corresponding boundary point of C_{k+1}, and let F_{k} denote the triangle with vertices 0, p_{k}, and p_{k+1}. Note that

ν(F_{k}) = \frac{1}{2}ρ_{k}ρ_{k+1} sin Δt_{k},  (7.104)

where, by (7.38) and (7.37) (with σ_{k} = 1/23),

Δt_{k} = t_{k} − t_{k+1} = \frac{|u(x^{(k)}, t_{k})|}{23 \sqrt{θ} ∥x_{1,2}^{(k)}∥}.  (7.105)

By construction of the triangle F_{k}, we have

F_{k} ⊂ C_{k} and C_{k+1} ⊂ C_{k} \ F_{k}^{∘},

and thus

ν(C_{k+1}) ≤ ν(C_{k}) − ν(F_{k}) = (1 − \frac{ν(F_{k})}{ν(C_{k})}) ν(C_{k}).  (7.106)

Next, recall the definition of d_{k} in (7.89), and set

q_{k} := d_{k} [ \begin{matrix}sin t_{k} \\ − cos t_{k} \end{matrix} ] , where d_{k} = \frac{13}{6} |u(x^{(k)}, t_{k})|.  (7.107)

Observe that, by construction of C_{k} as the intersection of the disk D and the stripes T_{j}, j = 0, 1, …, k, the angle of the set C_{k} at the vertex p_{k} is at most π/2, and hence

C_{k} ⊂ \{ z ∈ IR^{2} | z^{T} [ \begin{matrix}cos t_{k} \\ sin t_{k} \end{matrix} ] ≤ ρ_{k} \}.  (7.108)

Furthermore, the radius ρ_{k} of the vertex p_{k} is nonincreasing, i.e., ρ_{j} ≥ ρ_{j+1} for all j = 0, 1, …, implying that

C_{k} ⊂ { z ∈ IR^{2} | ∥z∥ ≤ ρ_{k} }.  (7.109)

In view of (7.108) and since C_{k} ⊂ T_{k}, the set C_{k} is contained in the rectangle R_{k} whose vertices are the points 0, p_{k}, p_{k} + q_{k}, and q_{k}. Moreover, one easily verifies that p_{k+1} is a boundary point of C_{k}, and that

C_{k} ⊂ R_{k} \ (ΔR_{k})^{∘} and ΔR_{k} ⊂ R_{k}.  (7.110)

Here, ΔR_{k} is the rectangle with vertices p_{k+1} and p_{k} +q_{k}, and edges parallel to those of R_{k}. (Thus, ΔR_{k} is contained in R_{k}.) It is straightforward to verify that the remaining vertices of ΔR_{k} are given by p_{k} + s_{k}q_{k}/d_{k} and c_{k}p_{k}/ρ_{k} + q_{k}, where

s_{k} := ρ_{k+1} sin Δt_{k} and c_{k} := ρ_{k+1} cos Δt_{k}.  (7.111)

From (7.110) and (7.111), it follows that

ν(C_{k}) ≤ ν(R_{k}) − ν(ΔR_{k})
= ρ_{k}d_{k} − (ρ_{k} − ρ_{k+1} cos Δt_{k})(d_{k} − ρ_{k+1} sin Δt_{k})
≤ ρ_{k}ρ_{k+1} sin Δt_{k} + d_{k}ρ_{k+1} cos Δt_{k}.

(7.112)

Note that, by (7.101) and (7.109), we have ∥x_{1,2}^{(k)}∥ ≤ ρ_{k}. Together with (7.105) and (7.107), it follows that

tan Δt_{k} ≥ Δt_{k} = \frac{6 d_{k}}{13 · 23 \sqrt{θ} ∥x_{1,2}^{(k)}∥} ≥ \frac{d_{k}}{50 \sqrt{θ} ρ_{k}} .  (7.113)

From (7.112), (7.104), and (7.113), we obtain

\frac{ν(C_{k})}{ν(F_{k})} ≤ 2 (1 + \frac{d_{k}}{ρ_{k} tan Δt_{k}} ) ≤ 2 (1 + 50 \sqrt{θ}) ≤ 102 \sqrt{θ}.

In combination with (7.106), this implies the desired estimate (7.103).

Note that C_{0} ⊆ D, and thus, by (7.91), we have

ν(C_{0}) ≤ ρ_{0}^{2} \frac{π}{4} , where ρ_{0} = (1 + \sqrt{2θ})^{2} ∥x_{1,2}(t_{0})∥.

Together with (7.103), it follows that

ν(C_{k}) ≤ (1 − \frac{1}{102 \sqrt{θ}} )^{k} ρ_{0}^{2} \frac{π}{4} for all k = 0, 1, ….

Therefore, the area ν(C_{k}), which measures the convergence of Algorithm 7.2, converges to zero at a polynomial rate.

A corollary of Theorem 7.8 is that the area ν(C_{k}) is reduced by 50 percent after at most 71\sqrt{θ} iterations. This is immediately obvious by solving (1−1/(102\sqrt{θ}))^{k} ≤ 1/2 for k. From k log(1 − 1/(102\sqrt{θ})) ≤ −k/(102\sqrt{θ}) ≤ log(1/2), we obtain that k ≥ 102\sqrt{θ} log 2 ≈ 70.7\sqrt{θ} is sufficient.

Finally, we give a proof of Proposition 7.1 stated in Section 7.1.2.

Proof of Proposition 7.1. Recall that in the x_{1}, x_{2}-space, the objective is constant on all half lines emanating from the origin. The origin, x^{opt}_{1,2}, and x_{1,2}^{(k)} form a triangle in IR^{2} that is contained in C_{k} (since C_{k} is convex). The area of the triangle is ∥h∥ · ∥x^{opt}_{1,2}∥/2 ≤ ν(C_{k}). Here, h ∈ IR^{2} is perpendicular to the line through the origin and x^{opt}_{1,2}, and it connects x_{1,2}^{(k)} to this line. The vector [ h^{T} 0 ⋯ 0 ]^{T} ∈ IR^{n} just yields the vector δx^{(k)}.

7.5 Stopping criterion

In this section, we discuss optimality conditions for fractional programs, and we propose a practical stopping criterion.

7.5.1 The KKT conditions

We now denote by c(x) := x_{2}/x_{1} the objective function of the fractional program (7.1). If there are feasible points x ∈ S with x_{1} = 0, then, in view of (7.17), c(x) = ∞ at any such point x. Consequently, all optimal solutions of (7.1) are contained in the set

S̃:= { x ∈ S | x_{1} > 0 },

and thus, we only need to discuss optimality conditions for points x ∈ S̃. We remark that, as with S, the set S̃ is also bounded and convex.

The objective function c of (7.1) is pseudo-convex [49] on S̃, i.e., for all x ∈ S̃, c is differentiable at x and

(∇c(x))^{T} (x̃ − x) ≥ 0, x̃ ∈ S̃ ⟹ c(x̃) ≥ c(x).

The pseudo-convexity of c guarantees that the Karush-Kuhn-Tucker conditions (referred to as KKT conditions in the sequel) are necessary and sufficient for optimality; see, e.g., Mangasarian [49]. The KKT conditions for (7.1) are satisfied if, and only if, one can find a point x ∈ IR^{n} and a vector y ∈ IR^{s} such that

∇c(x) + ∑_{i=1}^{s} y_{i}∇r_{i}(x) = 0  (7.114)

and, for all i = 1, 2, …, s,

r_{i}(x) ≤ 0, y_{i} ≥ 0,  (7.115)
y_{i} r_{i}(x) = 0.  (7.116)

7.5.2 A practical stopping criterion

At central points the KKT conditions are nearly satisfied in the following sense. Let x = x(t) be the analytic center of the set S(t). Let the vector y ∈ IR^{s} be defined by

y_{i} := \frac{u(x, t)}{θ x_{1}^{2} r_{i}(x)} (x_{1} cos t + x_{2} sin t) , i = 1, 2, …, s.  (7.117)

By definition of the center x, we have ∇_{x}φ(x, t) = 0 or, equivalently,

− ∑_{i=1}^{s} \frac{∇r_{i}(x)}{r_{i}(x)} = \frac{θ v(t)}{u(x, t)}.  (7.118)

Moreover, note that

∇c(x) = \frac{1}{x_{1}^{2}} [ −x_{2} ; x_{1} ; 0 ; ⋮ ; 0 ] and ∥∇c(x)∥ = \frac{∥x_{1,2}∥}{x_{1}^{2}}.  (7.119)

We now define the relative residual vector of the KKT equation (7.114) as follows:

r(x, t) := \frac{1}{∥∇c(x)∥} (∇c(x) + ∑_{i=1}^{s} y_{i}∇r_{i}(x)) .  (7.120)

Using (7.117)–(7.120), we get

∥r(x, t)∥ = \frac{1}{∥x_{1,2}∥} ∥[ \begin{matrix}−x_{2} \\ x_{1} \end{matrix} ] − (x_{1} cos t + x_{2} sin t) [ \begin{matrix}− sin t \\ cos t \end{matrix} ]∥ = \frac{|u(x, t)|}{∥x_{1,2}∥}.  (7.121)

By definition, x ∈ S^{∘}, and therefore, we have r_{i}(x) < 0 and y_{i} > 0. Hence the inequalities (7.115) are always fulfilled. It turns out that a suitably normalized version of the complementary slackness condition (7.116) is also satisfied up to a relative error of size ∥r(x, t)∥. Using the normalization factor 1/(∥x_{1,2}∥ · ∥∇c(x)∥), we obtain

\frac{|y_{i} r_{i}(x)|}{∥x_{1,2}∥ · ∥∇c(x)∥} = \frac{|u(x, t)|}{θ∥x_{1,2}∥} · \frac{x_{1} cos t + x_{2} sin t}{∥x_{1,2}∥} ≤ \frac{∥r(x, t)∥}{θ}.  (7.122)

We stress that this normalization is indeed natural for our problem: The normalized complementary slackness condition (7.122) is invariant under scaling of c by a positive constant (i.e., for a given x, condition (7.122) is the same for the problems “minimizec(x)” and “minimizeλc(x)” for any fixed λ > 0), while the condition y_{i}r_{i} ≤ ϵ derived from (7.115) is not invariant. Similarly, (7.122) is also invariant under scaling of x.

Since the kth iterate x^{(k)} is close to the analytic center x(t_{k}) of S(t_{k}), the relations (7.121) and (7.122) suggest the following practical stopping criterion:

\frac{|u(x^{(k)}, t_{k})|}{∥x_{1,2}^{(k)}∥} ≤ tol,  (7.123)

where tol is some convergence tolerance.

7.6 A sketch of a practical implementation

In an actual implementation of our interior-point method, we include a predictor step. Before outlining the resulting implementation, we briefly motivate the predictor step.

7.6.1 The predictor step

By the implicit function theorem, x(t) is a smooth curve in the variable t > t^{opt}. Furthermore, in view of (7.28), x(t) approaches an optimal solution as t → t^{opt}. Fiacco and McCormick [11] showed that these properties also hold for the curves x̃(t) where x̃(t) = arg min_{x} φ(x, t) + w^{T} x with some constant linear perturbation w^{T} x.

Instead of computing x^{(k+1)} by Newton’s method starting from x^{(k)}, this suggests to use

x̃^{(k+1)} := x^{(k)} − x̃′(t_{k})Δt_{k}  (7.124)

as a starting vector for Newton’s method. Here, the linear perturbation w defining x̃ is chosen as w = g(x^{(k)}, t_{k}) and depends on k. (It is chosen such that x̃(t_{k}) = x^{(k)}!) Recall from (7.29) that g(x, t) = ∇_{x}φ(x, t) denotes the gradient of φ(·, t). In order to be able to use (7.124), we need an approximation to the tangent vector x̃′(t_{k}) of the curve x̃(t) at t = t_{k}.

Note that, since x̃(t) minimizes φ(x, t) + w^{T} x, and by (7.29), we have

−w = g(x̃(t), t) = ∇ϕ(x̃(t)) − θ \frac{v(t)}{u(x̃(t), t)} for all t.  (7.125)

Differentiating (7.125) with respect to t, it follows that

0 = ∇_{x}^{2}ϕ(x̃(t))x̃′(t) − θ \frac{v′(t)}{u(x̃(t), t)} + θ \frac{v(t)}{(u(x̃(t), t))^{2}} ((x̃(t))^{T} v′(t) + (x̃′(t))^{T} v(t)) (7.126)

for all t. For t = t_{k}, the relation (7.126) can be rewritten in the form

(∇_{x}^{2}ϕ(x^{(k)}) + θ \frac{v(t_{k})v(t_{k})^{T}}{(u(x^{(k)}, t_{k}))^{2}} ) x̃′(t_{k}) = θ \frac{v′(t_{k})}{u(x^{(k)}, t_{k})} − θ \frac{v(t_{k}) (v′(t_{k})^{T} x^{(k)})}{(u(x^{(k)}, t_{k}))^{2}}.  (7.127)

Hence we can compute x̃′(t_{k}) by solving the nonsingular system of linear equations (7.127).

7.6.2 Implementable algorithm

An outline of our implementation is then as follows:

• Let an initial point x^{(0)} ∈ S^{o} be given.

• Determine t = t_{0} ∈ (0, π/2] such that ∥g(x^{(0)}, t)∥ is minimal.

• Use Newton’s method to minimize φ(x, t_{0}) and let the result be x^{(1)}. Set k = 1 and t_{1} = t_{0}.

As long as the stopping criterion is not satisfied repeat the following steps:

• Compute x̃′(t_{k}) by solving the linear system (7.127).

• Find Δt_{max} = max { Δt ∈ IR | x^{(k)} − Δt x̃′(t_{k}) ∈ S }.

• Choose a reduction factor τ_{k} ∈ (0, 1) and set Δt_{k} = τ_{k} Δt_{max}.

• Extrapolate x̃^{(k+1)} := x^{(k)} − Δt_{k} x̃′(t_{k}).

• Find t_{k+1} < t_{k} (e.g., by minimizing ∥g(x̃^{(k+1)}, t)∥).

• Use Newton’s method with line search and starting point x̃^{(k+1)} to minimize φ(x, t_{k+1}). Let the result be x^{(k+1)}.

• Set k = k + 1.

Technical details, such as the choice of τ_{k}, the choice of t_{k+1} < t_{k} such that a sufficient reduction in t can be guaranteed, or how to choose the stopping criterion for Newton’s method at each step of the loop, are omitted here (they are as described in [31]).

The extrapolation step is also called “affine scaling step” in the literature (for linear and convex programming), and Newton’s method bringing the extrapolated point closer to the point x(t_{k+1}) on the central curve is called corrector step or centering step.

Finally, we consider the special case (7.12) of minimizing the condition number of positive definite matrices M(x). Here, given a starting point x^{(0)} such that M(x^{(0)}) > 0, we initialized our program by setting α = λ_{min}(M(x))/2 and β = λ_{max}(M(x^{(0)}) + α. For this problem, minimizing the norm of the gradient can be done by a simple formula: Setting p := trace((M(x) − αI)^{−1}) and q := trace((βI − M(x))^{−1}), we obtain

t = arctan (\frac{pβ^{2} + qαβ + α}{pαβ − β + qα^{2}}) .

We stress that practical issues such us sparsity or numerical stability are not addressed in this article. In particular, the issue of numerical stability needs special attention in a more sophisticated implementation. Even for the small model problems considered here the condition numbers of the linear systems to be solved were of the order 10^{16} in some cases.

7.7 Numerical experiments

The algorithm sketched in the previous section has been implemented in MATLAB^{TM}. In this section, we present results of two numerical experiments to give some insight into the practical behavior of our method. The first example examines a simple modification of the SSOR preconditioner for a matrix A derived from an elliptic partial differential equation. The second example determines a completion with a minimal condition number of a nonsymmetric block matrix A with a prescribed zero structure. For both examples our algorithm shows the same convergence behavior—a very slow growth of the number of iterations with increasing dimension—as it has also been observed in the case of interiorpoint methods for solving linear and convex programs.

In all runs, we used the stopping criterion (7.123), i.e., the algorithm was terminated as soon as

ϱ_{k} := \frac{|u(x^{(k)}, t_{k})|}{∥x_{1,2}^{(k)}∥} ≤ tol  (7.128)

was reached. All our examples were run on an SGI with machine precision of order O(10^{−16}).

7.7.1 Example 1

The following example allows us to compute preconditioners M for a positive definite matrix A such that the condition number of M^{−1}A is minimal. Our method is certainly not practical if all we want is to solve a system Ax = b. However, it allows us to compute “optimal preconditioners” for small matrices A—and to look at them and to derive heuristics for larger matrices A.

Let A be a symmetric positive definite matrix. We assume that A has been scaled such that all its diagonal elements are equal to one. Then, denoting by L the strictly lower triangular part of A, we have the splitting

A = L + I + L^{T} .  (7.129)

of the matrix A. The splitting (7.129) suggests considering preconditioning matrices M of the form

M = M(D) = (I + L)(I + D)(I + L^{T}),

where D is a diagonal matrix with D > −I. For D = 0, we obtain the SSOR preconditioner [3] with relaxation parameter ω = 1. Our goal is to find the diagonal matrix D such that the Euclidean condition number of (M(D))^{−1}A is minimized. To this end, let A = R^{T} R, where R is upper triangular, be the Cholesky decomposition of A. Using

cond ((M(D))^{−1}A) = cond ((M(D))A^{−1})
= cond ((M(D))R^{−1}R^{−T} ) = cond (R^{−T} (M(D))R^{−1}) ,

we minimize the condition of R^{−T} (M(D))R^{−1}. Observe that the unknown variable D appears linearly in R^{−T} (M(D))R^{−1}. Thus, the resulting problem is of the type (7.10), and we can use the approach described in Section 1.2 for its solution.

In our examples below we considered matrices A resulting from a finite difference approximation of the Laplacian on an n × n grid. Note that the number of unknowns is of

Problem  n = 4 n = 6 n = 8 n = 10
Iterations  5      5      5      6
Newton steps  27     33     39     52

               Optimal condition 1.939  3.160  4.788  6.820
               SSOR condition  2.008  3.227  4.854  6.886

Table 7.1: Iterations counts and optimal and SSOR condition numbers for Example 1

Iteration  ϱ_{k}  tan t_{k} Newton steps

                    k=0    5.119e-02  14.96       13
                    k=1    1.024e-02  14.96       7
                    k=2    1.951e-03  7.565       6
                    k=3    2.309e-04  6.939       6
                    k=4    1.857e-05  6.833       6
                    k=5    1.033e-06  6.821       8
                    k=6    4.439e-08  6.820       6

Table 7.2: Convergence history for Example 1 with n = 10

order n^{2}. In Table 1, we list the total number of iterations and Newton steps of our algorithm, as well as the optimal condition number and the condition number of the SSOR preconditioner, for the cases n = 4, 6, 8, and 10. In all four runs of this example, the convergence tolerance tol = 1.0e-06 was used for the stopping test (7.128). It is worth noticing that the condition number of the SSOR preconditioner is almost optimal in all four cases.

For the case n = 10, the iteration number k, the value of ϱ_{k} from (7.128) at iteration k, the value of tan t_{k}, and the number of Newton steps needed at iteration k are listed in Table 2.

7.7.2 Example 2

Here we consider matrices A of the form

A = [ \begin{matrix}Ā & B̄ \\ 0 & X \end{matrix} ]

with given blocks Ā ∈ IR^{l×l} and B̄ ∈ IR^{l×m}. The problem is to find a matrix X ∈ IR^{m×m} that minimizes the Euclidean condition number of A. The problem can be transformed into the equivalent problem of minimizing the condition number of the matrix

M = A^{T} A = [ \begin{matrix}Ā^{T} Ā & Ā^{T} B̄ \\ B̄^{T} Ā & B̄^{T} B̄ + X^{T} X \end{matrix} ] .

We can substitute X^{T} X by another positive definite matrix Z and minimize the condition number of M with respect to Z. Note that M depends affinely on Z, and again, the

Problem  m = 2 m = 4 m = 6 m = 8 m = 10
Iterations  3      2      2      2       2
Newton steps  29     31     33     37      39

Table 7.3: Number of iterations and Newton steps for Example 2

approach described in Section 1.2 can be used to minimize the condition number of M. By taking square roots, we then obtain an optimal solution for the original problem.

In our example below, we chose an upper triangular matrix Ā of size l × l with random entries and a random matrix B̄ of size l × m. The entries of Ā and B̄ are uniformly distributed in (0, 1). As initial matrix Z such that M is positive definite, we chose Z = (∥Ā^{T} Ā∥ + ∥Ā^{T} B̄∥)I. In Table 3, we list the number of iterations and Newton steps for the five problems with m = 2, 4, 6, 8, and 10, and fixed l = 10. In all five runs of this example, the convergence tolerance tol = 1.0e-08 was used for the stopping test (7.128).

Concluding remarks

We presented an interior-point method for solving fractional programs over a convex set S. As our main result, we showed that the method converges at a polynomial rate if a self-concordant barrier function of S is known (and computable). Here, convergence is measured in terms of the area of suitably constructed two-dimensional sets that contain the leading two components of the iterates generated by our method, as well as the origin and the leading two components of all optimal solutions of the fractional program. We stress that the class of problems considered in this Chapter is not convex, but only pseudoconvex, and therefore, the existence of a polynomial method is somewhat surprising. We also discussed a practical implementation of the proposed method, and we presented results of a few numerical experiments. These results suggest that the proposed method is a viable approach for solving problem arising in practical applications.

Finally, we mention an extension of the method proposed here. The class of fractional programs considered here is the special case l = 1 of the more general problem of minimizing the maximum of l fractions over some convex set. Techniques similar to the ones in this chapter can be employed to devise and analyze an interior-point method for this more general problem. Numerical experiments for such a method as well as a theoretical analysis are given in [17].

Appendix, two basic lemmas

We begin this section by recalling a slightly generalized version of the well-known Cauchy-Schwarz inequality.

Generalized Cauchy-Schwarz Inequality. If A, M are symmetric matrices with |x^{T} Mx| ≤ x^{T} Ax, ∀x ∈ IR^{n}, then

(a^{T} Mb)^{2} ≤ a^{T} Aab^{T} Ab, ∀a, b ∈ IR^{n}.  (8.1)

Proof. Without loss of generality assume that A is positive definite. (Else A_{ε} := A + εI is

positive definite for all ε > 0, take the limit as ε → 0 for fixed a, b.) Assume further that a, b ≠ 0 and set μ := ^{4} \sqrt{(a^{T} Aa/b^{T} Ab)}, then it follows from

a^{T} Mb = \frac{1}{4}((a + b)^{T} M(a + b) − (a − b)^{T} M(a − b))

that

(a^{T} Mb)^{2} = \frac{1}{16}((a + b)^{T} M(a + b) − (a − b)^{T} M(a − b))^{2}
≤ \frac{1}{16}((a + b)^{T} A(a + b) + (a − b)^{T} A(a − b))^{2}
= \frac{1}{16}(2a^{T} Aa + 2b^{T} Ab)^{2} = \frac{1}{4}(a^{T} Aa + b^{T} Ab)^{2}.

When replacing a by a/μ and b by μb this implies

(a^{T} Mb)^{2} = ((\frac{a}{μ})^{T} M(μb))^{2} ≤ \frac{1}{4}(\frac{1}{μ^{2}} a^{T} Aa + μ^{2}b^{T} Ab)^{2}

= (a^{T} Aa)(b^{T} Ab).

The following estimate about the spectral radius for symmetric homogenous trilinear forms was stated in [63] and proved in [64, 30]. We give the proof of [30].

Spectral Radius for Symmetric Trilinear Forms.

Let a symmetric homogenous trilinear form M : IR^{n} × IR^{n} × IR^{n} → IR be given by its coefficient matrix M ∈ IR^{n×n×n}, let A ∈ IR^{n×n} define a symmetric bilinear form, and let further μ > 0 be a scalar such that

M[h, h, h]^{2} ≤ μA[h, h]^{3}, ∀h ∈ IR^{n},

then

M[x, y, z]^{2} ≤ μA[x, x]A[y, y]A[z, z], ∀x, y, z ∈ IR^{n}.

Proof. For x ∈ IR^{n} denote by M_{x} the (symmetric) matrix defined by y^{T} M_{x}z := M_{x}[y, z] := M[x, y, z], ∀y, z ∈ IR^{n}. Without loss of generality let μ = 1 (else substitute A by ^{3} \sqrt{μ}A). As in the proof of (8.1) assume again that A is positive definite. By substituting M̃[x, y, z] := M[A^{−1/2}x, A^{−1/2}y, A^{−1/2}z] we can further assume that A = I is the identity. Finally, it is sufficient to show that

|M[x, h, h]| ≤ ∥x∥_{2}∥h∥_{2}^{2}, ∀x, h ∈ IR^{n},

holds, provided that M[h, h, h]^{2} ≤ ∥h∥_{2}^{6}, ∀h ∈ IR^{n} is true. (The remaining part follows by applying the generalized Cauchy-Schwarz inequality (8.1) for fixed x to M_{x}!) Let

σ := max{M[x, h, h] | s.t. ∥x∥_{2} = ∥h∥_{2} = 1}

and let x̃, h̃ be the (not necessarily unique) corresponding arguments. The necessary conditions for a maximum (or a minimum if M[x̃, h̃, h̃] is negative) imply that

({Mh̃h̃ \atop 2Mh̃x̃}) = β({2x̃ \atop 0}) + ρ({0 \atop 2h̃}),

where β and ρ are the Lagrange multipliers. From this we deduce that β = σ/2 and ρ = σ (by multiplying from left with (x̃^{T} , 0) and (0, h̃^{T})) and therefore

(Mh̃)h̃ = σx̃, (Mh̃)x̃ = σh̃,

i.e. (Mh̃)^{2}h̃ = σ^{2}h̃. By symmetry of Mh̃, h̃ is eigenvector of Mh̃ to the eigenvalue ±σ, which implies that

σ = |h̃^{T} Mh̃h̃| = M[h̃, h̃, h̃],

and this completes the proof. ⊓ ⊔

Bibliography

[1] F. Alizadeh, “Optimization over the positive semi-definite cone: interior-point methods and combinatorial applications,” in: P. Pardalos, ed., Advances in Optimization and Parallel Computing (North-Holland, Amsterdam, 1992) pp. 1–25.

[2] K.M. Anstreicher, “A monotone projective algorithm for fractional linear programming,” Algorithmica 1 (1986) 483–498.

[3] O. Axelsson and V.A. Barker, Finite Element Solution of Boundary Value Problems (Academic Press, Orlando, 1984).

[4] S.P. Boyd, C.H. Barrat, “Linear controller design: Limits of performance”, Prentice-Hall Information and System Sciences Series, T. Kailath (ed.), New Jersey (1990).

[5] S. Boyd and L. El Ghaoui, “Method of centers for minimizing generalized eigenvalues,” Linear Algebra and Its Applications 188/189 (1993) 63–111.

[6] S.P. Boyd, Q. Yang, “Structured and simultaneous Lyapunov functions for system stability problems”, Int. J. Control, Vol. 49, No. 6, (1989) 2215–2240.

[7] I. Dikin, “Iterative solution of problems of linear and quadratic programming”, Soviet Mat. Doklady Vol. 8 (1967) 674–675

[8] D. den Hertog, F. Jarre, C. Roos and T. Terlaky, “A unifying investigation of interior-point methods for convex programming,” Preprint, Institut für Angewandte Mathematik und Statistik, Universität Würzburg (Würzburg, Germany, 1992), to appear in Mathematical Programming series B.

[9] D. den Hertog, C. Roos, T. Terlaky, “On the classical logarithmic barrier function method for a class of smooth convex programming problems”, J. Opt. Theory Appl., 73 (1992), pp. 1–25.

[10] L. Elsner, C. He and V. Mehrmann, “Minimization of the norm, the norm of the inverse and the condition number of a matrix by completion,” Numerical Linear Algebra with Applications, to appear.

[11] A.V. Fiacco and G.P. McCormick, Nonlinear Programming: Sequential Unconstrained Minimization Techniques (Wiley, New York, 1968).

[12] R. Fletcher, “Semi-definite constraints in optimization”, SIAM J. on Control and Optimization, Vol. 23 (1985) 493–513.

[13] R. Fletcher, “A new variational result for Quasi-Newton formulae” SIAM J. on Optimization, Vol. 1 (1991) 18–21.

[14] R. M. Freund, “A potential-reduction algorithm for solving a linear program directly from an infeasible “warm start”,” Mathematical Programming 52 (1992), pp. 441–466.

[15] R. Freund, “An infeasible-start algorithm for linear programming whose complexity depends on the distance from the starting point to the optimal solution,” Working paper 3559-93-MSA, Sloan School of Management, Massachusetts Institute of Technology, (Massachusetts, 1993).

[16] R.W. Freund and F. Jarre, “An interior-point method for fractional programs with convex constraints,” AT&T Numerical Analysis Manuscript No. 93–07, Bell Laboratories, (Murray Hill, NJ, 1993).

[17] R.W. Freund and F. Jarre, “An interior-point method for convex multi-fractional programming,” AT&T Numerical Analysis Manuscript No. 93–03, Bell Laboratories, (Murray Hill, NJ, 1993).

[18] R.W. Freund and F. Jarre, “A note on the solution of KKT systems arising from linear programs”, in preparation (1994).

[19] K. R. Frisch, “The logarithmic potential method of convex programming”, Technical Report, University Institute of Economics, Oslo, Norway, 1955.

[20] P. E. Gill, W. Murray, D. B. Ponceleon, M. A. Saunders, “Preconditioners for indefinite systems arising in optimization”, Report SOL 90-8 Dept. of Operations Research, Stanford University, Stanford, CA (1990).

[21] P. E. Gill, W. Murray, D. B. Ponceleón and M. A. Saunders, “Solving reduced KKT systems in barrier methods for linear and quadratic programming”, Report SOL91-7, Dept. of Operations Research, Stanford University (1991).

[22] P. E. Gill, W. Murray, M. A. Saunders and M. H. Wright, “Shifted barrier methods for linear programming”, Report SOL88-9, Dept. of Operations Research, Stanford University (1988).

[23] C.J. Goh, K.L. Teo, “On minimax eigenvalue problems via constrained optimization”, J. of Opt. Theory and Appl. Vol. 57 (1988) 59–68.

[24] C. Gonzaga, “An algorithm for solving linear programming problems in O(n^{3}L) operations”, in N. Megiddo, ed., Progress in Mathematical Programming: Interior-Point and related methods, Springer Verlag (1989) 1–28.

[25] A. Greenbaum and G.H. Rodrigue, “Optimal preconditioners of a given sparsity pattern,” BIT 29 (1989) 610–634.

[26] P. Huard, B. T. Liêu, “La méthode des centres dans un espace topologique”, Numerische Mathematik 8 (1966) 56–67.

[27] M. Iri and H. Imai, “A multiplicative barrier function method for linear programming”, Algorithmica 1 (1986), pp. 455–482.

[28] F. Jarre, “On the convergence of the method of analytic centers when applied to convex quadratic programs”, Report No. 35 (1987), Schwerpunktprogramm der DFG Anwendungsbezogene Optimierung und Steuerung.

[29] F. Jarre, “On the method of analytic centers for solving smooth convex programs,” in: Optimization (Varetz, 1988), Lecture Notes in Mathematics No. 1405 (Springer, Berlin, 1989) pp. 69–85.

[30] F. Jarre, “Interior-point methods for convex programming,” Applied Mathematics and Optimization 26 (1992) 287–311.

[31] F. Jarre, “An interior-point method for minimizing the maximum eigenvalue of a linear combination of matrices,” SIAM Journal on Control and Optimization 31 (1993) 1360–1377.

[32] F. Jarre, “Optimal ellipsoidal approximations around the analytic center,” Applied Mathematics and Optimization, to appear in Applied Mathematics and Optimization (1994).

[33] F. Jarre, “A new line-search step based on the Weierstrass ℘-function for minimizing a class of logarithmic barrier functions” to appear (1994).

[34] F. Jarre, M.A. Saunders, “An adaptive interior-point method for linear programming”, Report SOL 91-1, Dept. of Operations Research, Stanford University, Stanford, CA (1991).

[35] F. Jarre, M.A. Saunders, “An implementation of an interior-point method for convex programming”, Report SOL 91-9, Dept. of Operations Research, Stanford University, Stanford, CA (1991).

[36] F. Jarre, G. Sonnevend, J. Stoer, “An implementation of the method of analytic centers”, in A. Benoussan, J.L. Lions (eds.), Lecture Notes in Control and Information Sciences 111, Analysis and Optimization of Systems, Springer, New York (1988) 297–307.

[37] V.A. Kamenetskii, “Absolute stability and absolute instability of control systems with several nonlinear nonstationary elements” Automation and Remote Control 12 (1983) 1543–1552.

[38] V.A. Kamenetskii, E.S. Pyatnitskii, “Gradient method of constructing Lyapunov functions in problems of absolute stability” Automation and Remote Control 1 (1987) 1–9.

[39] N. Karmarkar, “A new polynomial-time algorithm for linear programming,” Combinatorica 4 (1984) 373–395.

[40] H.W. Knobloch and F. Kappel, Gewöhnliche Differentialgleichungen (Teubner Verlag, Stuttgart, 1974).

[41] M. Kojima, N. Megiddo and S. Mizuno, “A primal-dual infeasible-interior-point algorithm for linear programming,” Research Report RJ 8500, IBM Almaden Research Center (San Jose, CA, 1991), to appear in Mathematical Programming. 131–158.

[42] M. Kojima, N. Megiddo, T. Noma and A. Yoshise, “A unified approach to interior-point algorithms for linear complementarity problems”, G. Goos and J. Hartmanis, eds., Lecture Notes in Computer Science, Vol. 538, Springer, New York, 1991.

[43] M. Kojima, S. Mizuno and T. Noma, “A new continuation method for complementarity problems with uniform P-functions”, Mathematical Programming 43 (1989), pp. 107–130.

[44] M. Kojima, T. Noma and A. Yoshise, “Global convergence and detecting infeasibilty in interiorpoint algorithms”, Report, Dept. of Information Sciences, Tokyo Institute of Technology, Tokyo, Japan (1992).

[45] E. Kranich, “Bibliography about interior-point methods”, available via NETLIB, (1993).

[46] C. Lemarechal, R. Mifflin (eds.), “Nonsmooth Optimization”, Pergamon Press, Oxford, (1978).

[47] I. J. Lustig, R. E. Marsten and D. F. Shanno, “Computational experience with a primal-dual interior point method for linear programming,” Linear Algebra and Its Applications 152 (1991) 191–222.

[48] I. J. Lustig, R. E. Marsten and D. F. Shanno, “On implementing Mehrotra’s predictor-corrector interior-point method for linear programming”, SIAM J. Opt. 2 (1992), pp. 435–449.

[49] O.L. Mangasarian, Nonlinear Programming (McGraw-Hill, New York, 1969).

[50] N. Megiddo, “Pathways to the optimal set in linear programming”, in N. Megiddo (ed.), Progress in Mathematical Programming: Interior-Point and Related Methods, Springer Verlag, New York (1989) 131–158.

[51] S. Mehrotra, “On the implementation of a (primal-dual) interior-point method”, SIAM J. Opt. 2 (1992), pp. 575–601.

[52] R. Fourer and S. Mehrotra, “Solving symmetric indefinite systems in an interior-point method for linear programming” Mathematical Programming 62 (1993), pp. 15–39.

[53] S. Mehrotra and J. Sun, “An interior-point algorithm for solving smooth convex programs based on Newton’s method”, Mathematical developments arising from linear programming, 265–284, Contemporary Mathematics, Vol. 114, Amer. Math. Soc., Providence, RI, 1990.

[54] S. Mizuno, “Polynomiality of infeasible interior point algorithms for linear programming,” Technical Report No. 1006, School of Operations Research and Industrial Engineering, Cornell University (Ithaca, NY, 1992).

[55] S. Mizuno, M. Kojima, and M. J. Todd, “Infeasible-interior-point primal-dual potentialreduction algorithms for linear programming,” Technical Report No. 1023, School of Operations Research and Industrial Engineering, Cornell University (Ithaca, NY, 1990).

[56] C. Moler, J. Little, S. Bangert, “PRO-MATLAB user’s guide”, The Math-Works, Inc., Sherborn, MA (1987).

[57] W. Murray, M.L. Overton, “A projected Lagrangian algorithm for nonlinear minimax optimization”, SIAM J. on Scientific and Statistical Computing, Vol. 1 (1980) 345–370.

[58] W. Murray, M. H. Wright, “Efficient linear search algorithms for the logarithmic barrier function” Report SOL 76-18, Dept. of Operations Research, Stanford University, Stanford, CA (1976).

[59] W. Murray, M. H. Wright, “Line search procedures for the logarithmic barrier function” AT&T report 92-01, AT&T Bell Laboratories, (1992)

[60] J. L. Nazareth, “Lagrangian globalization: Solving nonlinear equations via constrained optimization”, Technical report DAMTP 1993/NA18, Dept. of Applied Mathematics and Theoretical Physics, Cambridge (1993).

[61] A.S. Nemirovsky, “On polynomiality of the method of analytic centers for fractional problems”, Technical Report, INRIA Rocquencourt, France (1993).

[62] J.E. Nesterov and A.S. Nemirovsky, “A general approach to polynomial-time algorithms design for convex programming,” Report, Central Economical and Mathematical Institute, USSR Acad. Sci. (Moscow, Russia, 1988).

[63] J.E. Nesterov and A.S. Nemirovsky, Self-concordant functions and polynomial-time methods in convex programming Report CEMI, USSR Academy of Sciences, Moscow (1989).

[64] J.E. Nesterov and A.S. Nemirovskii, Interior Point Polynomial Algorithms in Convex Programming (SIAM Studies in Applied Mathematics, Philadelphia, 1994), (revised version of [63]).

[65] J.E. Nesterov, A.S. Nemirovsky, “Optimization over positive semidefinite matrices”, Manual, Central Economical and Mathematical Institute, USSR Acad. Sci., Moscow, USSR (1990).

[66] J.E. Nesterov and A.S. Nemirovsky, “An interior-point method for generalized linear-fractional programming,” Mathematical Programming, Ser. B, to appear.

[67] J.E. Nesterov and A.S. Nemirovsky, “Technique for constrctiong self-concordant barriers”, Report No. 1993.17, Dept. of Economical and Social Sciences, University of Geneva, Switzerland (1993).

[68] M.L. Overton, “On minimizing the maximum eigenvalue of a symmetric matrix”, SIAM J. Matrix Anal. Appl. Vol. 9, No. 2 (1988) 256–268.

[69] M.L. Overton, “Large-scale optimization of eigenvalues”, NYU Computer Science Dept. Report No. 505, Courant Institute of Mathematical Sciences, New York Univerity (1990).

[70] M.L. Overton, R.S. Womersley, “On minimizing the spectral radius of a nonsymmetric matrix function—optimality conditions and duality theory”, SIAM J. Matrix Anal. Appl. Vol. 9 (1988) 473–498.

[71] E.R. Panier, “On the need for special purpose algorithms for minimax eigenvalue problems”, Report, Dept. of Elec. Eng., University of Maryland (1989).

[72] F. A. Potra, “An infeasible interior-point predictor-corrector algorithm for linear programming,” Report No. 26, Department of Mathematics, The University of Iowa (Iowa City, Iowa, 1992).

[73] F. A. Potra, “A quadratically convergent infeasible interior-point algorithm for linear programming,” Report No. 28, Department of Mathematics, The University of Iowa (Iowa City, Iowa, 1992).

[74] E.S. Pyatnitskii, V.I. Skorodinskii, “Numerical methods of construction of Lyapunov functions and absolute stability criteria in the form of numerical procedures”, Automation and Remote Control 11 (1983) 1427–1437.

[75] J. Renegar, “A polynomial-time algorithm based on Newton’s method for linear programming”, Math. Prog., 40 (1988), pp. 59–93.

[76] J. Renegar, “Linear programming, complexity theory and elementary functional analysis”, Technical Report, School of Operations Research and Industrial Engineering, Cornell University (1993).

[77] U.T. Ringertz, “Optimal design of nonlinear shell structures”, Report, Flygtekniska Försöksanstalten, The Aeronautical Research Institute of Sweden (1991).

[78] S. Schaible, “A survey of fractional programming,” in: S. Schaible and W.T. Ziemba, eds., Generalized Concavity in Optimization and Economics (Academic Press, New York, 1981) pp. 417-440.

[79] S. Schaible, “Bibliography in fractional programming,” Zeitschrift für Operations Research 26 (1982) 211–241.

[80] S. Schaible, “Fractional programming,” Zeitschrift für Operations Research 27 (1983) 39–54.

[81] H. Schramm, “Eine Kombination von Bundle und Trust-Region-Verfahren zur Lösung nichtdifferenzierbarer Optimierungsprobleme”, PhD thesis, Universität Bayreuth (1989).

[82] G. Sonnevend, “An ‘analytical centre’ for polyhedrons and new classes of global algorithms for linear (smooth, convex) programming,” in: System Modelling and Optimization (Budapest, 1985), Lecture Notes in Control and Information Sciences No. 84 (Springer, Berlin, 1986) pp. 866–875.

[83] G. Sonnevend, “A new method for solving a set of linear (convex) inequalities and its applications”, 5-th IFAC-IFORS Conference, Budapest 1986, Pergamon press (1987).

[84] G. Sonnevend, “Applications of analytic centers to feedback control systems”, Control of uncertain systems, Bremen, June 1989, D. Hinrichsen and B. Martensson (eds.), Birkhäuser (1990).

[85] G. Sonnevend and J. Stoer, “Global ellipsoidal approximations and homotopy methods for solving convex analytic programs,” Applied Mathematics and Optimization 21 (1990) 139–165.

[86] G. Sonnevend, J. Stoer and G. Zhao, “On the complexity of following the central path by linear extrapolation in linear programs”, Mathematical Programming (1991) series B., pp. 527–553.

[87] T. H. Southard, “Weierstrass elliptic and related functions” in M. Abramowitz, I. A. Stegun eds. Handbook of math. functions, Dover Publications, Inc., New York (1965) 627–671.

[88] G. W. Stewart, J.G. Sun, “Matrix Perturbation Theory”, Academic Press (1990).

[89] J. Stoer, “Über einen Dualitätssatz der nichtlinearen Programmierung” Numerische Mathematik 6 (1964) 55–58.

[90] J. Stoer, “The complexity of an exterior point path-following method for the solution of linear programs,” Working paper, Institut für Angewandte Mathematik und Statistik, Universität Würzburg, (Germany, 1992).

[91] T. Tsuchiya and M. Muramatsu, “Global convergence of a long-step affine scaling algorithm for degenerate linear progrmming problems” , Report No. 423, The Institute of Statistical Mathematics, Minami-Azabu, Tokyo 106, Japan, 1992.

[92] P. M. Vaidya, “An algorithm for linear programming which requires O(((m + n)n^{2} + (m + n)^{1.5}n)L) arithmetic operations”, Mathematical Programming 47 (1990), 175–202.

[93] R.J. Vanderbei, M.S. Meketon, B.A. Freedman, “A modification of Karmarkar’s linear programming algorithm”, Algorithmica Vol. 1 (1986) 395–407.

[94] M. H. Wright, “Numerical methods for nonlinearly constrained optimization”, Ph.D. Thesis, Computer Science Dept., Stanford University, Stanford, CA, 1976.

[95] M.H. Wright, “Interior methods for constrained optimization,” Acta Numerica 1 (1992) 341– 407.

[96] H. Yamashita, “A globally convergent primal-dual interior point method for constrained optimization”, Technical report, Mathematical Systems Institute, Tokyo (1992).

[97] Y. Ye, O. Güler, R.A. Tapia, Y Zhang, “A quadratically convergent O(\sqrt{n}L)-iteration algorithm for linear programming”, Report 91-26, Dept. of Management Sciences, The University of Iowa, Iowa City (1991).

[98] Y. Ye, M. J. Todd, and S. Mizuno, “An O(\sqrt{n}L)-iteration homogeneous and self-dual linear programming algorithm,” Technical Report No. 1007, School of Operations Research and Industrial Engineering, Cornell University (Ithaca, NY, 1992) to appear in Mathematics of Operations Research.

[99] J. Zowe, “The BT-algorithm for minimizing a nonsmooth functional subject to linear constraints”, manuscript, Universität Bayreuth (1989).